The Georgian-style house faces Richmond Hill and overlooks the Thames river. It is located near Richmond Park, the largest urban park in the United Kingdom. The home was built of brown brick with yellow window dressing. It includes four storeys and a basement with bays on either side and balconies above. It features modillion cornice and balustrading above and cast-iron balustrades on the second floor windows. Windows above the bays are round-headed, and others are square-headed sash windows. The porch has heavy entablature and Doric piers.
Richmond Hill has been part of the Royal Manor of Richmond since Domesday. Richmond Park was enclosed by King Charles I around 1635. The countryside was mostly agricultural land in the early 18th century, but Terrace Walk was laid out at the top of Richmond Hill in the later 18th century, followed by construction of a number of other houses, including Downe House. The house is thought to have been built around 1780, and at one time was owned by Richard Brinsley Sheridan.
Mick Jagger bought the house in the early 1990s where he lived with his wife, model Jerry Hall. After the two separated, Jerry Hall remained in Downe House while Jagger moved into a flat next door. It is listed as Grade II by English Heritage.