His son, the twenty four year-old Marquess of Chandos, took over the running of the estate with trustees, only to discover that his father had been trying to plunder his inheritance.

By 1848 land in Ireland, Hampshire and London was sold, as were most of the contents of Stowe. This did little to re-coup the Duke's £1 million debt.

The scandal rocked the English aristocracy and appalled and entertained the British public. The future of Stowe was in grave doubt. Remarkably, the future 3rd Duke put in place such an effective programme of retrenchment, shutting up the house, cutting staffing to the bone, selling the woods and leaving the garden buildings to their own devices, that by the 1860s he was able to live once more at Stowe. In contrast, the 2nd Duke died in relative poverty at the Great Western Hotel in Paddington.

The 3rd Duke eventually began to refurnish the house and repair the garden, with the Palladian Bridge, the Temple of Friendship and the Bell Gate receiving most attention and admitting visitors once more. The garden and museum were restocked. Yet his effort was undermined when he died in 1889 with no male heir, the dukedom dying with him. His eldest daughter, Lady Kinloss, used Stowe rarely. When her son, the heir to Stowe, died in the First World War, his younger brother decided to sell up.

In 1921, the house and garden, along with most of the contents and statues, were the property of Mr Harry Shaw for £50,000. The rest was sold the following year.