The 6th Baronet, often known as wicked Sir James had a rich inheritance, with estates in Stainton, Tunstall, Maltby and Sadberge, and houses at Thornton and Beverley, as well as at Ormesby. However, he spent large amounts of money on politics and gambling and became bankrupt in 1792. All the contents of Ormesby Hall were auctioned to pay off his creditors, and the house was shut up for sixteen years. The family fortunes never fully recovered from his recklessness, and his successors had limited resources to invest in their home.
His son, Sir William Pennyman, 7th Bt, reclaimed the Hall and built East Lodge in the 1820s, but made few other improvements to the property. He left most of his personal possessions to his sister’s family, but his heir, James White (Worsley) Pennyman, bought back several items of furniture for the hall.
James White started to lease land at the northern end of the estate for housing associated with the new town of Middlesbrough, to make the estate a more viable proposition. He and his son, James Stovin, made the final alterations to Ormesby Hall, adding the front porch, the Dining Room extension and the corridors connecting the service wing and main building.