Jim married again in 1926 to Ruth Knight and they lived happily together at Ormesby Hall, where Jim managed the Ormesby estate. They were actively involved with the local community and in the 1930s Jim and Ruth helped unemployed miners learn carpentry skills through the venture ‘Boosbeck Industries’ and set up market gardens and livestock schemes, these are just a few examples of their generous nature. During the Second World War he commanded a battalion of the National Defence Corps as Lieutenant Colonel, giving him the title Colonel James Pennyman, as he is now known. Jim continued with his estate work and responsibilities alongside Ruth whose love of the arts transformed Ormesby into a home of theatrical performances and workshops.
Jim and Ruth were the last Mr and Mrs Pennyman of Ormesby Hall and because they had no children the Pennyman line ended with them. Upon Jim’s death in 1961 he bequeathed Ormesby Hall, park and home farm to the duty of the care of the National Trust, with Ruth continuing to live at Ormesby until she passed away in 1983.