Has government learned the vital lessons of public service failures?

From the failure at Mid Staffordshire NHS foundation trust that led to the mistreatment of hundreds of patients, to the Rotherham children’s services scandal, we have seen been high-profile crises in public services that have had direct and disastrous effects on people’s lives.

Managing failure is a vital skill, and new research from the Institute for Government aims to understand how government can best respond to – and learn from – such events more effectively. It focuses on failures in four key public services, hospitals, local authorities, children’s services and schools, and ways to restore services quickly. A few common themes emerge:

The first lesson is to avoid some of the most common responses to failure, including blame and restructuring. It’s human instinct to find an individual to blame: 70%-80% of inquiries across a range of industries and professions attribute tragedies to the error of particular individuals. But this instinctive response usually fails to understand or to address why it was that an individual thought it best to act in a given way. The impulse to restructure in the wake of a crisis has a similar appeal – it feels like you are at least doing something – but will be effective only if linked to the actual causes of failure, which are frequently rooted in behaviour or culture.

One example is Doncaster metropolitan borough council, which was found to be failing by an Audit Commission inspection in 2010. by the Audit Commission. The communities secretary, John Denham subsequently appointed commissioners and service quality improved. They were withdrawn a year earlier than planned, in 2014.

Related: May’s Whitehall reshuffle shows disregard for evidence and expertise

In Doncaster, the turnaround was based in part on reconnecting the council with neighbouring authorities. It had become insular, and as it detached from its peers, it lost its sense of what good looked like.

This was a common phenomenon across our case studies; one teacher, appointed to a school just weeks before it was tipped into special measures, recalled that it was absolutely crystal clear the second they walked in that building that the school was failing. “What was terrifying was that it wasn’t clear to the people inside it.” To overcome this insularity, organisations began rebuilding links with their peers, including them on improvement boards or by making use of sector-specific services such as the Local Government Association’s peer challenge tool.

Doncaster also avoided the risk of pursuing an unsustainable recovery based on temporary appointments or relying on external support. The commissioners were clear that ownership should remain with the council. The test for ending the intervention was not whether services had returned to a good standard, but whether commissioners were confident that the council could reach that point without further external support. In contrast, in Tower Hamlets in 2014, the commissioners sent in by then-communities secretary Eric Pickles formed an executive board and replaced the council in the day-to-day running of services.

Failure is not going to go away, but at least we can minimise its impact if the government is prepared to learn lessons from the past. There is plenty of scope for improvement.

Oliver Ilott is a researcher at the Institute for Government.

Talk to us on Twitter via @Guardianpublic and sign up for your free weekly Guardian Public Leaders newsletter with news and analysis sent direct to you every Thursday.

Has government learned the vital lessons of public service failures?