The thermocline of truth

26 April 2021 Filed under: knowledge

The thermocline of truth

In organisations, reality is not always what it seems. Why is it that often things look rosy right up until they fall apart?

Over the last twenty years, one of the most Kafka-esque scandals in British history has played out in slow motion: that of the Post Office’s Horizon IT system. The story is far too dramatic to do justice to here, but the basic outline of events is straightforward:

In the early 2000s, the newly privatised Royal Mail introduced a new IT system, called Horizon, aimed at reconciling every transaction made in every Post Office up and down the UK, removing the possibility for errors and fraud.
Like all large-scale corporate IT projects, the system had a significant number of severe bugs, including some that resulted in duplicate transactions being logged – which meant that it appeared that cash was disappearing from some branches.
Rather than believe that their flagship new IT system had crippling flaws, the top brass at the Royal Mail instead chose to believe that hundreds of their previously loyal sub-postmasters were embezzling cash.
Royal Mail then spent nearly two decades pursuing those employees through the courts, bankrupting many and even sending some to prison, entirely on the basis of IT evidence and without any proof of wrongdoing.

This month, the Court of Appeal finally quashed the remaining criminal convictions in the case, but Royal Mail spent nearly two decades – and over £100 million – hounding loyal, and entirely innocent, employees through the courts and ruining their lives and livelihoods. It’s perhaps the most shocking and terrifying miscarriage of justice in recent years, and it exposes – if examined closely – a laundry list of organisational and cultural failures on the part of Royal Mail.

Perhaps the most fundamental, though, is the question of how Royal Mail leaders weren’t able to see the true source of the problems. How were they able to convince themselves almost to the last that their own systems couldn’t have been the source of the errors? How were those at the top seeing such a warped version of reality?

In the ocean, temperature decreases with depth: the deeper you go, the colder it gets. But sometimes, what’s called a thermocline forms: a temperature barrier, a point at which the temperature changes rapidly. Go above the thermocline and the water is warm; pass beneath it, and it’s suddenly cold. This can have huge ramifications for life in the ocean, preventing the passage of oxygen and nutrients past the barrier.

In a 2008 blogpost, legendary IT consultant Bruce F. Webster applied the idea of the thermocline to large-scale IT projects. Why was it, Webster asked, that so many projects seemed to be on-track until just before their launch date, at which point it became suddenly clear that they were miles behind schedule?

Webster observed that, generally speaking, those at the bottom of an organisation have a fairly accurate view of what’s going on. They’re close to the detail; they know whether their area of the project is on-track, and can infer from that the state of the wider project.

Those at the top, though, have no such first-hand knowledge. They rely on the bubbling-up of information from below, in the form of dashboards and status reports. But, Webster noticed, those status reports tend to produce a comically optimistic view of the state of the project. Individual contributors presented a rosy picture of what they were working on to their line managers; middle managers gave good news to their bosses; and senior managers, keen to stay on the promotion track and perhaps hopeful that other parts of the project would fail before theirs, massage the truth yet again.

The result is that there is a thermocline within the organisation: not of temperature, but of truth. There is a clear line in the org chart, below which the truth of the project’s disastrous state is known, but above which everything looks rosy. We might call this line a “vericline”, since it divides the organisation not on temperature (“thermos”) but on truth (“verus”).

As the project nears its supposed date of completion, it becomes harder and harder to hide the reality of just how screwed up the project is, and so the vericline moves upwards in the organisation. Eventually, the truth reaches the highest echelons of the organisation – be that days, weeks, or months before the deadline.

Vericlines, to be clear, are not the fault of subordinates. They emerge because of the culture set by the leadership of an organisation. They are the inevitable consequence of an organisation that puts more emphasis on reputation than truth; that rewards good news and punishes bad; that has a leadership team disconnected from the delivery of the actual work; that instils fear and compliance with process into its employees, rather than a desire to do the right thing.

As the Post Office scandal became messily public in the 2010s, it became possible to see from the outside the moving of the vericline within Royal Mail. Paula Vennells, who took over as Chief Executive in 2012 with, by all accounts, the best of intentions, found herself surrounded by a senior management team desperate to protect their own reputations and that of the Royal Mail. They told her that there simply couldn’t be problems with the IT system on the scale that was being suggested; it was much more likely that the sub-postmasters were colluding, lying about IT bugs in order to escape responsibility for what was at best incompetence and at worst theft.

Unable to determine the truth for herself – she could hardly delve into the source code of Horizon and debug it – she was forced to accept reality as it was presented to her, sitting as she did above the vericline. Unwilling or unable to create the conditions where the truth would emerge, she remained in the dark. And so Vennells led Royal Mail into a protracted legal battle to fight the claims of those whose lives had been ruined by Horizon, a position that had no factual, legal, or moral basis.

The result was an unmitigated disaster. As Mr. Justice Fraser summed up in his judgment:

“This approach by the Post Office has amounted, in reality, to bare assertions and denials that ignore what has actually occurred… It amounts to the 21st century equivalent of maintaining that the earth is flat.”

But moving the vericline upwards, and eventually disrupting it entirely, was not easy. It took the twenty-year tenacity of people like Alan Bates. Sub-postmaster at the Post Office in Craig-y-Don, in North Wales, Bates was accused of running up a cash deficit by the faulty IT system. But he refused to accept reality as it was presented to him: he diligently reconciled his accounts, discovering that the system was at fault; he had his contract terminated for his troubles, but persisted regardless. He set up a campaign group, he reported his findings to the magazine Computer Weekly, and he eventually fought Royal Mail through the courts.

By 2019, it had become impossible to deny the truth. The Post Office paid out £58 million to Bates and the other sub-postmasters who had been affected by the scandal. In 2020, they agreed not to oppose the criminal appeals of those who had been convicted, and in April 2021 the Court of Appeal finally quashed the remaining convictions.

The danger of the vericline is that, if you’re above it, it’s difficult to know it. Is the rosy picture of reality you’re hearing from subordinates genuine? Or are you heading for an iceberg that everyone can see but you?

This dynamic doesn’t emerge in every business, though. It’s not an organisational inevitability. It’s a cultural artefact; the question of whether your organisation suffers from a vericline is a question of how your organisation treats the truth more broadly.

Does your organisation value truth above all, even from those at the bottom of the org chart? Do you create an environment in which people feel safe challenging their superiors? Do you publicly praise those who give bad news, rather than admonishing them for demoralising the team? Do you use objective metrics to measure progress, or do you rely on measures that are capable of being gamed?

Without working hard to create an environment that nurtures the truth and allows it to filter upwards, you might find yourself swimming in lovely warm water now – but it’s only a matter of time before reality bites.

Roblog