Wednesday 25 August 2021

The Horizon Scandal: outsourcing doesn't have to be a minefield if it's properly managed.

I make no apology for returning to the subject of the Post Office and the Horizon Scandal. With the public enquiry into what went wrong and why (to say nothing of compensation and criminal charges against the people actually responsible) still in progress and some way from completion, the more experts in different fields are contributing to the debate.  And they are not jumping on a kudos bandwagon.  

We have already seen the lawyer's perspective into duties and ethics of the law profession, and on the same link the start of academic research into corporate governance, criminal justice, and professional regulation, as well as government and parliamentary accountability.

Now I can relay the view of the IT professional, more particularly concerning 'high-level access' to systems - that is, access by those not directly involved in entering transactions on a daily basis, nor those working from the outputs of the system in a routine way.  Those with high-level or 'privileged' access would be IT professionals.  

James Christie is a software testing consultant who previously worked for IBM. His main area of interest is the governance issues associated with testing.   He has written a blog about the Horizon computer system and its management by Post Office Ltd entitled:

“Privileged accesses” – an insight into incompetence at Fujitsu and the Post Office

As he mentions, the findings by Ernst & Young (later rebranded as just E&Y) in 2011 that there was "poor control over user IDs with high privilege levels. Not only did this highlight the need to improve Fujitu’s management of the IT service and the oversight provided by the Post Office, it also pointed to an ineffective internal audit function at the Post Office".

At one stage in his IBM career he was an information security manager working with new outsourced accounts.

All the issues relating to privileged access raised by E&Y in their management letter were within my remit. The others, mainly change management, were dealt with by the relevant experts. Each outsourcing contract required us to reach agreement on the full detail of the service by a set date, typically within a few months of the service cutover. In one case we had to reach agreement before service even started. On the service cutover date all staff transferring to IBM were required to continue working to exactly the same processes and standards until they were told to do something new.

I had to set up a series of meetings and workshops with the client and work through the detail of the security service. We would agree all the tedious but vital details; password lengths and formats, the processes required for authorising and reviewing new accounts and access privileges, logging and review of accesses, security incident response actions. It went on and on.

For each item we would document the IBM recommended action or setting. Alongside that we had to record what the client was currently doing. Finally we would agree the client’s requirement for the future service. If the future requirement entailed work by IBM to improve on what the client was currently doing that would entail a charge.

You can read the whole blog post, and earlier ones also relevant to the Horizon problems here. But his conclusions are inescapable:

Getting the basics correct is vital if corporations want to show that they are in control of their systems. If users have high privilege levels without effective authorisation, logging and monitoring then the corporation cannot have confidence in its data, which can be changed without permission and without a record of who took what action. Nobody can have confidence in the integrity of the systems. That has clear implications for the Horizon scandal. The Post Office insisted that Horizon was reliable when the reality was that Fujitsu did not apply the controls to justify that confidence.

Fujitsu may have failed to manage the service properly, but the Post Office is equally culpable. Outsourcing an IT service is not a matter of handing over responsibility then forgetting about it. The service has to be specified precisely then monitored carefully and constantly.

Why were the two corporations so incompetent and so negligent for so long? Why were the Post Office and Fujitsu so much less responsible and careful than IBM, AstraZeneca, Boots and Nokia?

Why did the Royal Mail’s and subsequently the Post Office’s internal auditors not detect problems with the outsourced service and force through an effective response?   

I know how these jobs should be done and it amazes me to see that one of our major rivals was able to get away with such shoddy practices at the very time I was in the outsourcing game. Fujitsu still has the Post Office contract. That is astonishing.

If you work or have worked in IT, and particularly in IT security the protocols should be familiar to you - and if you have any imagination so will the pitfalls that are there if the organisations don't get things right.

But it is doubtful whether you have been involved in a less competent organisation which impacted the lives of so many people at such a low operational level - the foot soldiers - than Post Office Ltd.


  1. I worked in IT for 35 years, and of those 27 were in the pharmaceutical sector where you REALLY want to have regulatory compliance. My primary function for much of that time was in software maintenance, and I can tell you that no large piece of software produced by anybody at any time has EVER been bug-free. I can also tell you that there is always an independent way of getting at data. Mostly this would be done to counteract the effects of misbehaving software or correcting a user error, but ALWAYS ALWAYS ALWAYS this should be in an audit trail.

    As for the auditors - well, they were either totally incompetent, and need to answer for that, or they were part of a corporate mission, and need to answer for that.

    Above all of that, did nobody involved in this sorry saga have a CONSCIENCE??? Where were the whistle-blowers when they were needed?

    1. Great comment Mike. I agree wholeheartedly. When I worked for IBM on the AstraZeneca account there were several references to the Thalidomide tragedy. The message was that it must never happen again, but if it did it was vital that AZ and IBM could demonstrate the integrity of the systems and data. It had to be possible to show that every server was configured to the correct, agreed, secure specification at any point in time, and that nothing could be changed or tampered with without a secure audit trail showing who took what action.
      It was taken for granted that we should work like that. People's lives were at stake. People's health, life savings, marriages, families, and yes, lives too, were also at stake in the Horizon scandal. But did anyone in senior positions at either the Post Office or Fujitsu care?

  2. Thank you Ian. I do find it infuriating that we were working hard, being responsible and we were being undercut by a cheaper competitor that was neglecting basic controls.

  3. The article talks about the "duties and ethics of the law profession". There are other professions involved in this. The BCS (aka British computer Society) which is a professional body for those working in the IT industry has a Code of Conduct for its members. Membership of the BCS is not a prerequisite for those working in IT but we might hope that those BCS members who worked on the Horizon system did so within the BCS's Code of Conduct. We might hope that the BCS will take action against members if it is shown that the did not follow the Code of Conduct.

  4. A light reading - these are on publicly accessible web-pages

    Relaunched Postmaster Support Guide out now

    Branch Operational Training catalogue

  5. The British Computer Society is the obvious choice for external computing audits.


Thank you for reading the blog and commenting: please use an identity (name or pseudonym) rather than being Anonymous; it helps us to know which 'anonymous' comments are from the same person to avoid confusion. Comments are moderated to avoid spam, but will be published as soon as possible.