Two leading statisticians have expressed ‘frustration’ at the failure of the 18 month long inquiry into the events that led to the convictions of Lucy Letby to get to grips with the analysis of data. Professor Jane Hutton, professor of statistics at the University of Warwick and Dr Amy Wilson, lecturer in Statistics at the University of Edinburgh, shared their response to the inquiry with the Justice Gap.
They argue that four of 13 recommendations require input from a statistician because of the analysis of data including those referring to the monitoring of deaths in hospital (6), the reporting of data (7), the rollout of a suspicion of deliberate harm protocol (9) and the establishment of a panel of independent experts to deal concerns about an individual causing harm to patients (10). ‘There is no mention of a statistician anywhere in these recommendations,’ say the pair. ‘Indeed, it is even implied that clinicians or managers would be the appropriate people to do and lead this task,’ Hutton and Wilson continue. ‘Statistics is difficult, and it can go badly wrong when done by people without proper training. Examples of proper training would be a degree in statistics, or chartered statistician status.’
Professor Jane Hutton was instructed by Cheshire Police to advise in its investigation of the murders – she warned the force that its approach on the stats was wrong (see below). She has long had a concern about the misuse of statistics in the court in healthcare cases. The academic was instructed and ready to give evidence in the the case of the nurse Ben Geen convicted of murdering two patients and harming 30 others. In that case, she took issue with the prosecution case that there was an ‘unusual pattern’ of collapses on the ward indicating the presence of a serial killer. She was prevented from giving evidence by the the Court of Appeal – as reported by the Justice Gap.
- You an read about the Lucy Letby case in the latest PROOF magazine including Stephen Phelps on the media, Prof Richard Gill and Dr Svilena Dimitrova on How to become a serial killer (without killing anyone) and an interview with the barrister Mark McDonald representing the former nurse
From PROOF issue 7
Prof Hutton has told a new investigation by the Sun newspaper broadcast on Channel 5 broadcast earlier this week that the investigation into the murders at the Countess of Chester Hospital ‘made just about every mistake in the book’. She now argues, with Dr Wilson, that those mistakes have been compounded by Lady Justice Thirwall’s failure to draw on proper statistical advice.
‘The importance of statistical input to these recommendations needs to be highlighted if they are to be done seriously,’ they write. ‘The lack of mention of statistics is particularly frustrating given the enormous amount of work done by statisticians to raise awareness of the importance of sound statistical analysis in the Letby trial.’
The Royal Statistical Society published a report tackling statistical bias in criminal trials – Healthcare serial killer or coincidence? – the month before the Letby trial started which was informed by concerns by statisticians involved in the Geen case including Professor Richard Gill and Sir David Spiegelhalter as well as the Dutch case of Lucia de Berk (see here). Lucia de Berk, a paediatric nurse, was found guilty of seven murders and three attempted murders of children in her care at Juliana Children’s Hospital in The Hague in 2003. She was reckoned to be the Netherlands’ most prolific serial killer. At least she was until she was exonerated in 2010. Now her case is recognised as one of the country’s gravest miscarriages of justice.
The pair highlight how the inquiry dealt with the evidence of the leading statistician Sir David Spiegelhalter. He looked at the the numbers of recorded deaths from 2010 to 2016 on the unit noting between 2010 and 2014 there was surprising consistency (one, three, three, two, three respectively). ‘[T]he probability of getting eight or more deaths in 2015 I assessed to be around 0.008 … that would generally be considered sufficient to trigger an alert signal, someone should look at this locally,’ Sir David said. ‘But not extreme enough to be considered an outlier and I think this is very useful to put this in perspective.’
Hutton and Wilson argue that Thjirwall misreads the 0.008 number ‘claiming it excluded other explanations for the deaths’. They continue: ‘It did not do this – the only explanation it considers is random chance (not, for example the medical condition of the babies, any differences in staffing, admissions or acuity, etc…). There is a failure to understand… that it is not possible to determine whether there has been an increase in deaths beyond that expected by random variation without a statistical analysis.’
The pair argue that it is important to understand whether the change in deaths is statistically significant ‘otherwise it might just be random fluctuation’. ‘The NHS would not be able to function if it had to investigate every increase in deaths, regardless of statistical significance (it would happen in roughly 50% of time periods considered),’ they say.
Reference is made in the inquiry report repeatedly to ‘the temporal association of collapses with Letby’s shifts’, as Hutton and Wilson put it: ‘as if this is fact’. ‘The report also refers to claims of the pattern of deaths changing when Letby moved from nights to day, and when Letby was moved off the neonatal unit, with no mention of the mitigating points there – e.g., that her leaving the neonatal unit coincided with the unit being downgraded. All of these points have been refuted publicly by statisticians many times.’
Thirlwall identifies ‘suggestions from different quarters outside the Inquiry’ that the testing for insulin, immune assay tests, were defective, asserting evidence to the inquiry was to the ‘contrary effect’. A relevant expert is quoted assuring the inquiry: ‘We wouldn’t have even measured the sample if we weren’t sure that the analyser was performing appropriately that day. And then with the results, we have to double-check that the — it’s a quality control procedure before we even release a result on to an electronic system to be communicated.’ There was ‘no reason to think that the machines were not properly calibrated or that there was any error in the process’, the inquiry notes.
All laboratory tests have ‘error in the process’, Hutton and Wilson argue. ‘The important thing is to understand what it is, and whether it would change the interpretation of the results. This was not explored, either at trial or in the Inquiry, because nobody asked the question of what the measurement uncertainty was (a statistician would ask this immediately). “Performing appropriately” was mistaken for “accurate”. ‘ They point out that that the current external quality control procedure for these tests allows for a +/- 25% average error, which is ‘very large by most standards’.
Hutton and Wilson are also critical of Thirlwall’s recommendations for suggesting ‘repeating work that has already been done – some of which was ignored in the Letby case’. For example a 2011 Department of health guidelines on detection and management of outliers which notes on its opening page: ‘The statistical analyses involved should be carried out by people with appropriate statistical expertise and experience’ (here).
Lucy Letby: Scapegoat?
Detectives approached the academic again after Letby had been charged in 2021 and arranged a video call. It was later cancelled. ‘We have had a further meeting this afternoon where we have informed the prosecutors that we were looking at the validity of statistical evidence again in the case,’ an officer emailed. ‘The prosecutor does not agree with our line of inquiry and has instructed us not to pursue this avenue, any further, at present.’
‘The investigation made just about every mistake in the book,’ Hutton says in the new documentary. ‘We know from an early version of the police shift chart I have that there were more places where suspicious events were identified and they’ve been excluded. When they presented the chart that supposedly showed she was always there when babies died they needed to explain – it includes other deaths she was not there for.’
Rob Pattinson claimed that when the Sun began its Letby investigation they ‘set out to show that she had actually done it’. ‘But the more that we’ve done, the more doubt we actually uncovered. And with everything that we found, it’s become increasingly difficult to believe that Lucy Letby killed those babies the way that the court said that she did, and if you take that away, what do you have left…’
In the film the vascular surgeon Dr Colin Ferguson meets with the outspoken prosecution expert Dr Dewi Evans over his now widely discredited theory of murder by injecting babies with air. ‘The way a death from air embolism is commonly diagnosed is by the presence of air in the heart at postmortem, and this can be seen on X-ray or it can be seen physically appearing as foam in a heart that’s examined during a postmortem,’ Ferguson explained.’ In the Lucy Letby case, detailed post-mortems were carried out by specialist neonatal pathologists in six of the seven babies on the murder environment. No air was found in the hearts of any of them.’
In his exchange with Evans, he says: ‘I know that these babies did not die from air embolisms. I know that for certain.’ Evans holds his ground, saying that the cases concerned ‘tiny amounts of air in tiny babies – whether it ended up in the heart in a large enough volume to be shown on X-ray is a matter for debate.’
The programme also features the world’s leading confession expert Dr Gisli Gudjonsson, a forensic clinical psychologist and professor emeritus at Kings College, London, who has been working in miscarriage of justice cases since the 1980s. He revisits the notorious post-it notes which included the seemingly damning scribbled phrases: “I am evil”; “I did this” and “I killed them on purpose because I’m not good enough to care for them and I am a horrible evil person.”’
Gudjonsson argues that the evidence ‘should not be taken as a confession to murder’. ‘When we look at the note it’s very important to look at it as a whole. You can’t just pick up one or two words,’ he said.
‘The impetus for writing these notes didn’t come from her. It came from the occupational health manager who was looking after her healthcare,’ Dr Gudjonsson explained. He suggested to her that her writing notes about her feelings, her thoughts, might help. She was making notes in order to sort out her psychological agony, partly because other people had been blaming her inadvertently or deliberately.’
Support the Justice Gap, buy PROOF
