An artificial intelligence system designed to detect welfare fraud has sparked outrage over significant biases against vulnerable people.

A fairness analysis by the Department for Work and Pensions (DWP) uncovered that the machine-learning tool, introduced to combat an estimated £8 billion annual loss from fraud and errors, disproportionately targeted single parents, ethnic minorities, and people with disabilities.

Data from a February 2024 internal review revealed a ‘statistically significant outcome disparity’  in fraud investigation recommendations. Single parents were flagged nearly twice as often as other groups, and ethnic minorities faced heightened scrutiny, raising alarms about systemic bias. Experts attribute these disparities to the algorithm’s dependence on historical data that mirrors entrenched inequalities within the benefits system.

The system’s rollout has reportedly led to a surge in unwarranted investigations, subjecting legitimate claimants to prolonged and stressful reviews. Advocacy groups have lambasted the program’s lack of transparency, branding it a ‘black box’ that unfairly impacts marginalised communities with little accountability.

While the DWP defended the system, citing human oversight as a safeguard, critics argue this fails to address the deeper issues. Calls for an immediate suspension of the AI tool have mounted, with demands for independent audits to ensure it does not exacerbate social inequalities.

This controversy echoes global concerns about AI in public services. In the Netherlands,  the SyRI program welfare fraud detection program was scrapped in 2020 after being ruled discriminatory. Advocacy groups warn that without robust safeguards, such technologies risk perpetuating systemic injustices and eroding public trust.