Bias In UK Welfare Fraud Detection AI Sparks Concerns Over Fairness

The UK government’s artificial intelligence system for detecting welfare fraud has been found to exhibit bias based on age, disability, marital status, and nationality, according to internal assessments, as reported today by The Guardian.

The system, used to assess universal credit claims across England, disproportionately flags certain groups for investigation, raising fears of systemic discrimination, said The Guardian.

The bias, described as a “statistically significant outcome disparity,” was revealed in a fairness analysis conducted by the Department for Work and Pensions (DWP) in February.

The analysis found that the machine-learning program selected people from some demographic groups more frequently than others when determining who should be investigated for potential fraud, reports The Guardian.

This disclosure contrasts with the DWP’s earlier claims that the AI system posed no immediate risks of discrimination or unfair treatment.

The department defended the system, emphasizing that final decisions are made by human caseworkers and arguing that the tool is “reasonable and proportionate” given the estimated £8 billion annual cost of fraud and errors in the benefits system, reported The Guardian.

However, the analysis did not explore potential biases related to race, gender, religion, sexual orientation, or pregnancy, leaving significant gaps in understanding the system’s fairness.

Critics, including the Public Law Project, accuse the government of adopting a “hurt first, fix later” approach, calling for greater transparency and safeguards against targeting marginalized groups, as reported by The Guardian.

“It is clear that in a vast majority of cases the DWP did not assess whether their automated processes risked unfairly targeting marginalised groups,” said Caroline Selman, a senior research fellow at the Public Law Project,as reported by The Guardian.

The findings come amid increasing scrutiny of AI use in public services. Independent reports suggest that at least 55 automated tools are in operation across UK public authorities, potentially affecting decisions for millions, says The Guardian.

Yet, the government’s official AI register lists only nine systems, revealing a significant oversight in accountability, says The Guardian.

Moreover, the UK government is facing criticism for not recording AI use on the mandatory register, sparking concerns about transparency and accountability as AI adoption grows.

The DWP redacted critical details from its fairness analysis, including which age groups or nationalities were disproportionately flagged. Officials argued that revealing such specifics could enable fraudsters to manipulate the system, noted The Guardian.

A DWP spokesperson emphasized that human judgment remains central to decision-making, stating, as reported by The Guardian. The revelations add to broader concerns about the government’s transparency in deploying AI, with critics urging stricter oversight and robust safeguards to prevent misuse.