Public sector ‘AI’ fail
Unsurprisingly – since too often ‘AI’ is being used in inappropriate ways to do things which are outside the realms of possibility – there are numerous instances of the systems being abandoned. Sometimes this is quietly, early on in the project, officials having interrogated the systems properly and consulted with the public. Other times the end has been in flames. Destruction of people’s lives, high profile job losses and public inquiries.
We need to listen to the evidence and the clear warnings which case studies from around the world tell us. While ‘AI’ involves very powerful computing and mathematics it cannot possibly work for uses such as risk prediction of people. Yet as authorities buy into the narrative of ‘doing more for less’ it is likely that we will see increasing numbers of these imploding systems.
“…the benefits are all speculative, all of the things that are talked about how they’re gonna make society better. I would say, fantasies. Whereas the harm that these things are able to do already evidenced, even in the minimal ways that real AI has been deployed in the world.”
Dr Dan McQuillan
Tech Won’t Save Us podcast: Why We Must Resist AI (2023)
“…findings suggest that good practice by government in datafied societies is tied to a politics of care, which involves taking the time to carefully consider the use of these systems before implementation.”
Professor Joanna Redden and Jessica Brand, Ina Sander and Harry Warne
Data Justice Lab and Carnegie UK: Automating Public Services: Learning from Cancelled Systems (2022)
“…the benefits are all speculative, all of the things that are talked about how they’re gonna make society better. I would say, fantasies. Whereas the harm that these things are able to do already evidenced, even in the minimal ways that real AI has been deployed in the world.”
Dr Dan McQuillan
Tech Won’t Save Us podcast: Why We Must Resist AI (2023)
‘Welfare cop’ implosion
Australia’s Robodebt scandal
Almost half a million Australians received letters telling them that they had weeks to pay back ‘debt’ creating huge distress and leading to deeply negative life impacts as they tried to comply.
They won a class action in 2019 which forced the government to pay out $1.8 billion in refunds. The Royal Commission set up to investigate was damning in its findings and recommended both civil and criminal proceedings be considered. While there were undoubtedly particular failures in the system, and failures of oversight the commission also criticised the aim of the project given that welfare fraud was only 0.1% of cases.
The UK should take note.
“The harms caused by robodebt are overwhelming, heartbreaking and horrific in both their scope and impact.”
Dr Honni van Rijswijk and Professor Penny Crofts
The Conversation. Explainer: what is the ‘tort of misfeasance’ and how might it apply in the case of robodebt? (2023)
“It is remarkable how little interest there seems to have been in ensuring the Scheme’s legality, how rushed its implementation was, how little thought was given to how it would affect welfare recipients and the lengths to which public servants were prepared to go to oblige ministers on a quest for savings.”
Commonwealth of Australia
Royal Commission into the Robodebt Scheme (2023)
“politicians need to lead a change in social attitudes to people receiving welfare payments. The evidence before the Commission was that fraud in the welfare system was miniscule, but that is not the impression one would get from what ministers responsible for social security payments have said over the years. Anti-welfare rhetoric is easy populism, useful for campaign purposes.”
Commonwealth of Australia
Royal Commission into the Robodebt Scheme (2023)
High-profile Dutch scandal led to vital privacy ruling
In 2013 the Dutch public sector used an algorithm to detect fraud by families claiming government support for childcare. Statistical correlations between various characteristics were used – without other evidence – to stop families payments and to accuse parents of fraud. Often the data used was false but families had no way of knowing that, or of challenging the errors as the government had removed any human oversight. Approximately 30,000 were accused, often being told to repay large sums.
Political fallout
A 3.7 million Euro fine was levied and the Prime Minister and his entire cabinet were forced to resign over the matter.
In 2020 – in a ruling with worldwide impacts – the District Court of the Hague ruled that the use of risk profiling such as SyRI, the system used in Holland, is unlawful as it violates the right to privacy.
“…core aspects of the right to private life encompass the right to a personal identity and protection against discrimination and stereotyping in the context of data processing.”
Professor Adamantia Rachovitsa and Niclas Johann
The Human Rights Implications of the Use of AI in the Digital Welfare State: Lessons Learned from the Dutch SyRI Case (2022)
Learning from abandoned systems
While the examples above are some of the most dramatic examples of failures there are a much larger number of systems which are abandoned for various reasons, which we can learn from.
Research from the Data Justice Lab is a valuable account of the harms which can be inflicted but also of trying to map why and at what stage systems have been abandoned. The researchers make recommendations for how public authorities can avoid the many pitfalls, focused around meaningful consultation and engagement, transparency and examining evidence.
“Our research identifies 61 ADS that have been cancelled in various countries. This research, along with efforts to map government uses of ADS, are an indication of how quickly the use of ADS is being introduced across public sectors. Algorithm Watch have highlighted that systems of oversight are not being adapted at the same pace as these new changes.”
Professor Joanna Redden and Jessica Brand, Ina Sander and Harry Warne
Data Justice Lab and Carnegie UK: Automating Public Services: Learning from Cancelled Systems (2022)
“There is a need to shift the burden of proof. By this, we mean that those introducing ADS should be required to prove that they have investigated the potential harms that could occur, but also fundamentally that they are able to demonstrate the effectiveness of the changes they are implementing.”
Professor Joanna Redden and Jessica Brand, Ina Sander and Harry Warne
Data Justice Lab and Carnegie UK: Automating Public Services: Learning from Cancelled Systems (2022)
“….a politics care approach is one that involves: ensuring time is taken to consult and investigate if and how ADS should be used; consultation involves extended stakeholders including affected communities; provides the potential for meaningful engagement and public review which includes being responsive to criticism to emerge as well as the option to refuse use, early on. This kind of effort, ideally, would lead to necessary improvements, cancellation or decisions about no go areas before scandal, harm and public protest later on.”
Professor Joanna Redden and Jessica Brand, Ina Sander and Harry Warne
Data Justice Lab and Carnegie UK: Automating Public Services: Learning from Cancelled Systems (2022)