Bad decisions
While machine learning and algorithms are brilliant in some areas they do not function well in uncertain environments such as those involving human behaviour.
That is known and well understood and has been proven in both academic studies and repeatedly in real life, where there have been devastating impacts on children and their families.
Indeed, the total failure of machine learning and big data in predicting risk should make us seriously question the fundamental assumptions the attempt is based on. These are the idea that an aim of social work should be to predict future risk and that it is useful to ‘flag’ children and their families based on different characteristics.
Robust evidence shows that even vast amounts of children’s information did not help identify the children who needed support. This failure must make us reject not only the flawed machine learning systems but also the constant push for ‘more data’.
Using these highly inappropriate systems not only leads to harm for children and families involved but it also further shapes society and social work towards policing families rather than providing universal support. The only winners in this system are those who stand to gain financially.
“The broader justice implications for replicating, rather than remedying, both social inequities and the known problems with child protection decisions in child protection system data are only just beginning to be fully understood.”
Dr Emily Keddell
Social Sciences: Algorithmic Justice in Child Protection: Statistical Fairness, Social Justice and the Implications for Practice (2019)
“We hear about highly accurate data and powerful predictions not always because they actually exist, but because these stories need to be told to sell products and stay in business.”
“Seldom discussed is how often the system is wrong, or how many families are falsely flagged as requiring intervention – and what effects these false positives have on affected families, and for the system.”
Professors Joanna Redden and Lina Dencik, and Harry Warne
Policy Studies: Datafied child welfare services: unpacking politics, economics and power (2020)
“The broader justice implications for replicating, rather than remedying, both social inequities and the known problems with child protection decisions in child protection system data are only just beginning to be fully understood.”
Dr Emily Keddell
Social Sciences: Algorithmic Justice in Child Protection: Statistical Fairness, Social Justice and the Implications for Practice (2019)
No amount of data helps
Research into the ability of machine learning models to predict life outcomes by hundreds of researchers using data from the Fragile Families study showed that the problem of machine learning in ‘understanding’ families is not one which can be solved by more data.
“Despite using a rich dataset and applying machine-learning methods optimized for prediction, the best predictions were not very accurate and were only slightly better than those from a simple benchmark model.”
Professor Matthew Salganik et al
Proceedings of the National Academy of Sciences: Measuring the predictability of life outcomes with a scientific mass collaboration (2020)
“The study really highlights that at the end of the day, machine-learning tools are not magic”
Alice Xiang, head of fairness and accountability research at Partnership on AI, quoted
Karen Hoa
MIT Technology Review: AI can’t predict how a child’s life will turn out even with a ton of data (2020)
It is widely accepted that AI is poor at risk prediction but it has often been suggested that perhaps the poor results are because we don’t have enough data and therefore need more. An American study of the predictive ability of different risk prediction models blew this idea out of the water.
The ‘Fragile Families and child well-being study’ tracked the children of 4,000 families from birth, resulting in a huge amount of data on each child. Almost 13,000 data points.
Hundreds of researchers – with different programmes, algorithms and models – were asked to predict six life outcomes. None succeeded and the most complicated machine-learning techniques did no better than far simpler methods.
The results were not good. The best model missed 4 out of 5 children considered ‘at risk’ and wrongly flagged 3 out of 5 children not considered in need of help.
“Our findings and the findings of the 160 teams suggest that it is very challenging to build models to predict outcomes well in children’s social care.”
What Works for Children’s Services
Machine Learning in Children’s Services summary report (2020)
“Our findings and the findings of the 160 teams suggest that it is very challenging to build models to predict outcomes well in children’s social care.”
What Works for Children’s Services
Machine Learning in Children’s Services summary report (2020)
“We can’t predict the future — that should be common sense. But we seem to have decided to suspend common sense when “AI” is involved.”
Professor Arvind Narayanan
Princeton University: How to Recognise AI Snake Oil
Big data and AI amplify problems
Big data and machine learning amplify the existing bias and inequality present in society and do enormous harm.
Firstly they are a distraction to seeking societal wide solutions and they are based on a misunderstanding of what risk means in an individual rather than population wide context. But also by using existing data to train from they literally amplify existing bias. Perhaps the clearest way to understand this is to see how machine learning produces biased imagery.
“We argue for policy debates to go beyond technical fixes and privacy concerns to engage with fundamental questions about the power dynamics and rights issues linked to the expansion of data sharing in this sector as well as whether predictive data systems should be used at all.”
Professors Joanna Redden and Lina Dencik, and Harry Warne
Policy Studies: Datafied child welfare services: unpacking politics, economics and power (2020)
Bias in ‘training’ data
As systems use existing data to train from they literally amplify existing bias. One of the clearest way to understand this is to see how machine learning produces biased imagery.
“Artificial intelligence image tools have a tendency to spin up disturbing clichés. These stereotypes don’t reflect the real world; they stem from the data that trains the technology. Grabbed from the internet, these troves can be toxic — rife with pornography, misogyny, violence and bigotry.”
Nitasha Tiku, Kevin Schaul, Szu Yu Chen
Washington Post: These fake images reveal how AI amplifies our worst stereotypes (2023)
“…judgments reflect the biases in the data-sets on which [machine learning] systems are trained, for example – which can make the technology an amplifier of inequality, racism or poverty.“
Professor John Naughton
Observer: From viral conspiracies to exam fiascos, algorithms come with serious side effects (2020)
Marginalised groups use public services more
Big data and machine learning works with the data it has. Marginalised groups have to interact more with public services, both because they need support such as around health and housing and because of entirely unwanted focus from authorities, such as stop and search of young Black men. These interactions mean that they are the ones whose data is in the programmes used and from which algorithms calculate risk factors.
“…poverty profiling targets individuals for extra scrutiny based not on their behaviour but rather on a personal characteristic: living in poverty. Because the model confuses parenting while poor with poor parenting, [it] view[s] parents who reach out to public programs as risks to their children”
Professor Virginia Eubanks
Automating Inequality (2018)
“In addition to these biases inherent in police data, individuals from disadvantaged sociodemographic backgrounds are likely to engage with public services more frequently, meaning the police often have access to more data relating to these individuals, which may in turn lead to them being calculated as posing a greater risk.”
Drs Alexander Babuta and Marion Oswald
Royal United Services Institute: Data Analytics and Algorithmic Bias in Policing (2019)
Biased policy amplified
Not only is the government using AI for areas it is poorly suited – human behaviour and risk prediction being an area it is very weak on – but there is a predictable pattern to who they use it on and the aim of such measures.
For example, while there is evidence that citizens have been harmed by the Department of Welfare and Pension underpaying claimants, with a high level of appeals found in claimant’s favour and also that tax evasion by corporations loses the government billions the focus of the government is instead on tracking possible fraud committed by benefits claimants. Not only have the figures on fraud been exaggerated by including error and ‘failure to provide evidence/fully engage in the process’ but there are also serious risks of harm to the people being incorrectly flagged.
Systems have been repeatedly shown to be flawed and there are serious risks of those who are entitled to support not receiving it because of errors. Examples of such fundamental errors include the failures in the Horizon system which was behind the Post Office scandal, England’s exam grades scandal blamed by then PM Boris Johnson on a “mutant algorithm”, the Dutch welfare scandal.
“Algorithmic fairness cannot be understood solely as a matter of data bias, but requires careful consideration of the wider operational, organisational and legal context, as well as the overall decision-making process informed by the analytics.”
Drs Alexander Babuta and Marion Oswald
Royal United Services Institute: Data Analytics and Algorithmic Bias in Policing (2019)
“Digital technologies are employed in the welfare state to surveil, target, harass and punish beneficiaries, especially the poorest and most vulnerable among them.”
Professor Philip Alston
UN Special Rapporteur on extreme poverty and human rights
Extreme poverty and digital welfare (2019)
“This means tens of thousands of people have been denied social security they were entitled to for months on end – all because of government error.”
Anela Anwar chief executive Z2K
Quoted in Guardian: ‘Unforgivable’: UK government errors denied thousands disability benefits (2022)
Reshape policy around aims of big data
Sharing information about children has become an aim in itself, increasingly unconnected to the question of what is good for children or their families. Identifying those who are poor and struggling has become an aim in itself, with little questioning over the fact that the help available to them once identified is very limited, or that this is distracting from the fact that struggle is increasingly widespread.
“The austerity agenda has been linked to long-standing neoliberal logics that promote reduced state services and individual versus collective responsibility. The use of data systems to target services reinforce neoliberal logics by individualizing social problems and directing attention away from structural causes of problems.”
Professors Joanna Redden and Lina Dencik, and Harry Warne
Policy Studies: Datafied child welfare services: unpacking politics, economics and power (2020)
“Risk scores impose a vision of the importance of contextual factors, even when the statistical correlation with maltreatment is barely significant.”
Dr Emily Keddell
Social Sciences: Algorithmic Justice in Child Protection: Statistical Fairness, Social Justice and the Implications for Practice (2019)
“…the digitization of welfare systems has been accompanied by deep reductions in the overall welfare budget, a narrowing of the beneficiary pool, the elimination of some services, the introduction of demanding and intrusive forms of conditionality, the pursuit of behavioural modification goals, the imposition of stronger sanctions regimes, and a complete reversal of the traditional notion that the state should be accountable to the individual.”
Professor Philip Alston
UN Special Rapporteur on extreme poverty and human rights
Extreme poverty and digital welfare (2019)