Artificial intelligence was supposed to be the great equaliser, a tool immune to human prejudice. Instead, researchers and communities are discovering that it can encode and amplify the worst of our biases, at breathtaking scale.
In 2015, a Black software engineer named Jacky Alciné noticed something alarming. Google Photos, the tech giant’s newly launched image-recognition app, had automatically labelled photos of him and his girlfriend as ‘gorillas’. Google apologised, called it ‘appalling and inexcusable’, and promised a fix. The fix, it later emerged, was to remove the word ‘gorilla’ from the classifier altogether, along with ‘chimp’ and ‘monkey’. The model hadn’t been taught to see Black faces correctly. It had simply been taught not to see certain things at all.
That incident (embarrassing, painful, and significant) was an early public signal of a problem that computer scientists and civil rights advocates had been warning about for years: artificial intelligence does not arrive into the world neutral. It is built by people, trained on data generated by people, and deployed into societies shaped by centuries of unequal power. The biases embedded in those foundations travel with it.
A decade on, the scale of that problem has grown exponentially. AI now determines who gets a job interview, who receives a loan, who is flagged by a predictive policing algorithm, and, increasingly, who is diagnosed with a medical condition. The stakes could not be higher.
What Is Algorithmic Bias, And Where Does It Come From?
Algorithmic bias refers to systematic errors in AI outputs that create unfair outcomes for particular groups, typically along lines of race, gender, age, disability, or socioeconomic status. It can emerge at almost any point in the development pipeline.
Training data is perhaps the most common culprit. AI systems learn patterns from historical data, and when that data reflects historical inequalities, as it almost always does, the model learns to reproduce those inequalities as if they were natural laws. A hiring algorithm trained on a company’s past successful employees will learn to favour candidates who look like the people that company historically promoted: often white, often male. Amazon famously scrapped such a tool in 2018 after discovering it was downgrading CVs from women.
Facial recognition provides some of the starkest examples. Research published by MIT Media Lab scholar Joy Buolamwini, who founded the Algorithmic Justice League, found that several leading commercial facial recognition systems had error rates of up to 34.7% for darker-skinned women, compared with just 0.8% for lighter-skinned men. The technology was primarily trained on faces that looked like the people who had historically been overrepresented in tech and academic datasets: overwhelmingly white, overwhelmingly male.
“The coded gaze reflects the priorities of those who code it. If you’re not at the table, you’re likely to be on the menu.”, Joy Buolamwini, MIT Media Lab
The High-Stakes Domains Where Bias Does Real Harm
The consequences of biased AI are not merely theoretical. In the United States, the use of predictive risk assessment tools in the criminal justice system (used to determine sentencing, parole, and bail) has been extensively scrutinised. An investigation by ProPublica found that one widely used tool, COMPAS, was nearly twice as likely to falsely flag Black defendants as future criminals compared to white defendants. The tool’s designers disputed the analysis, but the underlying controversy, using opaque algorithmic scores to help determine human liberty, has not gone away.
Healthcare is another arena where AI bias carries life-or-death implications. A landmark 2019 study published in Science found that an algorithm used by US hospitals to allocate healthcare resources to high-risk patients was significantly less likely to identify Black patients as needing additional care, despite their often having more complex health conditions. The reason? The algorithm used healthcare costs as a proxy for healthcare need, but because Black Americans have historically had less access to care and therefore spent less on it, the model systematically underestimated their needs.
In employment, mortgage lending, insurance pricing, and targeted advertising, similar dynamics play out, often invisibly, in systems that neither the people affected nor, sometimes, the companies deploying them fully understand.
The Problem of ‘Black Box’ Accountability
Part of what makes algorithmic bias so difficult to address is the opacity of many AI systems. Machine learning models, particularly large neural networks, can be extraordinarily complex. Their decision-making processes are often not readily interpretable even by the engineers who built them. When a loan application is rejected or a job candidate is filtered out, there may be no clear explanation. This creates what critics call the ‘black box’ problem: consequential decisions made by systems that cannot adequately account for themselves.
The EU’s General Data Protection Regulation (GDPR) introduced a right to explanation; the idea that individuals have a right to meaningful information about automated decisions that affect them significantly. But implementation is patchy, and experts argue that many explanations offered by companies fall far short of genuine transparency.
Regulation is beginning to catch up. The EU AI Act, passed in 2024, introduces risk-based obligations for AI systems used in high-stakes domains, including requirements for bias testing and human oversight. In the United States, the picture is more fragmented, with sector-specific guidance rather than comprehensive federal legislation.
Why This Is Everyone’s Problem
It is tempting to see AI bias as primarily a problem for the communities most immediately harmed by it, and those harms are very real and should be centred. But there are reasons why all of us have a stake in getting this right.
As AI systems become embedded in more areas of public and private life, from public service delivery to healthcare to the platforms that shape our information environment, the question of who they serve and whose assumptions they encode becomes a question about what kind of society we want to live in. Systems that are less accurate for large swathes of the population are also, by definition, less reliable systems. They are also systems that erode public trust, which is increasingly a precondition for the social licence AI companies need to operate.
The good news, if it can be called that, is that this is not an intractable problem. Bias in AI can be measured, audited, and mitigated; if there is the will and the investment to do so. Diverse development teams produce more robust systems. Inclusive datasets lead to more equitable models. Independent auditing creates accountability. None of this is impossible. It simply requires prioritising it.
What is clear is that the moment for claiming that this is a fringe concern, or a problem for later, has long passed. The decisions being made now (about how AI systems are built, tested, and governed) will shape the experience of billions of people for decades to come. That is a responsibility too important to leave only to the people currently in the room.