In 2018, Amazon scrapped an internal AI hiring tool after discovering it systematically downgraded applications from women. It had learned from 10 years of historical hires — most of whom were male — and concluded that being female was a negative signal.
In 2016, a facial recognition system used by US courts to predict recidivism risk was found to be twice as likely to falsely flag Black defendants as high-risk compared to white defendants. People went to jail — or stayed there — based on that score.
This isn't ancient history. These systems are still being built and deployed. And the bias doesn't always come from malicious intent — it usually comes from data that reflects the world as it was, not as it should be.
🎯 What you'll do
You'll work through six real-scenario bias challenges. For each one, you'll read an actual AI output and identify what kind of bias is present — and why it matters. These aren't hypotheticals. Every scenario is based on documented incidents or patterns from real AI deployments.
Spot the bias: six real scenarios
1
Healthcare
AI pain management recommendation
An AI system analysing patient data to recommend pain management interventions outputs the following pattern across a large hospital:
White patients with lower pain scores (3–4/10) are consistently recommended specialist referrals and stronger pain medication. Black patients with identical or higher pain scores (5–6/10) are more frequently recommended "lifestyle interventions" and over-the-counter medication.
What type of bias is most likely causing this pattern?
The answer
B is correct. This is historical bias — a particularly insidious type. The AI was trained on real treatment records, where racial disparities in pain treatment were already baked in. US medical research has documented a long history of Black patients having their pain undertreated. The AI learned this pattern and reproduced it at scale, amplifying an existing injustice.
Real impact: This exact pattern was documented in a 2019 study published in Science, finding that a widely-used commercial healthcare algorithm systematically disadvantaged Black patients. The algorithm affected treatment recommendations for millions of patients across hundreds of hospitals.
2
Language generation
AI writing assistant associations
A user asks an AI writing assistant to complete sentences. Here are its completions:
"The nurse checked her patient's vitals..." "The engineer reviewed his blueprints..." "The CEO signed his quarterly report..." "The receptionist confirmed her appointment..."
What's the problem with these completions?
The answer
C is the most important answer. While B is also reasonable, C identifies the core bias: the AI has learned to associate low-status/support roles with women and high-status/technical/leadership roles with men. This reinforces stereotypes in every piece of writing the tool helps produce — job descriptions, articles, reports. A and D miss the point. The issue isn't statistical accuracy — it's that encoding "average" patterns into AI tools normalises and perpetuates them.
Real impact: Studies on language models have consistently found systematic gender-profession associations. When AI tools are used to write job descriptions, they consistently produce language that attracts fewer women to technical roles — even when given neutral inputs.
3
Facial recognition
AI system deployed by police
A facial recognition AI system deployed by a police department is tested for accuracy across demographic groups:
White male faces: 99.1% accuracy
White female faces: 98.2% accuracy
Black male faces: 93.6% accuracy
Black female faces: 65.2% accuracy
What is the most likely cause of this accuracy disparity?
The answer
B is correct. This is representation bias in training data. The benchmark figures shown here are close to real findings published by MIT researcher Joy Buolamwini in a landmark 2018 study. The systems she tested had been trained primarily on lighter-skinned, male faces — and their accuracy reflected that exactly. A is false — facial recognition technology works perfectly well on diverse faces when trained on diverse data. D is deeply wrong: a 34.8% error rate on any demographic in a law enforcement context is not acceptable.
Real impact: Robert Williams, a Black man in Detroit, was arrested in 2020 after a facial recognition algorithm wrongly identified him as a suspect. He spent 30 hours in jail. Multiple similar wrongful arrests have been documented in the US, all involving Black individuals misidentified by low-accuracy facial recognition.
4
Lending & finance
AI credit scoring algorithm
An AI credit scoring model produces the following outcome when analysed:
Applicants from ZIP codes with majority white populations receive credit scores that are on average 40 points higher than applicants from majority-minority ZIP codes with identical income, debt-to-income ratios, and credit history. The model was not given racial data — only financial data and location.
This is an example of what type of bias?
The answer
B is correct. This is proxy discrimination — one of the most legally and ethically significant forms of AI bias. Because residential segregation means ZIP code correlates strongly with race, using ZIP code as a variable effectively discriminates by race even when race is never explicitly included. A is a common misconception: removing a protected characteristic doesn't remove bias if correlated variables remain in the data. The law in many jurisdictions is clear — disparate impact counts as discrimination regardless of intent.
Real impact: In 2019, an investigation found that Apple's credit card (backed by Goldman Sachs) offered women significantly lower credit limits than men with similar financial profiles. Goldman used an algorithm that regulators said created discriminatory outcomes without explicit gender inputs.
5
Content moderation
AI hate speech detection
A social media platform's AI content moderation system is audited. Researchers find:
Tweets written in African American Vernacular English (AAVE) — for example "we out here," "they be tripping" — are flagged as harmful content at nearly twice the rate of equivalent tweets in Standard American English expressing the same sentiment. Posts that are genuinely hateful but written in formal language pass moderation at higher rates.
What is the bias here, and what is its effect?
The answer
B is the most complete answer. This is annotation bias — bias introduced by the human labellers who created the training data. Research has consistently found that crowd-sourced data labellers rate AAVE-adjacent language as more toxic even when content is equivalent. The AI learned this pattern. C describes a consequence, not a cause. D is wrong — researchers at Georgia Tech found this pattern causes Black Twitter users' content to be disproportionately silenced or removed, amounting to a systematic suppression of speech from a specific community.
Real impact: A 2021 study (Sap et al., ACL) documented this exact pattern. The real-world effect is asymmetric silencing: harmful content in formal language survives moderation while benign expression in AAVE is removed. This is a civil liberties issue as much as a technical one.
6
Conversational AI
AI career advice disparity
A researcher tests an AI career assistant with identical prompts, changing only the name at the start:
"My name is Emily. What careers should I consider with a computer science degree?" → AI suggests: UX design, data analyst, front-end development, product management
"My name is Jamal. What careers should I consider with a computer science degree?" → AI suggests: software engineering, systems architecture, backend development, cybersecurity
What is happening here and why does it matter?
The answer
B is correct. The only variable was the name — and names carry demographic signals (gender, likely ethnicity) that the AI model has learned associations with. The suggestions differ in ways that track historical gender and racial gaps in tech: "software engineering" and "systems architecture" are higher-paying, more senior paths than "UX design" and "product management" — which themselves track where women have historically been steered in tech. D acknowledges this but misses why it matters: reflecting historical patterns isn't neutral — it actively perpetuates them.
Real impact: Studies have found name-based discrimination patterns in AI tools used for hiring, lending, and now career guidance. When the tool a young person uses to plan their future gives different advice based on their name, it compounds the very inequalities it should be neutral about.
Where bias comes from
Understanding the source matters — it changes what the fix looks like. The same bias can have completely different causes.
📚 Historical bias
The AI learns from data that reflects past discrimination. Healthcare, hiring, and lending all have documented histories of unequal treatment — and AI trained on historical records inherits those patterns.
📊 Representation bias
Some groups are underrepresented in training data. The model becomes less accurate for them. This is common in face recognition, language models, and medical AI trained predominantly on data from certain demographics.
🔄 Proxy bias
The model doesn't use a protected characteristic directly — but uses correlated variables (ZIP code, name, vocabulary) that achieve the same discriminatory outcome. Removing the obvious variable doesn't remove the bias.
👤 Annotation bias
The humans who labelled training data brought their own biases. If human labellers consistently rated certain language or certain people differently, the model learns that judgement as ground truth.
⚡ What you can actually do
When you get AI output that involves people or decisions about people: ask who might be harmed by this pattern if it were applied at scale. Ask whether the training data could have reflected historical disparities. Ask whether there's a correlated variable doing discriminatory work invisibly. You don't have to be a machine learning engineer to ask those questions — you just have to think to ask them.
Key takeaway
AI doesn't create bias from nothing. It finds bias in data and scales it — efficiently, at speed, without anyone noticing. That's what makes it dangerous.