AI bias: 12 documented examples and what actually causes it

AI bias is not a bug that better engineering simply deletes; it is what happens when optimization meets skewed data and unexamined target variables. Here are the documented cases every practitioner should know, then the five causes that keep producing them.

The canonical cases

• Gender Shades (2018). Commercial face-analysis systems misclassified darker-skinned women up to 34.7% of the time vs. under 1% for lighter-skinned men. The study created the modern algorithmic-audit genre and forced vendor improvements within a year.
• COMPAS (2016). Double the false-positive "high-risk" rate for Black defendants in recidivism prediction — and the proof that fairness metrics conflict mathematically.
• Amazon's recruiting engine (2018). Penalized the word "women's" on CVs after training on male-dominated hiring history.
• Optum's care algorithm (2019). Used healthcare cost as a proxy for need, making Black patients sicker before qualifying for help.
• Apple Card (2019). Gender-correlated credit limits despite gender not being an input — proxy discrimination in action.
• Dutch childcare benefits. Dual-nationality families flagged as fraud risks; a government fell over it.
• Wrongful arrests from face recognition. Documented US cases cluster overwhelmingly among Black men — accuracy gaps plus over-reliance in investigation.
• Mortgage-approval disparities. Investigations found algorithmic lenders more likely to deny minority applicants than white applicants with comparable finances.
• Image generators (2022– ). Early systems rendered "CEO" as white and male, "criminal" as dark-skinned; over-correction then produced its own controversies — demonstrating there is no neutral setting, only choices.
• Speech recognition gaps. Word-error rates for Black American speakers roughly double those for white speakers across major systems (2020 Stanford study).
• Medical devices. Pulse oximeters and dermatology models trained mostly on lighter skin perform worse on darker skin — bias inherited from clinical datasets.
• LLM stereotyping. Large language models associate names, dialects and demographics with occupations and traits in measurably skewed ways — subtler than a wrong arrest, but deployed at billions-of-interactions scale.

The five causes

• Unrepresentative training data. The system has simply seen fewer examples of some groups (Gender Shades, oximeters, speech recognition).
• Historical bias in the labels. The data faithfully records a biased world (Amazon: past hiring; predictive policing: past arrests). Perfectly accurate learning of an unjust pattern.
• Bad target variables. The thing you can measure stands in for the thing you mean (cost ≠ need at Optum; arrest ≠ crime). Often the deadliest and least visible choice.
• Proxy features. Removing the protected attribute does nothing when postcode, shopping patterns or word choice encode it (Apple Card).
• Feedback loops. The model's outputs shape the next round of training data (predictive policing; recommender systems).

What actually works

Per-group error reporting (not just aggregate accuracy), documented datasets and model cards, independent audits with publication rights, choosing target variables with domain experts, and — increasingly — law: the EU AI Act makes bias testing mandatory for high-risk systems from December 2027, and NYC's Local Law 144 already requires annual bias audits for hiring tools.

Fresh bias research lands almost daily — tracked in research → bias & fairness.