AI Bias and Fairness: Metrics, Testing, and Regulation
In 2019, researchers at UC Berkeley found that mortgage lending algorithms — systems with no explicit racial variables — charged Black and Latino borrowers higher interest rates than white borrowers with identical credit profiles. The excess cost: $765 million per year. The algorithms weren't programmed to discriminate. They didn't need to be.
AI bias isn't a bug in the code. It's a property that emerges from training data, design decisions, and deployment context. Addressing it requires understanding where bias enters, how to measure it, and what regulators demand.
The Taxonomy of Bias
Saying "the model is biased" is like saying "the patient is sick" — not actionable without specificity.
Historical bias exists before any data is collected. If women were historically underrepresented in engineering roles, a hiring model trained on that data learns gender predicts engineering aptitude. You can't fix historical bias by collecting more data from the same biased process.
Representation bias occurs when training data doesn't reflect the served population. Dermatology AI trained predominantly on light-skinned patients underperforms on darker skin — not because darker skin is harder to diagnose, but because the model hasn't seen enough examples. Groh et al. (2021) documented this exact disparity.
Measurement bias arises when the proxy variable doesn't equally reflect the true outcome across groups. The Optum algorithm used healthcare spending as a proxy for health need, systematically underestimating Black patients' severity — spending measured access, not health.
Aggregation bias happens when a pooled model fails to account for subgroup differences. A diabetes prediction model trained on both Type 1 and Type 2 patients without distinguishing between them performs poorly for both.
Evaluation bias occurs when benchmarks don't represent deployment context. A facial recognition system evaluated on a skewed benchmark shows artificially inflated accuracy.
Each type requires different mitigation. There is no single "debias" button.
Fairness Metrics and the Impossibility Theorem
The impossibility theorem, demonstrated independently by Chouldechova (2017) and Kleinberg, Mullainathan, and Raghavan (2016), proves that three commonly desired fairness properties cannot hold simultaneously for any imperfect classifier when base rates differ across groups:
Calibration (predictive parity): a score of "70% risk" means 70% for everyone. Balance for the positive class (equal true positive rates): each group has equal probability of correct identification. Balance for the negative class (equal false positive rates): each group has equal probability of being incorrectly flagged.
When base rates differ across groups — which they almost always do — you cannot satisfy all three. This is mathematical, not technical. Better models don't escape it.
The practical implication: your organization must choose which fairness definition to optimize for, and document that choice. Responsible AI programs treat metric selection as a governance decision. Demographic parity suits contexts where equal outcome rates matter. Equalized odds fits high-stakes decisions where both error types carry weight. Predictive parity applies when false positive costs dominate. No metric is universally correct.
Testing for Bias: Before and After Deployment
Pre-Deployment
Evaluate performance across all relevant demographic groups and intersectional subgroups. Compare outcomes using your chosen metric. The four-fifths rule (EEOC Uniform Guidelines) provides one reference: selection rates for any group should reach at least 80% of the highest-performing group's rate.
Include adversarial evaluation — inputs designed to expose failure modes along demographic dimensions. Document disparities found and mitigation steps taken.
Post-Deployment Monitoring
Models drift. Populations change. A fair system at launch can become discriminatory over time. Monitor outcome distributions continuously. Set automated alerts for drift in fairness metrics. Schedule periodic revalidation — quarterly for high-risk systems.
The healthcare sector learned this painfully. Clinical AI systems deployed without ongoing fairness monitoring produced documented racial disparities persisting for years before discovery.
Regulatory Requirements
Regulators have moved beyond "consider bias" to imposing specific obligations.
EU AI Act Article 10 requires that high-risk systems' training, validation, and testing data undergo governance practices including "examination in view of possible biases that are likely to affect the health and safety of persons, have a negative impact on fundamental rights or lead to discrimination." Enforceable with penalties up to 15 million euros or 3% of turnover.
CFPB fair lending guidance clarifies that AI models must meet the same fair lending obligations as traditional underwriting. The Winter 2025 Supervisory Highlights identified AI-driven disparities in credit scoring and confirmed that "black box" algorithms provide no exemption from adverse action notice requirements under ECOA and Regulation B. CFPB analysts demonstrated less discriminatory alternatives exist — establishing that failure to seek them constitutes a compliance failure.
NYC Local Law 144 requires annual independent bias audits for automated employment decision tools, with public disclosure of results. Algorithmic accountability legislation following this model is expanding to other jurisdictions.
From Awareness to Practice
Operational fairness means embedding bias testing into development workflows as a required step. Choose fairness metrics at design, not evaluation. Document rationale and thresholds in governance records. Monitor production with the urgency you bring to performance monitoring.
AI ethics statements mentioning fairness without specifying metrics, testing cadence, or accountability are governance in name only. The organizations that take fairness seriously can show test results, explain metric choices, and demonstrate what changed when disparities were found. Transparency about these processes builds the trust that vague commitments never will.
Build fairness testing into every stage of your AI lifecycle. Start with Starkguard or request a demo to see how automated assessments surface bias before it surfaces in production.