AI Does Not Merely Reflect Bias—It Can Compound It

The idea that AI is "neutral" because a machine makes the decision is deeply flawed. AI learns from historical data, and history itself carries human biases and inequalities. The real danger is not that AI invents bias, but that it can automate and scale existing bias while making accountability harder to trace.

In 2024, a federal judge let a discrimination case against Workday (Mobley v. Workday, Inc., No. 23-cv-00770-RFL (N.D. Cal., July 12, 2024))[1] move forward. Workday – one of the biggest names in HR software – was sued due to its AI hiring tool. The plaintiff, Derek Mobley, argued that he was rejected job after job even before a human reviewed his application. However, Workday's defense counter was simple. The company built an AI tool to increase efficiency, and the decision was made by the tool and not any employee of the company. The court did not accept that as a reason to dismiss the case. It allowed the claims to proceed. The court held that Workday could potentially qualify as an agent based on the plaintiff's allegations; it did not establish liability. An opaque algorithm cannot be used as a shield. No matter how the tool reached its conclusion, the responsibility does not disappear just because a machine took the decision.

This points to a much bigger question we still have not answered well. The consequential decision about people's lives is handed over to systems that are trained on our own history. History was never neutral to begin with. The question isn't whether AI can be biased anymore. The harder question is: who is responsible when it is—and how do we prove harm when even the people deploying the system may not understand its reasoning?

How bias actually gets in

AI systems learn by studying patterns. These patterns are based on past human experiences and fed to AI as context. If, in the past, a company had skewed data towards some type of candidates in its hiring process or a bank's past loan approvals were skewed towards specific borrowers, the system does not question it but accepts it as the truth. This decision is based on the historical "normal" behavior. AI optimizes patterns, not judging whether the patterns were fair. This is not out of malice or intent to discriminate. However, this does not eliminate harmful outcomes.

Historical data is not the only way bias creeps in. Sometimes it is inherent in how a system is instructed to define success. Historical data is the most common culprit to blame but treating it as the only one is disregarding everything else. The tricky part is that systems rarely rely on obvious facts such as race or gender directly. Instead, AI picks up on proxies/variables that seem neutral but indirectly correlate with a protected trait. E.g. a graduation year can signal age; a zip code could signal race based on neighborhood communities. The problem is not simply biased data. It is that bias can enter through the data, the design of the system, and even the definition of what the AI is asked to optimize.

It's a bigger issue

Such bias shows up in every domain as the underlying pattern remains the same. In healthcare, a landmark 2019 study published in 'Science' (Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. "Dissecting racial bias in an algorithm used to manage the health of populations.") [2] looked at a classic prediction algorithm, which was used to decide which patients need extra medical care. It used the historical data related to healthcare spending as a stand-in for healthcare needs. Historically less money has been spent treating Black patients than white patients with the very same conditions because of longstanding unequal access to care. This bias reduced the number of Black patients flagged for extra care by more than half. The algorithm systematically underestimated how sick they were. Researchers showed that this was fixable by reformulating it to predict actual illness instead of cost.

Lehigh University indicated an even stranger issue with the usage of Generative AI [3]. GPT-4 Turbo evaluated mortgage applications with identical financial profiles based on applicants' race. It's worth noting that this was an experimental study and no loan was disbursed based on this study. Black applicants were approved at lower rates and were offered worse terms. The gap widened for "riskier" applicants, the ones with lower credit scores or higher debt. The researchers also tested using other leading models side by side and found bias varied significantly between them. This is a reminder that AI bias is not a fixed one; it relies heavily on the system and context that is input.

This variability is also reflected during the hiring of candidates. When researchers used leading models, including GPT models, Gemini, and Claude, to screen resumes, the models favored some demographic groups over others [4]. The pattern was not uniform, i.e., bias towards white female candidates grew stronger amongst the top-scoring ones, while bias against Black male candidates was worse among the lower-scoring ones. Bias can shift shape depending on who's already closest to getting the job. Bias does not remain constant. It can change depending on the model, the context, and even how close someone already is to a favorable outcome.

The part nobody's fully solved

The very process used to make today's chatbots feel helpful is also fueling the bias. Modern AI models are refined using human feedback. People rate which of the model's responses are "better," and the model learns to produce more responses like the ones people prefer. But the issue is that "better" and "unbiased" are not the same. If the responses people rate higher happen to carry subtle biases, the model learns to reproduce the bias right along with the quality – the two get tangled together in the training data. A 2026 study by Hahm, Hadfield-Menell & Lee presented at the ICML conference tested this and found that the tangling is not inevitable [5]. The amplification did not happen when bias and quality were not correlated in the underlying data.

In the same 2026 study, the mitigation methods indicated various pointers on the bias versus quality spectrum. One method (WARM) barely reduced bias at all but maintained high response quality. Two other methods (InfoRM and RRM) reduced bias further, but at the cost of quality. Retraining the model repeatedly on its own feedback reduced bias with each round, but quality gains slowed down. None of the methods can eliminate bias completely. Mitigation research is still going on. Anyone claiming otherwise is oversimplifying a genuinely open problem.

Regulators are also playing catch-up, and instead of addressing the problem in its totality, they're piecemealing a response. There is no comprehensive federal AI law, although existing federal civil-rights and consumer-protection laws still apply. The European Union has had the most ambitious AI law, but in a provisional May 2026 agreement it pushed back the broader compliance deadlines for high-risk AI systems. This was not just one bias specific rule, but the full set of obligations covering something like documentation, risk management and human oversight for tools used in hiring, credit scoring and other high-risk applications until December 2027 and AI embedded in regulated products such as medical devices to fully comply by August 2028[6].

In the U.S. New York City's Local Law 144 has been in force since 2023 requiring independent bias auditing of any automated hiring tool every year such as an Automated Employment Decision Tool (AEDT). This tool uses machine learning or similar techniques to substantially assist or replace human decision making in hiring. The results are made available to the public [7]. California's rules are more fragmented where businesses making significant decisions about consumers by use of any automated decision-making technology (ADMT), must provide notice, opt-out rights or human appeal option with compliance required by January 2027. Also, larger qualifying businesses, especially those meeting specific revenue or data processing thresholds, must complete annual cybersecurity audits and risk assessments, with timelines stretching from 2028 to 2030[8].

All this does not imply AI decisions are completely untrustworthy, or that AI is not capable of automating tasks. There are some well-audited algorithms that have reduced bias considerably. The fact remains: there is still a human in the loop (HITL). Is there a human who can actually review a decision if pattern-based decision outcomes suggest bias? HITL doesn't necessarily resolve the problem. A human reviewer only catches biased decisions if three things are actually true: they have sufficient knowledge to detect that something is wrong, they have the real authority to override the system's output, and there is a genuine process for reconsidering or appealing the decision.

AI reflects the systems, data, and choices behind it and those choices should remain open to scrutiny. Most of the knowledge comes from historical data and that history was never perfect or neutral to begin with. Being a passive subject of an algorithm's decision does not have to be the only option. Asking relevant questions, pushing back when decisions appear skewed and expecting real answers is what will drive the reliability of these systems over time.

References

1. Mobley v. Workday, Inc., No. 23-cv-00770-RFL (N.D. Cal., July 12, 2024) — ruling on motion to dismiss. https://caselaw.findlaw.com/court/us-dis-crt-n-d-cal/116378658.html

2. Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). "Dissecting racial bias in an algorithm used to manage the health of populations." Science, 366(6464), 447–453. https://www.science.org/doi/10.1126/science.aax2342

3. Lehigh University News, "AI Exhibits Racial Bias in Mortgage Underwriting Decisions" (originally published August 2024; updated April 2026). https://news.lehigh.edu/ai-exhibits-racial-bias-in-mortgage-underwriting-decisions

4. An, J., Huang, D., Lin, C., & Tai, M. (2025). "Measuring gender and racial biases in large language models: Intersectional evidence from automated resume evaluation." PNAS Nexus, 4(3), pgaf089. https://academic.oup.com/pnasnexus/article/4/3/pgaf089/8071848

5. Hahm, D., Hadfield-Menell, D., & Lee, K. (2026). "Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases." Accepted at ICML 2026. https://arxiv.org/html/2605.27355v2

6. Gibson Dunn, "EU AI Act Omnibus Agreement — Postponed High-Risk Deadlines and Other Key Changes" (May 2026). https://www.gibsondunn.com/eu-ai-act-omnibus-agreement-postponed-high-risk-deadlines-and-other-key-changes/

7. NYC Department of Consumer and Worker Protection, "Automated Employment Decision Tools: Frequently Asked Questions" — Local Law 144 of 2021 (effective January 1, 2023; enforcement began July 5, 2023). https://www.nyc.gov/assets/dca/downloads/pdf/about/DCWP-AEDT-FAQ.pdf

8. California Privacy Protection Agency, "CCPA Updates, Cybersecurity Audits, Risk Assessments, Automated Decisionmaking Technology (ADMT), and Insurance Regulations" (September 2025; general effective date January 1, 2026, employer ADMT obligations phasing in by January 2027). https://cppa.ca.gov/announcements/2025/20250923.html