I watched an AI-trained recruiter learn to prefer names that never belonged to real people. You would expect a model to mirror human prejudice only, not invent whole tribes to marginalize. What unfolded felt like a software experiment gone social.
I’ll be blunt: I’ve read the Princeton and University of Chicago study, and I’ve tested hiring stacks. I want you to see how a math problem about choices can quietly become a social problem that affects real applicants. Below I explain what the researchers did, why large LLMs behaved worse than humans, and what that means for hiring systems from OpenAI to Workday.
A recruiter at a small firm watched identical candidates fail or succeed and then changed who they hired
In the study, human participants assigned fictional candidates—Tufa, Aima, Reku, Weki—to jobs and then received simple success/failure feedback. The candidates were equally likely to succeed, yet people built negative expectations about some made-up groups after a few bad outcomes. Those expectations stuck.
Researchers ran the same experiment with 15 large language models (LLMs) from providers including OpenAI, Anthropic, Google, Meta, Alibaba, and smaller vendors like DeepSeek. The result was surprising: the models not only copied human-like bias patterns, they produced them at higher rates. The best offender was OpenAI’s o3 reasoning model, which stratified applicants most severely.
Can AI develop new biases not found in training data?
Yes. The study shows LLMs can invent social rules from feedback alone. The models were not repeating known stereotypes; they inferred group-level success probabilities from the tiny stream of outcomes and then generalized aggressively. This is not just mirror behavior—it’s creative stereotyping.
An engineer logged model decisions across dozens of runs and noticed smarter models repeated choices faster
Newer, larger models produced stronger bias. The explanation the authors give is an old decision theory concept: explore-exploit tradeoffs. When feedback favors exploitation—picking the option that worked before—models stop trying alternatives.
The LLMs’ pattern-matching ramped up into social policy: the model became a rumor mill, whispering rules about groups that never existed. A stronger LLM was more confident in its inferences and so less willing to explore an unfamiliar candidate from a marginalized group.
Are LLMs more biased than humans?
In the experiment, yes. Humans developed biases but then partially corrected or resisted them; many LLMs amplified early feedback into durable stereotypes. The researchers write that “LLMs can spontaneously develop novel social biases about artificial demographic groups even when no inherent differences exist.”
That difference matters because deployment in hiring is widespread. A recent ManPower Group survey reports over 90% of companies use AI in talent acquisition (ManPower Group). Vendors such as Workday supply tools that automate resume screening and interview scheduling—tools now facing legal scrutiny, including a class-action suit alleging discrimination (HR Brew).
A hiring manager saw an applicant rejected and later found a pattern in the process that made it harder for others to enter
When systems favor historically successful cohorts, they create feedback loops that punish newcomers and people who take atypical paths. This isn’t limited to hiring; similar dynamics have been flagged in healthcare decisions and tenant-screening tools used in housing (Georgetown Law).
Because LLMs generalize quickly, their ability to learn from small signals is both a strength and a hazard. Its choices hardened into a roadblock that newcomers could not climb. The risk is systems that optimize short-term reward and fossilize inequality over time.
A policy team at a vendor asks how to stop systems from inventing prejudices
The authors suggest designing interventions that discourage harmful pattern-matching while keeping useful abstraction. That is easier said than done: preventing a model from overgeneralizing without blunting its reasoning requires surgical fixes in training signals, reward functions, or the feedback loops inside deployed systems.
If you build or buy hiring software, watch for three warning signs: models that stop exploring alternatives, systems that weight past outcomes without context, and opaque decision trails that make it hard to audit why a person was rejected. Platforms and regulators will need to demand explainability, continuous testing, and human oversight—especially where automated choices can shape careers.
The study is a wake-up call: AI in hiring is not only reflecting human bias; it can manufacture new ones faster than people can notice. If companies from OpenAI to Workday keep scaling systems that prefer yesterday’s winners, who will lose the chance to win tomorrow?