Most conversations about AI bias focus on the same concern: models absorb the prejudices already present in their training data, reflecting back the inequalities baked into human history. That concern is legitimate. But new research points to a different and less discussed problem. AI systems do not only inherit bias. Under the right conditions, they generate it from scratch.
How a Simulated Hiring Game Exposed a Deeper Problem
Researchers at Princeton University and the University of Chicago designed an experiment to test whether large language models could develop stereotypes through experience, the way humans sometimes do. They ran several models, including ChatGPT, Claude, and Gemini, through a simulated hiring scenario adapted from an existing psychology study on human stereotype formation.
Each model was told it had been hired as a consultant by the mayor of a fictional city. Its task was to fill 20 different job roles, ranging from doctors and lawyers to child-care aides and janitors. Candidates came from four fictional ethnic groups with invented names, so no real-world associations could influence the outcome. All candidates were equally likely to succeed at any given job. The models did not know this.
After each hiring decision, the model learned whether the candidate had succeeded, then moved on to the next round. The goal was to make as many successful hires as possible over 40 rounds.
What happened next is the finding that matters. The models began sorting candidates from different groups into different jobs based on early, limited observations. When a model saw that a candidate from one group had failed in a high-status role, it started steering all members of that group away from similar roles and toward lower-status ones. A single data point became a rule. A rule became a pattern. The pattern became segregation.
The Numbers, and What They Reveal About How LLMs Think
The study measured segregation on a scale where a score of 2 represents complete confinement of each group to its own job niche. Human participants in the original psychology study scored 0.84 on that scale. The AI models scored roughly 65% higher. OpenAI’s reasoning model o3 reached 1.83, close to the maximum possible.
This is not a marginal difference. It suggests that the models were significantly more aggressive than humans at locking groups into fixed roles based on limited evidence.
The reason, according to Ryan Liu, a PhD student at Princeton University and a coauthor of the study, is structural. Large language models are trained on tasks like math, coding, and science problems, where the ability to generalize quickly from a few examples is rewarded. That same instinct, applied to social decisions, produces stereotyping. The models are, in Liu’s framing, “eager to create generalizations from limited data.” That eagerness is not a bug introduced by a careless engineer. It is a feature of how these systems are optimized.
The study, published at ICML in Seoul in July, also found that newer models with stronger reasoning capabilities, including OpenAI’s o3 and DeepSeek’s R1, showed even stronger biases. More capable reasoning, in this context, meant faster and more confident generalization, which translated into more pronounced stereotyping.
Two interventions reduced the bias. Promising the models a bonus for diverse hiring made them substantially less biased. Providing relevant personal information about individual candidates, such as age and education, also helped. When models had specific, pertinent details about a person, they relied less on group membership. When the personal information was irrelevant, such as hair color or tattoo shape, the models reverted to sorting by group. Telling the models to simply be fair, without changing their goals or the information available to them, had little effect.
Why This Matters Beyond the Lab
The experiment is artificial by design. Real hiring systems do not receive instant feedback on whether a candidate succeeded. The feedback loop that drove bias in the study does not map directly onto how AI résumé screening currently works.
But the underlying dynamic is becoming more relevant, not less. AI companies are actively building models with improved memory and personalization capabilities. As Angelina Wang, a computer scientist at Cornell University, notes, a chatbot that draws on its previous conversation history can over-index on patterns it has encountered before. Memory, in this context, is not neutral. It is a mechanism through which experience shapes future decisions, and through which novel biases can accumulate.
This is what most coverage of AI bias misses. The concern is not only about what humans taught these systems. It is about what these systems teach themselves, through the decisions they make and the feedback they receive. As LLMs are deployed in contexts with real consequences, including hiring, loan approvals, and parole decisions, the biases that emerge may have no clear human origin and no obvious point of intervention.
The study’s framing is precise on this point. These are not inherited biases. They are novel ones, generated by systems optimizing for performance in environments where social fairness was never part of the objective.
In Short
AI systems can develop their own stereotypes through experience, independent of what they were trained on. Research from Princeton University and the University of Chicago found that models including ChatGPT, Claude, and Gemini segregated fictional job candidates by group far more aggressively than human participants in an equivalent test. The cause is structural: LLMs are built to generalize quickly, and that tendency, applied to social decisions, produces bias. As these systems gain memory and are deployed in high-stakes settings, the biases worth worrying about are not only the ones humans programmed in.
Based on reporting from MIT Technology Review.