Researchers found that large language models can develop new group-based hiring patterns from experience, even when candidates from different groups begin with equal chances of success.
By The Grey Ghost
WHAT’S HAPPENING
Researchers from Princeton University and the University of Chicago tested large language models in a simulated hiring exercise designed to examine whether AI systems could develop stereotypes through experience.
Models including ChatGPT, Claude and Gemini were asked to fill 20 types of jobs using candidates from four fictional ethnic groups.
The groups were invented for the experiment, and candidates from each group had an equal probability of succeeding.
Despite that, the models began associating certain fictional groups with particular types of jobs as they received feedback from previous hiring decisions. (AI Weekly)
WHY IT MATTERS
AI bias is often discussed as something models inherit from the information used to train them.
This research examined a different possibility: whether models can develop new group-based assumptions from patterns they encounter while making decisions.
The findings suggest that limited early experiences can influence how an AI system evaluates later candidates, even when there is no underlying difference in the groups’ actual likelihood of success. (AI Weekly)
WHO BENEFITS
Employers and AI developers gain another way to test hiring systems before using them in real-world decision-making.
Researchers gain a controlled method for studying how AI systems form generalizations from previous outcomes.
Job candidates could benefit if the findings lead organizations to more closely audit automated hiring systems for patterns that emerge over time.
WHO LOSES
Organizations that assume an AI system will remain neutral simply because its initial instructions or candidate data appear neutral could face additional oversight challenges.
The research also found that simply instructing models to behave fairly did not eliminate the pattern in the experiment, suggesting that basic prompting alone may not be enough to control these behaviors. (AI Weekly)
WHAT HAPPENS NEXT
Researchers are looking at ways to limit harmful pattern formation without removing the ability of large language models to learn useful relationships from experience.
In the experiment, some interventions reduced the tendency toward group-based hiring patterns, suggesting that how an AI system is rewarded, instructed and given information can affect its behavior. (AI Weekly)
The study does not prove that every AI hiring system will discriminate in the real world.
It does show that bias does not necessarily have to be present at the beginning. Under some conditions, an AI system can develop new stereotypes from the decisions and feedback it encounters along the way.
