Large language models mirror human biases only partially and sometimes invert them
Research report. Peer-review status is not recorded in the source records below; consult the linked publication.
Researchers evaluated social bias in eleven large language models using factorial survey experiments based on four different English and German experiments.
The large language models mirrored human biases only partially and sometimes inverted them.
While vignettes reduce social desirability bias in humans, this effect does not occur for large language models, where alignment-induced tendency towards socially desirable responses persists.
Researchers caution against applying bias measurement techniques from the social sciences developed for humans to measure attitudes and bias in large language models.
Sources behind this report
Analysis
Why it matters
Factorial survey experiments use descriptions of socially complex situations to elicit judgments such as social bias.
The context
The method reduces the effect of social desirability in self-reported social biases and makes it possible to separate effects of multiple sociodemographic attributes.
LLMs mirror human biases only partially and sometimes invert them, with alignment-induced tendency towards socially desirable responses carrying over directly while vignettes reduce such bias in humans.
Report history
No earlier revision dates are retained here. This does not establish that the report has never changed; read any dated updates and corrections.
Corrections
No corrections have been appended.