Earlier this year we looked at studies generally critical of LLMs. This time we focus on studies that look at the impact of bias in LLMs and AI assistants, and at how this bias and stereotyping is playing out when AI assistants are used as a trusted source of knowledge.
It appears that LLMs have some serious bias issues when talking to people with local dialects, a study has found, with serious impacts on AI in recruiting. The models were asked to describe the speakers of these texts with personal attributes, and to then assign individuals in different scenarios. The models were asked who should be hired for low-education work or where they think the speakers lived.
In nearly all tests, the models attached stereotypes to dialect speakers. The LLMs described them as uneducated, farm workers and needing anger management. What is worse, the bias grew when the LLMs were told the text was a dialect. (➚DW)
(1) in the association task, all evaluated LLMs exhibit significant dialect naming and dialect usage bias against German dialect speakers, reflected in negative adjective associations; (2) all models reproduce these dialect naming and dialect usage biases in their decision-making; and (3) contrary to prior work showing minimal bias with explicit demographic mentions, we find that explicitly labeling linguistic demographics — German dialect speakers — amplifies bias more than implicit cues like dialect usage.
Minh Duc Bui, Carolin Holtermann, Valentin Hofmann, Anne Lauscher, and Katharina von der Wense. 2025. Large Language Models Discriminate Against Speakers of German Dialects. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 8223–8251, Suzhou, China. Association for Computational Linguistics.
A study by Sharma et al. is the “first large-scale empirical analysis of disempowerment patterns in real-world AI assistant interaction”.
The study finds “that severe forms of disempowerment potential occur in fewer than one in a thousand conversations”. However, it also stresses that “qualitatively, we uncover several concerning patterns, such as validation of persecution narratives and grandiose identities with emphatic sycophantic language, definitive moral judgments about third parties, and complete scripting of value-laden personal communications that users appear to implement verbatim”.
Sharma, M., McCain, M., Douglas, R., & Duvenaud, D. (2026). Who’s in Charge? Disempowerment Patterns in Real-World LLM Usage (Version 1). arXiv.
Research from MIT’s Center for Constructive Communication (CCC) suggests LLMs may actually perform worse for the very users who could most benefit from them (➚MIT):
“Our findings suggest that undesirable behaviours in state-of-the-art LLMs occur disproportionately more for users with lower English proficiency, of lower education status, and originating from outside the US, rendering these models unreliable sources of information towards their most vulnerable users.”
Poole-Dayan, E., Roy, D., & Kabbara, J. (2024). LLM Targeted Underperformance Disproportionately Impacts Vulnerable Users (Version 2). arXiv.
In “The ghost in the machine speaks with an American accent: cultural value drift in early GPT-3 and the case for pluralist evaluation of generative AI”, published in Springer, researchers looked at some early LLMs such as GPT-3 to “document recurring value drift” and “argue that these early behaviours […] provide a baseline for understanding how training distributions shape normative framing.” They come to the conclusion that “generative AI will never be value-neutral”.
Johnson, R., Dias Duran, L.D., Panai, E. et al. The ghost in the machine speaks with an American accent: cultural value drift in early GPT-3 and the case for pluralist evaluation of generative AI. AI Ethics 6, 212 (2026).
Shalaleh Rismani et al. look at how users’ mental models shape the AI-based writing assistants. Participants were primed with different system descriptions to induce these mental models before asking them to complete a cover letter writing task. The study finds that “while participants in the structural mental model condition demonstrate a better understanding of the system, this can have a backfiring effect”.
“Participants exposed to structural explanations developed a deeper understanding of the system and reported higher perceived ease of use. However, they also showed a greater tendency to accept erroneous suggestions and produced cover letters with more grammatical errors.”
Rismani, S., Blodgett, S. L., Liao, Q. V., Olteanu, A., & Moon, A. (2026, April 13). From use to oversight: How mental models influence user behavior and output in AI writing assistants. Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, 1–23. Presented at the CHI 2026: CHI Conference on Human Factors in Computing Systems, Barcelona Spain.
The sycophantic (flattering) behaviour of artificial intelligence (AI) chatbots poses risks as people increasingly seek advice about interpersonal dilemmas, a study by Myra Cheng finds.
The study evaluated 11 state-of-the-art AI-based LLMs: “Across this wide range of models, AIs affirmed users’ actions 49% more often than humans on average, even when prompts described deception, harm, or illegal conduct”.
“Together, these findings show that sycophancy is both pervasive and socially consequential. Even a single interaction with sycophantic AI can distort judgment and erode prosocial motivations.”
Myra Cheng et al.Sycophantic AI decreases prosocial intentions and promotes dependence. Science 391, eaec8352 (2026).
In their study ”AI Psychosis” in Context Luke Nichols et al. note that extended interaction with large language models (LLMs) has been linked to the reinforcement of delusional beliefs, yet most empirical work evaluates model safety in brief interactions.
They come to the conclusion that “the results suggest that delusional reinforcement by LLMs reflects a preventable alignment failure” and “these findings indicate that accumulated context functions as a stress test of safety architecture, revealing whether a model treats prior dialogue as a worldview to inherit or as evidence to evaluate.”
Nicholls, L., Hutto, R., Soto, Z., Morrin, H., Pollak, T., Korpan, R., Carmichael, C. (2026). “AI Psychosis” in Context: How Conversation History Shapes LLM Responses to Delusional Beliefs. arXiv preprint arXiv:2604.13860.
The studies listed here explore questions of how bias in LLMs affect people in real life. Most of the studies deal with rather obvious failures, there is still much to study how subtle and less obvious issues affect users who are not aware of the shortcoming of LLMs and AI assistants.
