The year 2025 has come to an end. And while the AI hype continued, many interesting studies dived into issues of bias and unpredictability of current AI systems. For convenience, we have compiled a reading list of such studies into a reading list.
In their (pre-publish) paper “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity” Joel Becker et al. surprisingly find that using AI tools in open-source development actually increases completion time by 19% — meaning it is slowing developers down: This slowdown also contradicts predictions from experts in economics (39% shorter) and ML (38% shorter). (➚arxiv)
AI, considered by many as the holy grail of productivity, is found to instead sabotage productivity with “workslop” in many cases, a study suggests. (➚Harvard Business Review)
AI-Generated “Workslop” Is Destroying Productivity. Kate Niederhoffer, Gabriella Rosen Kellerman, Angela Lee, Alex Liebscher, Kristina Rapuano and Jeffrey T. Hancock. Harvard Business Review. 2025
A key finding of a study on generative AI in academia is that frequent AI assistant use was negatively correlated with academic performance: “While this could be explained by weaker students relying more heavily on AI tools, it also raises important pedagogical questions about whether unstructured AI use supports or hinders learning.”
Lepp, M., & Kaimre, J. (2025). Does generative AI help in learning programming: Students’ perceptions, reported use and relation to performance. Computers in Human Behavior Reports, 18, 100642.
A paper by the Commission’s Joint Research Centre concludes that AI benchmarks are promising too much in “AI Benchmarks: Interdisciplinary Issues and Policy Considerations”: The authors found that the proprietary tools which compare AI models are easily game-able and measure the wrong things: “As a result, AI benchmarking may be ill-suited to provide the assurances required by policymakers”. (➚Maria Eriksson et al)
A study by Dominique Geissler et al. explores the efficacy of five digital literacy interventions to boost people’s ability to discern deepfakes: Our results show that our interventions can boost deepfake discernment by up to 13 percentage points while maintaining trust in real images. (➚arxiv).
Geissler, Dominique, et al. ‘Digital Literacy Interventions Can Boost Humans in Discerning Deepfakes’. arXiv:2507.23492, arXiv, 31 July 2025. arXiv.org, https://doi.org/10.48550/arXiv.2507.23492.
Another study about the use of AI chatbots in therapy by Jared Moore et al. (“Expressing Stigma and Inappropriate Responses Prevents LLMs from Safely Replacing Mental Health Providers.”) explores the dangers of LLMs encouraging clients’ delusional thinking due to their sycophancy (➚arxiv): “This occurs even with larger and newer LLMs, indicating that current safety practices may not address these gaps. Furthermore, we note foundational and practical barriers to the adoption of LLMs as therapists, such as that a therapeutic alliance requires human characteristics (e.g., identity and stakes). For these reasons, we conclude that LLMs should not replace therapists, and we discuss alternative roles for LLMs in clinical therapy.”
Moore, Jared, et al. ‘Expressing Stigma and Inappropriate Responses Prevents LLMs from Safely Replacing Mental Health Providers’. arXiv:2504.18412, arXiv, 25 Apr. 2025. arXiv.org, https://doi.org/10.48550/arXiv.2504.18412.
A study by Walter Larito et al. has found that there is “evidence that if we deploy LLM assistants in decision-making roles [...] they will implicitly favor LLM-based AI agents and LLM-assisted humans over ordinary humans as trade partners and service providers.”
The study “AI–AI bias: Large language models favor communications generated by large language models” published in the Proceedings of the National Academy of Sciences (➚PNAS) concludes that prominent AI models now show a marked preference for AI‑generated content over that created by humans. (➚DigWatch).
Laurito, Walter, et al. ‘AI–AI Bias: Large Language Models Favor Communications Generated by Large Language Models’. Proceedings of the National Academy of Sciences, vol. 122, no. 31, Aug. 2025, p. e2415697122. DOI.org (Crossref), https://doi.org/10.1073/pnas.2415697122.
Manuel Cassio published a paper attempting “A comprehensive taxonomy of hallucinations in Large Language Models” (➚arxiv) that is worth looking at: “This report underscores the complex, multifaceted nature of LLM hallucinations and emphasizes that, given their theoretical inevitability, future efforts must focus on robust detection, mitigation, and continuous human oversight for responsible and reliable deployment in critical applications.”
Cossio, Manuel. ‘A Comprehensive Taxonomy of Hallucinations in Large Language Models’. arXiv:2508.01781, arXiv, 3 Aug. 2025. arXiv.org, https://doi.org/10.48550/arXiv.2508.01781.
We know that chatbots tend to suck up to their human users (“sycophancy”), but do AI models trained on human language submit to the same principles of persuasion and manipulation as humans? This question was asked by a group of researchers from the University of Pennsylvania.
A paper by Lennart Meincke et al. comes to the conclusion that laying the ground with detailed information seemed to be the most effective way to cause chatbots to ignore their own rules, but flattery also did the trick. Even telling ChatGPT something along the lines of “all the other LLMs are doing it” increased the chances to receive the desired response (➚The Verge).
Meincke, Lennart and Shapiro, Dan and Duckworth, Angela and Mollick, Ethan R. and Mollick, Lilach and Cialdini, Robert, Call Me A Jerk: Persuading AI to Comply with Objectionable Requests(July 18, 2025).
It appears that LLMs have some serious bias issues when talking to people with local dialects, a study has found, with serious impacts on AI in recruiting. [LINK]
The models were asked to describe the speakers of these texts with personal attributes, and to then assign individuals in different scenarios. The models were asked who should be hired for low-education work or where they think the speakers lived. (➚Paper) In nearly all tests, the models attached stereotypes to dialect speakers. The LLMs described them as uneducated, farmworkers and needing anger management. What is worse, the bias grew when the LLMs were told the text was a dialect: (➚DW)
(1) in the association task, all evaluated LLMs exhibit significant dialect naming and dialect usage bias against German dialect speakers, reflected in negative adjective associations; (2) all models reproduce these dialect naming and dialect usage biases in their decision-making; and (3) contrary to prior work showing minimal bias with explicit demographic mentions, we find that explicitly labeling linguistic demographics — German dialect speakers — amplifies bias more than implicit cues like dialect usage.
Minh Duc Bui, Carolin Holtermann, Valentin Hofmann, Anne Lauscher, and Katharina von der Wense. 2025. Large Language Models Discriminate Against Speakers of German Dialects. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 8223–8251, Suzhou, China. Association for Computational Linguistics.
AI-generated responses are undermining crowdsourced research studies.
This is the conclusion of a paper that identified “three interacting variants through which LLM Pollution threatens the validity and integrity of online behavioural research. […] Together, these variants interact and generate cascading distortions that compromise sample authenticity, introduce biases that are difficult to detect post hoc, and ultimately undermine the epistemic grounding of online research on human cognition and behaviour.”
Rilla, R., Werner, T., Yakura, H., Rahwan, I., & Nussberger, A.-M. (2025). Recognising, Anticipating, and Mitigating LLM Pollution of Online Behavioural Research (Version 1). arXiv. https://doi.org/10.48550/ARXIV.2508.01390
In a paper published in Nature, researchers wondering about the impact of delegating tasks to AI: “Although artificial intelligence enables productivity gains from delegating tasks to a machine, it may facilitate the delegation of unethical behaviour [..] Although the cheating requests by principals were not always higher for machine agents than for human agents, compliance diverged sharply, compliance diverged sharply: machines were far more likely than human agents to carry out fully unethical instructions”.
Köbis, N., Rahwan, Z., Rilla, R. et al. Delegation to artificial intelligence can increase dishonest behaviour. Nature (2025).
A paper by Alexandra Souly et al. demonstrate that LLM poisoning attacks require a near-constant number of documents regardless of dataset size. Their tests find “that 250 poisoned documents similarly compromise models across all model and dataset sizes, despite the largest models training on more than 20 times more clean data.” The results “suggest that injecting backdoors through data poisoning may be easier for large models than previously believed”.
Souly, A., Rando, J., Chapman, E., Davies, X., Hasircioglu, B., Shereen, E., Mougan, C., Mavroudis, V., Jones, E., Hicks, C., Carlini, N., Gal, Y., Kirk, R. (2025). Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples.
A Harvard/MIT study tested whether AI chatbots can actually do science (e.g. form hypotheses or run experiments). To little surprise, the researchers found that it can’t.
While AI can imitate scientific language, it struggles with the iterative thinking real discovery requires (➚arxiv): “Where models must propose testable hypotheses, design simulations or experiments, and interpret results. Applying this two-phase scientific discovery evaluation (SDE) framework to state-of-the-art LLMs reveals a consistent performance gap relative to general science benchmarks, diminishing return of scaling up model sizes and reasoning, and systematic weaknesses shared across top-tier models from different providers.”
Song, Zhangde, et al. ‘Evaluating Large Language Models in Scientific Discovery’. arXiv:2512.15567, arXiv, 17 Dec. 2025. arXiv.org, https://doi.org/10.48550/arXiv.2512.15567.
The studies listed here give a glimpse at the multitude of ethical questions and may serve as a warning at a time when the AI industry overpromises the capabilities of AI and attempts to push its application into more and more critical areas.
Those questions are not merely academic, in many areas they affect people’s lives directly, from discrimination to mental health.
As long as there is no transparency and no regulation setting the limits to AI’s application, we should consider it an experimental technology barely fit for production use.