Suggesting glue as a pizza topping or recommending eating a rock a day for health benefits may have been the highlight of generative AI failures in 2024 but it was also symptomatic for a series of products rushed to market. It may hint at fundamental issues with the development culture of Big Tech companies who long have outsourced quality management to their user base.
In 2024, no developer conference of any of the Big Tech companies could do without the introduction of some new generative AI service. Google presented a flurry of features at its developer conference in May last year¹ , most importantly adding generative AI into its core product: Google Search².
Google Search has long been on the slippery slope of integrating more and more content into what once was a list of links. What started with Wikipedia snippets (which are coming from human-curated articles) a few years back expanded to an often slightly nonsensical, sometimes extremely off-topic questions and answers section. The new feature, dubbed “AI Overview”, attempts to deliver information (normally found in the provided links) directly to the user, powered by Google’s Gemini AI model, and appears prominently on top of the traditional search results (at first for US users, then to more countries from October 2024 ³).
One would think that no one has a data corpus at hand better suited for this than Google. Google has been crawling millions of websites and extracting text data from them for decades, and while Google has been rather numb about how much text/data quality plays a role in search ranking, it is easy to expect that the company has become rather good at it.
Only two months before the rollout of “AI Overviews”, in February 2024, Google announced it had struck a deal with Reddit⁴ to use their content for generative AI training, one of the first large training data for cash deals that have since then become more common.
The test rollout of Google’s “AI Overview” did not go smoothly: Users quickly discovered a plethora of false information, most famously the recommendation to use “glue as a pizza topping”⁵, the advice that humans should eat “at least one small rock per day”⁶ and the “fact” that former US president Obama was the “first Muslim president” of the United States⁷.
Some of those “AI Overviews” (like the glue pizza recipe) could be directly tracked back to a specific comment in a subreddit⁸ clearly derived from the acquired dataset.
The examples above are hilarious because their are obvious. But this is not the issue. The issue here is the uncountable number of misinformed overviews that are not obvious but instead present subtle mistakes. And the fact that Google does not seem even have guardrails for sensitive topics such as health (i.e. in the “rocks” incident), politically charged misinformation (i.e. the Obama example) is deeply worrying.
In a prominent recent case OpenAi’s ChatGPT falsely named a Georgetown University researcher as the accused in a sexual harassment case⁹. A mismatch like this would certainly not be noticed by most users and identifying it requires vast amounts of cross referencing.
Meanwhile Apple took a more conservative path when rolling out their new AI features for ios and macos, adding summaries to users’ push and email — not without issues either: In one recently reported instance the AI informed a user with “Terrorism resurgence. Trump plans legal challenge. Nikki Glaser killed at Golden Globes“.¹⁰ (Note: Nikki Glaser of course is alive and well).
The problem in Google’s case are not AI hallucinations, rather it is the result of a botched approach to model training and a lack of data quality control and oversight.
The Reddit dataset is obviously fundamentally different from Wikipedia data. Insider jokes, sarcasm, internet slang, memes and irony — all common on Reddit — not to mention content by hordes of playful bots, do escape self-supervised training.
Reddit data is not a source of reliable information, nor do upvotes on Reddit carry any relation to correctness: “give me the answer that was popular on Reddit” is not identical with “give me a correct answer”. This seems trivial but appears to have been misunderstood by Google.
Large Language Models (LLM) by nature are well suited for summarizing text or retrieve “some” information from a large corpus. Their ability to create a reasonable chain of tokens (text) that looks like it has been written by a human makes their output easily consumable. But while they are good at summarizing a source it does not mean they are good in evaluating the source.
LLMs are on the other hand extremely bad for identifying sarcasm or irony unless a wider context is known or it is precisely labelled, a fact that is not helped that a defining feature of sarcasm is that it sounds plausible unless context is known.
Google’s subsequent handling of the “glue pizza” incident and other issues with the “AI Overviews” is telling: Some users who tried to reproduce the “AI Overview” results found that the queries quickly resulted in no overview at all, a clear sign that certain search keywords or queries had simply been exempt from the generative AI feature.
Considering that there are possibly tens of thousands cases similar to the “glue pizza” that could be derived from e.g. the Reddit dataset this feels like a hopeless enterprise: It’s playing whack-a-mole with whatever bug goes viral, and while it achieves a short term goal it does not tackle the underlying misconception and fundamental issues.
The enormous cost of training recent models which in turn demands their widest possible rollout, even where inappropriate
It’s easy to blame the enormous cost of training recent models which in turn demands their widest possible rollout to retain any chance of a return on the investment, and fear of missing out in an over-hyped market but it is worth looking what product development culture at the large AI service provider is prevalent:
Large language model based services — when provided responsibly —demand a different quality cycle:
If 2024 is symptomatic for hasty roll-outs of unfinished and untested products, it might get only worse in 2025. Cost pressure will certainly increase. Meanwhile in the European Union the AI Act that has come into force — but it will remain to be seen what change in development culture that will cause in jurisdictions outside the EU.
¹ Google rolls out AI-generated, summarized search results in US — The Guardian
² I will go into the details of how web search engines have changed into knowledge engines — and not in a good way — in an upcoming article.
³ Google’s AI Overview is rolling out worldwide — Mashable
⁴ Reddit in AI content licensing deal with Google — Reuters
⁵ Google promised a better search experience — now it’s telling us to put glue on our pizza — The Verge
⁶ Casey Newton on Threads : “at least one rock per day”
⁷ Google Search’s AI falsely said Obama is a Muslim. Now it’s turning off some results — CNN
⁸ Google Is Paying Reddit $60 Million for Fucksmith to Tell Its Users to Eat Glue — 404 Media
⁹ It’s not a lie, says generative AI: untruths, defamation and the use of ChatGPT — Lexology
¹⁰ Chris Messina on Threads : “Apple Intelligence and idiomatic expressions don’t mix.”
¹¹ Meta drops fact-checking, loosens its content moderation rules — Techcrunch