The importance of training data for LLM output can’t be overstated. And with the public internet having been crawled to the end, the large model providers exceedingly look for other data sources to add to their corpus. How much do we know about what LLMs are trained on?

Do the companies even know any more, or do they blindly trust their tool chains to sort it out? Should we know? What arguments can be made for a more rigorous transparency? Can generative AI ever be “trustworthy” without it?

This article will argue that total and fine-grained transparency about the sources of training material would greatly benefit the ability to judge models.

Garbage in

In recent weeks, the question “What was your model trained on?” has emerged in two very different cases:

In the “Navier-Stoke affair” over OpenAI’s claim to have solved a longstanding math problem, a mathematician wondered if OpenAI accessed his material on OpenAI Codex for training purposes. When confronted, OpenAI just stated ominously that “can’t rule it out”.

Meanwhile, in a lawsuit against Elon Musk’s SpaceX AI, a plaintiff alleges that Grok was trained using pictures of her abuse to generate new illegal pornographic images (Nick Robins-Early, The Guardian 2026). More generally, the lawsuit alleges that the chatbot has derived its capability to generate non-consensual sexual material by being trained on such material because xAI has no filter on its training material in place:

xAI’s publicly stated datafiltering practices name violent content as an example of what’s excluded from training, but do not specify CSAM, NCII, or NSFW material as excluded categories. (Case 5:26-cv-09016-NC)

Now it could be that OpenAI is just playing it safe and attempts to keep it vague for legal reasons, but it is also possible that the companies have lost the ability to monitor and review training material due to its sheer size and time pressure.

The privacy question

At this point, the publicly accessible web has been largely sucked dry by AI crawlers, which has caused them to a) turn to offline material such as physical books and documents and b) turn their attention to non-public data such as chats or cloud storage.

All large model providers train their models on their clients chatbot conversations by default (at least outside the EU), a feature that users can turn off in their privacy settings, but unlikely to do so. In fact, a survey found that 53% of users are not aware that their chatbot conversations become training material (Valdes 2026).

The motivation behind this “default” setting is obvious, as I can’t think of no reason why someone would deliberately enable this feature without gaining any advantage from it.

It is interesting that all large model providers make exceptions for their business plans: If their tools are so reliable in filtering out personal data, why is there a concern for business? The answer is simply that enterprise customers will have the legal firepower when information leaks, something the average user lacks.

What we want to know, and why?

Model providers should be mandated to provide a complete list of documents used for training (source, type, access time). In addition, companies should be mandated to lay out what percentage of their training data falls into certain categories, such as: Public information, offline information, private information (such as chatbot session, cloud storage etc), and this information should be public to at least a certain extent.

The benefits are plenty:

1. It would tell us a lot more about data governance at the large model providers, and the transparency would in turn put pressure on the providers.
2. It would tell us to what extent privacy is respected, and allow for a detailed assessment if data is appropriately cleaned.
3. It would give outside researchers a change to review the training material for obvious sources of bias, and disclose if model providers deliberately or accidentally train on biased input, poisoned or manipulated data sets.
4. It would help to assess to what extent models are trained on other models output.
5. It would help legal discovery in cases of privacy and copyright
6. Analysing the extent to which “semantic interference” occurs, without relying on a later explanation from the provider.
7. Identifying what weight (by percentage) is given to sources, what impact they have on the overall corpus.

This is not a conclusive list.

Before we deal with the arguments the AI industry will bring up against this, let’s look at current regulations that have (or plan to have) some form of transparency requirements when it comes to training data.

Current regulatory requirements are falling short

The EU AI Act mandates GPAI providers in Article 53.1(d) to “draw up and make publicly available a sufficiently detailed summary about the content used for training of the general-purpose AI model, according to a template provided by the AI Office.”

The template mentioned has recently been specified in the “Explanatory Notice and Template for the Public Summary of Training Content”. It specifies the data as “all sources and types of data, regardless of whether the data are protected or not, including by an intellectual property right.” (EU Explanatory Notice 2026)

[..] this section requires disclosure of the main datasets that were used to train the model, such as large private or public databases, and a comprehensive narrative description of the data scraped online by or on behalf of the provider (including a summary of the most relevant domain names scraped) and a narrative description of all other data sources used (e.g. user data or synthetic data) to ensure completeness of the summary regarding the content used for the model training (EU Explanatory Notice)

While those requirements are extensive, they may not be detailed enough for the purposes we outlined. More importantly, there appears to be no mechanism for anyone outside the regulator to access this information.

The lack of access to a complete list of datasets diminishes the ability of parties with legitimate interests to exercise their rights under EU law, including the protection of copyright and personal data. As a result, this approach would lead to reduced transparency and accountability of GPAI model providers. (Warso 2025)

Furthermore, without an audit of the submitted reports and no information on the impact a source had on the training corpus, the information provides only some but limited insight.

There is no audit. Every flag is self-asserted; nothing external confirms that a “no” on crawled data or user data is accurate. And the template asks whether a source was used, never how much. (Maynard 2026)

Meanwhile, just last month, the Japanese government approved the draft of the “AI Panel’s” rules on disclosing training data. Again though, the focus appears to be on copyright violation, not on data quality or privacy.

More concerningly, the Japanese government appears to favour a “comply or explain” approach, and make the rules a nonbinding code, “allowing them to choose either to comply with the code or publicly explain why they will not comply” (The Japan Times 2026).

It is to be feared that such a quasi-voluntary approach will do little in an industry driven by time and efficiency pressure.

The industry has, and will continue, to push back

So, how much should regulators or independent researchers be allowed to know? The AI industry that has already lobbied against self-assessed reports in the past would surely drag out the same defences against a detailed and public reporting requirement.

• It will argue that the burden of transparency will “stifle innovation”, however accuracy in training data governance can hardly be called anti innovation.
• It will claim that training data sources are a trade secret. That, too, is a questionable argument. While the vast amount of public information has already been accessed by everyone else, the non-public data sources can’t be accessed even if the sources are disclosed. It would therefore not be of much benefit to competitors.

Conclusion

Of course AI regulation should not stop there, but improving on training data transparency would go a long way in accountability.

The EU AI Act moves into the right direction, but the lack of publicly available data and lack of audit is worrying. A system that only looks at the submitted data in case of incidents will do little in an industry that is hungry for data and considers privacy as a probability vector only.

If the black box is to be made transparent, we need to start at the source.