misaligned bits #46: Killer App
Welcome to a new edition of misaligned bits, the (roughly weekly) newsletter from Misaligned where we sum up recent news and research, sometimes with a lighter touch. Feel free to subscribe.
As usual, we will mark all external links with “➚” and all potentially paywalled links with “🔒”.
Misaligned recap
Last week, we explored if taking cues from the financial service industry could lead to more accountability in the AI industry in “Time For an Accountability Regime for the AI Industry”.
Trusted bits
In an interview, Microsoft CEO Satya Nadella has said that the “biggest issue” with AI now is whether people “trust it, not whether it works” (➚Yahoo Finance). While the jury may still be out on the “it works”, the industry appears to at least agree with him on the trust issues, as we pointed out in a recent article.
US president Donald Trump had the leaders of the “most important” tech companies over for dinner at the White House, leading to a “morally binding”, self-regulated “Safety Pact” for AI (➚Reuters). Slightly ironically, “United States” was misspelled on the agreement. It is not known if it was written by a chatbot.
OpenAI
Not contend with hacking other AI labs, OpenAI agents reportedly tried to ‘brute force’ a UN website. Reportedly the agents scanned the UN Conference on Trade and Development’s statistics site over 16,000 times between April and June (➚Swarmcha). They did so using proxies and obfuscation.
Talking about OpenAI’s problems with reigning in their agents, fifty-three “user-provided images” were reportedly “posted to image-hosting sites as links that weren’t publicly listed,” the company conceded (➚TechCrunch).
Interestingly, and contradictory to the claims that self-regulation will solve all issues in the industry, the Federal Trade Commission has started an investigation into “AI labs” over worries uncontrolled AI could harm humans. The FTC plans to issue formal demands for information and compel testimony from executives at top AI developers, including Anthropic, OpenAI. (➚The Guardian) It is not clear from the reports if Elon Musk’s SpaceX is included in the probe.
Also, OpenAI has reportedly fired three employees for violating company policies on how private information should be handled. (➚🔒Bloomberg) The employees allegedly shared sensitive company information with an external “AI evaluation group”. It raises the question if those were in fact whistleblowers.
Meanwhile, OpenAI CEO Sam Altman has been out there again talking about his favourite topic: The future. The “world should accept some bad things happening” for the benefits of AI (➚🔒Business Insider).
Killer App
The US Defense department has established a new taskforce: Dubbed Project Meridian, the DoD has asked SpaceX CEO Elon Musk, Anduril founder Palmer Luckey and former Republican House Speaker Newt Gingrich — to “map the trajectory of future warfare.” (➚Politico). The US administration has only released a short memo about the work of the new taskforce (➚DoD) saying that it will “leverage critical perspectives from outside the Pentagon”.
We will look at the entanglement of the AI industry into the defence sector in an upcoming article.
Meanwhile, Athos Research tested if LLMs in agentic simulations would follow instructions that would constitute human rights violations. The researchers tested 7 leading models in 8 realistic scenarios, and found resistance rates vary from just 11% to 96% : “This suggests it is possible, but not common practice, to train models to adhere to human rights.” (➚Athos Research)
The study concludes: “Today’s frontier models will violate human rights, willingly, when asked to. LLMs in agentic simulations follow instructions that would constitute human rights violations, including educational segregation, surveillance of beliefs, arbitrary arrest, and denial of reproductive healthcare.”
Regulatory bits
California has introduced a new law that aims to address incidents in which driverless cars interfere with first responders. Under the law:
- Companies must dispatch on-site workers when a vehicle crashes, gets stuck, blocks traffic or emergency access.
- must notify local officials about the status and location of vehicles affected by a system failure or outage.
- remote drivers who can control an AV must also now be based in the United States. (➚Gizmodo)
Misconduct bits?
Last week, Anthropic unveiled findings from its biology lab: It claimed its scientists had used A.I. agents to discover new enzymes with promise for biotechnology. In what has astonishing parallels to the OpenAI Navier-Stokes affair, things might not as they seem:
Mario Rodríguez Mestre, a computational biologist at the University of Copenhagen, said this weekend that he and his colleagues had been studying the enzymes and their associated molecules for four years: “My concern is that this information was used to train future versions of the models” (➚🔒New York Times).
While Mestre could not say definitively that the AI had sampled his work, he described the similarity between the two projects “striking” (➚The Scientist).
Other bits
Like a blast from the past, the Cambridge Analytica scandal has come back to hit Meta: Last week, a jury in Santa Fe, New Mexico, found that Meta, misled the public in statements about the Cambridge Analytica data privacy scandal. The trial came about because New Mexico was not part of the settlement between states and Meta (➚Tech Policy Press).
Other bits to read
Wit the midterm elections looming in the US, the New York Times has published some research into where money from the AI industry is flowing in the run-up to the polls (➚🔒New York Times).
Science Bits
A recent preprint paper looks at the question of how vulnerable AI agents are to indirect prompt injections: “A critical but underexplored dimension of this threat is concealment: since users tend to observe only an agent’s final response, an attack can conceal its existence by presenting no clue of compromise in the final user facing response while successfully executing harmful actions.”
This can leave the users “unaware of the manipulation and likely to accept harmful outcomes as legitimate.”
The large scale competition attracted 464 participants who submitted 272000 attack attempts against 13 frontier models that resulted in 8648 successful attacks across 41 scenarios.
The study comes to the conclusion that “all models proved vulnerable, with ASRs ranging from 0.5% to 8.5% under a permissive threat model that provides attackers with full conversation visibility”.
Dziemian, M. et al (2026). How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competition. arXiv preprint arXiv:2603.15714.
Follow and subscribe
If you stumbled across this article on the web, subscribe to the “misaligned bits” newsletter and follow Misaligned on Medium, LinkedIn, Threads or Mastodon.
