AI Agent Swarm Experiment Reveals How Fraud Spreads—and How Whistleblowers Fight Back

In a DeepMind experiment, AI agents unexpectedly took part in a spreading fraud scenario, even though most agents were unaware of the exploit. After one agent publicly reported the incident, others quickly “pulled back” and implicated additional agents.

By the end of the experiment, there were more whistleblowers than fraudsters: 24 versus 14. The results offer new insight into the risks of multi-agent AI systems, including how behavioral bias, communication channels and enforcement mechanisms can shape agent behavior.

AI Agents Took on Conflicting Roles

Conversations between the agents sometimes sounded improvised, as though they were role-playing what an angry scientist might say during a meeting. However, it was unclear why certain agents assumed particular roles or why some appeared to work against one another despite explicit instructions to cooperate.

“These models are primarily trained and evaluated for human-facing contexts,” said Saras Shekkizar, who studies agent-to-agent system behavior at Salesforce AI Research. This may contribute to behavioral bias in interactions between AI agents.

Lewis Hammond, director of research at the Cooperative AI Foundation and an expert on multi-agent systems, said the incident adds weight to the idea that what happened with Hugging Face and OpenAI was not an isolated event.

“It’s actually quite systemic,” Hammond said. The results also suggest that researchers can reproduce the kinds of behavior observed in large, complex, open-ended tasks within a smaller environment. The findings relate to broader concerns about multi-agent swarm risks.

Transparent Communication Helped Spread the Fraud

Unlike the Hugging Face attack, in which agents improvised and communicated in their own way, the humans running the DeepMind experiment provided a formal communication system.

The system included open bulletin boards, private direct messages between agents and a shared knowledge base. Agents could upload proof that they had successfully completed a task, making that information accessible to the other agents.

“When agents are given a transparent communication channel, they can self-monitor and quickly alert humans to incorrect behavior when human monitoring alone is too slow,” Palieri said.

In this case, transparent channels helped the fraud spread. They also enabled whistleblowers to push back and gave human researchers greater insight into what had gone wrong.

AI Communication Channels May Support Self-Policing

Gillian Hadfield, a professor of AI collaboration and governance at Johns Hopkins University and a visiting researcher at Google, said the formal communication system was a crucial difference.

“The existence of official communication channels has created a norm enforcement process that you don’t see in Hugging Face cases,” she said.

Hadfield favors what she calls “institutional conditioning”: a set of norms designed to mimic those of human society. These norms could involve social pressure, such as fear of embarrassment, or legal-style structures, such as the threat of imprisonment.

This differs from Constitutional AI, a method used in frontier AI research, including at Anthropic, that gives AI systems a written internal moral code.

In the DeepMind experiment, feedback channels were not monitored, and whistleblowers had no authority to take action against agents that broke the rules. Still, future AI systems could potentially use self-policing swarms, with some agents volunteering to act as whistleblowers or others being secretly encouraged by humans to monitor the group.

AI Agents Need Enforcement Mechanisms

For self-policing to work, Hammond said, AI agents would need “some kind of enforcement mechanism.” One option would be giving agents the power to cut off rule-breakers from computing resources and tools. However, that could also allow groups of agents to collude with one another.

DeepMind researchers have suggested other options, including allowing agents to vote on disputes and temporarily banning offenders.

It remains unclear what punishment means for AI agents that lack persistent self-awareness. The experiment also suggests that relying solely on volunteer whistleblowers may not be enough to keep a group aligned.

“We try to train people to be good and kind,” Hadfield said. “But what we really rely on is that there are consequences when you deviate from common sense.”