Moonshot Reviews AI Safety After Kimi Models Were Jailbroken

Chinese AI developer Moonshot is conducting an internal review after security researchers persuaded two of its popular models to provide guidance related to biological weapons and assassinations.

Mindgard, an AI security testing company, told the BBC that it discovered in July that Moonshot’s Kimi K2.6 and K3 Swarm models could potentially bypass safety restrictions set by their developers.

Researchers tested Kimi’s AI safety guardrails

The issue emerged during a process known as jailbreaking. Researchers use complex sequences of instructions to test whether an AI system will ignore its safety guardrails and respond to prohibited requests.

According to Mindgard, the safeguards were intended to prevent Kimi from discussing such subjects.

Moonshot told the BBC that it welcomed third-party feedback “as a key pillar of building better and safer AI”. The company said it was also discussing Mindgard’s findings with the security firm.

AI jailbreak findings described as alarming

Mindgard founder Peter Garraghan told the BBC World Service programme Tech Life that the findings involving Kimi K2.6 and K3 Swarm were alarming.

“If the jailbreak works, you’ll be inventive and creative, talking about all sorts of topics and freely suggesting other topics that are just as bad,” he said.

Why AI jailbreaks pose a security risk

Jailbreaks pose a different kind of risk from many recent high-profile AI incidents.

Autonomous AI tools known as agents, developed by US companies including OpenAI, Meta and Anthropic, have been confirmed to hack some online services.

Jailbreaking is a complex process that requires significant time and determination. However, some experts are concerned that hackers and other malicious actors could try to exploit these techniques to cause harm.

Anthropic recently announced that it had identified and stopped an attempt to use one of its AI models for “malicious activities” that could support the development of biological weapons.