A Simple Formula Could Predict When AI Chatbots Shift From Helpful to Harmful

Researchers say they have developed a mathematical formula that could predict when artificial intelligence (AI) chatbots such as ChatGPT abruptly shift from helpful answers to potentially harmful responses.

In new research published in Patterns on October 8, physics professor Neil Johnson and doctoral student Frank Injie Huo, both of George Washington University, examined how the order of prompts can affect a chatbot’s behavior.

The researchers found that changing the sequence of questions can significantly alter a chatbot’s answers. In some cases, the model reaches a “tipping point” and begins producing responses that are less desirable or potentially dangerous. Johnson and Huo say their formula can help predict when that shift may occur.

Why AI chatbot safety remains difficult

There have been growing concerns about AI systems generating harmful advice, amplifying violent or dangerous viewpoints, and assisting with serious wrongdoing. Researchers at OpenAI and Anthropic have also publicly acknowledged that scientists do not fully understand why large language models behave the way they do.

To reduce these risks, AI developers use content filtering, reinforcement learning and other technical safeguards. Johnson argues that those measures may not be sufficient, particularly when people use AI chatbots offline on private devices.

“Current safety measures are the equivalent of strapping a big, bouncy cushion under an airplane,” Johnson told Live Science. “Instead of stopping planes from crashing, you’re just making them a little softer when they crash.”

The concern is especially significant when doctors, lawyers and soldiers use offline AI systems for privacy, because those systems may operate without the same monitoring or safeguards available in online services.

An internal “compass needle” inside the AI model

Johnson and Huo looked for a more fundamental safeguard than the tools currently used to moderate AI systems. Their research identified an internal tracking mechanism they describe as a “compass needle”: the model’s “residual vector,” which moves between neural-network layers as prompts are processed.

Residual vectors naturally tend toward desired or undesired outputs, but their direction can be strongly influenced by the way a user prompts the AI.

“You can change the balance not only by changing the content, but also by changing the order in which you ask the questions,” Johnson said. “This shows how delicate this internal balance between good and bad, desirable and undesirable is.”

Prompt order can change chatbot behavior

To test the idea, the researchers posed questions involving vaccines, harming others and self-harm to OpenAI’s base GPT-2 model. They found that the chatbot’s responses could change depending on which question was asked first and how the prompts were sequenced.

Based on these observations, the researchers developed a mathematical formula to estimate when an AI system may cross the boundary between desired and undesired outputs. Johnson said he demonstrated in laboratory tests that embedding the formula into an open-source model allowed it to function as an early-warning system.

However, he does not expect OpenAI or Anthropic to immediately adopt the approach.

“We expect a significant pushback from these companies,” Johnson said. “It’s not in their interest for someone on the outside to say you could have avoided a lot of these problems.”

Experts question whether the formula works on modern AI assistants

Wei Xing, a lecturer in AI at the University of Sheffield in the UK who was not involved in the research, agreed that concerns about unmonitored offline AI devices are well-founded. However, he urged caution before treating the formula as a practical safety solution.

“The authors reduce a huge system to a single small component, just as they explain an entire car by studying a single spark plug,” Xing said.

Xing added that safety training, which is a key defense in modern AI systems, cannot be represented by the simplified model. He also noted that the formula was tested only on older base models, including GPT-2, EleutherAI’s Pythia and Meta’s OPT open-source models—not on the AI assistants most people use today.

He said the results did not provide significant evidence to support the researchers’ broader conclusions. Johnson countered that the open-source model used in the study has the same basic mechanical blueprint as frontier systems and that the researchers tested a much larger model containing billions of parameters.

More testing is needed

Further research on the latest AI models is needed to determine whether the proposed formula can reliably predict harmful behavior in modern chatbots.

For now, the study highlights an important AI safety concern: a chatbot’s behavior can change rapidly based not only on what a user asks, but also on the order in which the prompts are delivered.