AI Text Watermarks Can Change How Models Respond to Harmful Prompts, Study Finds

Google’s SynthID-Text watermarking system may affect more than the detectability of AI-generated text. New research suggests that watermarking can also change how large language models respond to harmful requests—and, in some cases, how AI agents use tools.

How SynthID-Text watermarking works

One of SynthID’s key techniques is tournament sampling. The system evaluates a large number of possible next-word tokens and assigns each a probability score using a private key. Tokens compete in multiple rounds, with the highest-scoring token advancing until a final token is selected.

Because the private key influences token selection, the watermarking process can subtly alter the model’s generated response. More information about tournament sampling is available here and here.

Watermarks changed harmful-request behavior

Siposova tested SynthID-Text’s “undistorted” configuration with an unmodified version of Hugging Face’s SynthIDTextWatermarkLogitsProcessor. The experiment used harmful prompts against six open-weight models, comparing responses generated with and without watermarks.

The results showed that watermarking changed responses to malicious requests, particularly when the prompts also used prompt-injection techniques. In some models, watermarking increased the likelihood that the system would answer a harmful request it would normally reject.

“Watermarks change rejection behavior for harmful requests, but the effect is more pronounced when the same requests are combined with prompt injection techniques,” Siposova wrote. “For some models, watermarking increases the likelihood that the model will respond to harmful requests that it would normally deny.”

Why the effect matters for AI agents

The issue extends beyond the model’s text response. AI agents rely on sampled tokens to decide which tools to invoke and which arguments to pass. A small change in token selection can therefore alter the agent’s actions, even when the model’s overall accuracy score changes only slightly.

The researchers describe this phenomenon as “drifting”: watermarking can influence both what a model says and what an agent does. Prompt injection connects the two problems, because weakening a model’s refusal behavior becomes more consequential when the model can operate through external tools.

Different private keys produced different results

The researchers also found that model behavior varied depending on the private key used by the watermarking system. Some keys increased harmful compliance compared with the unwatermarked case, while others decreased it.

Study limitations

The research did not test how Claude’s responses change under watermarking. Instead, the researchers tested six open-weight models with access to token sampling that could be enabled or disabled during tournament sampling while other settings remained fixed.

The experiment also examined Hugging Face’s implementation of SynthID-Text tournament sampling, rather than the specific implementation used by Claude.

Watermarking needs safety testing, not just detection testing

Despite these limitations, the findings suggest that some watermarking approaches can affect both model safety and AI-agent behavior. Red-team exercises should therefore test whether SynthID and similar systems behave as expected when deployed, especially in applications that use prompt-sensitive models to call external tools.