Reproducing Claude's Watermark Locally — SynthID-Text on an Open Model, With Detection and Removal Attacks

Reproducing Claude's Watermark Locally -- SynthID-Text on an Open Model, With Detection and Removal Attacks

Part 1 explained how Claude's text watermark works. This time, instead of explaining, we run it. Claude itself can't be tested -- only Anthropic holds the key -- but the algorithm Anthropic says it adopted, SynthID-Text, is already implemented in Hugging Face `transformers`. So we apply the same algorithm to a local open model (Gemma 2 2B) and measure: how well detection works, how many tokens it needs, and what it takes to erase it.

This is Part 2 of a 2-part series on Claude's watermark.

• [Part 1](/post/claude-watermark-part1-en): How the watermark works -- sampling, the secret key, tournaments, detection, and its limits
• [Part 2](/post/claude-watermark-part2-en) (this post): Applying the same algorithm to a local open model and running generation, detection, and removal attacks

All code is in the companion notebook (claude-watermark-part2-tutorial.ipynb). Everything reproduces in about 30 minutes on a single A100.

1. Experimental Setup

Two reasons for the plain-mean detector. First, it makes Part 1's "fight against 0.5" literally visible. Second, it is more conservative than the paper's trained Bayesian detector, so the detection rates here are a floor. Anthropic's real detection API will do better.

The code is startlingly short. Turning on the watermark is one extra argument to generate().

from transformers import (AutoTokenizer, AutoModelForCausalLM,
                          SynthIDTextWatermarkingConfig, SynthIDTextWatermarkLogitsProcessor)

cfg = SynthIDTextWatermarkingConfig(keys=KEYS, ngram_len=5)   # KEYS: 20 integers
out = model.generate(ids, do_sample=True, temperature=1.0, top_k=40,
                     max_new_tokens=220, watermarking_config=cfg)  # this line

The detector builds a LogitsProcessor from the same key and recomputes g-values. No model needed.

lp = SynthIDTextWatermarkLogitsProcessor(**cfg.to_dict(), device="cuda")

def score(ids):
    g    = lp.compute_g_values(ids).float()                        # [1, T, depth]
    mask = lp.compute_context_repetition_mask(ids).float()[..., None]
    return ((g * mask).sum() / (mask.sum() * g.shape[-1])).item()  # 0.5 = unmarked

2. Result 1: With the Key, Perfect Separation. Without It, Nothing.

The two distributions do not overlap at all. The lowest watermarked score (0.549) sits well above the highest plain score (0.517). At ~200 tokens, a plain mean is enough for perfect separation.

The orange bars are the more important ones. Shift every integer in the key by one and try to detect: the watermarked texts average 0.501, indistinguishable from plain text. Part 1's claim that "detection without the key is impossible" isn't theory -- this is what it looks like. It is why nobody but Anthropic can build a Claude detector.

3. Result 2: How Many Tokens -- and the Trap

We scored only the first N tokens of each text.

Reading only the blue line, "82% at 10 tokens" sounds great. That's half the story. The threshold 0.514 was calibrated on 200-token texts. A 10-token sample has high variance, so 38% of plain texts also cross it. Apply a fixed threshold to short text and false positives explode.

The practical conclusion: the detection threshold must scale with length, and in this setup you need roughly 150 tokens for 1%-false-positive confidence. Pointing a watermark detector at a two-sentence email or a comment is, in principle, a bad idea. The paper's Bayesian detector narrows this considerably but does not eliminate it.

4. Result 3: Lower Temperature, Fainter Watermark

One of Part 1's central claims was that the watermark only lives where the model had a choice. The most direct test is lowering the sampling temperature: candidates collapse to one token, the tournament pits identical tokens against each other, and nothing happens.

Going from 1.0 to 0.3 cuts the signal (the margin above 0.5) from 0.074 to 0.018 -- a factor of four. At 0.3, even 200-token texts are missed one time in four.

Two ways to read this:

• It is exactly the phenomenon Anthropic describes as the watermark being "sparser on factual passages" and code. Lowering temperature and writing text with a single right answer are the same thing from the watermark's point of view.
• An API user calling with temperature=0 leaves essentially no room for a watermark. Whether Anthropic enforces a floor on temperature internally is not stated in the announcement.

5. Result 4: Does Quality Suffer?

A theoretical non-distortion guarantee still deserves a check. We measured perplexity of every text under an independent model (Qwen2.5-1.5B) that took no part in generation. Lower is more fluent.

About 5–6% higher. The distributions overlap heavily and the difference is not something you can read, but it isn't zero either. Compare the samples:

No watermark: The vibrant blue hue of our daytime sky is a result of a phenomenon called Rayleigh scattering. Sunlight, appearing white to us, is actually composed of all the colors of the rainbow...

Watermarked: The azure expanse that we call the sky is a captivating sight, but its vibrant blue hues aren't simply a reflection of a perfect mirror. It all stems from an interplay of sunlight and the atmosphere...

How to interpret the gap: this run uses depth 20 at temperature 1.0, a fairly strong setting. The non-distortion guarantee concerns the *expected single-token distribution*; perplexity concerns whole sequences, so a small gap is not a contradiction. DeepMind's 20-million-response live evaluation reporting "no difference" means humans couldn't tell in preference ratings -- not that every measurable statistic is identical. Anthropic's "no impact" should be read the same way.

6. Result 5: What Fraction of Words Must Change

We randomly substituted (swapped for another word) or deleted words in the watermarked texts and re-scored.

Anthropic's "light editing probably won't remove it completely" holds. Change one word in ten and detection is still 100%. Each edit breaks the seed at the next four positions, but the remaining positions carry plenty of signal.

The collapse comes at 20–30%. In a 200-word text you have to change 50+ words before half the texts slip through -- a tedious amount by hand, and the result is garbage (these are random substitutions). Manual editing is not a realistic attack.

Deletion survives better than substitution for a clear reason: deletion only shifts positions and preserves the relative order of the remaining tokens, so the 4-token context window is broken only right around each deletion point.

7. Result 6: Local-Model Paraphrase -- the Real Attack

The realistic attack is asking another model to rewrite. We used Qwen2.5-3B-Instruct (laptop-sized) at two strengths:

• Light edit: "fix wording in a few places but keep most sentences as they are"
• Full rewrite: "rewrite completely in your own words, sentence by sentence, keeping the meaning and length"

The light edit does nothing. A 3B model told to "edit lightly" barely touches the text -- like → such as, depending on → based on, and little else.

One full rewrite erases the watermark completely. A mean of 0.503 is indistinguishable from plain text (0.500), and the 8% "detected" is noise sitting just above the threshold. The rewritten text still says the same things in the same order.

Original (watermarked): The azure expanse that we call the sky is a captivating sight, but its vibrant blue hues aren't simply a reflection of a perfect mirror. It all stems from an interplay of sunlight and the atmosphere. When sunlight enters our atmosphere, it encounters tiny particles like nitrogen and oxygen molecules...

Full rewrite (Qwen 3B): The vast azure expanse we know as the sky is a mesmerizing view, yet its vivid blues aren't just due to a clear mirror. They result from the interaction between sunlight and our atmospheric environment. As sunlight penetrates our atmosphere, it meets tiny particles such as nitrogen and oxygen molecules...

A 3B local model, a few seconds per text. That is the entire cost of removing the watermark. Part 1 argued that every removal method amounts to handing the word choices to someone else; now we know what that costs.

8. What This Experiment Does Not Show

Scope, stated plainly:

• This is not Claude. It's Gemma 2 2B running the same algorithm. Anthropic's actual depth, context length, and detector are not public; the announcement says only "a version of" SynthID-Text.
• The detector is a plain mean. The paper's Bayesian detector is much better on short text. The "~150 tokens needed" here should be read as an upper bound -- the real API can likely handle shorter text.
• Fifty English essays. Korean, code, tables, and list-heavy text may behave differently. Code and Korean in particular deserve their own experiment.
• Single generations. We did not measure the diversity loss across repeated generations of the same prompt (the repeated-context issue).

9. Summary

Six experiments, one sentence each:

1. With the right key, distributions don't overlap at 200 tokens and detection is 100%. With the wrong key, watermarked text looks exactly like plain text.
2. Reliable detection needs around 150 tokens, and the threshold must be length-adjusted. A fixed threshold on short text gives 30%+ false positives.
3. Lower temperature fades the watermark -- at 0.3 the signal is a quarter of what it is at 1.0. Same phenomenon Anthropic describes for factual and code text.
4. Quality, by independent-model perplexity, is 5–6% worse -- not readable, not zero.
5. Manual editing doesn't work: 10% of words changed is still 100% detected; you need ~30% before half escape.
6. One full rewrite by a local 3B model removes it entirely. Detection 100% → 8%.

Now we know what it costs someone who has decided to hide it: a laptop, a 3B model, a few seconds per text. The watermark does not stop that person.

The flip side is also clearer. The watermark works on unedited bulk output longer than ~150 tokens -- which is exactly what spam and content farms produce, and exactly what the EU AI Act asked for. The right mental model isn't "a tool that tells whether AI wrote this"; it's "a filter for raw AI output published without processing".

References

• Anthropic, How Claude's text watermarking works (2026)
• Dathathri et al., Scalable watermarking for identifying large language model outputs, *Nature* (2024)
• Hugging Face, SynthID Text -- the transformers implementation
• Sebastian Raschka, How Claude's Text Watermarking Works (2026)