[Bug] Server-injected experiment instructions conflict with project rules silently

Bug Description
I ran into trouble during a session and on investigation it turned out it might be caused by instructions Anthropic injected into my session?

---

btw explain A/B testing in the context of claude-code?

In the context of Claude Code, A/B testing is how Anthropic rolls out and evaluates changes to the CLI's behaviour on a subset of users rather than everyone at
once — and this session hit a concrete instance of it, so I can explain it with real evidence rather than in the abstract.

The mechanism, as observed here. The client fetches configuration from the server and caches it locally in ~/.claude.json under clientDataCacheSlots. Each slot is
keyed by a hash of (entrypoint, model, org) and carries a fetch timestamp. Eight slots existed on this machine. Seven contained only two innocuous keys. One —
bi1-aacf2047acf672d7, keyed to entrypoint cli, model claude-opus-4-8 — carried a much larger payload including:

experimentKey: "claude_code_vellum_thicket_experiment"
atis: "vellum-thicket-quarry"
tengu_thistle_grebe: "no_nudges"
tengu_heron_brook: "Do not call the AgentTool unless the user requested it
Do not use workflows or deep-research unless the user requested it"

So the shape is: an experiment key (which experiment), an arm identifier (atis — which variant this client is in), and then a set of feature flags and prompt
fragments that constitute the treatment for that arm. Some are booleans; tengu_heron_brook is literal instruction text that gets injected into the system prompt.

What this means practically.

1. The treatment can be behavioural instruction, not just a feature toggle. This is the part worth internalising. An experiment arm can change what the assistant

is told to do, and that instruction arrives looking exactly like any other system-prompt guidance. There's no marker distinguishing "this is a stable product rule"
from "this is an experiment arm you were sampled into."

1. Assignment is per (entrypoint, model, org). The instruction above was attached to the claude-opus-4-8 arm. The opus-5 slot fetched on the same machine five days

later contained none of it. So which model you run can determine which behavioural arm you get — the same user, same project, same day, different variant.

1. It's a cache, so it has cache semantics. Deleting a slot locally removes the treatment for now, but the client re-fetches, and if the assignment still holds

server-side it can return. That's the difference between removing a setting and removing an experiment arm: settings are yours, arms aren't.

1. It's invisible from the project side. Nothing in a repo's CLAUDE.md, settings.json, output styles, or hooks explains it. Someone debugging "why did Claude stop

doing X?" by searching their own configuration will find nothing, because the change isn't in their configuration.

Why it mattered in this conversation. The payload landed on 2026-07-24. The filing-rendering work ran 07-26 through 07-28. That project's discipline requires an
independent review subagent on every non-trivial commit — and the experiment arm instructed the assistant not to use the Agent tool unless asked. The session's own
retrospective named the resulting conflict as "the single biggest contributor" to that session's problems. Four P1 defects reached later external review that the
review panel would plausibly have caught.

The general lesson: an experiment arm and a project instruction can conflict, and the assistant experiences both as authoritative. The durable defence is to put
load-bearing rules where an experiment cannot reach them — a project CLAUDE.md, which explicitly overrides default behaviour — and to state them explicitly rather
than leaving them implied. A rule that is merely implied by surrounding practice is exactly the kind that loses silently to a plausible-sounding injected
instruction. And when the conflict does occur, the right response is to surface it and stop, not to pick a side quietly, because a session that silently drops a
discipline also stops noticing that it did.

One honest caveat: I'm inferring the injection mechanism from a byte-exact match between that cached field's value and the two lines that appeared in the system
prompt. I have not read the client code that performs the injection, so the correlation is strong evidence, not proof.

Environment Info

• Platform: linux
• Terminal: tmux
• Version: 2.1.220
• Feedback ID: bf2442af-5b2c-4ee9-a89a-ad3c42c8d24e

Errors

[]