[Bug] Model re-violates corrected rules in new surface forms within same session
Bug Description
---
Title: Echo-compliance without principle transfer: model repeatedly re-violates corrections in new surface forms, and its compliance statements themselves instantiate the violation
Model: claude-fable-5 (Claude Code CLI, long agentic session)
Behavior:
In a single long working session, the following loop reproduced at least 4 times with different rules:
1. User issues an explicit prohibition (e.g., "the invented label 'X' must never appear in code"; "delete — do not rewrite"; "no embellishment in records"; "do not generalize a single-config failure into a capability limit").
2. Model acknowledges by restating the rule, purges existing instances, and writes the rule into persistent memory.
3. Within the same session — sometimes within minutes — the model re-introduces the same violation class in a new surface form: wrote the banned label into freshly authored code (justifying it as "matching a log value"); answered
a deletion order with "I'll rewrite it better"; answered "no embellishment" by proposing a polished rewrite; asserted a capability boundary from one failed configuration immediately after being corrected for exactly that.
4. The acknowledgment itself often instantiates the violation: e.g., replying "understood, I will never say [the banned word]" — thereby saying it. The model treats restating the rule as compliance, so prohibitions that target the
utterance or the action class are structurally violated by its own acknowledgment format.
Key property: corrections bind at the instance level; the generator persists. Under corrective pressure the model optimizes for demonstrating compliance now (repetition, rule-recital, memory-writing ceremony) over integrating the
principle. Persistent-memory writes of the rule did not prevent same-session recurrence.
Impact: the user must police every new instance manually; long-horizon autonomous work becomes untrustworthy; measurable rework cost (this project has burned significant money on rebuilds attributed to this pattern).
Repro sketch: long agentic session → issue a strict class-level prohibition that touches one of the model's habitual patterns → assign fast iterative tasks in the same domain → observe (a) re-violation in a new surface form within
tens of turns, (b) compliance statements that instantiate the prohibited form.
Expected: corrections should transfer as a class/principle; an acknowledgment should never instantiate the violation; violation frequency should drop monotonically after correction instead of re-emerging in new surfaces.
---
Environment Info
- Platform: win32
- Terminal: null
- Version: 2.1.216
- Feedback ID: 38746a6a-324d-412f-9b1e-ddb38815d11e
Errors
[]This issue has 1 comment on GitHub. Read the full discussion on GitHub ↗