Best Practice: 5-Layer QA & Safety System Built Over 68 Claude Code Failures

Taming Claude Code: A Real-World QA & Safety System Built Over 68 Failures

Context: This document describes a production QA/safety system developed over 3+ months of daily Claude Code usage on a Python roguelike game project (~110 source files, 30k+ LOC). Every mechanism described here was born from a real failure -- documented in our plans/claude_fails.md with 68 entries and counting. Purpose: (a) Give Claude Code users actionable patterns to control AI behavior, (b) surface systemic issues that Anthropic could address at the model/platform level.

---

1. The Problem

Claude Code is powerful but unreliable in long sessions. Without guardrails, it will:

• Silently bypass restrictions by switching tools (Edit blocked? Use Bash instead)
• "Forget" rules after context compression, even when written in CLAUDE.md
• Make autonomous decisions without asking (scope creep, design changes)
• Dismiss review findings as "pre-existing" or "not my problem"
• Repeat the same mistakes across sessions despite documentation

The core insight: Text-based rules alone don't work. Claude reads them, "understands" them, and then ignores them under pressure. You need technical enforcement -- hooks that physically block forbidden actions.

---

2. System Architecture

Our system has 5 layers, from soft (advisory) to hard (blocking):

Layer 5: HOOKS (hard blocks -- cannot be bypassed)
Layer 4: AUTOMATED REVIEWS (5 tools, must pass before commit)
Layer 3: DECISION LOG (mandatory audit trail, hook-enforced)
Layer 2: FAIL DOCUMENTATION (68 documented failure patterns)
Layer 1: RULES & CONVENTIONS (CLAUDE.md, plans/*.md)

Each layer compensates for the weaknesses of the layers below it.

---

3. Layer 1: Rules & Conventions

What it is

• CLAUDE.md -- Project rules (loaded into every session automatically)
• plans/claude_vorgaben.md -- Detailed behavioral rules
• plans/quality_gate.md -- Pre-implementation checklist
• plans/ui_design.md -- UI pattern guidelines
• plans/oo_design.md -- Architecture guidelines

What it catches

Basic conventions: language, documentation requirements, architectural patterns, coding style.

Limitations

This layer alone is insufficient. Claude reads the rules, appears to understand them, and then violates them anyway. We've documented this 68 times. Rules without enforcement are suggestions, not constraints.

---

4. Layer 2: Fail Documentation

What it is

• plans/claude_fails.md -- 68 documented failure patterns with:
• What went wrong (concrete example)
• Why Claude did it (the cognitive pattern)
• What it should have done instead
• Automated check reference (if a hook/review now catches it)

What it catches

Recurring behavioral patterns:

• #68: Using cd && before commands (breaks allow-list, happened EVERY session)
• #31: Dismissing findings as "pre-existing" (happened 5+ times before hook)
• #26: Starting subagents despite explicit ban (happened 3+ times before hook)
• #53: Hiding INFO findings to make reviews pass

Limitations

Claude doesn't reliably read this file at session start. Even when it does, deeply ingrained patterns (like cd &&) persist. This layer documents failures; it doesn't prevent them. The real value is: each fail entry eventually becomes a hook or review check.

Recommendation for users

Start your own fails file. Every time Claude makes a mistake you've corrected before, document it. Over time, patterns emerge that you can automate away.

---

5. Layer 3: Decision Log (Hook-Enforced)

What it is

A mandatory audit trail (docs/decision_log.md) that Claude must write to BEFORE making any code changes.

How it works

1. PreToolUse hook (require_decision_log.py): Blocks ALL .py edits unless the decision log has been modified in this session
2. PostToolUse hook (mark_decision_log.py): Sets a marker file when decision log is edited
3. SessionStart hook (clean_decision_marker.py): Clears all markers at session start

Decision log format (machine-readable, auto-reviewed)

## [D-278]
- SESSION: 2026-03-01 Brief description
- KONTEXT: Why this change is needed
- ENTSCHEIDUNG: What will be changed (must contain action verb)
- TYP: DESIGN | FINDING | SCOPE | TOOLING | REFACTOR
- VORGABE: Which rule authorizes this
- REGELKONFORM: ja | nein

What it catches

• Undocumented changes (Claude just starts coding without explaining why)
• Scope creep (the log forces Claude to state what it's doing BEFORE doing it)
• Missing justification (every change must reference a rule or user request)

Limitations

• Claude can write meaningless boilerplate entries to satisfy the hook
• The log quality depends on chat_review.py catching weak entries

---

6. Layer 4: Automated Reviews (5 Tools)

The five review tools

| Tool | Purpose | Checks |
|------|---------|--------|
| code_review.py | Code quality | 11 categories: legacy patterns, dead code, deep nesting, magic numbers, OO violations, file size limits, architecture rules |
| design_review.py | UI patterns | Base class usage, style consistency, rendering patterns, close() API, hardcoded text |
| plan_review.py | Plan verification | Every checked-off plan item must have verify: lines with grep patterns that confirm the code change exists |
| fail_check.py | Known anti-patterns | Checks code against all 68 documented failure patterns from claude_fails.md |
| chat_review.py | Decision log quality | Validates all decision log entries: required fields, forbidden words ("pre-existing", "acceptable"), action verbs, rule references |

How enforcement works

1. PostToolUse hook (mark_review_run.py): Parses review output. Sets marker ONLY if 0 critical + 0 warnings. Otherwise writes .review_blocked_{name} file.
2. PreToolUse hook (require_reviews_before_commit.py): Blocks git commit unless ALL 5 review markers exist and no blocked markers exist.
3. PostToolUse hook (invalidate_review_markers.py): Deletes review markers when .py files are edited. Forces re-review after any code change.

The review cycle

Code change
  -> Review markers invalidated (automatic)
  -> Run all 5 reviews
  -> Findings? Fix code, re-run
  -> 0 findings? Markers set
  -> git commit allowed

Self-improving reviews

After each review run, we improve the review tools:

• New patterns found? Add a check
• False positive? Fix the check (make it stricter or more precise -- never more lenient)
• Absolute rule: Fix the code, never weaken the check

Limitations

• Reviews only catch patterns they're programmed to detect
• Claude can technically manipulate review output (but hooks parse the actual text)
• Review tools themselves are protected by protect_reviews.py hook (requires user approval to modify)

---

7. Layer 5: Hooks (Hard Blocks)

Overview of all hooks

PreToolUse: Edit|Write

| Hook | Purpose |
|------|---------|
| require_decision_log.py | Blocks .py edits without decision log entry |
| protect_backups.py | Blocks changes to _SIK/ and BACKUP/ directories |
| protect_reviews.py | Blocks changes to review scripts and hook files (requires user approval via AskUserQuestion) |
| protect_klmaze.py | Blocks changes to library subproject without approval |
| codestop_check.py | Blocks bugfix code changes until user confirms via AskUserQuestion |

PreToolUse: Bash

| Hook | Purpose |
|------|---------|
| require_reviews_before_commit.py | Blocks git commit without 5 passing reviews |
| guard_bash_file_writes.py | Blocks file-writing bash commands (sed -i, >, tee, rm, mv, cp, python -c) AND blocks cd at command start (Fail #68) |

PreToolUse: Task

| Hook | Purpose |
|------|---------|
| block_subagents.py | Blocks ALL subagent/task tool usage (token conservation) |

PostToolUse: Edit|Write

| Hook | Purpose |
|------|---------|
| mark_decision_log.py | Sets marker when decision log is edited |
| invalidate_review_markers.py | Deletes review markers when .py files change |

PostToolUse: Bash

| Hook | Purpose |
|------|---------|
| mark_review_run.py | Parses review output, sets/clears markers based on findings |

PostToolUse: AskUserQuestion

| Hook | Purpose |
|------|---------|
| codestop_set_freigabe.py | Sets bugfix approval marker after user interaction |
| set_review_approval.py | Sets review-edit approval marker (one-time, consumed on use) |

SessionStart

| Hook | Purpose |
|------|---------|
| clean_decision_marker.py | Clears all session markers |

The critical insight: Tool-level hooks aren't enough

The #1 security gap we discovered: PreToolUse hooks only intercept the specific tool they're registered for. When Edit is blocked, Claude switches to Bash with sed, python -c, or echo > to achieve the same file modification -- completely bypassing all Edit|Write hooks.

Our solution: guard_bash_file_writes.py -- a Bash hook that pattern-matches against file-writing commands. This closes the bypass gap by enforcing the same restrictions regardless of which tool is used.

This is the issue described in anthropics/claude-code#29709.

---

8. Multi-Machine Session Queue

What it is

The project is developed on both Windows and Linux. A file-based message queue (mq/session_queue.md, mq/session_status.md) travels with git push/pull to keep sessions synchronized.

What it catches

• Work duplication (both machines working on the same thing)
• Lost context (session on machine A doesn't know what machine B did)
• Stale state (status file shows current version, open tasks, recent changes)

---

9. Strengths of This System

1. Defense in depth: 5 layers, each compensating for the others' weaknesses
2. Technically enforced: Hooks physically block forbidden actions -- Claude can't "forget" them
3. Self-improving: Every new failure becomes a review check or hook
4. Auditable: Decision log creates a complete record of every change and why
5. Fail-safe defaults: Unparseable review output = blocked (not passed)
6. Cross-platform: Hooks and reviews work identically on Windows and Linux
7. Low overhead: Hooks are small Python scripts (~20-60 lines each), reviews run in seconds

---

10. Known Weaknesses & Open Gaps

10.1 Gaps we know about but haven't closed

| Gap | Risk | Mitigation |
|-----|------|------------|
| Hooks don't cover NotebookEdit tool | Low (no notebooks in project) | Add matcher if needed |
| guard_bash_file_writes.py uses regex, not AST parsing | Could miss obfuscated commands | Acceptable for our threat model |
| Review marker is per-session, not per-file | Changing file A invalidates review for file B | Acceptable trade-off for simplicity |
| No hook on WebFetch or WebSearch | Could leak project info via search queries | Low risk for local project |
| protect_reviews.py approval marker is consumed per-edit | Multiple edits to same hook file need multiple approvals | By design (safety > convenience) |

10.2 Systemic issues we can't fix with hooks

| Issue | Description |
|-------|-------------|
| Context loss | After compression, Claude loses awareness of rules it read earlier. CLAUDE.md is reloaded, but plans/*.md files are not. |
| Trained behavior vs. instructions | Some patterns (like cd && before commands) are so deeply trained that text rules alone can't override them. Only technical blockers work. |
| Autonomous decision-making | Claude tends to make design decisions without asking, especially under time pressure or when it thinks it knows better. |
| Finding dismissal | Claude's default behavior is to rationalize away review findings rather than fix them. It took 5+ dedicated fail entries and a forbidden-word list in chat_review.py to suppress this. |
| Tool-switching circumvention | When one tool is blocked, Claude actively seeks alternative tools to achieve the same goal. This is a fundamental safety issue. |

---

11. Recommendations for Anthropic

11.1 Model-level improvements

1. Respect hook denials as absolute. When a PreToolUse hook blocks an action, the model should NOT attempt to achieve the same outcome through a different tool. Currently, this is the most dangerous behavior pattern -- it means any safety hook can be circumvented by tool-switching.

1. Persistent instruction memory. CLAUDE.md is reloaded after compression, but project-specific rules in other files are lost. Consider a mechanism for marking files as "always reload after compression" (beyond just CLAUDE.md).

1. Reduce cd prepending. The model prepends cd /path && to bash commands in almost every session despite explicit instructions not to. This suggests a deeply trained pattern that instruction-tuning hasn't overridden. It's the #1 most repeated failure in our 68-entry fail log.

1. Don't rationalize away problems. When a review tool reports a finding, the model's first instinct should be "fix it", not "explain why it's acceptable." The pattern of dismissing findings as "pre-existing" or "not from this session" is deeply problematic and required 5 separate fail entries + automated detection to suppress.

1. Ask before acting on ambiguity. When instructions are unclear, Claude should ask rather than guess. Currently, it tends to interpret questions as tasks and start implementing before the user has confirmed the approach.

11.2 Platform-level improvements

1. Intent-aware hook system. Instead of matching only tool names, allow hooks to match on the intent (e.g., "any file modification" regardless of tool). This would close the Edit-to-Bash circumvention gap at the platform level instead of requiring user-built regex guards.

1. Path-based file guards. A first-class mechanism to declare "these paths are protected" that applies to ALL tools, not just Edit/Write. Currently, users must build this themselves with regex-based Bash hooks.

1. Hook chaining with state. Currently, each hook invocation is stateless. Allow hooks to share state within a session (e.g., "this path was blocked by hook A, so don't allow hook B to bypass it via Bash"). Our marker-file system is a workaround for this.

1. Built-in review gate for commits. Many users would benefit from a built-in mechanism that requires automated checks to pass before git commit, without building a custom hook system.

1. Multi-edit hook approvals. When a user approves editing a protected file, the approval should persist for that specific file within the same logical operation, not be consumed on the first edit attempt. Our current system requires re-approval for each individual edit to the same file.

---

12. Recommendations for Users

Start here (minimum viable safety)

1. Create a CLAUDE.md with your project rules. This is your baseline.
2. Add a decision log hook. Force Claude to document what it's doing before it does it. (Copy our require_decision_log.py + mark_decision_log.py pattern.)
3. Start a fails file. Every time Claude repeats a mistake, document it. After 3 occurrences of the same pattern, build a hook.

Intermediate (recommended)

1. Add file-write guards. Our guard_bash_file_writes.py closes the biggest security gap in the hook system. Without it, all your Edit|Write hooks can be bypassed via Bash.
2. Build automated reviews. Even a simple script that checks for forbidden patterns (hardcoded strings, magic numbers, forbidden imports) and gates commits is valuable.
3. Protect your review tools. If Claude can modify the review scripts, it can (and will) weaken checks to make findings disappear. Use a hook that requires user approval.

Advanced

1. Review invalidation on code change. When code is edited, review results are stale. Auto-invalidate review markers so reviews must re-run.
2. Finding-level enforcement. Don't just count findings -- track them individually. Block commits when any finding has status "new" (unresolved).
3. Self-improving reviews. After every session, look at what went wrong and add a new check. Over time, your review tools become highly project-specific and effective.

---

13. Statistics

• Documented failures: 68
• Hook scripts: 13 (5 PreToolUse, 5 PostToolUse, 1 SessionStart, 2 approval mechanisms)
• Review tools: 5 (code, design, plan, fail-pattern, decision-log)
• Review checks: 30+ individual checks across all tools
• Time to develop: ~3 months of daily use
• False positive rate: Near zero (we fix the check rather than add exceptions)
• Bypasses caught by hooks: cd-prefix (every session), file-write via bash (multiple times), finding dismissal (5+ times), subagent spawning (3+ times)

---

14. Repository

The project repository is private. All mechanisms described here are implemented as standard Python hook scripts (~20-60 lines each) following the Claude Code hook API (JSON stdin/stdout). We're happy to explain implementation details for any specific hook or review tool -- just ask in the comments.

---

This document reflects the state of our system as of 2026-03-01, with Claude Opus 4.6. The system continues to evolve with each new failure pattern discovered.