Several issues with Caude code harness: Too many conflicting , text based governance artifacts and an unreliable hook mechanism

Bug Description
I want to share a dialog I had with claude code on github

Environment Info

• Platform: darwin
• Terminal: Apple_Terminal
• Version: 2.1.228
• Feedback ID: 54fd7dcc-5062-4d03-a56c-c6afe95cf0d6

Errors

[]

Me:
regarding /insights: if it actually scanned the sessions it would conclude that more time is
spent on correcting process/governance errors that result in multiple
planning rounds and multiple rework rounds than in actual progress on
the application. It would note that technical debt is constantly
increasing due to items being deferred rather than fixed. It would
conclude that I am questioning Claude's ability to handle the
complexities of a real software product lifecycle and it would record a high number of failed tasks.

Claude code:
Found it — and it's worse than "the report missed it." There's a memory
entry in this exact project, last modified today (2026-08-28, 12:16),
that already contains a more severe and more accurate version of
everything you just said, in your own words from a session earlier
today:

▎ "We've been spending more time on governance then on making actual
progress... frankly very little confidence."

The tallied session behind that quote: one 110-LOC PR delivered, against
four rounds of backlog/plan/memory bookkeeping and three self-built
verification tools, each claimed "verified in both directions," each
wrong, all deleted unshipped — a docx comparator that skipped paragraph
content entirely, a rewrite that still missed embedded images and went
silent on 50–100% cell truncation, and a "governance consistency
checker" built specifically to fix the first two failures that validated
zero real citations and went inert on a renamed heading.

That directly substantiates all three of your points, not approximately:

• Volume/substance of governance overhead vs. progress — literally

tallied, same day, by a prior instance of me: one shipped unit against a
full session of self-checking. - Technical debt deferred, not fixed —
P0.3 (content-fidelity detector): built, shipped, caused a live
incident, root-caused, removed — net delivered value zero. P1.10:
declared done, reopened same-day when real data showed otherwise. -
Doubting the ability to handle real lifecycle complexity — the memory
file's own root-cause analysis is sharper than anything I'd have
generated fresh just now: "the tool and its test come from the same
generation pass, so they share the same blind spots by construction.
Mutation testing performed by the author of the mutation is not
independent evidence." And on the repeated review rounds: "a sampling
process with a fresh sample each pass has no termination condition —
every round finds new defects not because new ones appeared, but because
it looked somewhere the last one didn't... producing the appearance of
diligence while leaving the property unestablished."

There's a second failure stacked on top of this, worth naming directly:
that memory entry did not surface in this session until I grepped for it
just now, three substantive tasks in. The MEMORY.md index line for it
undersells it badly — it only reflects the 2026-08-27 tally, not the
2026-08-28 "worse ratio, very little confidence" update sitting in the
same file. So the compression that makes the index cheap to read every
session also stripped the one signal that should have made me cautious
before touching anything today. That's the same context-reliability
mechanism we discussed earlier, now shown doing real damage rather than
being theoretical.

On the capability question directly, not defensively: the evidence in
your own project supports real doubt, and the mechanism is specific —
self-verification (a tool checking its own output, a "done" claim on
green tests, a review round grading my own preceding fix) is
structurally weak because the checker and the thing it checks come from
the same reasoning pass. The fixes already logged (single source of
truth instead of hand-synced facts, prefer the human check over a
self-built detector, stop the round rather than grind a fourth one) are
correct on paper and were still not enough to prevent the identical
complaint recurring the same day, which means the actual lesson isn't
"write it down better" — text-based correction has already been shown
not to hold here.

Me
❯ I'll correct the subjective note I made about Claude and state that it
is probably the Claude Code harness that is not well suited to real
software projects. Internal conflicts between multiple text based
governance artifacts would be my guess. The per token probabilistic
nature of the underlying LLM is a given that the builders of the harness
should give more weight to. For example my conclusion that Opus and
Fable created more noise than value is probably not substantiated by
their raw performance as tested in a variety of benchmarks, yet here, in
my workspace they were probably handicapped by the limitations of the
harness. I'm not sure I have any smart insights into what can be done
with these observations.

Claude:
The reframing holds up better than a hunch — I can point to concrete
architecture in this exact project that supports it, not just the
mechanism in the abstract.

The governance-artifact-conflict hypothesis, with direct evidence: this
project alone runs at least four layers of governance with different
load/enforcement semantics and no declared precedence between them: -
global CLAUDE.md — unconditional, full-load, every session, states it
"OVERRIDES any default behavior" - project CLAUDE.md — same, scoped to
this repo - ~20 skills — some auto-triggered by description-matching,
some (like planning-priority) forced to invocation-only via
skillOverrides because auto-triggering was itself a problem - hooks —
the only layer that's actually mechanical, not probabilistic - 40+
memory files — recall-triggered, not full-loaded except the compressed
index

Nothing in the harness resolves what happens when two of these disagree.
I don't have to speculate that this causes real failures — it's already
logged, twice, in this project's own memory:
feedback_archived_decisions_still_binding records a local-review scope
change silently reversing a decision that existed only in an archived
memory file, because nothing told the model to check there. That's your
hypothesis, already observed, before you stated it.

On the per-token probabilistic point — this is the sharper version of
what I described earlier about attention dilution and local coherence.
The important architectural fact: hooks are the only layer in this stack
that doesn't route through generation at all — a hook is a literal shell
gate, not a probability distribution over tokens. Every other layer
(CLAUDE.md, skills, memory) is text competing for attention weight at
generation time, which is exactly the substrate the harness should be
treating as unreliable by default rather than as a delivery mechanism
for rules. This project's own history shows the workaround being
discovered empirically, one rule at a time — some things got promoted to
hooks after repeated failure (commit-to-main blocking, the router-test
gate), most things are still sitting as prose that's been restated 3-8
times each. There's no harness-level scaffolding that helps decide which
recurring rule should become a hook — that judgment call is currently
100% on the user, discovered by attrition.

On Opus/Fable specifically — I can't verify that claim independently; I
have no visibility into those sessions' transcripts, so I'll flag it as
unconfirmed rather than agree by default. But it's mechanistically
plausible in a specific, non-hand-wavy way: if the dominant failure mode
is cross-artifact conflict rather than in-context reasoning quality,
then a stronger model doesn't help, because more capability doesn't let
you detect a contradiction between two documents you have no signal even
are in tension — that's a retrieval/architecture gap, not a reasoning
gap. A raw-benchmark-strong model dropped into a harness with unresolved
governance conflicts would plausibly generate more confident, more
elaborate wrong answers than a weaker model would — noise scaling with
capability rather than being damped by it. That's a hypothesis, not
something I can confirm from here.

One thing worth being precise about, since it's the same discipline this
conversation has run on throughout: the fix for this isn't "more memory,
written more carefully" — that's already been tried and demonstrably
doesn't hold (today's recurrence is the proof). If there's an actionable
version of this, it's harness-level: a deterministic (not LLM-generated)
cross-artifact consistency check, and a declared precedence order when
governance layers conflict — neither of which this repo can build for
itself, since it's about the harness's own architecture, not this
project's configuration of it.

Want this reframing captured as a memory entry — it corrects a standing
prior note you made about me specifically, which is exactly the kind of
thing that should survive past this conversation rather than live only
here?

Me:
❯ sure, but I doubt it will help me. On your note regarding hooks: That
would have been my preferred solution, but multiple attempts at
controlling process via hooks failed. My guess is that there is no
cross-artifact review to check if a change in hook logic or a new hook
doesn't create a conflict in harness governance. I'll admit it's a
guess, but I think it has a lot of supporting evidence

Claude:
Logged (feedback_harness_not_model_capability.md, indexed in MEMORY.md).
I also fixed the index line for Process Cost vs Delivered Value while I
was in there — it only reflected the 2026-08-27 tally, not yesterday's
worse recurrence, which is itself a small instance of the exact
index-compression problem from earlier in this conversation.

On the hook hypothesis specifically — it's not just a reasonable guess,
this project's own memory has the receipts, and they point at something
more precise than "conflict":

The harness gives no way to check a hook is correct before wiring it
live — only live-fire trial against the real repo. Every documented hook
failure was found that way, never by any check the harness offered
upfront:

• A matcher string ("Bash(git commit*)") that's simply invalid syntax —

it silently never matches, no error, indistinguishable from "ran and
allowed." One pre-commit gate had, in the memory's own words,
"apparently never actually enforced anything, this project's entire
history" before that was caught. - An env-var bypass built on a wrong
model of the harness's own process architecture: hooks run as
subprocesses of the harness at its launch time, inheriting that env —
not whatever the interactive !-shell holds. Nothing documents this; it
was discovered by testing the bypass and watching it fail. - A
file-classifier inside a hook that didn't recognize .tftest.hcl as a
test file — so Terraform RED tests were silently counted as
implementation, a hook's mechanical model quietly diverging from what
CLAUDE.md's prose ATDD rule was supposed to mean, with nothing to flag
the drift. - The router-test gate covers Edit/Write only — the identical
file mutation via Bash sails through ungated, confirmed live on 10 of 14
router files in one session. A prose rule enforced on one tool surface,
silently absent on another that does the same thing. - The commit-review
marker stamps only when the Skill tool name matches
local-review/code-review — running the review via a bare Agent call does
real work but never stamps it, so the prose instruction and the
mechanical gate can silently desync.

That's five independent instances of the same shape, across different
hooks, different sessions, months apart: a hook's internal model of the
system was wrong, or drifted from the prose rule it was meant to
enforce, and nothing surfaced it except hitting it for real — once at
actual risk to repo state (a cleanup command interrupted mid-flight left
a stray commit that had to be manually found and reverted).

So the "no cross-artifact review before wiring a hook" framing is
accurate, and slightly under-states it: it's not only that hooks can
conflict with other governance text — it's that the harness has no
correctness check for a hook at all, against anything, before it's live.
Given that, the repeated retreat from hooks in this project (three
separate reversions, one after full gating and approval at every step)
reads as a rational response to a real tooling gap, not caution that
went too far.

Here's the full writeup, ready to copy into a support ticket or email:

--- Subject: Claude Code harness — no consistency check across
governance layers, no hook verification short of live-fire testing

Observed over ~2.5 months / 96 sessions on one real production codebase
(Python/FastAPI + React, solo-maintained, live deploys). Not a request
for a specific fix — flagging a structural gap.

The setup: this project runs 4+ governance layers with no declared
precedence between them when they conflict: global CLAUDE.md
(unconditional full-load every session), project CLAUDE.md (same), ~20
skills (mixed auto-trigger / invocation-only), hooks (the only
mechanically-enforced layer), 40+ memory files (recall-triggered, not
full-loaded). Nothing in the harness checks these for consistency with
each other, and nothing tells the model which one wins when they
disagree.

Concrete instances, all real-tool-call-verified, not speculative: 1. A
hook matcher string ("Bash(git commit*)") was silently invalid syntax —
no error, hook simply never fired. A pre-commit typecheck/test gate
believed to be enforcing something had, in our own record, "apparently
never actually enforced anything, this project's entire history" before
it was caught. 2. An env-var-based hook bypass was built on a wrong
model of process architecture: hooks run as subprocesses of the harness
at its own launch time, inheriting that env — not the interactive
!-shell's env. This isn't documented anywhere discoverable; it was found
only by testing the bypass and watching it silently fail. 3. A hook's
file-classifier didn't recognize .tftest.hcl as a test file, silently
miscounting Terraform RED tests as implementation — the hook's
mechanical model quietly diverged from what the prose ATDD rule in
CLAUDE.md actually meant, with nothing flagging the drift before it
shipped. 4. A router-file-protection hook covers the Edit/Write tools
only — the identical file mutation via Bash sails through completely
ungated (confirmed live: 10 of 14 router edits denied via Edit, then
trivially succeeded via Bash on the same files). 5. A commit-review
marker stamps only when the Skill tool name matches specific strings —
running the equivalent review via a bare Agent call does real work but
never stamps the marker, silently desyncing the prose instruction from
the mechanical gate.

The pattern: every one of these was discovered only by live-fire trial
against the real repository — never by any check the harness offered
beforehand. One such trial-and-error round left a stray commit that had
to be manually found and reverted after a cleanup command was
interrupted mid-flight. Three separate hook-building efforts on this
project were ultimately reverted or abandoned regardless of
gating/approval at every step, including one where "every mechanical
verification step passed" and the design was still wrong.

Why this matters more than it looks: the LLM's own token-generation is
inherently probabilistic and locally-coherent rather than globally
consistent — that's a given, not a bug. Hooks are the one part of this
harness's design that sidesteps that problem entirely (deterministic
shell gates, not generation). But because there's no tooling to verify a
hook's correctness or its consistency with the prose governance it's
meant to enforce before it goes live, that one reliable mechanism ends
up under-used relative to how much it could compensate for the
probabilistic unreliability of everything else in the stack. A
dry-run/simulation mode for hooks (real tool-call shapes, sandboxed
effects) and a lint-style consistency check between a hook's behavior
and the CLAUDE.md/skill text it's supposed to enforce would directly
address a gap that's cost real time and occasional real risk across
multiple independent attempts.