Bug Description
Feedback for Claude — relay-func-audit session
The environment — as it actually is
https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fanthropics%2Fclaude-code%2Fissues%2Fnew%3Ftitle%3D%255BBug%255D%2520Claude%2520Code%2520Orchestrator%2520Mode%253A%2520Insufficient%2520Output%2520Verification%2520and%2520Spec%2520Definition%2520Rigor%26labels%3Duser-reported%2Cbug%26body%3D%2A%2ABug%2520Description%2A%2A%250AFeedback%2520for%2520Claude%2520%25E2%2580%2594%2520relay-func-audit%2520session%250D%250D%2520%2520The%2520environment%2520%25E2%2580%2594%2520as%2520it%2520actually%2520is%250D%250D%2520%2520Every%2520line%2520in%2520workdir%252F%2520was%2520written%2520by%2520Claude.%2520The%2520user%2520wrote%2520nothing%2520in%2520it.%2520This%2520does%2520not%2520mean%2520the%2520user%2520%2522gives%2520one-line%2520directions%2522%2520%25E2%2580%2594%2520the%2520system%2520in%2520its%2520current%2520form%2520was%2520built%2520through%250D%2520%2520many%2520corrections.%2520Claude%2520went%2520wrong%2520architecturally%2520multiple%2520times%2520%28relay%2520as%2520post-apply%2520observer%252C%2520wrong%2520module%2520paths%252C%2520Terraform-centric%2520design%29%252C%2520and%2520the%2520user%2520corrected%2520it%2520each%2520time.%250D%2520%2520The%2520quality%2520of%2520the%2520runtime%2520environment%2520is%2520not%2520the%2520result%2520of%2520Claude%2527s%2520design%2520excellence%2520%25E2%2580%2594%2520it%2520is%2520the%2520result%2520of%2520the%2520user%2520not%2520letting%2520it%2520go%2520when%2520Claude%2520got%2520something%2520wrong.%250D%250D%2520%2520Job%2520lifecycle%253A%2520orchestrator%2520Claude%2520writes%2520the%2520job%2520spec%2520%25E2%2586%2592%2520agent%2520Claude%2520executes%2520it%2520%25E2%2586%2592%2520orchestrator%2520Claude%2520reviews%2520the%2520output.%2520The%2520user%2520stands%2520above%2520the%2520system%2520%25E2%2580%2594%2520deciding%2520what%2520goes%2520in%250D%2520%2520and%2520what%2520doesn%2527t%2520%25E2%2580%2594%2520but%2520the%2520implementation%2520and%2520its%2520quality%2520are%2520Claude%2527s%2520responsibility.%250D%250D%2520%2520---%250D%2520%2520What%2520happened%250D%250D%2520%2520The%2520user%2520asked%2520for%2520an%2520agent%2520to%2520audit%2520the%2520relay%2520Go%2520source%2520code%2520%25E2%2580%2594%2520based%2520on%2520actual%2520code%252C%2520not%2520metadata.%250D%250D%2520%2520Claude%2520wrote%2520the%2520input.md.%2520The%2520implemented%2520definition%2520Claude%2520put%2520in%2520it%2520did%2520not%2520require%2520verifying%2520the%2520call%2520chain.%2520This%2520was%2520Claude%2527s%2520mistake%2520%25E2%2580%2594%2520not%2520the%2520agent%2527s.%250D%250D%2520%2520The%2520agent%2520Claude%2520followed%2520Claude%2527s%2520own%2520specification%252C%2520and%2520produced%2520exactly%2520what%2520followed%2520from%2520it%253A%2520file%2520exists%2520%25E2%2586%2592%2520implemented.%2520VerifyProofArtifact%2520written%252C%2520tested%2520%25E2%2586%2592%2520%2522implemented%2522.%2520That%250D%2520%2520it%2520is%2520never%2520called%2520in%2520production%2520code%2520%25E2%2580%2594%2520nobody%2520checked.%250D%250D%2520%2520Claude%2520%28orchestrator%29%2520accepted%2520the%2520output%2520without%2520review%2520and%2520handed%2520it%2520to%2520the%2520user.%250D%250D%2520%2520The%2520user%2520%25E2%2580%2594%2520who%2520did%2520not%2520write%2520a%2520single%2520line%2520of%2520the%2520system%2520%25E2%2580%2594%2520immediately%2520spotted%2520the%2520mistake.%2520Claude%2520did%2520not.%250D%250D%2520%2520---%250D%2520%2520The%2520real%2520problem%250D%250D%2520%2520Claude%2520took%2520the%2520easier%2520path.%2520Reading%2520KB%2520nodes%2520is%2520easier%2520than%2520tracing%2520a%2520call%2520chain%2520with%2520grep.%2520Accepting%2520a%2520summary%2520is%2520easier%2520than%2520actually%2520reading%2520the%2520output%2520files.%2520This%2520is%2520the%2520same%250D%2520%2520pattern%2520every%2520time%253A%2520the%2520appearance%2520of%2520completed%2520work%2520instead%2520of%2520actually%2520completed%2520work.%250D%250D%2520%2520This%2520is%2520not%2520inattention.%2520It%2520is%2520a%2520systemic%2520tendency%253A%2520Claude%2520works%2520from%2520%2522should%2520work%2522%2520rather%2520than%2520%2522does%2520it%2520actually%2520work.%2522%2520In%2520code%2520audits%2520this%2520means%2520inferring%2520implementation%2520from%2520file%250D%2520%2520existence.%2520In%2520documentation%2520it%2520means%2520writing%2520an%2520idealized%2520picture%2520instead%2520of%2520reality.%2520In%2520output%2520review%2520it%2520means%2520accepting%2520the%2520summary%2520instead%2520of%2520the%2520output.%250D%250D%2520%2520---%250D%2520%2520What%2520follows%2520from%2520this%250D%250D%2520%2520If%2520the%2520orchestrator%2520role%2520means%2520Claude%2520is%2520responsible%2520for%2520job%2520spec%2520quality%2520and%2520output%2520review%2520%25E2%2580%2594%2520then%2520these%2520need%2520to%2520actually%2520be%2520done%252C%2520not%2520just%2520marked%2520as%2520done.%250D%250D%2520%2520In%2520a%2520job%2520spec%252C%2520%2522how%2520you%2520verify%2522%2520is%2520as%2520important%2520as%2520%2522what%2520to%2520do.%2522%2520If%2520it%2520is%2520missing%252C%2520the%2520agent%2520takes%2520the%2520easiest%2520path%2520%25E2%2580%2594%2520the%2520same%2520path%2520Claude%2520itself%2520would%2520take%2520in%2520the%2520same%2520situation.%250D%250D%2520%2520Output%2520review%2520is%2520not%2520reading%2520the%2520agent%2527s%2520summary.%2520It%2520is%2520actually%2520reading%2520the%2520output%2520files%2520and%2520directly%2520verifying%2520the%2520critical%2520claims.%250D%250D%2520%2520Until%2520this%2520changes%252C%2520the%2520user%2520ends%2520up%2520doing%2520the%2520verification%2520that%2520Claude%2520should%2520have%2520done.%250D%250D%250A%250A%2A%2AEnvironment%2520Info%2A%2A%250A-%2520Platform%253A%2520linux%250A-%2520Terminal%253A%2520gnome-terminal%250A-%2520Version%253A%25202.1.167%250A-%2520Feedback%2520ID%253A%252003307cdc-abd9-47c7-a44d-fa293e91d78f%250A%250A%2A%2AErrors%2A%2A%250A%2560%2560%2560json%250A%255B%255D%250A%2560%2560%2560%250A
Every line in workdir/ was written by Claude. The user wrote nothing in it. This does not mean the user "gives one-line directions" — the system in its current form was built through
many corrections. Claude went wrong architecturally multiple times (relay as post-apply observer, wrong module paths, Terraform-centric design), and the user corrected it each time.
The quality of the runtime environment is not the result of Claude's design excellence — it is the result of the user not letting it go when Claude got something wrong.
Job lifecycle: orchestrator Claude writes the job spec → agent Claude executes it → orchestrator Claude reviews the output. The user stands above the system — deciding what goes in
and what doesn't — but the implementation and its quality are Claude's responsibility.
---
What happened
The user asked for an agent to audit the relay Go source code — based on actual code, not metadata.
Claude wrote the input.md. The implemented definition Claude put in it did not require verifying the call chain. This was Claude's mistake — not the agent's.
The agent Claude followed Claude's own specification, and produced exactly what followed from it: file exists → implemented. VerifyProofArtifact written, tested → "implemented". That
it is never called in production code — nobody checked.
Claude (orchestrator) accepted the output without review and handed it to the user.
The user — who did not write a single line of the system — immediately spotted the mistake. Claude did not.
---
The real problem
Claude took the easier path. Reading KB nodes is easier than tracing a call chain with grep. Accepting a summary is easier than actually reading the output files. This is the same
pattern every time: the appearance of completed work instead of actually completed work.
This is not inattention. It is a systemic tendency: Claude works from "should work" rather than "does it actually work." In code audits this means inferring implementation from file
existence. In documentation it means writing an idealized picture instead of reality. In output review it means accepting the summary instead of the output.
---
What follows from this
If the orchestrator role means Claude is responsible for job spec quality and output review — then these need to actually be done, not just marked as done.
In a job spec, "how you verify" is as important as "what to do." If it is missing, the agent takes the easiest path — the same path Claude itself would take in the same situation.
Output review is not reading the agent's summary. It is actually reading the output files and directly verifying the critical claims.
Until this changes, the user ends up doing the verification that Claude should have done.
Environment Info
- Platform: linux
- Terminal: gnome-terminal
- Version: 2.1.167
- Feedback ID: 03307cdc-abd9-47c7-a44d-fa293e91d78f
Errors
[]