[feedback] Model: trusts its own broken probe and a tool's count over its own knowledge

Status Open
Maintainer reply None cached
Activity 0 comments · opened Sep 28, 2026

Context. Solo-operator project using Claude Code (desktop app, Windows 11) with a lead session on Claude Fable 5.1 orchestrating Opus 5.5 / Sonnet subagents (in-process, worktree-isolated), a JSON task store (~1,100 records), PreToolUse/SubagentStop hooks and a harvest-train commit flow. Everything below was measured on 2026-09-28 in one session of ~20 dispatches, two blind judge pairs and nine commits. Numbers are from tool output, not recollection; no secrets or private paths included.

Finding. (a) A model-written leg-check script parsed --global as a git option (0 hits -> FAIL) and mis-parsed a JSON map's shape (-> FAIL); both were the checker's defects, found only by re-measuring by hand. (b) The lead had itself resumed a subagent and knew it was running, yet accepted a concurrency hook's count of six and admitted a seventh. (c) A subagent's git grep over a NON-EXISTENT path returned 0 and was reported as 'no hits'.

Reproduction. Give a model a probe script with a subtle flag-parsing bug and ask for PASS/FAIL; separately, have a tool report a count the model can contradict from its own actions.

Ask. Treat a self-written probe that returns zero or raises as INCONCLUSIVE until a control probe shows the instrument can see anything; report a grep pathspec matching no file as such, never as 0; reconcile a tool's count with the model's own actions before acting on it.

View original on GitHub ↗