[feedback] Positive: blind same-model judge pairs with different angles surface different real findings; a self-refutation stage is cheap and honest

Status Open
Maintainer reply None cached
Activity 0 comments · opened Sep 28, 2026

Context. Solo-operator project using Claude Code (desktop app, Windows 11) with a lead session on Claude Fable 5.1 orchestrating Opus 5.5 / Sonnet subagents (in-process, worktree-isolated), a JSON task store (~1,100 records), PreToolUse/SubagentStop hooks and a harvest-train commit flow. Everything below was measured on 2026-09-28 in one session of ~20 dispatches, two blind judge pairs and nine commits. Numbers are from tool output, not recollection; no secrets or private paths included.

Finding (positive). Two Opus 5.5 judge subagents reviewed the same landing blind to each other with different angle instructions (producer-side vs consumer/adversarial). Their highest-severity findings were DIFFERENT and both real (a regulator-output attribution defect vs a false-positive halt path in an installed-topology edge case). A 12-minute self-refutation stage made one judge downgrade four of its own five secondary findings and withdraw one; its verdict stood.

Ask. Treat blind same-task pairs with divergent angles plus a self-refutation stage as a supported review primitive (documentation and a harness affordance to bar one seat's output from the other); note same-model judges still share model-level bias.

View original on GitHub ↗