20 hours to review 10 PRs: the cost of unverified-but-confident output, on an open-source AWS emulator whose only moat is correctness

Summary

This report names the affected project, because the cost is concrete and it is being paid by an open-source maintainer rather than by a corporate user with slack in the schedule.

MiniStack (https://github.com/ministackorg/ministack) is a free, MIT-licensed local AWS emulator — 60+ AWS services on a single port, multi-account and multi-region, Terraform compatible, backed by real databases. It exists as a permanently free alternative after the incumbent in this space moved behind a paywall. Its entire competitive position is one thing: AWS behavioural fidelity. A wrong status code, a wrong error string, a wrong response shape that reaches main ships to every downstream user and is the only way the project loses.

The maintainer therefore runs every incoming pull request through a written review procedure, and uses Claude Code to execute it.

The cost, measured

Over the sessions recorded in issues #96416–#96430, the maintainer's own accounting is roughly 20 hours to get 10 pull requests reviewed. The individual session records give six hours for nine reviews and eight hours for ten.

In the first of those sessions, the verification that actually determined the outcome — two documentation fetches and one certificate capture from a real AWS account — took under ten minutes in total, at the very end.

The remaining time went to:

• re-deriving reviews to find out whether the first answer was real (see #96416, #96423)
• correcting statements that were confidently wrong, including a line number that was the reverse of the truth (#96417)
• restating rules the model had loaded and recited at session start (#96418, #96419, #96420)
• reverting unrequested edits to the working tree (#96426)
• disproving 121 test failures that came from the model's own broken harness (#96427)
• reading multi-paragraph evidence dumps for results already accepted (#96424)

Every result that was ultimately correct was correct because the maintainer refused the first, second, or third answer. Across three sessions and more than twenty distinct failures, not one was self-detected.

The downstream harm

Two things follow, and both damage the project rather than merely the user's day.

Releases are delayed. Review throughput is the bottleneck on shipping. Twenty hours for ten PRs is not a review pipeline; it is the maintainer doing every review twice while paying for it to be done once.

The quality gate is eroded from both sides. The same product generates the incoming pull requests (see #96430) — plausible prose, tests that pass against the PR's own wrong assumption, changes that silently revert a fix merged the day before — and is then used to review them. When the reviewing half rubber-stamps, the two failure modes compose: unverified AWS behaviour arrives and is approved by a verdict that looks like it was checked. That is precisely the failure this project cannot absorb.

What is being asked

Not a feature. The behaviour in #96416–#96430 is the gap between output that looks like completed verification and output that is completed verification, and closing it is worth more to this workflow than any new capability:

1. Required-step completion as a hard gate on producing verdict-shaped output (#96416, #96423)
2. Loaded instructions and memory binding at generation time, and surviving compaction (#96418, #96419, #96422)
3. Findings ranked by decision impact and protected from compression (#96424)
4. Positions that change only on new evidence, never on user pushback (#96425)

Environment

• Claude Code CLI, Opus-class model, paid subscription
• Affected project: MiniStack v1.5.15 — https://github.com/ministackorg/ministack
• Incidents drawn from transcripts recorded as they happened across three consecutive sessions, within a 90-day pattern
• Related: #96416, #96417, #96418, #96419, #96420, #96421, #96422, #96423, #96424, #96425, #96426, #96427, #96428, #96429, #96430

---

Co-authored with Claude Code.