/ultrareview consumes free run on Find-phase timeout failure for oversized diffs

Resolved 💬 1 comment Opened Apr 29, 2026 by leandrogg Closed May 31, 2026

Summary

/ultrareview consumes a free run from the 3-of-3 quota even when the cloud review fails before producing any findings. Specifically observed: a Find-phase timeout on an oversized diff failed the run, but the quota was still decremented from 3 to 2.

Reproduction

  1. Confirm 3 free runs remaining (/extra-usage).
  2. Check out main of a project and create a branch pointing at the project's earliest commit (or any base producing a very large diff).
  3. Run /ultrareview <large-base-branch> against a diff that's far above the typical PR size — in my repro, 433 files / 241,017 insertions / ~417 commits.
  4. Confirm the launch succeeded (CLI shows \"Ultrareview launched...\", session URL provided).
  5. Wait for the task to complete.

Observed

  • Session UI (example layout) shows:
  • Setup: ✅ pass
  • Find: ❌ fail (timeout/heartbeat)
  • Verify: ⏳ never ran
  • Dedupe: ⏳ never ran
  • CLI returns `<remote-review>[]</remote-review>` (zero findings) — easily mistaken for \"no bugs found\" rather than \"review failed before finding anything\".
  • /extra-usage shows 2 of 3 remaining — the failed run was charged.

Expected

A Find-phase failure (especially a timeout for being too large to process) should either:

a. Auto-refund the free run, since the user got nothing actionable out of it.
b. Pre-flight reject the diff with a clear \"diff too large, narrow the scope\" message before consuming a run.

Why this matters

  • The CLI output [] looks identical to \"clean review, no bugs found\" — a user could legitimately mistake this for a successful clean pass and merge / ship without a real review having happened. The session UI does show the failure, but only if you click through.
  • Free runs are scarce (3 total per Pro/Max account, one-time, expiring May 5, 2026). Burning one on a no-output failure is high-friction.

Suggested fix

  • Add a pre-flight diff-size check at launch time. If files-changed × insertions-added is above the cloud fleet's effective processing window, refuse with a clear \"scope is too large; pass a more recent base branch or split into surface-specific reviews\" message.
  • Or: don't consume the free run unless Find phase produces output (verified or not).
  • Make the CLI output for a failed Find phase distinct from a successful zero-findings result. [] should not be the only signal for both states.

Environment

  • Claude Code CLI on Linux
  • Anthropic Pro/Max subscription, extra-usage enabled
  • Private GitHub repo (org-owned)

View original on GitHub ↗

This issue has 1 comment on GitHub. Read the full discussion on GitHub ↗