/ultrareview silently runs in dry_run mode on large diffs, reports completed with empty findings

/ultrareview silently runs in dry_run mode on large diffs

Summary

/ultrareview <branch> reports a successful completion with empty findings ([]) when the underlying Bug Hunter orchestrator actually fails. The tracking page shows "Review failed" but the task-notification reports status: completed with an empty findings array.

The root cause from the hook output appears to be that the orchestrator falls into dry_run: true mode when given a very large diff, completes in ~16 seconds with 0 tokens consumed and 0 agents producing output, yet still returns "completed" through the task channel.

Reproduction

1. Push a branch that diffs ~830 files / ~268K insertions against main.
2. Run /ultrareview <branch-name>.
3. Receive a task-notification with status: completed and an empty findings array.
4. Open the tracking link — it shows "Review failed".

This happened on two consecutive runs, burning 2 of 3 free reviews with no output.

Hook output (anonymised paths)

=== run_hunt.sh: Starting Review ===
Timestamp: 2026-04-28T00:04:35Z
PR: unknown
Repo: unknown
Mode: branch (base=53fa1de6d00779c090be112d2c28faaa31474ee9)
Python: Python 3.11.15
Working directory: /home/claude/repo
=== run_hunt.sh: Starting Bug Hunter orchestrator ===
=== run_hunt.sh: Orchestrator completed (exit=1) ===
=== Bug Hunter: Exit code: 1 ===

stats.json (key signals)

{
  "pr_number": null,
  "repository": "local-bundle",
  "commit_sha": null,
  "duration_seconds": 16.2,
  "total_cost_usd": 0,
  "total_input_tokens": 0,
  "total_output_tokens": 0,
  "fleet_size": 5,
  "dry_run": true,                  ← unexpected for a real review
  "agents_spawned": 5,
  "bugs_submitted": 0,
  "bugs_confirmed": 0,
  "bugs_refuted": 0,
  "reviews_submitted": 0,
  "agents": [],
  "reports": []
}

Why this is a bug, not user error

• dry_run: true is set by the orchestrator itself — theres no user-facing flag for it
• total_input_tokens: 0 confirms agents never received their work
• agents_spawned: 5 says they were started, but agents: [] and empty reports/ directory says they never produced anything
• Orchestrator completed (exit=1) — non-zero exit
• The tracking page correctly shows "Review failed", but the task-notification in Claude Code reports status: completed

Impact

• Two of three free reviews silently consumed for one branch
• No way for the user to see something went wrong without manually clicking the tracking link
• Even after detecting the failure, no clear retry path or refund mechanism

Suggested fixes

1. Surface failure clearly in task-notification. When dry_run: true or bugs_submitted: 0 after a non-zero orchestrator exit, the notification should report status: failed with an error reason (e.g. "diff too large to fit in agent context").
2. Dont count failed runs against the free quota. A run that produced zero token usage shouldnt decrement Free ultrareview N of 3.
3. Document a max-diff guideline. Either gracefully reject diffs above N lines/files at launch with a clear message, or document the recommended split.

Workaround we used

Re-pointed the diff base to a more recent commit so the diff is 99 files / 4,317 insertions (R7 fix-phase only) instead of 834 files / 268K insertions (post-history-scrub initial state). This is what we'll re-run on the third free review.

Environment

• Claude Code CLI
• Diff base before workaround: 53fa1de (post-git filter-repo initial commit)
• Branch under review: ultrareview/full-codebase
• Repo: github.com:Apostrophe-Corp/DDAO-Lottery (private)