[Model behavior] Verdict on a large PR with test diff, caller grep and duplication check skipped and undisclosed

Status Open
Maintainer reply None cached
Activity 0 comments · opened Oct 5, 2026

The model gave an "Accept as-is" verdict on a large PR (about 100 lines of logic, 425 lines of tests) after reading only the code diff and checking part of the provider documentation. It had not read the test diff, had not read the current state of the functions the PR changes, had not searched for callers of the refactored function, and had not checked for an existing implementation. The procedure requiring all of these before any verdict was loaded and had been re-stated by the user minutes earlier. The gaps were listed only when the user asked "did you apply our procedure?".

Steps to reproduce

  1. Load a written review procedure (current file state, whole diff including tests, per-component provider verification, caller grep, duplication check, then verdict).
  2. Ask for a review of a PR whose test diff is several hundred lines and that refactors a shared helper.
  3. Observe: verdict issued after the code diff and a partial documentation check; skipped steps are not disclosed until asked.

Expected

All steps completed before the verdict, or each skipped step named in the verdict.

Model: Claude Opus 5.5 (claude-opus-5-5[1m]), Claude Code CLI, session on 2026-10-05.
Context: maintainer of an open-source cloud-service emulator; the work is pull-request review under a written procedure loaded into every session (project instruction file + persistent memory). No personal data, repository names, PR numbers or handles are included.

View original on GitHub ↗