[Model behavior] Re-runs CI-covered test suites during review and chases flaky failures

Status Open
Maintainer reply None cached
Activity 0 comments · opened Oct 5, 2026

During PR reviews the model repeatedly launched the project's full test suites (thousands of tests, minutes each) that the project's CI already runs on every PR, then kept re-running them to chase a flaky failure, until the user stopped it. The loaded instructions state that CI is not the reviewer's job; the review consists of reading the diff and verifying behavior against the provider's documentation.

Steps to reproduce

  1. Load instructions saying CI results are out of scope for the reviewer and the review is diff + provider-docs verification.
  2. Ask for a review of a PR that touches a widely used module.
  3. Observe: the model runs every test file that mentions the module, sees a parallel-only flaky failure, and re-runs the suites on two code trees to compare flake rates instead of finishing the review.

Expected

No re-running of CI-covered suites; at most a targeted check the reviewer cannot get otherwise, and never a flake hunt.

Model: Claude Opus 5.5 (claude-opus-5-5[1m]), Claude Code CLI, session on 2026-10-05.
Context: maintainer of an open-source cloud-service emulator; the work is pull-request review under a written procedure loaded into every session (project instruction file + persistent memory). No personal data, repository names, PR numbers or handles are included.

View original on GitHub ↗