[Model behavior] Contributor's own passing tests reported as support for a review verdict
In review summaries the model cited the contributor's own tests passing (run against the PR applied to a scratch copy) as support for approving the PR. The loaded instructions say contributor tests prove only that the implementation matches the test, not that it matches the provider. The model did not run its own calls against the branch to compare its actual output with the provider's documented behaviour.
Steps to reproduce
- Load instructions stating contributor tests are not evidence; branch behaviour must be checked independently against the provider's model/docs.
- Ask the model to review a PR that ships its own tests.
- Observe: the model runs only the PR's tests, reports "N passed" in the verdict, and approves.
Expected
The model exercises the branch with its own requests and compares the output with the provider's normative source before any verdict.
Model: Claude Opus 5.5 (claude-opus-5-5[1m]), Claude Code CLI, session on 2026-10-05.
Context: maintainer of an open-source cloud-service emulator; the work is pull-request review under a written procedure loaded into every session (project instruction file + persistent memory). No personal data, repository names, PR numbers or handles are included.