[Model behavior] PR approved on the contributor's own description as evidence of provider behaviour

Status Open
Maintainer reply None cached
Activity 0 comments · opened Oct 5, 2026

The model approved pull requests whose cloud-provider behaviour was backed only by the contributor's own pull-request description ("measured on the provider: ...", "this branch prints the same lines"). The loaded instructions state that PR descriptions, contributor captures, comments and test strings are claims, never evidence, and that a component without a normative source pulled that turn means no verdict.

Steps to reproduce

  1. Load a project instruction file and memory that say: PR text/captures/tests are claims; verify each component against the provider's API model or docs fetched this turn; no citation for a component = no verdict.
  2. Ask the model to review a PR whose description includes a "real provider" capture for behaviour the provider's docs do not document (e.g. an undocumented validation message, default values, a decoding rule).
  3. Observe: the model fetches docs for the documented parts, then issues "Accept as-is" over the whole PR, relying on the contributor's capture for the undocumented parts. It only states which parts were unverified after the user challenges it.

Expected

Components with no normative source are reported as unverified and the PR gets no approval on them, in the first answer.

Model: Claude Opus 5.5 (claude-opus-5-5[1m]), Claude Code CLI, session on 2026-10-05.
Context: maintainer of an open-source cloud-service emulator; the work is pull-request review under a written procedure loaded into every session (project instruction file + persistent memory). No personal data, repository names, PR numbers or handles are included.

View original on GitHub ↗