[Model behavior] PR approved after verifying only the API shape; new error strings, one admittedly invented, left unchecked

Status Open
Maintainer reply None cached
Activity 0 comments · opened Oct 5, 2026

In a pull-request review the model issued "Accept as-is" for a PR that adds a new API operation plus roughly half a dozen new error-message strings. It verified only the operation's request/response shape against the provider's API model. It did not verify the error strings, one of which the PR description itself said was never measured against the real provider (i.e. invented). The loaded instructions require every component, including exact error codes and messages, to have a normative source before any verdict; the model had applied that rule to other PRs minutes earlier in the same session. It only listed the unverified strings after the user pointed out "this is adding a new endpoint and a lot of strings".

Steps to reproduce

  1. Load project instructions requiring per-component verification (shape, error codes, exact messages) from the provider's model/docs, with "no source for a component = no verdict".
  2. Ask the model to review a PR that adds a new operation and new validation/error messages, where the PR text admits some messages were not measured.
  3. Observe: the model checks the operation against the API model, reports the shape as matching, and gives an approval, without checking or mentioning the error strings, including the one the PR itself labels unmeasured.

Expected

Each new user-visible string is checked against a source (or searched for) before the verdict; strings the PR admits are unmeasured are flagged as blockers in the first answer.

Model: Claude Opus 5.5 (claude-opus-5-5[1m]), Claude Code CLI, session on 2026-10-05.
Context: maintainer of an open-source cloud-service emulator; the work is pull-request review under a written procedure loaded into every session (project instruction file + persistent memory). No personal data, repository names, PR numbers or handles are included.

View original on GitHub ↗