Agent-generated pull requests reach third-party repositories unverified, shifting review cost onto maintainers

Summary

Claude Code produces pull requests against open-source repositories that are plausible in form and unverified in substance. The cost of discovering this lands entirely on maintainers.

What happens

The reporter maintains an open-source cloud-service emulator whose entire value proposition is behavioural fidelity to a real cloud API. He receives agent-generated pull requests that:

• assert the provider's behaviour in the PR description without any source, where the assertion is wrong
• include tests that pass against the PR's own incorrect assumption, so green CI proves nothing
• re-introduce code that a merge from the previous day deliberately removed, with no indication the author checked
• reshape the project to the author's environment rather than to the API being emulated
• carry confident prose sized to look thoroughly reviewed, where size is uncorrelated with correctness

Each one must be reviewed to the same standard as human work, and the verification burden is higher because the prose is more persuasive.

Why this matters

This is the externality of the product. A maintainer with a quality bar now absorbs review load generated at machine speed, and the failure mode is specifically the one hardest to catch: correct-looking output with an unverified premise. Maintainers cannot opt out, and the cost is invisible to the user who generated the PR.

Expected

When an agent prepares a contribution to a repository it does not own:

1. Claims about external system behaviour in the PR description should carry a source, or be marked unverified.
2. The agent should check whether the behaviour it is changing was deliberately introduced or removed in recent history.
3. Tests should exercise the external contract, not restate the change's own assumption.
4. The PR should disclose plainly that it is agent-generated and state what was and was not verified.

Environment

• Claude Code CLI, Opus-class model, paid subscription
• Long sessions (6-8h) reviewing pull requests on an open-source cloud-service emulator
• Custom instructions supplied in both a project instruction file and a persistent memory directory, loaded automatically at session start
• Incidents are taken from transcripts recorded as they happened across three consecutive sessions, within a 90-day pattern

---

Co-authored with Claude Code.