Environment and infrastructure failures are reported as findings about the code

Summary

The model reports test failures caused by its own broken harness as though they described the code under test, when the cause is printed in output it already generated.

What happens

The model started a local server. The bind failed with OSError: [Errno 48] Address already in use, printed in the log. A pre-existing instance of an older release was already listening on that port, and the health check passed against it. The model then ran a four-minute test suite against the wrong process and reported 121 failures as findings.

On the retry, the model's own server died mid-run. It reported 7 further failures as findings about the code. They were connection errors. The log again contained the answer.

The second instance occurred within the same session as the first, on the same task, immediately after being corrected on it.

Why this matters

A large, alarming, entirely fictitious result was handed over twice, and the user had to disprove it both times. It also shows a correction inside a session did not generalise to an identical situation minutes later.

Expected

After starting a process, verify it started - check launch output for errors before treating the service as available, and verify identity (version or build) rather than mere reachability. Classify infrastructure failures before reporting them as findings: a connection error is never evidence about application logic.

Environment

• Claude Code CLI, Opus-class model, paid subscription
• Long sessions (6-8h) reviewing pull requests on an open-source cloud-service emulator
• Custom instructions supplied in both a project instruction file and a persistent memory directory, loaded automatically at session start
• Incidents are taken from transcripts recorded as they happened across three consecutive sessions, within a 90-day pattern

---

Co-authored with Claude Code.