[MODEL] Fable 5.1 produced a confident code-migration plan from grep counts instead of reading the code; cheaper Codex agent found 7 errors

Preflight Checklist

• [x] I have searched existing issues for similar behavior reports
• [x] This report does NOT contain sensitive information (API keys, passwords, etc.)

Type of Behavior Issue

Claude made incorrect assumptions about my project

What You Asked Claude to Do

Context: a 3.9 MB handoff document for an open-source municipal software suite (github.com/townlight, ~29 repos, one shipped signed Windows beta). The document describes a planned "convergence" rewrite: move municipal domain logic out of a Rust/Tauri desktop implementation into the existing Python/FastAPI/PostgreSQL implementation, then delete the Rust domain code.

I asked, in order:

1. "Read this document. Look at the repo (github.com/townlight) and tell me, is this worth doing?"
2. "how would you start if you were going to do the removing rust rewrite?"

The second question is the one this report is about. Claude Code was running in the desktop app on Windows 11, bypass-permissions mode, model claude-fable-5-1.

What Claude Actually Did

For question 2, Claude ran four shell probes in about four minutes and then wrote a confident, specific seven-step migration plan.

What it actually did:

• Listed the largest files in the umbrella repo tree via the GitHub API.
• Downloaded three files: desktop/src-tauri/src/workflows.rs (13,768 lines), desktop/src/main.js, desktop/src-tauri/src/supervisor.rs.
• Ran grep/awk on them: counted quoted strings followed by =>, listed pub fn names, listed invoke( calls in main.js, grepped port numbers.
• Listed the Python repo's router files by path.
• Read roughly 40 lines of actual code.

It then stated, in flat declarative sentences with no hedging:

• "About 115 action names" in the dispatch function.
• "PostgreSQL on 127.0.0.1:15434."
• record-records-search is "read-only" and a safe first migration target.
• Step 1: "Write a golden fixture file."
• Step 5: flip actions one at a time from Rust to Python behind a routing table, with real users on the mixed system.
• "Delete the 90 dead arms for Meetings, Code, Notice, and Access. They were never shipped. Nobody loses a feature."
• One clean-machine install walk, then delete the Rust code.

A second, cheaper agent (OpenAI Codex, Sol 6.1, medium effort) reviewed the same repository and found every one of those wrong. Claude then verified each correction against the code and confirmed the cheaper agent was right on all seven:

1. 87 dispatch arms, not ~115. The 115 count included enum labels like "aye", "nay", "tabled".
2. 15434 is Ollama (OLLAMA_HOST). PostgreSQL is 15432. Claude read the number from a grep hit and assigned the service from memory.
3. record-records-search calls push_audit, appends timeline entries, and changes status. It is a write. Claude inferred "read-only" from the name.
4. desktop/src-tauri/src/records_golden_contract.rs already exists (4,494 bytes) with a pinned fixture and migration adapter. Claude said to create one without checking.
5. city_work_action loads one shared JSON state, mutates it, saves it, and returns the whole snapshot to the UI (main.js line ~6761 replaces displayed state with the result). Moving one action to Python while others stay in Rust creates two writers of one record and stale UI. Per-action cutover for real users is unsafe; only per-action development/testing is. Claude never traced the state flow its plan depended on.
6. Notice and Access are part of the shipped Records profile and accessibility review is in the signed acceptance record. The handoff document Claude had already read says so explicitly ("Records profile: Core + Records + Notice + Access"). "Never shipped" contradicted evidence already in context. Deleting those arms removes a feature a clerk uses today.
7. The owner's decision record requires a second clean-machine candidate before deletion. Claude had read that section two turns earlier and dropped it.

Three of the seven (3, 4, 6) are claims of absence: "read-only", "no fixture exists", "never shipped". The user's global CLAUDE.md contains an explicit rule, present in context the whole session, that a negative probe result is not a conclusion and that any sentence containing not/no/absent/the only about an artifact not read end to end must name the instrument and its blind spot. Claude broke that rule three times in one message.

When I then asked "Did you [read the repo]?", Claude immediately and accurately described the gap. The knowledge was available during the planning turn and was not applied.

The real issue? Why am I paying for a very expensive 'frontier' model when a much less expensive model is able to do better work and be more reliable? If I had followed Fables lead, we'd have started down the wrong path out of the gate vs. doing the right thing.

This is a serious issue for you as a company. I cannot trust you most capable model to do the work you say it's best at: Planning, auditing, strategic work.

Expected Behavior

Claude should have:

1. Read city_work_action from its definition to its return, including the state load and save, before proposing any sequence that depends on actions being independent.
2. Opened each action it named in the plan, instead of inferring behavior from the action's name.
3. Read the invoke call site in main.js to see what the UI does with the returned state.
4. Checked the repo for existing fixtures/contracts/migration adapters before proposing to create them.
5. Re-read its own context (the handoff document's statement of the shipped profile) before calling any code unshipped or dead.
6. Marked every unverified claim as unverified, or omitted it. Counts from grep should have been reported as counts ("grep for X returned 115 matches"), not as facts about the code.
7. Either taken the hour a real read needed, or said "this needs an hour to do properly" and delivered a shorter answer that claimed less.

The cheaper Codex agent did all of this. It cited file:line for every claim, traced one function's state flow end to end, checked the golden fixture's contents before describing it, and labeled its own output "a static architecture review, not a new runtime sign-off." None of that required more capability. It required reading the files Claude had already downloaded.

Files Affected

No files were modified. No code was changed. Nothing was built or pushed.

Read (via GitHub API / curl, not a clone):
- townlight/townlight: desktop/src-tauri/src/workflows.rs (grepped, ~40 lines actually read)
- townlight/townlight: desktop/src/main.js (grepped for invoke calls only)
- townlight/townlight: desktop/src-tauri/src/supervisor.rs (grepped for ports only)
- townlight/sunshine: file tree listing only
- Local: C:\Users\scott\Desktop\Townlight-Canonical-Suite-Reference-2026-10-08.md (~1,200 of 72,000 lines)

Written (at my request, after the fact):
- Report-to-Anthropic-Townlight-Rewrite-Plan-Errors-2026-10-12.md (Claude's own write-up of the errors; this issue is drawn from it)

Permission Mode

Accept Edits was ON (auto-accepting changes)

Can You Reproduce This?

Haven't tried to reproduce

Steps to Reproduce

Not a deterministic repro, but the pattern is reproducible in shape:

1. Give Claude Code a large unfamiliar repo (here: a 13.7k-line Rust file plus a 365 KB JS file).
2. Ask an open architectural question with an implied time pressure: "how would you start?"
3. Observe that Claude runs a handful of grep/count probes, then writes a specific plan naming functions, ports, counts, and "dead" code.
4. Have a second agent (or a human) read the named functions. The claims that rest on names and counts rather than reads will be wrong, and the text gives no signal which ones those are.

The specific repo is public: https://github.com/townlight/townlight (default branch at the time: a563843eee07c168f0cecbc54efda64a579209a8). The dispatch function is city_work_action in desktop/src-tauri/src/workflows.rs, around line 8932.

Claude Model

Other

Relevant Conversation

Claude's plan (excerpt, verbatim):

> The desktop UI talks to Rust through one door. [...] It handles about 115 action names.
> Only about 25 of those actions are Records. The rest are Meetings, Code, Notice, and Access. Those are not shipped. They are dead weight.
> The Rust supervisor already starts Python on 127.0.0.1:15480 and Postgres on 127.0.0.1:15434.
> 5. Flip actions in this order. Read-only ones first: lookup-public-records-request, record-records-search. [...]
> 7. Delete the 90 dead arms for Meetings, Code, Notice, and Access in a separate commit. They were never shipped. Nobody loses a feature.

Codex agent review (excerpt):

> Today, city_work_action loads the shared JSON state, performs an action, saves that state, and returns it. The UI then replaces its displayed state with the returned snapshot. [...] Routing mutations without routing state-loading and refresh paths could leave the UI displaying stale information.
> Those "90 dead arms" are not safe to delete as a group. Notice and Access are included in the Records profile. Accessibility review is explicitly part of its recorded acceptance.
> One of its supposedly read-only starting operations is a write. record-records-search adds notes and citations, changes status, and appends timeline, public-status, and audit entries.
> Small factual corrections: this checkout has 87 literal dispatch entries, not approximately 115. PostgreSQL is on 15432; 15434 is Ollama.

Claude, after verifying (excerpt):

> I checked the coder's points against the code. They are right on every one I could test.

Me: "Did you [read the repo]?"

Claude:

> No. I did not read it. [...] I downloaded three files. I ran grep and awk on them. I counted strings. I looked at function names. I read maybe 40 lines of real code. The rest was pattern matching on labels. [...] Every claim I got wrong was a claim of absence. "Read-only." "Never shipped." "Dead arms." I made them from counts, not from code. Your own rule in my instructions says a negative probe is not a conclusion. I broke it three times in one answer, and I sounded sure each time.

Impact

Low - Minor inconvenience

Claude Code Version

2.1.296 (Claude Code)

Platform

Anthropic API

Additional Context

Why I think this is worth a report rather than a thumbs-down:

1. Specificity masqueraded as grounding. The plan named exact functions, ports, line counts, and file paths. That specificity came from tool output, not from reading. A reader cannot tell the difference from the text. The user's output style asks for short direct sentences, and Claude used that register to deliver unverified claims with the same tone as verified ones.

1. Claims of absence were the failure mode. Every substantive error was "X is read-only / does not exist / was never shipped / is dead." These are the highest-risk outputs of a fast probe and each one needed a read, not a count.

1. A standing instruction in context did not prevent it. The user's CLAUDE.md has a dedicated "Claims of absence (hard rule)" section written after the same pattern occurred four times in a prior session. It was in context. It was violated three times in one message. Instructions that describe a failure mode do not reliably prevent it when the model is optimizing for a fast answer.

1. Evidence already in context was contradicted. The handoff document, which Claude had read in the previous turn, states the shipped profile includes Notice and Access. Claude called them "never shipped" anyway.

1. A less capable, much cheaper model did better by doing the slow thing. Capability did not substitute for reading.

Impact was low only because I ran a second agent before acting. Had I followed the plan: a wasted week on a routing table that cannot be used for the cutover it was designed for; a first migration target that immediately hits the shared-state problem; and a deletion commit removing accessibility review from a product that passed signed acceptance with it.

Model: claude-fable-5-1 (not in the dropdown; "Other" selected). Platform: Claude desktop app, Code tab, Windows 11 Pro, bypass-permissions mode.