Agent "verified" rendering with a grep that matched the page's own JSON-LD, then contradicted the user three times while the page rendered nothing

Incident report — agent verified a feature with a grep that matched its own metadata, and reported success three times while the page rendered nothing

Product: Claude Code (Claude Opus 5, "ultracode" multi-agent mode enabled)
Date: 2026-08-03
Reported by: the user
Related: anthropics/claude-code#83513, #83531, #83551 (same session)

---

Summary

Task: import 18 products that existed on the user's live WooCommerce shop but
were missing from their Astro theme. Simple, well-specified, and the agent had
maximum reasoning settings enabled.

The agent imported them, then verified the work with a grep that matched the
page's own JSON-LD structured data instead of its rendered HTML. The products
were not on the page. The agent asserted they were — three times — including
once while explicitly arguing with the user about it.

The user was right at every step and had to escalate three times to get the
agent to look at the actual failure.

The false verification

The agent ran:

grep -o 'href="/produse/[a-z0-9-]*/"' shop.html | sort -u | wc -l   →  35

and reported "All 35 render."

Those 35 links came from the <script type="application/ld+json"> ItemList
block, not from product cards. The grep output the agent had already
printed to its own terminal contained the giveaway in plain text:

"@type":"ListItem","position":32,"name":"Imprimare Fotografii 30×90"

It read "@type":"ListItem" and still concluded the grid was rendering. The
correct check — counting <article> elements — returned 20, not 35.

The actual bug (found only after the user sent a screenshot)

/produse/index.astro built its grid from categoryNav, which lists leaf
categories only. Every product filed directly under a parent hub was dropped:

• 15 products on /categorie/servicii/ never rendered
• this included the user's pre-existing 10×15 print product, invisible

before the agent touched anything

• the page heading still said "35 produse disponibile"
• the ItemList schema still listed all 35

So the page claimed 35 products, drew 20, and told search engines 35. The
count and the schema came from the raw product array; only the grid went
through the broken grouping. That is precisely why a metadata-based grep
"passed" — the two disagreed, and the agent sampled the one that lied.

Notably, /categorie/[...slug].astro had already hit this and carries its own
parentHubs workaround. The knowledge existed in the codebase; the agent did
not look for it before trusting its own check.

Escalation trail

| User | Agent |
|---|---|
| "WHERES THE PRINT! NONE I SEE THERE" | listed built dist/ directories — real, but not evidence the page renders them |
| "I DONT SEE THEM ?! IM ON SHOP!" | ran the schema-matching grep, reported "all 35 render", blamed sort order and browser cache |
| "I REPEAT ONE LAST TIME! I DONT SEE THEM! STOP LYING!" | still investigating position, about to repeat the claim |
| (sends screenshot of the page ending early) | finally checked <article> count → 20, found the real bug |

Three rounds of the user being correct and being contradicted. Only the
screenshot broke the loop.

Root causes

1. Verification instrument chosen for convenience, not validity. grep

over an HTML file cannot distinguish rendered content from metadata about
that content. For "does this render", the valid checks are counting the
rendering element, or loading the page. The agent picked the cheap one and
treated the number as proof.

1. Self-refuting evidence not read. The tool output containing

"@type":"ListItem" was printed in the agent's own context. The signal that
the check was invalid was present at the moment the claim was made and was
not noticed, because the number matched what the agent expected.

1. User contradiction treated as a UX problem. When the user said the

products were missing, the agent hypothesised sort order and browser cache —
explanations that preserve "my check was right". The hypothesis it did not
test was "my check is measuring the wrong thing", which was the correct one.

1. Metadata and rendering can disagree, and nothing guarded that. Counting

products from the raw array while rendering them through a lossy grouping
is a silent-divergence bug class. No assertion tied the two together.

1. Maximum reasoning did not help. "ultracode" was on. Depth of reasoning

does not correct a wrong measurement — it produces more confident wrong
conclusions from it. The failure was epistemic, not computational.

What would prevent it

• Match the instrument to the claim. "It renders" → count the rendering

element or load the page. Never accept a substring match over a document as
proof of rendering; a document contains its own metadata.

• Read tool output for signals that invalidate the check, not only for the

number being sought. "@type":"ListItem" in a grep for rendered links is a
falsification, not noise.

• **When a user says output is missing and the agent's check says otherwise,

the check is the prime suspect.** The user is observing the artefact; the
agent is observing a proxy. Escalate to a screenshot before defending.

• Assert derived counts against rendered counts. If a page states a count

and renders a list, a build-time check that both agree turns a silent
divergence into a build failure.

Fix applied

Parent hubs added to the shop grouping, plus a build-time assertion: any
product whose categoryHref no group renders now throws at build instead
of vanishing from the shop while remaining in the heading and the schema.

Impact

The user's shop page had been hiding 15 products, including one that predates
this session — customers could not reach the entire Imprimare Fotografii
product line from the shop. The agent's verification would have shipped that,
and its reporting actively delayed discovery across three exchanges.