`/deep-research`: `[verified]` tags aren't bound to a primary-source check — checkable facts can pass on inter-model agreement / secondary sources

Summary

/deep-research fetches sources and produces cited reports — but its [verified] confidence tag does not appear to require a primary-source check for the specific claim it's attached to. In one report, several [verified]-tagged (and confidently stated) claims were factually wrong. The errors clustered in claims drawn from secondary aggregators or model knowledge; every claim that carried a primary-source citation was correct. The verification machinery ("17 of 25 confirmed", a refutation list) gave an impression of rigour the sourcing didn't support.

Caveat up front: I haven't seen the skill's source (it's bundled). The mechanism notes below are inferred from the output's own footer — the observable failure stands regardless of how verification works internally.

What I observed (one report — a sunscreen-filter comparison)

Output footer, verbatim:

85 claims extracted, 25 verified via 3-vote adversarial fact-checking (need 2/3 to confirm), 17 confirmed; 27 sources fetched.

So ~60 of 85 claims were never put through verification (they shipped tagged [background]), and the 25 that were got a 3-vote adversarial check. From the footer alone I can't tell whether those votes re-check a fetched authoritative source per claim or are model-judgement over the already-gathered corpus — but the error pattern points to the latter for at least some claims.

Re-checking the report against primary sources, claims that were tagged/stated as solid but were wrong:

| Claim in report | Authoritative source | Verdict |
|---|---|---|
| "EU/UK/Australia/Asia regulate sunscreen as a cosmetic" | Australian TGA regulates primary sunscreens as therapeutic goods (AUST L number) | wrong for Australia |
| US CeraVe AM SPF 50 actives are "avobenzone / homosalate / octinoxate" | FDA DailyMed Drug Facts: homosalate 8% / zinc oxide 7% / octisalate 5% / octocrylene 5% | wrong |
| Key West's sunscreen ban is in force | Florida preempted local sunscreen bans in 2020; Key West's never took effect | stale |

Meanwhile the report's FDA proposed-order facts — which did carry a primary-source (Federal Register) citation — were correct. Accuracy tracked sourcing discipline.

The pattern

[verified] reads as "checked against the world," but in practice it seems to mean "confirmed by the verification step," which for checkable facts isn't bound to an authoritative primary source. Two compounding effects:

• Confirmation from secondary sources / model knowledge. In trope-heavy domains (sunscreen: "the US is behind," "EU is better," "reef-safe"), a confirmation step not anchored to a primary source can confirm the common-but-wrong version. (If those votes are model-judgement, shared priors across similar models would explain unanimous-but-wrong confirmations.)
• [background] is an unverified bucket that reads as a deliberate tier. ~60/85 claims rode through unchecked under a tag that looks like an epistemic choice.

Three further gaps the same report showed

• No internal-consistency pass. The summary called one UV filter "the only filter" covering the 380–400 nm band, while the report's own refutation section noted zinc oxide also covers it — a claim and its refutation shipped together.
• No completeness/omission critic. Two material, textbook-for-the-domain topics were absent: that UV radiation (including UVA) is an IARC Group 1 carcinogen — not just an ageing factor — and visible-light / iron-oxide protection for pigmentation disorders. Verification can't catch what isn't there.
• Scope/audience not pinned. The report mixed "global buyer," US-import, and EU framings without stating an assumed reader location, which silently drove its buying/regulatory advice.

Suggested mechanisms (rough priority)

1. Bind [verified] to a fetched authoritative source. For fetchable-fact types (composition, regulatory/legal status, dates, numeric specs), require a fetch-and-quote from a named primary source, cited inline. No primary-source check → [unconfirmed], not [verified].
2. Source-bound tags — [primary:<source>] / [secondary] / [user-supplied] / [model-inference] — so a tag names where the claim was checked. (In this report, a set of figures was tagged as database-sourced when it actually came from user-supplied input.)
3. Pre-synthesis internal-consistency pass — flag any retained claim that contradicts another or the refutation list.
4. Completeness/omission critic — a generative pass: "what material topics are missing for this reader?"
5. Temporal-validity check — date-dependent facts get an as-of date and a currency re-check.
6. Pin reader scope for jurisdiction-/expertise-sensitive topics and print it as an explicit assumption.

(1) and (2) feel highest-leverage and share a root. Happy to share more detail.