deep-research workflow: adversarial verifier conflates "qualifies" with "contradicts", wrongly refuting accurate claims

Component: Built-in deep-research Workflow tool (multi-agent research harness: Scope → Search → Fetch → Verify → Synthesize)

Bug 1 — verifier treats "heavily qualifies" as grounds to refute an accurate claim

In the Verify phase, each claim gets a 3-vote adversarial panel. The verifier prompt's checklist asks:

"2. WebSearch for contradicting evidence — does any credible source dispute or heavily qualify this?"

and the decision rule is:

refuted=true if: unsupported by quote / contradicted / low-quality source for strong claim / outdated / marketing fluff.

"Dispute" and "heavily qualify" are bundled into one yes/no check that feeds directly into refuted. But they're different things: a contradiction means the claim's fact is wrong; a qualification is context needed to interpret an otherwise-correct fact (e.g., "this figure uses a new accounting methodology," "this assumption was later partially superseded by real-world events"). A claim can be 100% accurate and still have a qualification attached — under the current prompt, verifiers refute it anyway.

Concrete effect (measured): In a sibling internal workflow with the identical verifier design, a claim stating both an accurate GAAP and non-GAAP EPS figure was refuted 0-3 by three independent verifiers, even though all three explicitly confirmed the numbers were correct — they refuted because the source also disclosed an unrelated accounting-methodology change. Meanwhile a narrower claim citing only the GAAP figure (which didn't trigger the methodology caveat) survived 3-0. The claim being more complete and precise caused it to fail verification. This is backwards — it penalizes precision.

Bug 2 — refuted claims' verdict reasoning never reaches the Synthesize step

const toRefuted = c => ({ claim: c.claim, vote: ..., source: c.sourceUrl })
// killedBlock also only lists claim + source + vote — no verdict.evidence

The Synthesize agent is shown the list of refuted claims (for transparency) but never the judges' actual reasoning for why they were refuted. When the synthesis model chooses to explain a refutation in its output (e.g., contrasting it against a similar confirmed claim), it has no grounded basis to do so — it can only guess. In testing, this produced a confidently-stated but factually wrong explanation for a refutation, invented by the synthesis model with no visibility into the real cause. For a workflow whose entire value proposition is verified, hallucination-resistant claims, having the final synthesis step itself fabricate an explanation is a meaningful integrity gap.

Recommended fix

For Bug 1, split the verifier's decision criteria into two explicit categories instead of one bundled check:

A CONTRADICTION disputes the specific fact the claim asserts (wrong number/date, a source
says the opposite, the quote is misused/out of context, the source is unreliable/fabricated).
A QUALIFICATION is separate context that does NOT make the claim's own wording untrue (e.g. a
methodology change, a scope caveat, a later development) — it explains how to interpret an
accurate fact, it doesn't dispute the fact.

refuted=true if: the quote doesn't support the claim / a genuine CONTRADICTION of the specific
fact asserted / source quality too weak / outdated / fabricated.
refuted=false if: well-supported and no source disputes the SPECIFIC fact — even if a
QUALIFICATION was also found. Put the qualification in `evidence`; don't refute over it.

For Bug 2, thread the actual refuting verdict's evidence text through to both the refuted-claims list shown to Synthesize and the tool's final output, and instruct Synthesize to use only that stated reason (never invent one) if it references why a claim failed:

const refutingReason = c => (c.verdicts.find(v => v.refuted) || {}).evidence || "(no reason recorded)"
const toRefuted = c => ({ claim: c.claim, vote: ..., source: c.sourceUrl, reason: refutingReason(c) })
// killedBlock: append "\n  Refuting judge's stated reason: " + refutingReason(c) to each entry
// Synthesize instructions: "If you reference why a refuted claim failed, use its stated
// reason verbatim/faithfully — never invent or guess one."

Validation

Both fixes were implemented in a sibling custom workflow with an identical verifier architecture and A/B tested on the same research question, run twice, before and after. Confirmed claims went from 15/25 to 23/25 on the identical question, with the newly-surviving claims correctly annotated as having a qualification rather than a contradiction. The 2 claims still refuted post-fix were both genuine contradictions (one a fabricated/misattributed source citation, one a real factual dispute) — confirming the fix didn't just loosen verification, it corrected a specific false-positive mode.