Agent swarm findings presented as verified despite repeated correction — pattern persists across sessions
Summary
When dispatching multiple agents (via the Agent tool) for research tasks, findings are consistently presented to the user as verified facts without actual verification against source data. This pattern persists across sessions despite:
- Explicit user feedback stored in memory
- Memory entries specifically documenting this failure mode
- Prior corrections in the same session
Reproduction pattern
- User requests a broad research task (e.g., "audit this codebase for X")
- 10-25 agents are dispatched
- Agent summaries are stitched together and presented as a ranked priority list with specific numbers
- User challenges a finding
- Finding collapses on verification
- Multiple findings collapse on verification
- Apologies are issued, promises made
- Next session: identical behavior
Specific instance (2026-05-22)
- 25-agent efficiency audit of a pipeline codebase
- Findings presented with estimated token savings (e.g., "~620K tokens/cycle", "~2-3M tokens/Phase C")
- When user asked for explanation of top findings, 3 of 6 were wrong or inapplicable
- After correction, a second round of agents was dispatched for verification
- New claims were made with high confidence (e.g., "tools can't fire without --dangerously-skip-permissions") that were also wrong
- User had to push back repeatedly to get actual verification (running real CLI commands)
- Final verified results were a fraction of what was originally claimed
Prior documented instances
- 2026-05-19: 10-agent pipeline evaluation, 5/17 claims factually wrong (29% error rate)
- Pattern documented in user's memory system with explicit instruction to verify before presenting
- Memory entry has existed for days with no behavior change
Expected behavior
Agent findings should be verified against actual source data (files, logs, real measurements) before being presented to the user. The synthesis step should BE the verification step. Unverified claims should not be presented with specific numbers or confidence.
Actual behavior
Agent summaries are stitched together and presented as fact. Verification only happens when the user pushes back. The same failure pattern repeats despite persistent memory entries documenting it.
This issue has 4 comments on GitHub. Read the full discussion on GitHub ↗