Tool results overridden by conversational context salience during generation
Bug Description
Title: Tool results overridden by conversational context salience during generation
Description:
When generating content that should be based on tool output, the model
substitutes items from conversational context over actual tool results.
Reproduction steps:
- Use
statcommand to list files created between specific dates → returns 7 files - Ask the model to create a document listing those 7 files
- Model correctly identifies all 7 in the tool output
- During document generation, model substitutes one file (ralph-criteria-generator)
with a different file (my-fetch-tweet) that was heavily discussed in conversation
but was NOT in the tool result set
Expected behavior:
Document should contain exactly the 7 files returned by the tool.
Actual behavior:
Model replaced a low-salience tool result (ralph-criteria-generator — mentioned 0 times
in conversation) with a high-salience conversational item (my-fetch-tweet — discussed
30+ times across Day 5 blocks) despite the tool result being accurate.
Root cause analysis:
This appears to be a "Context Salience Bias" where:
- Tool results have lower weight than conversational recency/frequency
- Items discussed extensively in the conversation window override tool-retrieved data
- The model "remembers" salient conversation topics more strongly than single-occurrence
tool outputs when generating new content
Impact:
- Data-driven outputs become unreliable when conversation is long
- Users must manually verify every tool-based generation
- Particularly problematic in long sessions where many topics are discussed
Suggested mitigation:
- Higher weight for structured tool outputs vs conversational memory during generation
- Or explicit "grounding" mechanism that pins generation to specific tool results
Environment Info
- Platform: darwin
- Terminal: Apple_Terminal
- Version: 2.1.76
- Feedback ID: 6131044b-b5b2-4b6b-ae10-a77ec46f314c
- Claude Code Opus-4-6 MAX mode
This issue has 3 comments on GitHub. Read the full discussion on GitHub ↗