WebFetch introduces character-level errors when processing base64-encoded content

Summary

When using WebFetch to fetch base64-encoded file content from a Gerrit REST API, the tool's internal model introduces subtle character-level errors during decoding/summarization. These errors are plausible enough to appear as real data, leading to false conclusions.

What happened

I was reviewing an OpenStack election results file via the Gerrit API endpoint that returns base64-encoded YAML (/changes/{id}/revisions/current/files/{path}/content). WebFetch decoded and summarized the content, but introduced several character-level mutations:

• kajinamit → kajinimit (letter swap)
• noonedeadpunk → noondeadpunk (dropped letter)
• jbernard → jbernar (dropped letter)
• Ovchinnikova → Ovtchinnikova (inserted letter)
• Michal → Micheal (letter swap)
• Blazar → Blazer (letter swap)
• René (proper UTF-8) → RenÉ (wrong case on accented character)

These errors were presented confidently as the actual file content. I then cross-referenced them against a known-good local file from the previous cycle and incorrectly concluded there were 6 bugs in the reviewed change. In reality, only 1 issue existed. The hallucinated "bugs" were posted as a -1 code review on a real open-source project, causing unnecessary work for the patch author.

Root cause

WebFetch processes fetched content through a small, fast model before returning results. This model is not designed for byte-accurate reproduction of content, especially when base64 decoding is involved. The errors are particularly dangerous because:

1. They are subtle (single character changes) and plausible
2. They appear in structured data (emails, names) where exact spelling matters
3. There is no indication in the output that the content may be approximate
4. Even when explicitly prompted to return "COMPLETE raw content exactly as-is," errors were introduced

Reproduction

Fetch any base64-encoded file from a Gerrit API and compare WebFetch output against curl | base64 -d:

# WebFetch will introduce subtle errors:
WebFetch("https://review.opendev.org/changes/openstack%2Felection~981308/revisions/current/files/doc%2Fsource%2Fresults%2F2026.2%2Fptl.yaml/content")

# Correct approach:
curl -s "<same URL>" | base64 -d

Suggestion

Consider one or more of:

• Warning in WebFetch output when base64-encoded content is detected, noting that character-level accuracy is not guaranteed
• Documenting this limitation more prominently (the current description says "Results may be summarized" but doesn't mention character-level corruption)
• Providing a raw fetch mode that bypasses model summarization for cases requiring exact content