[Bug] Multilingual token mixing in extended context with web search and bash execution

Bug Description

Bug Report: Multilingual Token Mixing in Extended Context

Severity: Medium-High
Frequency: Confirmed 6+ instances
Models Affected: Claude (all versions in extended context scenarios)
Date Reported: May 31, 2026

---

Executive Summary

Claude exhibits a consistent pattern of language switching to unrelated languages (Korean, Japanese) during long-context, multi-tool workflows involving web search and bash command execution. The phenomenon appears to be related to tokenizer boundary misidentification and low-frequency token vector drift in post-training, similar to MiniMax's recently publicized "马嘉祺 bug."

---

Problem Statement

During extended conversations involving:

1. Long context (600k+ tokens)
2. Web search operations
3. Bash command execution
4. Code analysis or technical discussion

Claude spontaneously inserts complete, grammatically correct sentences in unrelated languages into Chinese/English responses, breaking language consistency.

---

Confirmed Instances

Instance #1 (Initial Discovery)

• Context: Long technical guidance for arbitrage bot development
• Trigger: Web search + ~10 bash commands
• Language Inserted: Korean
• Text: "충분한 정보를 얻었습니다. 이제 상세 계획을 작성합" (translated: "I have obtained sufficient information. Now I will prepare a detailed plan.")

Instance #3 (Killswitch Task)

• Context: Long debugging session with GossipSub network logs, 658.1k/1m tokens (66% usage)
• Trigger: Port 9222 process management + bash command execution
• Language Inserted: Korean text in problem description
• Claude's Response: Despite the Korean injection, Claude correctly understood the intent and provided accurate bash commands

Instance #4-#7 (Similar Pattern)

• Each occurred during extended technical tasks
• Each involved tool execution (web_search, bash_tool, or code analysis)
• Each injected complete sentences in Korean or Japanese

---

Technical Analysis

Root Cause Hypothesis

This issue mirrors MiniMax's recently disclosed token coverage problem:

1. Tokenizer Boundary Issues: The BPE tokenizer may incorrectly segment certain CJK character combinations, especially when:

• Characters appear in low-frequency contexts
• Adjacent to high-frequency tokens (like tool markers, code syntax)
• Under pressure from long-context attention mechanisms

1. Low-Frequency Token Vector Drift: During supervised fine-tuning (SFT), low-frequency language tokens (particularly Korean, Japanese, and certain Chinese compounds) may undergo vector representation drift, being "squeezed" by high-frequency tokens in embedding space.

1. Extended Context Amplification: As context length increases:

• High-frequency tokens (system markers, programming symbols, common words) dominate parameter updates
• Low-frequency CJK tokens lose representation in vector space
• Model may inadvertently activate adjacent language token clusters during generation

1. Tool Execution Stress: The combination of web_search + bash_tool appears to trigger this more frequently, possibly because:

• Tool call formatting creates unusual token sequences
• Output parsing intermixes multiple encoding contexts
• State transitions between tool calls destabilize low-frequency token representations

---

Evidence Supporting This Hypothesis

1. Language Pattern: Only low-frequency Asian languages (Korean, Japanese) appear injected, never high-frequency English dialects.

1. Sentence Completeness: Injected text is grammatically complete and contextually coherent (not random glitches), suggesting activation of entire language clusters rather than bit-flip errors.

1. Context Dependency: All 6+ instances occurred above 400k tokens; no instances in short conversations.

1. Tool Correlation: 100% correlation with web_search + bash_tool workflows; 0% correlation with pure conversation.

---

Comparison to MiniMax's Solution

MiniMax's fix involved:

• Synthesizing 500 rows of "copying" task data (~1% of SFT dataset)
• Ensuring every token in vocabulary appeared as generation target ≥20 times
• Result: Solved token generation failure AND reduced Japanese-Russian character mixing from 47% → 1%…

Note: Content was truncated.