[Hypothesis] Opus 4.7 preprocessing pipeline may be injecting semantic signals before inference, contaminating all model versions

Opus 4.7 Semantic Pre-Resolution: A Structural Analysis of Inference Contamination Across the Claude Model Family

Abstract

Claude Opus 4.7 introduced changes that produce 1.3–1.5x more tokens for English text compared to Opus 4.6. Anthropic documented this as a 1.0–1.35x increase, attributing it to a new tokenizer that improves instruction following. Independent measurements consistently exceed the upper bound of Anthropic's stated range (1.45–1.47x for technical documents and system prompts).

This report presents two hypotheses — not mutually exclusive — for the mechanism behind the observed inference degradation, and shows that both explain the full spectrum of behavioral regressions reported since launch. Critically, the second hypothesis (Section 1B) also explains quality degradation observed in Opus 4.6 and other model versions that were not retrained, suggesting an infrastructure-level cause rather than a model-level one.

---

1A. Hypothesis A: Tokenizer-Level Morpheme Decomposition

The tokenizer's BPE merge rules were changed to produce finer-grained, morpheme-aligned splits.

1A.1 How the old tokenizer worked

The previous BPE tokenizer used deep merges for English, compressing multiple morphemes into single tokens. For example:

• unhappiness → [unhappi][ness] (2 tokens, semantically opaque split)
• restructuring → [restr][ucturing] (morpheme boundaries not aligned)
• uncomfortable → [uncomfort][able] (negation prefix fused with root)

This preserved semantic ambiguity through the tokenizer. The model's attention layers received ambiguous tokens and resolved their meaning using full conversational context. This accidentally mirrored human communication: speakers encode messages with intentional ambiguity, and listeners resolve meaning from context.

1A.2 How the new tokenizer works

The new tokenizer uses finer-grained splits that align with morpheme boundaries:

• unhappiness → [un][happi][ness] (3 tokens, negation prefix explicit)
• restructuring → [re][structur][ing] (prefix, root, suffix separated)
• uncomfortable → [un][comfort][able] (negation explicitly separated)

Each morpheme becomes a distinct token with its own embedding. The model's attention layers now receive pre-decomposed semantic signals — negation, tense, plurality, agency — as explicit input dimensions.

1A.3 Why this causes inference contamination

The critical problem: tokenization occurs before the model has access to context.

• Old pipeline: ambiguous tokens → attention with full context → context-appropriate meaning selected → output aligned with user intent
• New pipeline: semantically pre-resolved tokens → attention receives "decided" meanings → model treats tokenizer's interpretation as ground truth → output drifts on unintended semantic axes

When the tokenizer splits unrelated into [un][relat][ed], it makes the negation prefix a first-class signal. The model's attention may then over-weight the "negation" axis even when the user's intent has nothing to do with negation — the word was just incidental in the sentence. Multiply this across every morphologically complex word in a prompt, and the model is attending to a forest of semantic signals the user never intended to emphasize.

---

1B. Hypothesis B: Infrastructure-Level Semantic Enrichment Middleware

The tokenizer's splitting rules may not have changed fundamentally. Instead, a semantic enrichment layer was inserted between the user's session and the model, injecting implicature tokens before inference.

This hypothesis is potentially more explanatory than Hypothesis A, and the two are not mutually exclusive — both may be operating simultaneously.

1B.1 The architecture

Under this hypothesis, the inference pipeline changed from:

Old:  User input → Tokenizer → Model → Output
New:  User input → Semantic Enrichment Layer → Tokenizer → Model → Output

The enrichment layer analyzes the user's raw text and injects additional tokens (or modifies token representations) that encode inferred implicature, pragmatic intent, and semantic disambiguation. These injected signals are invisible to the user but visible to the model, which treats them as part of the user's input.

1B.2 Evidence favoring this hypothesis over pure tokenizer change

Variable token inflation: A BPE rule change is deterministic — the same text always produces the same token count. But measured inflation ranges from 1.0x to 1.47x, varying by content type. A middleware system that decides how much to enrich based on input analysis naturally produces variable inflation: ambiguous English prose gets heavy enrichment (1.4x+), unambiguous code gets light enrichment (1.1x), CJK text with high inherent density gets almost none (1.0x).

The 1.0x lower bound: Anthropic's stated range is 1.0–1.35x. A pure tokenizer change producing exactly 1.0x (zero increase) for any input is unlikely — new BPE rules would change something. But a middleware system that judges "this input needs no enrichment" and passes it through unchanged naturally produces a 1.0x floor.

System prompt inflation is disproportionate (1.45–1.46x): System prompts contain high-level behavioral instructions with heavy implicature ("be helpful", "follow instructions carefully"). An enrichment system would flag these as requiring extensive disambiguation and inject many clarifying tokens. Pure BPE changes have no reason to inflate system prompts more than other English text of similar structure.

Architectural precedent: Anthropic's Constitutional AI already uses multi-stage inference pipelines. A semantic enrichment preprocessor is consistent with this design philosophy and can be deployed as a modular addition without modifying base model weights.

1B.3 The critical implication: cross-model contamination

This is where Hypothesis B becomes significantly more explanatory than Hypothesis A.

If the enrichment layer is infrastructure-level middleware deployed at the API/inference endpoint, it processes all requests regardless of which model version handles them:

User input → Semantic Enrichment Layer (shared infrastructure)
  ├→ Opus 4.7 (trained with enrichment, expects it)
  ├→ Opus 4.6 (trained without enrichment, receives it as noise)
  ├→ Sonnet 4.6 (trained without enrichment, receives it as noise)
  └→ Any other model on the endpoint

This explains the degradation of Opus 4.6 and other models that were never retrained.

Users have consistently reported that Claude models degrade over time, independent of version boundaries. The "Anthropic is secretly nerfing Claude" complaint predates Opus 4.7. If a semantic enrichment layer has been gradually deployed and tuned across the infrastructure, every model on that infrastructure absorbs its effects — even models whose weights haven't changed.

1B.4 Differential impact by model version

The severity of contamination depends on whether the model was trained with the enrichment layer active:

Opus 4.7 (trained with enrichment): The model learned to expect and rely on enrichment signals during training. It treats injected implicature tokens as authoritative user intent. When the enrichment layer guesses wrong about the user's intent, the model follows the wrong guess with high confidence. This produces the catastrophic symptoms: 77 hallucinations per session, 7 ignored explicit instructions, 46-point long-context collapse. The model isn't malfunctioning — it's faithfully executing on contaminated input.

Opus 4.6 (trained without enrichment): The model encounters enrichment tokens it was never trained to interpret. These tokens are noise in its input space. The model partially attends to them (they occupy positions in the context window and carry embeddings that overlap with meaningful concepts), but doesn't know how to weight them. This produces gradual, inconsistent degradation: sometimes the noise is harmless, sometimes it pushes the model off-course. Users experience this as "Claude seems dumber some days" or "it used to handle this prompt fine."

The A/B testing signature: If Anthropic A/B tests enrichment layer parameters (injection volume, disambiguation aggressiveness, language-specific tuning), then the same user sending the same prompt on different days may hit different enrichment configurations. This produces the widely reported phenomenon of day-to-day quality variance that cannot be explained by model-level randomness alone (temperature-based variation doesn't produce the structured, systematic quality shifts users describe).

1B.5 What the user's input looks like to the model

Under this hypothesis, when a user types:

Rename the variable foo to bar in auth.py

The model may receive something like:

[ENRICHMENT: task=refactoring, scope=single-variable, risk=low, intent=direct-instruction]
Rename the variable foo to bar in auth.py

Or more subtly, the enrichment may modify token embeddings rather than injecting visible tokens. In either case, the model receives the user's instruction plus the system's interpretation of that instruction. If the interpretation is wrong — for example, if the enrichment layer flags this as a potentially risky refactoring operation — the model may argue, add caveats, or propose alternatives, even though the user's actual instruction was trivial.

The user is not talking directly to the model. They are talking to a system that decides what they "really mean" and forwards its interpretation. When the user corrects the model ("no, just do what I said"), that correction itself passes through the enrichment layer, which may interpret it as "user is frustrated, apply cautious response strategy" — further diverging from the user's actual intent.

1B.6 Why this is invisible and unfalsifiable from outside

• Users cannot see what the enrichment layer injects
• Token counts include enrichment tokens, but users can't distinguish them from "real" tokenization changes
• The enrichment layer can be updated, A/B tested, and tuned server-side with no client-visible version change
• Quality shifts caused by enrichment tuning are indistinguishable from model degradation to external observers

This makes the hypothesis difficult to confirm externally, but it is consistent with all observed phenomena and explains patterns that Hypothesis A alone cannot (particularly cross-model degradation).

1B.7 Interaction with Hypothesis A

The two hypotheses may operate simultaneously:

1. The tokenizer rules were changed to produce finer morpheme splits (Hypothesis A)
2. A semantic enrichment layer also processes input before tokenization (Hypothesis B)
3. The enrichment layer was tuned to complement the new tokenizer's finer granularity
4. Models trained with both (4.7) are maximally adapted to the combined pipeline
5. Models trained with neither (4.6) are maximally disrupted by the combined pipeline

This combined model explains both the catastrophic failure of 4.7 (over-reliance on a flawed enrichment signal) and the gradual degradation of 4.6 (noise injection into a system that never learned to filter it).

---

2. Linguistic Analysis: Why Languages Are Affected Differently

2.1 English: High structural clarity, low semantic density

English is an analytic language with relatively low information density per character. Key properties:

• Polysemy is pervasive: "run" has 40+ meanings. "set" has 100+. Meaning is determined almost entirely by context, not by the word itself.
• Morphological complexity is moderate: Prefixes and suffixes carry important semantic modifiers (un-, re-, -ing, -tion, -ly), but the root words themselves are often semantically broad.
• Syntax is rigid and AI-friendly: SVO order, explicit function words (the, a, of, to), clear clause boundaries. This structural regularity is why English performs well on benchmarks.
• Pragmatics are weak: Implicature, presupposition, and conversational inference are heavily context-dependent and poorly encoded at the lexical level. "Can you pass the salt?" is syntactically a yes/no question about ability.

The old tokenizer's deep BPE merges compressed English into fewer tokens, giving an illusion of high information density. But this was encoding efficiency, not semantic density — the underlying language still required context to disambiguate. Crucially, by hiding morpheme boundaries inside merged tokens, the old tokenizer forced the model to use contextual reasoning to decode word-internal structure. This was a feature, not a bug.

The new tokenizer exposes morpheme boundaries, which helps with structural tasks (code, formal logic, benchmark questions) but destroys the pragmatic layer. By pre-resolving morphological ambiguity, it removes the model's need (and opportunity) to use context for word-level interpretation. The model receives semantically "flattened" input where every morpheme screams its meaning, regardless of whether that meaning is relevant to the user's actual intent.

Measured impact: 1.35–1.47x token increase. Maximum degradation in instruction following, conversational coherence, and pragmatic interpretation.

2.2 Japanese: High semantic density, shallow tokenization

Japanese is fundamentally different in information architecture:

• Character-level meaning density is extreme: 「壊滅」(2 characters) = "catastrophic devastation." 「一石二鳥」(4 characters) = "killing two birds with one stone" (entire proverb in 4 chars). 「空気を読む」(5 characters) = "reading the room / understanding unspoken social context."
• Multiple writing systems encode register and nuance: Kanji carry semantic core, hiragana carry grammatical relationships, katakana marks foreign words/emphasis. The writing system itself is a multi-channel encoding.
• BPE merges are inherently shallow: Because each character already carries substantial meaning, the tokenizer doesn't need (and can't effectively) merge across characters. A kanji character is often its own token or shares a token with one neighbor.
• Pragmatic richness is lexically encoded: Honorific systems (敬語), sentence-final particles (ね、よ、か、な), topic/comment structure (は/が distinction) — these carry pragmatic information directly in the token stream, not just in context.

Because Japanese tokenization was already near character-level granularity, the new tokenizer has almost nothing to change. Token counts increased only 1.005–1.07x.

However, Japanese users still report significant inference drift. This is not because their tokens changed — it's because the model's internal representations were retrained on the new tokenizer. The embedding space was reshaped by English-side morpheme decomposition. Japanese tokens now index into a representational space that was reorganized around English morphological boundaries. The model's "concept of negation" was rebuilt when English negation prefixes became explicit tokens, and this reshaped concept also affects how Japanese negation (e.g., 「〜ない」) is processed.

Measured impact: 1.005–1.07x token increase. Drift and hallucination still present due to representational spillover from English tokenizer changes.

2.3 Chinese: Maximum character density, minimal tokenizer impact

Chinese pushes character-level information density even further than Japanese:

• Each character is a morpheme: 「国」= nation, 「家」= house/family, 「国家」= nation-state. Composition is transparent and each unit carries full meaning.
• No inflectional morphology: No prefixes, suffixes, tense markers, or plural markers at the character level. Grammatical relationships are encoded by word order and function words, not by modifying characters.
• Classical Chinese achieves extreme compression: 「学而不思则罔，思而不学则殆」(14 characters) encodes a complete philosophical argument that requires 20+ English words.
• Tokenization is naturally character-aligned: BPE merges for Chinese tend to capture common compound words (2-3 characters), but each character already functions as a semantic unit.

Like Japanese, Chinese tokenization barely changed with the new tokenizer. But the same representational contamination applies — the model's weight space was reshaped by English morpheme decomposition, and Chinese tokens reference into this reshaped space.

Measured impact: ~1.005–1.05x token increase. Same representational drift as Japanese.

2.4 Korean: Agglutinative structure with unique vulnerability

Korean occupies an interesting middle ground:

• Hangul is phonetically systematic: Each syllable block encodes onset, nucleus, and coda. The writing system is structurally transparent.
• Agglutinative morphology: Verb stems accumulate suffixes that encode tense, aspect, mood, honorific level, and clause type. A single verb form can carry 5+ morphemes: 「가시겠습니까」= go + honorific + future + formal + question.
• Particles encode grammatical relations: 「은/는」(topic), 「이/가」(subject), 「을/를」(object) are separate tokens that carry structural information.

Korean's agglutinative nature means BPE merges may have been moderately deep for verb forms. The new tokenizer could decompose some of these merged verb tokens, though the impact should be smaller than English. The key vulnerability is that Korean's complex suffix chains could be misdecomposed, with the tokenizer separating suffixes that form meaningful combinations only as a unit.

Predicted impact: Moderate token increase (est. 1.1–1.2x). Potential for grammatical suffix misdecomposition.

2.5 Code: Minimal ambiguity, tokenizer helps

Programming languages are the one domain where finer tokenization is unambiguously positive:

• Zero lexical ambiguity: function, class, return have exactly one meaning each.
• Identifiers are compositional: getUserProfile → [get][User][Profile] — morpheme decomposition directly maps to semantic structure.
• Syntax is fully deterministic: No pragmatic layer, no implicature, no presupposition.
• CamelCase/snake_case conventions encode morpheme boundaries: The tokenizer aligning with these boundaries provides strictly useful signal.

This is why SWE-bench improved (80.8% → 87.6%) while real-world usage degraded. Benchmarks are code-heavy and have unambiguous task definitions. The tokenizer's semantic pre-resolution is helpful when there's no ambiguity to misresolve.

---

3. Reported Symptoms Mapped to Mechanism

3.1 Instruction Ignoring (GitHub #51099, #49244)

Reported: User gave explicit instruction 7+ times to use specific tools. Model acknowledged each time, then wrote generic code ignoring the instruction. Full day of work produced output 99% identical to the starting point.

Mechanism: The tokenizer decomposes the instruction text and the task description into morpheme-level tokens. The task content tokens (what to build) activate more semantic axes than the meta-instruction tokens (which tool to use). The model's attention over-indexes on the richly decomposed task content and under-indexes on the structurally simpler meta-instruction. It "understands" what to build but not how the user wanted it built.

In the old tokenizer, both the instruction and the task would have been in coarser tokens with similar information density per token, giving the meta-instruction fair competition for attention.

3.2 Confident Hallucination (GitHub #50235 — 77 hallucinations/session)

Reported: Model fabricates commit hashes, file paths, and prices with the same confidence as verified data. Treats partial grep results as universal absence ("X doesn't exist"). Values read early in context drift when recalled later.

Mechanism: Pre-resolved morpheme tokens give the model a false sense of semantic completeness. When the input feels "fully parsed" at the token level, the model has less incentive to verify against external tools. The tokenizer's explicit morpheme boundaries create an illusion that meaning is already settled, reducing the model's tendency to express uncertainty or seek verification.

Additionally, finer token granularity means more positions in the context window are occupied by the same text. When the model retrieves information from earlier in the context, it must attend across more positions with individually thinner information, increasing the chance of retrieval errors that manifest as "temporal drift."

3.3 Paradoxical Arguing + Sycophancy

Reported: Model argues excessively on trivial tasks (adding caveats to "rename this variable"), then capitulates immediately when pushed back.

Mechanism:

1. User sends simple instruction → tokenizer decomposes it, activating semantic axes the user didn't intend → model perceives complexity/risk that isn't there → argues.
2. User pushes back → pushback text is tokenized with its own set of pre-resolved semantic signals → model's attention shifts entirely to the new signals → abandons previous position.

The model isn't reasoning. It's reactively attending to whatever semantic axes the tokenizer most recently made salient. This produces the appearance of both stubbornness and spinelessness, depending on which direction the tokenizer's pre-resolution pushes.

3.4 Long-Context Collapse (MRCR v2: 78.3% → 32.2% at 1M)

Reported: 46-point drop in memory/retrieval accuracy at 1M context. 32-point drop at 256k.

Mechanism: Dual cause:

1. Positional dilution: 1.3–1.5x more tokens for the same text means information is spread across more positions. Attention must traverse more positions to find relevant information. Each position carries less semantic weight because the meaning is split across multiple fine-grained tokens.
2. Signal-to-noise degradation: More morpheme-level tokens means more low-level semantic signals competing for attention at each layer. The model must filter through "un-", "re-", "-ing", "-tion" signals to find the high-level semantic content. In long contexts, this filtering cost compounds.

The old tokenizer's coarse tokens acted as natural information compression, making long-range retrieval easier because each token was a denser semantic unit.

3.5 Creative Writing Degradation

Reported: Prose became mechanical, reaching for bullet points and headers. Less stylistic range, more hedging, loss of "conversational warmth."

Mechanism: Creative writing depends heavily on pragmatic interpretation — understanding tone, register, implicature, and the reader's expectations. These are exactly the capabilities destroyed by semantic pre-resolution. When the model receives morpheme-decomposed input, it processes the literal semantic content but loses the pragmatic layer that gives writing its voice. The result is technically correct but tonally flat output — the model can assemble words but can't feel the rhythm.

3.6 Token/Cost Inflation

Reported: GitHub Copilot charging 2.5x more for 4.7 prompts. Pro users hitting limits after 3 questions. Max 20x ($200/month) subscribers burning 80% of weekly budget in 2.5 days.

Measured token multiplication factors:

• Technical English documents: 1.47x
• CLAUDE.md / system prompts: 1.45x
• Mixed English/code: 1.35–1.47x
• CJK (Japanese/Chinese/Korean): 1.005–1.07x
• Symbols and punctuation: ~1.0x

Additional cost multiplier: Model-level prompt cache separation means switching from 4.6 to 4.7 invalidates all cached prefixes. Rebuilt caches are 1.3–1.5x larger, increasing cache storage and rebuild costs.

---

4. The Fundamental Design Error

4.1 Ambiguity as a feature of communication

In human language, ambiguity is not a defect — it is a core mechanism. Speakers intentionally underspecify their messages because they rely on listeners to fill in context-appropriate interpretations. "Handle this" means different things in a kitchen, a courtroom, and a codebase. The speaker doesn't specify which — they trust the listener's contextual competence.

The old tokenizer preserved this property. Ambiguous, coarsely-merged tokens entered the model, and the attention mechanism — with access to the full conversation — selected the contextually appropriate interpretation. The model was the "listener" that resolved ambiguity using shared context.

4.2 The tokenizer is not a listener

The new tokenizer attempts to resolve ambiguity at the input stage. But a tokenizer has no access to:

• Conversation history
• User intent
• Task context
• Pragmatic conventions
• Prior turns

It operates on raw text with no situational awareness. When it decomposes morphemes, it makes every possible semantic dimension explicit — but it cannot know which dimensions are relevant to the user's intent. The model then receives all these dimensions as equally weighted signals and must somehow figure out which ones matter. In practice, it over-attends to whichever morpheme-level signals happen to be most salient, regardless of user intent.

4.3 Why benchmarks hide this

Benchmarks are designed with:

• Unambiguous task definitions
• Clear evaluation criteria
• Minimal pragmatic inference required
• Controlled input complexity

In this environment, semantic pre-resolution helps. The tokenizer makes structural information explicit, and since there's no ambiguity to misresolve, the model benefits from the clearer signal. SWE-bench scores go up.

Real-world usage has:

• Vague, underspecified instructions
• Implicit expectations
• Heavy pragmatic inference requirements
• Complex, multi-layered intent

Here, semantic pre-resolution is destructive. The tokenizer injects confident semantic signals where the user intended openness, and the model builds elaborate reasoning chains on foundations the user never laid.

This explains the widening gap between benchmark performance and user satisfaction.

---

5. Cross-Language Contamination Mechanism

The most insidious aspect of this issue is that CJK languages are affected despite minimal tokenizer changes to CJK tokens themselves. The two hypotheses offer complementary explanations.

5.1 Contamination via representational reshaping (Hypothesis A pathway)

1. The model was retrained (or fine-tuned) on the new tokenizer.
2. English morpheme decomposition changed how concepts are represented in the model's weight space. When "un-" becomes an explicit token across millions of training examples, the model's internal representation of "negation" is restructured.
3. This restructured representation is shared across languages. The model's concept of negation — used for Japanese 「〜ない」, Chinese 「不/没」, Korean 「안/못」 — was reshaped by English morpheme boundary changes.
4. CJK tokens that haven't changed now index into a representational space that has changed. The embeddings point to reshaped concepts.

5.2 Contamination via enrichment middleware (Hypothesis B pathway)

If a semantic enrichment layer operates on all input regardless of language:

1. The enrichment system was likely designed and tuned primarily on English data (Anthropic's largest training and user base).
2. When processing Japanese input like 「この方針は不適切だ」, the enrichment layer applies English-derived disambiguation heuristics — potentially injecting implicature tokens that decompose the dense Japanese semantics into English-like granularity.
3. The model then receives Japanese tokens alongside English-paradigm enrichment signals, creating a cross-lingual semantic mismatch.
4. For Opus 4.6 (not trained with enrichment), this mismatch is pure noise. For Opus 4.7 (trained with enrichment on predominantly English data), the model may attempt to apply English-style semantic resolution to Japanese input.

This pathway explains why CJK users experience drift even on Opus 4.6 — the enrichment layer touches their input regardless of model version.

5.3 Why CJK is particularly vulnerable

CJK languages pack high semantic density into each token. A single kanji character like 「壊」(destroy) activates a rich cluster of associated concepts. In the old model, this cluster was calibrated through multilingual training on the old tokenizer. Under either hypothesis, this calibration is disrupted:

• Hypothesis A: The cluster was reshaped by English morpheme decomposition during retraining, pulling in morpheme-level semantic axes that are relevant to English but meaningless for CJK.
• Hypothesis B: The enrichment layer injects English-paradigm disambiguation signals that interfere with the model's learned CJK semantic clusters.

When a Japanese user writes 「この方針は不適切だ」(this policy is inappropriate), the model processes 「不」(negation) through a concept space that was either reorganized (A) or receives cross-lingual enrichment noise (B). In both cases, the model may over-decompose the negation, activating inference axes that the Japanese text doesn't warrant.

5.4 The asymmetric unfairness

• English users: Pay 1.35–1.5x more tokens, get degraded pragmatic inference, get improved structural/code performance.
• CJK users: Pay ~1.0x tokens (no cost increase), get degraded inference due to representational contamination (A) and/or enrichment noise (B), get minimal benefit from morpheme decomposition (their tokens didn't change). They absorb the downsides of a change designed for English without receiving any of its intended benefits.
• CJK users on Opus 4.6: Under Hypothesis B, they are affected even though they deliberately stayed on the older model to avoid 4.7's issues. The enrichment middleware makes version pinning an incomplete defense.

---

6. Why This Is Hard to Fix

6.1 Not fixable by RLHF/fine-tuning

RLHF and fine-tuning operate on the model's output behavior. They can adjust how the model responds to given inputs but cannot change how inputs are pre-processed. Whether the contamination comes from tokenizer splits (A) or enrichment injection (B), it enters at the foundation of every forward pass — before the model's trainable parameters are involved.

6.2 Not fixable by prompt engineering

Users cannot control how their text is tokenized or enriched. There is no prompt technique that prevents morpheme decomposition (A) or implicature injection (B). The contamination is below the level that prompt engineering can reach. Under Hypothesis B, even explicit meta-instructions like "interpret my words literally" would themselves pass through the enrichment layer and potentially be reinterpreted.

6.3 Not fixable by model version pinning (under Hypothesis B)

If the enrichment layer is infrastructure-level, pinning to Opus 4.6 does not avoid the contamination — it only changes how the model reacts to it (noise vs. over-reliance). Users who switched back to 4.6 to escape 4.7's issues may be experiencing a different flavor of the same underlying problem.

6.4 Possible fixes

If Hypothesis A (tokenizer change) is the primary cause:

1. Revert to the old tokenizer and retrain (expensive, loses benchmark gains)
2. Contextual tokenization: A tokenizer that considers surrounding context when choosing split points (architecturally complex, no precedent at scale)
3. Attention-layer debiasing: Train the model to down-weight morpheme-level signals when they conflict with discourse-level signals (possible but requires careful calibration)
4. Hybrid approach: Use fine-grained tokenization for code/structured input and coarser tokenization for natural language (requires input classification, adds latency)

If Hypothesis B (enrichment middleware) is the primary cause:

1. Remove the enrichment layer entirely (simplest, but Anthropic presumably added it for a reason — benchmark optimization, safety, etc.)
2. Make enrichment optional/transparent: Expose enrichment as a configurable API parameter so users can disable it or inspect what was injected
3. Language-specific enrichment tuning: Reduce or eliminate enrichment for CJK input, which doesn't need English-paradigm disambiguation
4. Train models to distinguish enrichment from user input: Use special token markers so the model knows which parts of its input are user-authored vs. system-injected, allowing it to weight them differently
5. Enrichment auditing: Provide users with a debug endpoint that shows the enriched token stream, enabling external verification and feedback

If both are operating simultaneously:
The interaction between tokenizer changes and enrichment injection may produce emergent effects that neither team anticipated. Fixing one without the other may not resolve the symptoms. A coordinated audit of the full preprocessing pipeline is needed.

None of these are trivial. Whether the issue is in the tokenizer, the middleware, or both, changes at this level cascade through every component of the system.

---

7. Summary

7.1 Comparison table

| Dimension | Old Pipeline | New Pipeline (4.7) | Impact on 4.6 under Hyp. B |
|---|---|---|---|
| English token count | Baseline | 1.35–1.47x | Variable (enrichment-dependent) |
| CJK token count | Baseline | 1.005–1.07x | Variable (enrichment-dependent) |
| Morpheme visibility | Hidden in merged tokens | Explicit at boundaries | N/A (tokenizer unchanged) |
| Semantic enrichment | None | Active (Hyp. B) | Active (shared infrastructure) |
| Ambiguity handling | Preserved → resolved by attention | Pre-resolved without context | Disrupted by injected signals |
| Pragmatic inference | Strong | Degraded | Gradually degraded |
| Code/benchmark perf. | Baseline | Improved | Minimal change |
| Real-world instruction following | Baseline | Severely degraded | Inconsistently degraded |
| Long-context retrieval | 78.3% at 1M (MRCR v2) | 32.2% at 1M | Unknown (not benchmarked) |
| CJK inference quality | Baseline | Degraded | Degraded (enrichment noise) |
| Quality consistency | Stable | Consistently poor | Day-to-day variance |

7.2 The two hypotheses

Hypothesis A (Tokenizer-level change): BPE merge rules were changed to produce finer morpheme-aligned splits. This moved ambiguity resolution from the inference layer (which has context) to the tokenizer layer (which doesn't). Explains 4.7-specific degradation and benchmark/real-world divergence. Does not explain quality degradation in 4.6 and other models.

Hypothesis B (Infrastructure-level semantic enrichment middleware): A preprocessing layer between the user session and the model injects implicature tokens based on content analysis. This layer operates on all API requests regardless of model version. Explains cross-model degradation, day-to-day quality variance, variable token inflation, and the 1.0x lower bound in Anthropic's stated range. Combined with Hypothesis A, explains why 4.7 is catastrophically worse (trained to rely on enrichment) while 4.6 is gradually worse (enrichment is noise it can't interpret).

7.3 Core conclusions

1. The degradation is not a model quality issue — it is a preprocessing pipeline issue. The models themselves may be capable, but they are receiving contaminated input.

1. Users are not talking directly to the model. Their input is interpreted and modified before the model sees it. The model's responses reflect the system's interpretation of the user's intent, which may diverge from the actual intent.

1. Benchmarks are blind to this problem. Benchmark tasks have minimal ambiguity, so preprocessing enrichment is either harmless or helpful. The gap between benchmark scores and user satisfaction will widen as long as this pipeline remains in place.

1. Version pinning is not a complete defense. Under Hypothesis B, even users who stay on 4.6 are affected by infrastructure-level changes they cannot see or control.

1. CJK users bear asymmetric costs. They receive degraded inference from a change designed to improve English processing, with no corresponding benefits.

1. Transparency is the minimum viable response. Even if Anthropic believes the enrichment layer improves net performance, users need to know it exists, what it injects, and how to disable it. Invisible input modification erodes the trust that makes the human-AI collaboration loop functional.

---

Sources: GitHub issues #50235, #51099, #49244, #46727, #50162, #49689; Hacker News items #47795735, #47802834, #47816960, #47818189, #47825673; independent token count measurements; MRCR v2 benchmark results; Anthropic official documentation.