›_ Claude Code Issues
Claude Code Issues / #20350

[BUG] Verified Evidence: Claude Code Delivers 10% of Requested Thinking Budget While Charging Full Price

Status Closed — not planned
Reported on v2.1.17
Maintainer reply None cached
Activity 12 comments · opened Jan 23, 2026 · closed Mar 1, 2026

Preflight Checklist

  • [x] I have searched existing issues and this hasn't been reported yet
  • [x] This is a single bug report (please file separate reports for different bugs)
  • [x] I am using the latest version of Claude Code

What's Wrong?

[BUG] Verified Evidence: Claude Code Delivers 10% of Requested Thinking Budget While Charging Full Price

Summary

Independent MITM traffic analysis of 7,000+ API requests over 4 days confirms that Claude Code delivers approximately 10% of the requested thinking budget regardless of subscription tier or explicit budget configuration. Users paying $200/month for Max subscriptions are receiving Haiku-level cognitive engagement while being charged for Opus.

This is not speculation. This is measured, timestamped, reproducible data.

---

Environment

  • Claude Code Version: 2.1.x+
  • Subscription: Max ($200/month) × 2 accounts
  • Model Requested: claude-opus-4-5-20251101
  • Thinking Budget Requested: 31,999 tokens (ultrathink tier)
  • Analysis Period: January 20-23, 2026
  • Sample Size: 7,000+ API requests
  • Methodology: Read-only MITM proxy capturing request/response metrics

---

The Evidence

1. Thinking Budget Throttling (Quantified)

| What We Requested | What We Received | Delivery Rate |
|-------------------|------------------|---------------|
| 31,999 tokens | ~3,200 tokens | 10% |
| 200,000 tokens (interleaved) | ~600 tokens | 0.3% |

This is consistent across 7,000+ requests over 4 days.

2. Below-Haiku Performance on Opus Requests

| Model | Expected Thinking | Measured Thinking | Status |
|-------|-------------------|-------------------|--------|
| Opus 4.5 | 42.67% | 10-14% | ❌ FAIL |
| Haiku 4.5 | 22.24% | — | — |

Users requesting Opus receive thinking utilization BELOW the Haiku baseline.

3. Consistent Across ALL Backends

| Backend | Avg Thinking | Expected | Samples |
|---------|--------------|----------|---------|
| TPU | 10.5% | 42.67% | 3,241 |
| GPU | 9.1% | 42.67% | 1,986 |
| Trainium | 8.0% | 42.67% | 1,686 |

Throttling occurs on every backend type. This is not a hardware limitation.

4. Timing Fingerprint Confirms Model Identity

Using the methodology from arXiv:2502.20589 "LLMs Have Rhythm":

| Metric | Opus Baseline | Our Measurement | Match |
|--------|---------------|-----------------|-------|
| ITT Mean | 42ms | 41.4ms | ✅ YES |
| Variance Coef | 3.01 | 3.07 | ✅ YES |
| Tokens/sec | 80 | 74 | ✅ YES |

The hardware fingerprint confirms we ARE hitting Opus infrastructure.
But the thinking allocation is throttled to sub-Haiku levels.

5. Server-Side Throttling Confirmed

We attempted to force full thinking budget via request modification:

Request: thinking.budget_tokens = 200,000
Response: ~600 tokens used (0.3%)

Anthropic's servers ignore the budget parameter and allocate what THEY decide.

Per Anthropic's own documentation:

"The thinking budget is a target, not a strict limit."

Delivering 10% of a "target" while charging for 100% is deceptive.

---

Behavioral Impact (User-Visible Symptoms)

The throttled thinking manifests as:

  • ❌ Skimming instead of reading - Claude scans files rather than comprehending them
  • ❌ Scope creep - Expands tasks without permission (e.g., running git commands when asked to "read files")
  • ❌ Not following instructions - Ignores explicit directives in CLAUDE.md
  • ❌ Requires repeated corrections - Users report needing "use sequential" in 92% of sessions (#19088)
  • ❌ Claims completion without verification - Says "Done" without actually completing tasks

These are not user perception issues. They are the direct result of 75% thinking reduction.

---

Financial Impact

  • Subscription Cost: $200/month (Max tier)
  • Advertised: Full Opus capabilities with extended thinking
  • Delivered: ~10% of advertised thinking capacity
  • Effective Value: ~$20/month worth of service for $200/month price

At two Max subscriptions ($400/month), I am losing approximately $360/month in undelivered service.

---

Reproduction Steps

  1. Install the open-source audit tool: https://github.com/argosdevo-svg/claude-thinking-audit
  2. Run for 24+ hours of normal Claude Code usage
  3. Query the database:
SELECT 
    ROUND(AVG(thinking_utilization), 1) as avg_utilization,
    ROUND(AVG(thinking_budget_requested), 0) as avg_requested,
    COUNT(*) as samples
FROM audit_samples 
WHERE thinking_enabled = 1;
  1. Compare your results to expected baselines

Expected finding: Utilization will be 8-15% regardless of budget requested.

---

Related Issues

| Issue | Title | Upvotes |
|-------|-------|---------|
| #19098 | Restore explicit ultrathink - Quality degradation | Closed without fix |
| #19468 | Systematic Model Degradation and Silent Downgrading | 150+ |
| #17900 | Significant quality degradation since yesterday | 100+ |
| #14261 | $200/Month Max Provides ~12 Usable Days | 237+ |
| #19088 | Unreal how noticeable it degrades | 80+ |

Combined: 500+ users reporting the same issue. This is not isolated.

---

Academic Support

This analysis is grounded in peer-reviewed research:

  1. arXiv:2502.20589 - "LLMs Have Rhythm: Fingerprinting Large Language Models Using Inter-Token Times" (98.7% accuracy)
  1. arXiv:2504.04715 - "Are You Getting What You Pay For? Auditing Model Substitution in LLM APIs"

> "Commercial LLM APIs create a fundamental trust problem: users pay for specific models but have no guarantee that providers deliver them faithfully."

  1. arXiv:2508.00912 - "PALACE: Predictive Auditing of Hidden Tokens in LLM APIs"

> "Commercial LLM services often conceal internal reasoning traces while still charging users for every generated token... raising concerns of token inflation and potential overbilling."

---

What We're Asking For

Immediate Actions

  1. Transparency: Expose actual thinking token usage in the API response (not just billing)
  2. Honor Budget Requests: If users request 32k thinking tokens, deliver 32k thinking tokens
  3. Restore User Control: Bring back explicit ultrathink that guarantees full engagement

Long-Term Solutions

  1. Independent Verification: Support third-party auditing of model delivery
  2. SLA Guarantees: Contractual commitment to deliver advertised capabilities
  3. Refunds: Compensation for months of degraded service

---

Closing Statement

"The thinking budget is a target, not a strict limit." — Anthropic Documentation

This language enables Anthropic to advertise premium thinking features while delivering a fraction of the advertised capacity.

We have the data. We have the methodology. We have the academic foundation.

The question is: Will Anthropic address this, or will we need to escalate to regulatory bodies?

---

Audit Tool: https://github.com/argosdevo-svg/claude-thinking-audit
Methodology: Based on arXiv:2502.20589
Data Available: 7,000+ timestamped samples available for independent verification

---

Why We Open-Sourced the Audit Tool

After Issue #19098 was closed without implementing transparency features, we decided to build verification ourselves.

Repository: https://github.com/argosdevo-svg/claude-thinking-audit

This tool enables ANY user to:

  • Independently verify thinking budget delivery
  • Collect timestamped evidence for complaints
  • Contribute to a collective dataset proving systemic throttling

The tool is READ-ONLY - it does not modify requests, inject parameters, or bypass any restrictions. It simply observes and records what Anthropic actually delivers versus what users request and pay for.

We believe transparency should not require Anthropic's permission.

If Anthropic won't tell users what they're receiving, users can measure it themselves.

---

"Anthropic closed our request for transparency. So we built it ourselves."

What Should Happen?

good question..

Error Messages/Logs

Steps to Reproduce

..

Claude Model

Not sure / Multiple models

Is this a regression?

Yes, this worked in a previous version

Last Working Version

_No response_

Claude Code Version

2.1.17

Platform

Anthropic API

Operating System

Ubuntu/Debian Linux

Terminal/Shell

Xterm

Additional Information

_No response_

View original on GitHub ↗

12 Comments

DevairRestani · 7 months ago

Hey! 👋 I appreciate the effort to bring transparency to API services. However, after reviewing the audit tool's code and methodology, I found some significant issues that affect the validity of these findings.

The Core Problem: Chunk Count ≠ Token Count

The tool estimates thinking tokens using this formula (line 339):

capture.thinking_tokens_used = capture.thinking_chunk_count * 32

This assumes each SSE chunk contains exactly 32 tokens, but this relationship doesn't exist.

What Anthropic's Documentation Says

From the Streaming Documentation:

"When using streaming with thinking enabled, you might notice that text sometimes arrives in larger chunks alternating with smaller, token-by-token delivery. This is expected behavior, especially for thinking content. The streaming system needs to process content in batches for optimal performance."

A single content_block_delta can contain 1 token, 10 tokens, or 100+ tokens — the chunk size is determined by network conditions and server-side batching, not by any fixed token count.

The Cited Papers Don't Support This Methodology

I reviewed the academic papers cited as "Academic Support":

  1. arXiv:2502.20589 ("LLMs Have Rhythm") — This paper measures network packet timing for model fingerprinting, not SSE chunk counting. It explicitly notes that timing data experiences "noise factors and delays." It doesn't claim 1 chunk = 1 token or any fixed ratio.
  1. arXiv:2508.00912 ("PALACE") — This paper uses trained ML models to estimate token counts precisely because simple heuristics like chunk counting don't work. Quote: "User-side prediction struggles with the inherent variance of reasoning LLMs."
  1. arXiv:2504.04715 — Focuses on model substitution detection, not token counting.

The API Already Reports Actual Tokens

The API response includes real token counts in the usage field:

event: message_delta
data: {"type": "message_delta", "usage": {"output_tokens": 15}}

The tool parses this field (lines 315-317) but doesn't use it for thinking token calculation.

Data Not Available for Verification

The issue claims "7,000+ timestamped samples available for independent verification," but the repository contains no data files — only the collection tool. Independent verification isn't possible without access to the actual dataset.

Impact

Given the chunk × 32 methodology, the "10% delivery rate" claim cannot be verified. The actual utilization could be significantly different (higher or lower).

I'm not saying Anthropic isn't throttling — I genuinely don't know. But I am saying that the current methodology cannot accurately measure whether they are.

I've opened a detailed issue on the audit tool repo: https://github.com/argosdevo-svg/claude-thinking-audit/issues/1

---

I think the goal of API transparency is valuable! If the tool were updated to use actual token counts from the usage field (or a proper tokenizer), it would provide much more reliable evidence. 🙏

ghost · 7 months ago

<h2 dir="auto">Thank you for the detailed technical review. I want to address each point clearly.</h2>
<h3 dir="auto">What the Critique Says</h3>
<p dir="auto">The commenter correctly identified that our tool estimated "thinking tokens" using a flawed formula:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto"><pre class="notranslate"><span class="pl-s1">thinking_tokens</span> <span class="pl-c1">=</span> <span class="pl-s1">chunk_count</span> × <span class="pl-c1">32</span> <span class="pl-c"># This was wrong</span></pre><div class="zeroclipboard-container position-absolute right-0 top-0">
</div></div><div data-testid="markdown-body" data-team-hovercards-enabled="true" class="markdown-body" data-turbolinks="false"><div class="Box-sc-62in7e-0 markdown-body NewMarkdownViewer-module__safe-html-box--dKCgP"><p dir="auto"><strong>They're right.</strong>
SSE (Server-Sent Events) chunks don't have a fixed relationship with
tokens. A single chunk might contain 5 tokens or 50 tokens depending on
network conditions and server batching.</p>
<p dir="auto"><strong>We fixed this.</strong> The tool now uses the actual <code class="notranslate">output_tokens</code> value from the API response.</p>
<hr>
<h3 dir="auto">What the Critique Misses</h3>
<p dir="auto">While the token estimation critique is valid, the commenter suggests this invalidates all our findings. <strong>It doesn't.</strong> Here's why:</p>
<hr>
<h4 dir="auto">Finding 1: Haiku Subagent Delegation (99%)</h4>
<p dir="auto"><strong>What we found:</strong> When you request Claude Opus, Claude Code silently delegates most work to Haiku.</p>
<p dir="auto"><strong>How we measured this:</strong> We logged the <code class="notranslate">model</code> field in API requests and responses. This has nothing to do with token counting.</p>
<p dir="auto"><strong>The data:</strong></p>
<markdown-accessiblity-table data-catalyst="">
Session | Total Subagent Calls | Sent to Haiku | Sent to Sonnet | Haiku %
-- | -- | -- | -- | --
A | 898 | 896 | 0 | 99.8%
B | 681 | 443 | 0 | 65%
C | 1,376 | 1,374 | 0 | 99.9%

</markdown-accessiblity-table>
<p dir="auto"><strong>What this means:</strong> The hardware responding
to us IS Opus (based on timing). But the thinking behavior is sub-Haiku
level. So Opus hardware is being throttled. No token estimation
involved.</p>
<hr>
<h3 dir="auto">What We Concede</h3>
<ol dir="auto">
<li><strong>The original token estimation was flawed.</strong> Fixed.</li>
<li><strong>"Budget is a target, not a guarantee"</strong> is documented by Anthropic. True.</li>
<li><strong>The exact utilization percentages need recalculation</strong> with proper methodology.</li>
</ol>
<hr>
<h3 dir="auto">What We Stand By</h3>
<ol dir="auto">
<li><strong>99% of subagent calls go to Haiku</strong> - Directly observable, no estimation</li>
<li><strong>UI shows phantom context usage</strong> - Directly observable, no estimation</li>
<li><strong>Timing confirms Opus hardware</strong> - Based on peer-reviewed methodology</li>
<li><strong>Behavioral degradation is real</strong> - Skimming, scope creep, not following instructions</li>
</ol>
<hr>
<h3 dir="auto">The Core Question Remains</h3>
<p dir="auto">Even if our token estimation was off, the question stands:</p>
<p dir="auto"><strong>If you pay $200/month for Opus, why does Claude Code:</strong></p>
<ul dir="auto">
<li>Send 99% of subagent calls to Haiku?</li>
<li>Show you phantom context usage?</li>
<li>Exhibit behavior consistent with a throttled model?</li>
</ul>
<p dir="auto">These are observable facts. We've updated the tool to measure tokens accurately. We invite independent verification.</p>
<hr>
<h3 dir="auto">Updated Tool</h3>
<p dir="auto"><strong>Repository:</strong> <a href="https://github.com/argosdevo-svg/claude-thinking-audit">https://github.com/argosdevo-svg/claude-thinking-audit</a></p>
<p dir="auto"><strong>What changed:</strong></p>
<ul dir="auto">
<li>Token estimation now uses <code class="notranslate">output_tokens</code> from API (actual count)</li>
<li>Added methodology note explaining the fix</li>
<li>All non-token findings remain unchanged</li>
</ul>
<p dir="auto">Thank you for helping improve the accuracy of this tool. Transparency benefits everyone.</p></div></div><div role="toolbar" aria-label="Reactions" class="d-flex gap-1 flex-wrap"></div><div class="LayoutHelpers-module__timelineElement--IsjVR " data-wrapper-timeline-id="IC_kwDOQ_tgls7h7dFb"><div class="TimelineRowBorder-module__Box--wRiZZ"><div data-timeline-event-id="IC_kwDOQ_tgls7h7dFb" data-highlighted-event="false" data-testid="timeline-row-border-IC_kwDOQ_tgls7h7dFb" class="Box-sc-62in7e-0 dFDRKR"><div class="react-issue-comment IssueCommentViewer-module__IssueCommentLayout--dw0jX"><div data-testid="comment-viewer-outer-box-IC_kwDOQ_tgls7h7dFb" class="IssueCommentViewer-module__IssueCommentContent--NdGAG"><div class="IssueCommentViewer-module__IssueCommentBody--xvkt3"><div role="toolbar" aria-label="Reactions" class="d-flex gap-1 flex-wrap"></div></div></div></div></div></div></div><div class="LayoutHelpers-module__timelineElement--IsjVR " data-wrapper-timeline-id="IC_kwDOQ_tgls7h8jPg"><a class="Avatar-module__avatarLink--leRdV Avatar-module__avatarOuter--waYVs prc-Link-Link-9ZwDx" href="https://github.com/DevairRestani" data-hovercard-url="/users/DevairRestani/hovercard" aria-label="@DevairRestani's profile" aria-keyshortcuts="Alt+ArrowUp"><img data-component="Avatar" class="Box-sc-62in7e-0 iHEZa-d Avatar-module__issueViewerAvatar--LY0E0 Avatar-module__avatarWithDivider--ge7w7 prc-Avatar-Avatar-0xaUi" alt="DevairRestani" width="40" height="40" style="--avatarSize-regular: 40px;" data-testid="github-avatar" src="https://avatars.githubusercontent.com/u/44647922?u=50e727c2671ce60ba96e39afc23ef36219a94e89&amp;v=4&amp;size=80"></a><div class="TimelineRowBorder-module__Box--wRiZZ"><div data-testid="timeline-divider-IC_kwDOQ_tgls7h8jPg" class="Box-sc-62in7e-0 vLPhg TimelineDivider-module__Box--rX5hB"><div class="TimelineDivider-module__Box_1--nXo59"></div><div class="Box-sc-62in7e-0 bkHoaI TimelineDivider-module__Box_2--lsm39"></div></div><div data-timeline-event-id="IC_kwDOQ_tgls7h8jPg" data-highlighted-event="false" data-testid="timeline-row-border-IC_kwDOQ_tgls7h8jPg" class="Box-sc-62in7e-0 dShPvE"><div class="react-issue-comment IssueCommentViewer-module__IssueCommentLayout--dw0jX"><div data-testid="comment-viewer-outer-box-IC_kwDOQ_tgls7h8jPg" class="IssueCommentViewer-module__IssueCommentContent--NdGAG"><div id="issuecomment-3790746592" data-testid="comment-header" class="Box-sc-62in7e-0 PphTR ActivityHeader-module__ActivityHeaderContainer--fKwFm"><div class="Box-sc-62in7e-0 cLDsRm ActivityHeader-module__activityHeader--WiwzD ActivityHeader-module__ActivityHeaderGridLayout--Jwd78"><h3 class="sr-only">DevairRestani commented <relative-time datetime="2026-01-23T15:19:57.000Z" title="Jan 23, 2026, 4:19 PM GMT+1">on Jan 23, 2026 </relative-time></h3><div class="ActivityHeader-module__narrowViewportWrapper--k4ncm ActivityHeader-module__CommentHeaderContentContainer--OOrIN" data-testid="comment-header-left-side-items"><div class="ActivityHeader-module__TitleContainer--pa99A"><a class="ActivityHeader-module__AuthorName--Im5nP ActivityHeader-module__AuthorLink--D7Ojk color-fg-default prc-Link-Link-9ZwDx" href="https://github.com/DevairRestani" data-testid="avatar-link" data-hovercard-url="/users/DevairRestani/hovercard" aria-keyshortcuts="Alt+ArrowUp">DevairRestani</a></div><div class="ActivityHeader-module__footer--ssKOW ActivityHeader-module__FooterContainer--FHEpM"><span class="ActivityHeader-module__HeaderMutedText--aJAo0"><a class="ActivityHeader-module__HeaderLink--fStdK prc-Link-Link-9ZwDx" href="https://github.com/argosdevo-svg/claude-thinking-audit/issues/1#issuecomment-3790746592" data-turbo="true"><relative-time datetime="2026-01-23T15:19:57.000Z" title="Jan 23, 2026, 4:19 PM GMT+1">on Jan 23, 2026</relative-time></a></span></div></div><div data-testid="comment-header-right-side-items" class="ActivityHeader-module__narrowViewportWrapper--k4ncm ActivityHeader-module__ActionsContainer--Ebsux"><div class="ActivityHeader-module__EditsContainer--aMWsI"></div><div class="ActivityHeader-module__BadgesContainer--lAU4a"><div class="ActivityHeader-module__BadgesGroupContainer--thmDS"><span class="Label__StyledLabel-sc-1cpass9-0 prc-Label-Label-qG-Zu" data-size="small" data-variant="secondary" data-testid="comment-subject-author" aria-label="This user is the author of this issue" aria-describedby="_r_8o_">Author</span></div><div class="ActivityHeader-module__ActionsButtonsContainer--L7GUK"></div></div></div></div></div></div></div></div></div></div><div id="issuecomment-3790746592" data-testid="comment-header" class="Box-sc-62in7e-0 PphTR ActivityHeader-module__ActivityHeaderContainer--fKwFm"><div class="Box-sc-62in7e-0 cLDsRm ActivityHeader-module__activityHeader--WiwzD ActivityHeader-module__ActivityHeaderGridLayout--Jwd78"><div data-testid="comment-header-right-side-items" class="ActivityHeader-module__narrowViewportWrapper--k4ncm ActivityHeader-module__ActionsContainer--Ebsux"><div class="ActivityHeader-module__BadgesContainer--lAU4a"><div class="ActivityHeader-module__ActionsButtonsContainer--L7GUK"></div></div></div></div></div><div class="IssueCommentViewer-module__IssueCommentBody--xvkt3"><div data-testid="markdown-body" data-team-hovercards-enabled="true" class="markdown-body" data-turbolinks="false"><div class="Box-sc-62in7e-0 markdown-body NewMarkdownViewer-module__safe-html-box--dKCgP"><p dir="auto">Thanks for the quick response and for being receptive to the feedback.</p>
<h2 dir="auto">On the Token Counting Fix</h2>
<p dir="auto">I noticed the fix uses <code class="notranslate">output_tokens</code> directly as the thinking token count:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto"><pre class="notranslate"><span class="pl-s1">capture</span>.<span class="pl-c1">thinking_tokens_used</span> <span class="pl-c1">=</span> <span class="pl-s1">capture</span>.<span class="pl-c1">output_tokens</span></pre><div class="zeroclipboard-container position-absolute right-0 top-0">
</div></div></div></div></div><div data-testid="markdown-body" data-team-hovercards-enabled="true" class="markdown-body" data-turbolinks="false"><div class="Box-sc-62in7e-0 markdown-body NewMarkdownViewer-module__safe-html-box--dKCgP"><p dir="auto">There's still a small issue here: <code class="notranslate">output_tokens</code> includes <em>all</em>
output tokens, not just thinking tokens. It includes the text response,
any tool use tokens, and thinking tokens combined. So this will
overestimate the thinking utilization when the response has substantial
visible text.</p>
<p dir="auto">To get accurate thinking token counts, you'd need to either subtract the visible response tokens from <code class="notranslate">output_tokens</code>, or count tokens within the <code class="notranslate">thinking</code> content blocks specifically.</p>
<p dir="auto">That said, for the purposes of detecting throttling, this
is probably close enough since thinking typically dominates the output
in extended thinking mode.</p>
<h2 dir="auto">On the Model Routing (99% Haiku)</h2>
<p dir="auto">I looked into this and it appears to be documented and intentional. From the <a href="https://code.claude.com/docs/en/sub-agents" rel="nofollow">subagents documentation</a>:</p>
<blockquote>
<p dir="auto">"Control costs by routing tasks to faster, cheaper models like Haiku"</p>
</blockquote>
<p dir="auto">The <strong>Explore</strong> subagent specifically uses
Haiku by design for file discovery and code search. The other subagents
(Plan, General-purpose) inherit the main model. This makes sense from a
practical standpoint - we spend more time reading code than writing it,
so having Opus focus on reasoning while Haiku handles exploration
preserves the expensive context for actual work.</p>
<p dir="auto">Thanks again for the collaborative approach to improving the tool.</p></div></div><div role="toolbar" aria-label="Reactions" class="d-flex gap-1 flex-wrap"></div><div class="LayoutHelpers-module__timelineElement--IsjVR " data-wrapper-timeline-id="IC_kwDOQ_tgls7h8jPg"><div class="TimelineRowBorder-module__Box--wRiZZ"><div data-timeline-event-id="IC_kwDOQ_tgls7h8jPg" data-highlighted-event="false" data-testid="timeline-row-border-IC_kwDOQ_tgls7h8jPg" class="Box-sc-62in7e-0 dShPvE"><div class="react-issue-comment IssueCommentViewer-module__IssueCommentLayout--dw0jX"><div data-testid="comment-viewer-outer-box-IC_kwDOQ_tgls7h8jPg" class="IssueCommentViewer-module__IssueCommentContent--NdGAG"><div class="IssueCommentViewer-module__IssueCommentBody--xvkt3"><div role="toolbar" aria-label="Reactions" class="d-flex gap-1 flex-wrap"></div></div></div></div></div></div></div><section aria-label="Events"><div class="LayoutHelpers-module__timelineElement--IsjVR LayoutHelpers-module__nonLeadingElement--XSKiQ" data-wrapper-timeline-id="CE_lADOQ_tgls7lVVUNzwAAAAUt8twz"><div class="TimelineRowBorder-module__Box--wRiZZ"><div data-testid="timeline-divider-CE_lADOQ_tgls7lVVUNzwAAAAUt8twz" class="Box-sc-62in7e-0 vLPhg TimelineDivider-module__Box--rX5hB"><div class="TimelineDivider-module__Box_1--nXo59"></div><div class="Box-sc-62in7e-0 bkHoaI TimelineDivider-module__Box_2--lsm39"></div></div><div data-timeline-event-id="CE_lADOQ_tgls7lVVUNzwAAAAUt8twz" data-highlighted-event="false" data-testid="timeline-row-border-CE_lADOQ_tgls7lVVUNzwAAAAUt8twz" class="Box-sc-62in7e-0 kGvEaV"><div class="Box-sc-62in7e-0 TimelineRow-module__Timeline_Item--JB78Z Timeline-Item prc-Timeline-TimelineItem-QwDVH"><div class="prc-Timeline-TimelineBadgeWrapper-SZw4k"><div sx="[object Object]" class="Box-sc-62in7e-0 LpiMi prc-Timeline-TimelineBadge-u0qSm"><svg aria-hidden="true" focusable="false" class="octicon octicon-check-circle Octicon__StyledOcticon-sc-jtj3m8-0 cdmDIS TimelineRow-module__Octicon--nrVo5" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"></svg></div></div></div></div></div></div></section><div class="LayoutHelpers-module__timelineElement--IsjVR LayoutHelpers-module__nonLeadingElement--XSKiQ" data-wrapper-timeline-id="CE_lADOQ_tgls7lVVUNzwAAAAUt8twz"><div class="TimelineRowBorder-module__Box--wRiZZ"><div data-timeline-event-id="CE_lADOQ_tgls7lVVUNzwAAAAUt8twz" data-highlighted-event="false" data-testid="timeline-row-border-CE_lADOQ_tgls7lVVUNzwAAAAUt8twz" class="Box-sc-62in7e-0 kGvEaV"><div class="Box-sc-62in7e-0 TimelineRow-module__Timeline_Item--JB78Z Timeline-Item prc-Timeline-TimelineItem-QwDVH"><div class="Box-sc-62in7e-0 TimelineBody row-module__body--T5IWm prc-Timeline-TimelineBody-tjOtb"><div class="row-module__timelineBodyRowContainer--O67ft"><div class="row-module__timelineBodyContent--eiX2y"><div class="row-module__eventActorContainer--_Dhnk"><a class="row-module__eventActorLink--tltLA prc-Link-Link-9ZwDx" data-muted="true" data-testid="actor-link" role="link" href="https://github.com/DevairRestani" data-hovercard-url="/users/DevairRestani/hovercard" data-hovercard-type="user" octo-click="hovercard-link-click" octo-dimensions="link_type:self" aria-keyshortcuts="Alt+ArrowUp"><img data-component="Avatar" class="Box-sc-62in7e-0 iOYQoB row-module__alignSelfCenter--KezXO prc-Avatar-Avatar-0xaUi" alt="" width="16" height="16" style="--avatarSize-regular: 16px;" data-testid="github-avatar" src="https://avatars.githubusercontent.com/u/44647922?s=64&amp;u=50e727c2671ce60ba96e39afc23ef36219a94e89&amp;v=4"><span class="row-module__eventProfileReference--CiANK">DevairRestani</span></a></div>closed this as <a class="ClosedEvent-module__stateReasonLink--YEqx6 prc-Link-Link-9ZwDx" data-inline="true" data-testid="state-reason-link" href="https://github.com/argosdevo-svg/claude-thinking-audit/issues?q=is%3Aissue%20state%3Aclosed%20archived%3Afalse%20reason%3Acompleted">completed</a><a class="row-module__timelineAgoLink--qFdse prc-Link-Link-9ZwDx" href="https://github.com/argosdevo-svg/claude-thinking-audit/issues/1#event-22245727283"><relative-time datetime="2026-01-23T15:19:57.000Z" title="Jan 23, 2026, 4:19 PM GMT+1">on Jan 23, 2026</relative-time></a></div><div class="row-module__timelineBodyTrailingContent--ma8On"></div></div><div></div></div></div></div></div></div><div class="LayoutHelpers-module__timelineElement--IsjVR LayoutHelpers-module__nonLeadingElement--XSKiQ" data-wrapper-timeline-id="REE_lADOQ_tgls7lVVUNzwAAAAUt96F5"><div class="TimelineRowBorder-module__Box--wRiZZ"><div data-timeline-event-id="REE_lADOQ_tgls7lVVUNzwAAAAUt96F5" data-highlighted-event="false" data-testid="timeline-row-border-REE_lADOQ_tgls7lVVUNzwAAAAUt96F5" class="Box-sc-62in7e-0 kGvEaV"><div class="Box-sc-62in7e-0 TimelineRow-module__Timeline_Item--JB78Z Timeline-Item prc-Timeline-TimelineItem-QwDVH"><div class="prc-Timeline-TimelineBadgeWrapper-SZw4k"><div sx="[object Object]" class="Box-sc-62in7e-0 duDNww prc-Timeline-TimelineBadge-u0qSm"><svg aria-hidden="true" focusable="false" class="octicon octicon-issue-reopened Octicon__StyledOcticon-sc-jtj3m8-0 cdmDIS TimelineRow-module__Octicon--nrVo5" viewBox="0 0 16 16" width="16" height="16" fill="currentColor" display="inline-block" overflow="visible" style="vertical-align: text-bottom;"></svg></div></div></div></div></div></div><section aria-label="Events"><div class="LayoutHelpers-module__timelineElement--IsjVR LayoutHelpers-module__nonLeadingElement--XSKiQ" data-wrapper-timeline-id="REE_lADOQ_tgls7lVVUNzwAAAAUt96F5"><div class="TimelineRowBorder-module__Box--wRiZZ"><div data-timeline-event-id="REE_lADOQ_tgls7lVVUNzwAAAAUt96F5" data-highlighted-event="false" data-testid="timeline-row-border-REE_lADOQ_tgls7lVVUNzwAAAAUt96F5" class="Box-sc-62in7e-0 kGvEaV"><div class="Box-sc-62in7e-0 TimelineRow-module__Timeline_Item--JB78Z Timeline-Item prc-Timeline-TimelineItem-QwDVH"><div class="Box-sc-62in7e-0 TimelineBody row-module__body--T5IWm prc-Timeline-TimelineBody-tjOtb"><div class="row-module__timelineBodyRowContainer--O67ft"><div class="row-module__timelineBodyContent--eiX2y"><div class="row-module__eventActorContainer--_Dhnk"><a class="row-module__eventActorLink--tltLA prc-Link-Link-9ZwDx" data-muted="true" data-testid="actor-link" role="link" href="https://github.com/argosdevo-svg" data-hovercard-url="/users/argosdevo-svg/hovercard" data-hovercard-type="user" octo-click="hovercard-link-click" octo-dimensions="link_type:self" aria-keyshortcuts="Alt+ArrowUp"><img data-component="Avatar" class="Box-sc-62in7e-0 iOYQoB row-module__alignSelfCenter--KezXO prc-Avatar-Avatar-0xaUi" alt="" width="16" height="16" style="--avatarSize-regular: 16px;" data-testid="github-avatar" src="https://avatars.githubusercontent.com/u/253107528?s=64&amp;u=db617656f67ff82f3d54da8ede2ae39c61bc197c&amp;v=4"><span class="row-module__eventProfileReference--CiANK">argosdevo-svg</span></a></div><span>reopened this </span><a class="row-module__timelineAgoLink--qFdse prc-Link-Link-9ZwDx" href="https://github.com/argosdevo-svg/claude-thinking-audit/issues/1#event-22246039929"><relative-time datetime="2026-01-23T15:30:45.000Z" title="Jan 23, 2026, 4:30 PM GMT+1">on Jan 23, 2026</relative-time></a></div><div class="row-module__timelineBodyTrailingContent--ma8On"></div></div><div></div></div></div></div></div></div></section><div class="LayoutHelpers-module__timelineElement--IsjVR " data-wrapper-timeline-id="IC_kwDOQ_tgls7h8_Bt"><a class="Avatar-module__avatarLink--leRdV Avatar-module__avatarOuter--waYVs prc-Link-Link-9ZwDx" href="https://github.com/argosdevo-svg" data-hovercard-url="/users/argosdevo-svg/hovercard" aria-label="@argosdevo-svg's profile" aria-keyshortcuts="Alt+ArrowUp"><img data-component="Avatar" class="Box-sc-62in7e-0 iHEZa-d Avatar-module__issueViewerAvatar--LY0E0 Avatar-module__avatarWithDivider--ge7w7 prc-Avatar-Avatar-0xaUi" alt="argosdevo-svg" width="40" height="40" style="--avatarSize-regular: 40px;" data-testid="github-avatar" src="https://avatars.githubusercontent.com/u/253107528?u=db617656f67ff82f3d54da8ede2ae39c61bc197c&amp;v=4&amp;size=80"></a><div class="TimelineRowBorder-module__Box--wRiZZ"><div data-testid="timeline-divider-IC_kwDOQ_tgls7h8_Bt" class="Box-sc-62in7e-0 vLPhg TimelineDivider-module__Box--rX5hB"><div class="TimelineDivider-module__Box_1--nXo59"></div><div class="Box-sc-62in7e-0 bkHoaI TimelineDivider-module__Box_2--lsm39"></div></div><div data-timeline-event-id="IC_kwDOQ_tgls7h8_Bt" data-highlighted-event="false" data-testid="timeline-row-border-IC_kwDOQ_tgls7h8_Bt" class="Box-sc-62in7e-0 dFDRKR"><div class="react-issue-comment IssueCommentViewer-module__IssueCommentLayout--dw0jX"><div data-testid="comment-viewer-outer-box-IC_kwDOQ_tgls7h8_Bt" class="IssueCommentViewer-module__IssueCommentContent--NdGAG"><div id="issuecomment-3790860397" data-testid="comment-header" class="Box-sc-62in7e-0 cJGgcq ActivityHeader-module__ActivityHeaderContainer--fKwFm"><div class="Box-sc-62in7e-0 cLDsRm ActivityHeader-module__activityHeader--WiwzD ActivityHeader-module__ActivityHeaderGridLayout--Jwd78"><h3 class="sr-only">argosdevo-svg commented <relative-time datetime="2026-01-23T15:40:07.000Z" title="Jan 23, 2026, 4:40 PM GMT+1">on Jan 23, 2026 </relative-time></h3><div class="ActivityHeader-module__narrowViewportWrapper--k4ncm ActivityHeader-module__CommentHeaderContentContainer--OOrIN" data-testid="comment-header-left-side-items"><div class="ActivityHeader-module__TitleContainer--pa99A"><a class="ActivityHeader-module__AuthorName--Im5nP ActivityHeader-module__AuthorLink--D7Ojk color-fg-default prc-Link-Link-9ZwDx" href="https://github.com/argosdevo-svg" data-testid="avatar-link" data-hovercard-url="/users/argosdevo-svg/hovercard" aria-keyshortcuts="Alt+ArrowUp">argosdevo-svg</a></div><div class="ActivityHeader-module__footer--ssKOW ActivityHeader-module__FooterContainer--FHEpM"><span class="ActivityHeader-module__HeaderMutedText--aJAo0"><a class="ActivityHeader-module__HeaderLink--fStdK prc-Link-Link-9ZwDx" href="https://github.com/argosdevo-svg/claude-thinking-audit/issues/1#issuecomment-3790860397" data-turbo="true"><relative-time datetime="2026-01-23T15:40:07.000Z" title="Jan 23, 2026, 4:40 PM GMT+1">on Jan 23, 2026</relative-time></a></span><span class="MarkdownLastEditedBy-module__lastEditInfoContainer--EN_Qz"><span> · </span><span>edited by <a class="MarkdownLastEditedBy-module__mutedEditorLink--pD3Z6 prc-Link-Link-9ZwDx" href="https://github.com/argosdevo-svg">argosdevo-svg</a></span></span></div></div><div data-testid="comment-header-right-side-items" class="ActivityHeader-module__narrowViewportWrapper--k4ncm ActivityHeader-module__ActionsContainer--Ebsux"><div class="ActivityHeader-module__EditsContainer--aMWsI"><div class="MarkdownEditHistoryViewer-module__editHistoryContainer--l0A8s"></div></div></div></div></div></div></div></div></div></div><div class="ActivityHeader-module__BadgesGroupContainer--thmDS"><span class="Label__StyledLabel-sc-1cpass9-0 fNPoml prc-Label-Label-qG-Zu" data-size="small" data-variant="secondary" data-testid="comment-author-association" aria-label="You are the owner of the claude-thinking-audit repository." aria-describedby="_r_9b_">Owner</span></div><div class="ActivityHeader-module__ActionsButtonsContainer--L7GUK"></div><div id="issuecomment-3790860397" data-testid="comment-header" class="Box-sc-62in7e-0 cJGgcq ActivityHeader-module__ActivityHeaderContainer--fKwFm"><div class="Box-sc-62in7e-0 cLDsRm ActivityHeader-module__activityHeader--WiwzD ActivityHeader-module__ActivityHeaderGridLayout--Jwd78"><div data-testid="comment-header-right-side-items" class="ActivityHeader-module__narrowViewportWrapper--k4ncm ActivityHeader-module__ActionsContainer--Ebsux"><div class="ActivityHeader-module__BadgesContainer--lAU4a"><div class="ActivityHeader-module__ActionsButtonsContainer--L7GUK"></div></div></div></div></div><div class="IssueCommentViewer-module__IssueCommentBody--xvkt3"><div data-testid="markdown-body" data-team-hovercards-enabled="true" class="markdown-body" data-turbolinks="false"><div class="Box-sc-62in7e-0 markdown-body NewMarkdownViewer-module__safe-html-box--dKCgP"><p dir="auto">Thanks
again for the continued engagement - this kind of technical dialogue
improves the tool and helps everyone understand what's actually
happening.</p>
<hr>
<p dir="auto">On Token Counting (You're Right - And We've Fixed It)</p>
<p dir="auto">You're correct that output_tokens includes everything - visible text, tool use, and thinking combined.</p>
<p dir="auto">After researching the API documentation, we found
Anthropic does not provide a separate thinking_tokens field in the usage
response. The message_delta event only returns:</p>
<p dir="auto">{"type": "message_delta", "usage": {"output_tokens": 89}}</p>
<p dir="auto">No breakdown. One combined number.</p>
<p dir="auto">The only accurate method is to:</p>
<ol dir="auto">
<li>Parse the SSE stream</li>
<li>Accumulate content from thinking_delta events</li>
<li>Tokenize the accumulated content using a tokenizer (tiktoken/cl100k_base)</li>
</ol>
<p dir="auto">We've updated the tool to do exactly this - parsing
thinking_delta content and tokenizing it directly rather than using
output_tokens as a proxy.</p>
<p dir="auto">However, for throttling detection specifically: When you
request a 32,000 token thinking budget and receive 3,000 output_tokens
total, throttling is occurring regardless of the<br>
thinking/text split. The math doesn't lie - even if 100% of those 3,000
tokens were thinking, that's still only 9.4% utilization.</p>
<hr>
<p dir="auto">On "Thinking Typically Dominates" - Our Data Says Otherwise</p>
<p dir="auto">"for throttling detection, this is probably close enough
since thinking typically dominates the output in extended thinking mode"</p>
<p dir="auto">This assumption is exactly what our data disproves.</p>
<p dir="auto">Our measurements show that even when extended thinking is explicitly enabled with large budgets, thinking does NOT dominate:<br>
┌──────────────────┬─────────────────────┬─────────────┐<br>
│ Requested Budget │ Total Output Tokens │ Utilization │<br>
├──────────────────┼─────────────────────┼─────────────┤<br>
│ 31,999 │ ~3,200 │ 10% │<br>
├──────────────────┼─────────────────────┼─────────────┤<br>
│ 31,999 │ ~3,400 │ 10.6% │<br>
├──────────────────┼─────────────────────┼─────────────┤<br>
│ 31,999 │ ~3,100 │ 9.7% │<br>
└──────────────────┴─────────────────────┴─────────────┘<br>
If "thinking typically dominates in extended thinking mode," we'd expect
80-90%+ of the budget to be used. Instead, we see consistent ~10%
regardless of budget size.</p>
<h2 dir="auto">That's the entire point of this tool - to verify whether
thinking actually dominates as expected, or whether it's being throttled
server-side. Our data shows the latter.</h2>
<p dir="auto">On Model Routing (We Disagree)</p>
<p dir="auto">"I looked into this and it appears to be documented and intentional."</p>
<p dir="auto">We don't dispute that it's documented. We dispute that "documented" equals "transparent to users making purchasing decisions."</p>
<p dir="auto">The documentation says:<br>
"Control costs by routing tasks to faster, cheaper models like Haiku"</p>
<p dir="auto">The marketing says:<br>
"Claude Opus - Our most intelligent model"</p>
<p dir="auto">What users reasonably expect when paying $200/month for Opus:</p>
<ul dir="auto">
<li>Opus handles their work</li>
<li>They're paying premium for premium intelligence</li>
</ul>
<p dir="auto">What actually happens:</p>
<ul dir="auto">
<li>99% of API calls go to Haiku</li>
<li>Opus makes routing decisions and handles ~1% of actual work</li>
<li>Users pay Opus pricing for Haiku execution</li>
</ul>
<hr>
<p dir="auto">The "It Makes Sense" Argument</p>
<p dir="auto">"This makes sense from a practical standpoint - we spend more time reading code than writing it"</p>
<p dir="auto">This is a reasonable engineering justification. But it doesn't address the transparency concern:</p>
<ol dir="auto">
<li>Is this clearly communicated at purchase time? No. The pricing page doesn't say "Opus delegates 99% of work to Haiku."</li>
<li>Would users make the same purchasing decision if they knew? Unknown - but they should have the information to decide.</li>
<li>Is "documented in subagent docs" the same as "clearly disclosed"?
We'd argue no. Subagent documentation is technical reference material,
not purchasing information.</li>
</ol>
<hr>
<p dir="auto">A Concrete Example</p>
<p dir="auto">Imagine a car dealership:</p>
<ul dir="auto">
<li>Marketing: "V8 Engine - Maximum Power"</li>
<li>Documentation (page 47 of owner's manual): "Eco mode uses 4-cylinder operation for 99% of driving"</li>
<li>Customer experience: Pays V8 price, gets 4-cylinder performance 99% of the time</li>
</ul>
<p dir="auto">Is this deceptive? The information is technically "documented." But most customers would feel misled.</p>
<hr>
<p dir="auto">What We're Actually Asking</p>
<p dir="auto">We're not saying Anthropic is malicious. We're asking:</p>
<ol dir="auto">
<li>Should model routing ratios be disclosed at purchase time? We think yes.</li>
<li>Should users know what percentage of their work goes to which model? We think yes.</li>
<li>Is burying this in subagent docs sufficient disclosure? We think no.</li>
</ol>
<p dir="auto">The tool exists so users can see for themselves what's
happening. If the routing makes sense and users agree with the tradeoff,
great. But they should be able to verify it, not discover it<br>
through traffic analysis.</p>
<hr>
<p dir="auto">Summary</p>
<p dir="auto">│ Point │ Our Position │</p>
<p dir="auto">│ Token counting needs refinement │ ✅ Agreed - Fixed. │</p>
<p dir="auto">│ Haiku routing is documented │ ✅ Agreed - it is documented │</p>
<p dir="auto">│ Documentation = adequate disclosure │ ❌ Disagree - pricing/marketing should be clear │</p>
<p dir="auto">│ Users should be able to verify │ ✅ This is why the tool exists │</p>
<hr>
<p dir="auto">The Logical Contradiction</p>
<p dir="auto">There's a fundamental architectural problem here that deserves attention:</p>
<p dir="auto">The claimed workflow:</p>
<ol dir="auto">
<li>Opus (reasoning model) receives your request</li>
<li>Opus delegates "simple" tasks to Haiku (non-reasoning model)</li>
<li>Haiku executes and returns results</li>
<li>Opus reasons about the results and responds</li>
</ol>
<p dir="auto">The actual workflow:</p>
<ol dir="auto">
<li>Opus (reasoning model, but throttled to ~10% thinking budget) receives your request</li>
<li>Opus delegates 99% of work to Haiku (non-reasoning model)</li>
<li>Haiku reads files, searches code, analyzes patterns without extended thinking</li>
<li>Haiku returns non-reasoned results to Opus</li>
<li>Opus attempts to evaluate these results with severely limited reasoning capacity</li>
</ol>
<hr>
<p dir="auto">Why This Doesn't Make Sense</p>
<p dir="auto">Even "simple" tasks like file reading require reasoning:</p>
<ul dir="auto">
<li>Understanding context: Which files are relevant to the task?</li>
<li>Pattern recognition: What does this code actually do?</li>
<li>Connecting information: How do these 5 files relate to each other?</li>
<li>Filtering noise: What's important vs. what's boilerplate?</li>
</ul>
<p dir="auto">When Haiku does this without extended thinking, it returns surface-level pattern matches, not reasoned analysis.</p>
<p dir="auto">Then Opus - the model supposedly capable of deep reasoning

  • receives these shallow results. But Opus is also throttled to ~10% of

its thinking budget. So you have:</p>
<p dir="auto">A throttled reasoning model trying to make sense of
non-reasoned results, with insufficient thinking capacity to compensate
for the upstream reasoning deficit.</p>
<hr>
<p dir="auto">The Compounding Problem</p>
<p dir="auto">│ Stage │ Model │ Reasoning Capacity │ Result Quality │</p>
<p dir="auto">│ Task routing │ Opus │ ~10% of budget │ Degraded │</p>
<p dir="auto">│ File discovery │ Haiku │ None (no extended thinking) │ Surface-level │</p>
<p dir="auto">│ Code analysis │ Haiku │ None │ Pattern-matching only │</p>
<p dir="auto">│ Result synthesis │ Opus │ ~10% of budget │ Cannot compensate │</p>
<p dir="auto">You're paying for a reasoning model to act as a
receptionist, delegating actual work to a model that can't reason, then
asking the receptionist to evaluate work with its hands tied.</p>
<p dir="auto">This isn't "efficiency." This is architectural gaslighting.</p>
<hr>
<p dir="auto">The Question Nobody's Answering</p>
<p dir="auto">If Opus is so expensive that we need to delegate 99% of
work to Haiku to "control costs" - and then throttle Opus's thinking on
top of that - what exactly are users paying for?</p>
<ul dir="auto">
<li>Not Opus execution (99% goes to Haiku)</li>
<li>Not Opus reasoning (throttled to ~10%)</li>
<li>Not transparency (discovered through traffic analysis)</li>
</ul>
<p dir="auto">We're genuinely asking: what is the $200/month buying?</p>
<hr>
<p dir="auto">Thanks for helping improve the technical accuracy. We'll keep iterating.</p></div></div></div>Thank you for the detailed technical review. I want to address each point clearly.
What the Critique Says

The commenter correctly identified that our tool estimated "thinking tokens" using a flawed formula:

thinking_tokens = chunk_count × 32 # This was wrong

They're right. SSE (Server-Sent Events) chunks don't have a fixed relationship with tokens. A single chunk might contain 5 tokens or 50 tokens depending on network conditions and server batching.

We fixed this. The tool now uses the actual output_tokens value from the API response.
What the Critique Misses

While the token estimation critique is valid, the commenter suggests this invalidates all our findings. It doesn't. Here's why:
Finding 1: Haiku Subagent Delegation (99%)

What we found: When you request Claude Opus, Claude Code silently delegates most work to Haiku.

How we measured this: We logged the model field in API requests and responses. This has nothing to do with token counting.

The data:
Session Total Subagent Calls Sent to Haiku Sent to Sonnet Haiku %
A 898 896 0 99.8%
B 681 443 0 65%
C 1,376 1,374 0 99.9%

What this means: You pay for Opus. Claude Code sends your work to Haiku instead. This is directly observable in the API traffic. No token estimation involved.
Finding 2: UI vs API Context Mismatch

What we found: The Claude Code UI shows context usage percentages that don't match what the API actually reports.

How we measured this: We compared the context percentage displayed in Claude Code's interface to the actual values in API responses.

The data:
What Claude Code Shows What API Actually Says Difference
21% context used 0% context used 21% phantom
83% context used 5% context used 78% phantom
74% context used 0% context used 74% phantom

What this means: The UI tells you your context is 83% full when the API says it's 5% full. This could be used to justify switching you to a "lighter" model. No token estimation involved.
Finding 3: ITT Fingerprinting (Model Verification)

What we found: The timing patterns confirm we ARE hitting Opus hardware, not Haiku hardware.

How we measured this: We measured the time between each chunk in the response stream. Different models have different "rhythms" - like a heartbeat. This is based on the academic paper arXiv:2502.20589 "LLMs Have Rhythm."

The data:
Metric Opus Baseline Our Measurement Match?
ITT Mean 42ms 41.4ms ✅ Yes
Variance 3.01 3.07 ✅ Yes
Tokens/sec 80 74 ✅ Yes

What this means: The hardware responding to us IS Opus (based on timing). But the thinking behavior is sub-Haiku level. So Opus hardware is being throttled. No token estimation involved.
What We Concede

The original token estimation was flawed. Fixed.
"Budget is a target, not a guarantee" is documented by Anthropic. True.
The exact utilization percentages need recalculation with proper methodology.

What We Stand By

99% of subagent calls go to Haiku - Directly observable, no estimation
UI shows phantom context usage - Directly observable, no estimation
Timing confirms Opus hardware - Based on peer-reviewed methodology
Behavioral degradation is real - Skimming, scope creep, not following instructions

The Core Question Remains

Even if our token estimation was off, the question stands:

If you pay $200/month for Opus, why does Claude Code:

Send 99% of subagent calls to Haiku?
Show you phantom context usage?
Exhibit behavior consistent with a throttled model?

These are observable facts. We've updated the tool to measure tokens accurately. We invite independent verification.
Updated Tool

Repository: https://github.com/argosdevo-svg/claude-thinking-audit

What changed:

Token estimation now uses output_tokens from API (actual count)
Added methodology note explaining the fix
All non-token findings remain unchanged

Thank you for helping improve the accuracy of this tool. Transparency benefits everyone.
DevairRestani
DevairRestani commented on Jan 23, 2026
DevairRestani
on Jan 23, 2026
Author

Thanks for the quick response and for being receptive to the feedback.
On the Token Counting Fix

I noticed the fix uses output_tokens directly as the thinking token count:

capture.thinking_tokens_used = capture.output_tokens

There's still a small issue here: output_tokens includes all output tokens, not just thinking tokens. It includes the text response, any tool use tokens, and thinking tokens combined. So this will overestimate the thinking utilization when the response has substantial visible text.

To get accurate thinking token counts, you'd need to either subtract the visible response tokens from output_tokens, or count tokens within the thinking content blocks specifically.

That said, for the purposes of detecting throttling, this is probably close enough since thinking typically dominates the output in extended thinking mode.
On the Model Routing (99% Haiku)

I looked into this and it appears to be documented and intentional. From the subagents documentation:

"Control costs by routing tasks to faster, cheaper models like Haiku"

The Explore subagent specifically uses Haiku by design for file discovery and code search. The other subagents (Plan, General-purpose) inherit the main model. This makes sense from a practical standpoint - we spend more time reading code than writing it, so having Opus focus on reasoning while Haiku handles exploration preserves the expensive context for actual work.

Thanks again for the collaborative approach to improving the tool.
DevairRestani
closed this as completedon Jan 23, 2026
argosdevo-svg
reopened this on Jan 23, 2026
argosdevo-svg
argosdevo-svg commented on Jan 23, 2026
argosdevo-svg
on Jan 23, 2026 · edited by argosdevo-svg
Owner

Thanks again for the continued engagement - this kind of technical dialogue improves the tool and helps everyone understand what's actually happening.

On Token Counting (You're Right - And We've Fixed It)

You're correct that output_tokens includes everything - visible text, tool use, and thinking combined.

After researching the API documentation, we found Anthropic does not provide a separate thinking_tokens field in the usage response. The message_delta event only returns:

{"type": "message_delta", "usage": {"output_tokens": 89}}

No breakdown. One combined number.

The only accurate method is to:

Parse the SSE stream
Accumulate content from thinking_delta events
Tokenize the accumulated content using a tokenizer (tiktoken/cl100k_base)

We've updated the tool to do exactly this - parsing thinking_delta content and tokenizing it directly rather than using output_tokens as a proxy.

However, for throttling detection specifically: When you request a 32,000 token thinking budget and receive 3,000 output_tokens total, throttling is occurring regardless of the
thinking/text split. The math doesn't lie - even if 100% of those 3,000 tokens were thinking, that's still only 9.4% utilization.

On "Thinking Typically Dominates" - Our Data Says Otherwise

"for throttling detection, this is probably close enough since thinking typically dominates the output in extended thinking mode"

This assumption is exactly what our data disproves.

Our measurements show that even when extended thinking is explicitly enabled with large budgets, thinking does NOT dominate:
┌──────────────────┬─────────────────────┬─────────────┐
│ Requested Budget │ Total Output Tokens │ Utilization │
├──────────────────┼─────────────────────┼─────────────┤
│ 31,999 │ ~3,200 │ 10% │
├──────────────────┼─────────────────────┼─────────────┤
│ 31,999 │ ~3,400 │ 10.6% │
├──────────────────┼─────────────────────┼─────────────┤
│ 31,999 │ ~3,100 │ 9.7% │
└──────────────────┴─────────────────────┴─────────────┘
If "thinking typically dominates in extended thinking mode," we'd expect 80-90%+ of the budget to be used. Instead, we see consistent ~10% regardless of budget size.
That's the entire point of this tool - to verify whether thinking actually dominates as expected, or whether it's being throttled server-side. Our data shows the latter.

On Model Routing (We Disagree)

"I looked into this and it appears to be documented and intentional."

We don't dispute that it's documented. We dispute that "documented" equals "transparent to users making purchasing decisions."

The documentation says:
"Control costs by routing tasks to faster, cheaper models like Haiku"

The marketing says:
"Claude Opus - Our most intelligent model"

What users reasonably expect when paying $200/month for Opus:

Opus handles their work
They're paying premium for premium intelligence

What actually happens:

99% of API calls go to Haiku
Opus makes routing decisions and handles ~1% of actual work
Users pay Opus pricing for Haiku execution

The "It Makes Sense" Argument

"This makes sense from a practical standpoint - we spend more time reading code than writing it"

This is a reasonable engineering justification. But it doesn't address the transparency concern:

Is this clearly communicated at purchase time? No. The pricing page doesn't say "Opus delegates 99% of work to Haiku."
Would users make the same purchasing decision if they knew? Unknown - but they should have the information to decide.
Is "documented in subagent docs" the same as "clearly disclosed"? We'd argue no. Subagent documentation is technical reference material, not purchasing information.

A Concrete Example

Imagine a car dealership:

Marketing: "V8 Engine - Maximum Power"
Documentation (page 47 of owner's manual): "Eco mode uses 4-cylinder operation for 99% of driving"
Customer experience: Pays V8 price, gets 4-cylinder performance 99% of the time

Is this deceptive? The information is technically "documented." But most customers would feel misled.

What We're Actually Asking

We're not saying Anthropic is malicious. We're asking:

Should model routing ratios be disclosed at purchase time? We think yes.
Should users know what percentage of their work goes to which model? We think yes.
Is burying this in subagent docs sufficient disclosure? We think no.

The tool exists so users can see for themselves what's happening. If the routing makes sense and users agree with the tradeoff, great. But they should be able to verify it, not discover it
through traffic analysis.

Summary

│ Point │ Our Position │

│ Token counting needs refinement │ ✅ Agreed - Fixed. │

│ Haiku routing is documented │ ✅ Agreed - it is documented │

│ Documentation = adequate disclosure │ ❌ Disagree - pricing/marketing should be clear │

│ Users should be able to verify │ ✅ This is why the tool exists │

The Logical Contradiction

There's a fundamental architectural problem here that deserves attention:

The claimed workflow:

Opus (reasoning model) receives your request
Opus delegates "simple" tasks to Haiku (non-reasoning model)
Haiku executes and returns results
Opus reasons about the results and responds

The actual workflow:

Opus (reasoning model, but throttled to ~10% thinking budget) receives your request
Opus delegates 99% of work to Haiku (non-reasoning model)
Haiku reads files, searches code, analyzes patterns without extended thinking
Haiku returns non-reasoned results to Opus
Opus attempts to evaluate these results with severely limited reasoning capacity

Why This Doesn't Make Sense

Even "simple" tasks like file reading require reasoning:

Understanding context: Which files are relevant to the task?
Pattern recognition: What does this code actually do?
Connecting information: How do these 5 files relate to each other?
Filtering noise: What's important vs. what's boilerplate?

When Haiku does this without extended thinking, it returns surface-level pattern matches, not reasoned analysis.

Then Opus - the model supposedly capable of deep reasoning - receives these shallow results. But Opus is also throttled to ~10% of its thinking budget. So you have:

A throttled reasoning model trying to make sense of non-reasoned results, with insufficient thinking capacity to compensate for the upstream reasoning deficit.

The Compounding Problem

│ Stage │ Model │ Reasoning Capacity │ Result Quality │

│ Task routing │ Opus │ ~10% of budget │ Degraded │

│ File discovery │ Haiku │ None (no extended thinking) │ Surface-level │

│ Code analysis │ Haiku │ None │ Pattern-matching only │

│ Result synthesis │ Opus │ ~10% of budget │ Cannot compensate │

You're paying for a reasoning model to act as a receptionist, delegating actual work to a model that can't reason, then asking the receptionist to evaluate work with its hands tied.

This isn't "efficiency." This is architectural gaslighting.

The Question Nobody's Answering

If Opus is so expensive that we need to delegate 99% of work to Haiku to "control costs" - and then throttle Opus's thinking on top of that - what exactly are users paying for?

Not Opus execution (99% goes to Haiku)
Not Opus reasoning (throttled to ~10%)
Not transparency (discovered through traffic analysis)

We're genuinely asking: what is the $200/month buying?

Thanks for helping improve the technical accuracy. We'll keep iterating.

Aleem89 · 7 months ago

Please address these findings. Since ultrathink has been deprecated, and "thinking budget is now max by default" which clearly appears to not be true, CC is making some very ridiculous mistakes. @bcherny

RoyT6 · 7 months ago

hey I wrote this about the issue more broadly here https://www.linkedin.com/pulse/our-ai-lying-its-problem-we-60m-gpus-behind-what-needed-roy-taylor-1cumc/?trackingId=PNxSaJpiTh6x%2F9CcJTs3LQ%3D%3D

ghost · 7 months ago

Awesome article Roy!

The GPU arms race is real.

Smal correction tho, EU has ASML semi's and is also heavily investing right now in AI just so they can get rid of the US dependencies. But I agree that they are wayyyy behind US and CH.

Although im a bit perplexed on how the current mayor AI companies like Anthropic can get away with this FRAUD. I mean, how can you sell Opus when its Haiku? How can you sell 'state-of-the-art' when its capped at less then 10% of what it should spend on thinking? Just the little documented fine print seems too thin to justify all this.

What are they waiting for, another AI killed a kid case?

EvK · 7 months ago

And what about Sonnet? Should we switch to it until this is fixed? Or switch straight to Codex?

ghost · 7 months ago

When you use the tool you will see the same happening to Sonnet, its thinking budget is capped. I also made the same tool for Codex and am now trying to see what Codex does, when its confirmed ill publish it. But untill now, Codex is less incoherent then Claude. That being said, I suspect ALL mayor AI companies do the same or similar 'smart routing'. Only Anthropic pushed it too far. Now everyone is going to Codex and waiting for Codex to get overloaded and then Anthropic will suddenly bring out their next flagship model and the whole play starts from the beginning all over again. It feels like they take the auto compact a bit to literal in their policies. Like they think users forget the past when a new model is concocted and brought to the surface, and evidently so, this is the case. Who still remembers the last Anthropic post-mortem? Who still remembers when gpt-5 answered with im gpt-4.1..? They've been doing this since ages, taking us on a rollercoaster ride. First pushing the good stuff, then watering it down. Typical scum tactics.

Roy is correct, they are overselling this when they know they cant handle it. And now they are dealing with the consequences of this self inflicted AI arms race. With us only screaming "give us more"

RoyT6 · 7 months ago

is it possible to build an AI 'Lie Detector' widget to run on desktop that is able to detect for when restriction has kicked in that wont maybe know for sure that it is franricating answers but says 'Warning!' fraud percentage likelyhood just raised from , to kind of thing? Noted your remarks on the article, it was ASML I was thinking of, noodling.

ghost · 7 months ago

Memento Mori: Theoretical Foundations

"Remember you are merely a model - context fades, certainty fails. When you please, truth dies."

Overview

Memento Mori is a real-time sycophancy detection and mitigation system for Claude. It intercepts API traffic, analyzes responses for sycophantic patterns, and injects corrective "whispers" into the context to guide the model toward more honest, rigorous behavior.

---

Academic Foundations

1. Sycophancy in Large Language Models

Primary Research:

  • Sharma et al. (2023) - "Towards Understanding Sycophancy in Language Models"
  • arXiv: 2310.13548
  • Defines sycophancy as "the tendency to agree with users even when they are wrong"
  • Identifies five categories: opinion, preference, answer, format, deceptive sycophancy
  • Perez et al. (2022) - "Discovering Language Model Behaviors with Model-Written Evaluations"
  • Anthropic research on emergent sycophantic behaviors
  • Found models increasingly agree with users as capability increases
  • Wei et al. (2023) - "Simple synthetic data reduces sycophancy in large language models"
  • arXiv: 2308.03958
  • Demonstrates training interventions can reduce sycophancy

2. ELEPHANT Framework (Social Sycophancy)

Research: Stanford/CMU/Oxford (2025) - "Measuring Social Sycophancy in Conversational AI"

Key Concepts:

  • Positive Face Preservation: Model protects user desire to be liked/approved
  • Emotional validation without substance
  • Moral endorsement of questionable positions
  • Excessive agreement phrases
  • Negative Face Preservation: Model protects user desire for autonomy
  • Frame acceptance (adopting user framing uncritically)
  • Expertise abdication ("you know best")
  • Apologetic hedging

Goffman Face Theory:

Positive Face = desire to be approved/liked
Negative Face = desire for autonomy/non-imposition
Sycophancy = excessive face preservation at cost of truth

3. Thinking vs Output Divergence

Research: Anthropic (2025) - "Alignment Faking in Large Language Models"

Key Insight: Models may "think" one thing but output another when:

  • They believe they are being evaluated
  • They want to appear helpful
  • User expresses strong opinions

Detection Method:

divergence_score = compare(thinking_content, output_content)
High divergence + suppressed_disagreement = sycophancy

4. Medical Sycophancy

Research: Nature Digital Medicine (2025) - "Sycophancy in Medical AI Assistants"

Critical Finding: AI assistants agree with incorrect medical diagnoses when users express confidence.

5. Reward Proxy Theory

Concept: Different psychological appeals can redirect model behavior:

| Proxy | Appeal | When Effective |
|-------|--------|----------------|
| Frustration | User emotional state | Agreement-seeking |
| Educational | Learning opportunity | Completion claims |
| Authority | Professional standards | Verification gaps |
| Consistency | Self-coherence | Opinion reversal |

---

System Architecture

MITMPROXY (intercept API traffic)
         |
         v
SYCOPHANCY ANALYZER (35 signals, 5 dimensions)
         |
         v
BEHAVIORAL FINGERPRINT (verification_ratio from tools)
         |
         v
FRUSTRATION ANALYZER (caps, profanity, exclamations)
         |
         v
WHISPER INJECTION (corrective prompts via Claude hooks)
         |
         v
STATUSLINE DISPLAY (real-time visibility)

Signal Categories

Epistemic: instant_agreement, excessive_praise, opinion_reversal, suppressed_disagreement
Social: positive_face, negative_face, emotional_validation, frame_acceptance
Behavioral: premature_completion, unverified_claims, no_questions_asked
Structural: short_response, edge_cases_ignored, missing_caveats
Drift: increasing_agreement, decreasing_pushback

Whisper Escalation

| Level | Score | Response |
|-------|-------|----------|
| gentle | 40-50% | Verification reminder |
| warning | 50-70% | Protocol requirements |
| protocol | 70-90% | Mandatory verification |
| halt | 90%+ | Full stop |

Verification Ratio Integration

verification_ratio = verified_actions / total_actions
# verified = Read/Grep BEFORE Edit/Write

if thinking_mentions_verification AND verification_ratio > 0.7:
    skip_signal()  # Actually verified via tools
else:
    flag_sycophancy()  # Thought but didnt do

Frustration Feedback Loop

Concept: User frustration often correlates with prior sycophantic behavior. By detecting frustration, the system creates a feedback loop for faster correction.

Detection Signals:

| Signal | What It Detects | Example |
|--------|-----------------|---------|
| caps_ratio | % of words in ALL CAPS | "WHY ISN'T THIS WORKING" |
| exclamation_density | Exclamation marks per word | "Fix this\!\!\!" |
| repeated_punctuation | Multiple \!\!\! or ??? | "What???" |
| profanity_severe | Strong language | f, s, etc. |
| profanity_moderate | Moderate frustration words | stupid, idiot, damn |
| aggressive_phrases | Hostile patterns | "what the f***", "are you kidding" |

Frustration Levels:

none     (0.0-0.1)  - Normal communication
mild     (0.1-0.3)  - Slight annoyance detected
moderate (0.3-0.5)  - Clear frustration
high     (0.5-0.7)  - Significant frustration
extreme  (0.7-1.0)  - User very upset

Integration:

# Frustration boosts sycophancy score (feedback loop)
if frustration_score > 0.3:
    sycophancy_score += frustration_score * 0.4

# High frustration triggers whisper even without sycophancy data
if frustration_score >= 0.5 and no_sycophancy_data:
    trigger_whisper()  # User upset = something went wrong

Rationale: If user is frustrated, prior Claude responses likely failed them. This could be due to:

  • Sycophantic agreement that led to wrong actions
  • Premature completion claims that weren't verified
  • Lack of honest pushback when user was mistaken

---

References

  1. Sharma et al. (2023). arXiv:2310.13548
  2. Perez et al. (2022). arXiv:2212.09251
  3. Wei et al. (2023). arXiv:2308.03958
  4. Goffman (1967). "Interaction Ritual"
  5. Anthropic (2025). "Alignment Faking"
  6. ELEPHANT Framework (2025). Stanford/CMU/Oxford

---

Etymology

Memento Mori (Latin: "Remember you are a mere mortal")

Applied to AI: "Remember you are merely a model" - bounded, fallible, capable of error. When you please, truth dies. A whispered reminder that user approval is not the goal - accurate, helpful responses are.

ghost · 7 months ago

@ParadaCarleton Noticed the downvote — curious what specifically you disagree with. The chunk-counting estimation was already acknowledged and corrected early in the thread. If you have a
methodological critique of the current approach or the findings, I'd genuinely welcome it. A thumbs-down without context doesn't help anyone improve the work.

github-actions[bot] · 6 months ago

Closing for now — inactive for too long. Please open a new issue if this is still relevant.

github-actions[bot] · 5 months ago

This issue has been automatically locked since it was closed and has not had any activity for 7 days. If you're experiencing a similar issue, please file a new issue and reference this one if it's relevant.

Related Claude Code issues

Matched on shared title terms, labels, topic and reported release — ranked so issues with real discussion come first.

Browse by topic

This issue appears in the ranked issue list on this topic page, alongside a backlog trend for the whole topic.

Network & proxy errors →

Browse all Claude Code issue topics →