[BUG] getContextUsage fans out one billed Haiku inference per context item on Bedrock application inference profiles

Status Open

Reported on v2.1.232

Maintainer reply None cached

Activity 3 comments · opened Aug 14, 2026

Summary

On Bedrock, CountTokens rejects application-inference-profile/* ARNs. Claude Code's token counter falls back to a real messages.create call with max_tokens: 1 purely to read usage.input_tokens back — and the context-usage breakdown does this once per context item (each category, plus each MCP tool, memory file, agent and skill), with the memoisation wrapper disabled.

One SDK getContextUsage() call therefore costs 26–40 billed Haiku inferences and ~110k–170k uncached input tokens. SDK clients that render a context bar call it on every turn, so this repeats per turn for the whole session.

This is invisible on the Anthropic API, where count_tokens is free — it only bills on Bedrock with application inference profiles.

Environment

Claude Code CLI 2.1.232 (native installer)
@anthropic-ai/claude-agent-sdk 0.3.232
Node v24.14.1, macOS (darwin 25.6.0)
CLAUDE_CODE_USE_BEDROCK=1, AWS_REGION=<REGION>
ANTHROPIC_MODEL and ANTHROPIC_DEFAULT_HAIKU_MODEL both set to application inference profile ARNs

Reproduction

Setup — any Bedrock account reproduces this

The trigger is purely that the configured models are application inference profiles rather than foundation-model IDs or system-defined (cross-region) profiles. Create two throwaway profiles in your own account:

# one for the main model, one for the small/fast model
aws bedrock create-inference-profile \
  --inference-profile-name counttokens-repro-main \
  --model-source copyFrom=<ARN of a system-defined inference profile or foundation model, e.g. a Sonnet/Opus one>

aws bedrock create-inference-profile \
  --inference-profile-name counttokens-repro-haiku \
  --model-source copyFrom=<ARN of the Haiku foundation model or its cross-region profile>

Each returns an inferenceProfileArn of the form
arn:aws:bedrock:<REGION>:<ACCOUNT_ID>:application-inference-profile/<PROFILE_ID>. Point Claude Code at them:

export CLAUDE_CODE_USE_BEDROCK=1
export AWS_REGION=<REGION>
export ANTHROPIC_MODEL=<main profile ARN>
export ANTHROPIC_DEFAULT_HAIKU_MODEL=<haiku profile ARN>

Substituting a foundation-model ID or a system-defined inference profile ARN for ANTHROPIC_MODEL makes the bug disappear — CountTokens succeeds and no fallback inference is issued. That contrast is the cleanest confirmation of the cause.

The script

repro.mjs runs one trivial turn and calls getContextUsage() when the result message arrives — what an SDK client does to render a context bar. Pass no as the second argument to skip that call as a negative control.

import { query } from "@anthropic-ai/claude-agent-sdk";

let release;
const keepOpen = new Promise((r) => (release = r));

async function* prompts() {
  yield {
    type: "user",
    parent_tool_use_id: null,
    message: { role: "user", content: "Reply with exactly: OK. Do not use any tools." },
  };
  await keepOpen; // hold the session open so the control request can be answered
}

const q = query({
  prompt: prompts(),
  options: {
    permissionMode: "bypassPermissions",
    extraArgs: { "debug-file": process.argv[2] },
  },
});

for await (const msg of q) {
  if (msg.type === "result") {
    if (process.argv[3] !== "no") {
      const u = await q.getContextUsage();
      console.log(`getContextUsage -> ${u.totalTokens} tokens across ${u.categories.length} categories`);
    }
    release();
    break;
  }
}
await q.return?.();

node repro.mjs "$PWD/with-usage.txt"        # calls getContextUsage
node repro.mjs "$PWD/without-usage.txt" no  # negative control

for f in with-usage.txt without-usage.txt; do
  echo "== $f"; grep -o 'source=[a-z_0-9]*' $f | sort | uniq -c
done

Results

| Run | getContextUsage() | source=count_tokens requests |
|---|---|---|
| with-usage.txt | called | 26 |
| without-usage.txt | not called | 0 |

The two runs are otherwise identical — same prompt, same settings, same session shape. Every one of the 26 is a separate POST /model/<haiku-profile-arn>/invoke, fired as one concurrent burst at turn end, each preceded by:

[ERROR] Bedrock CountTokens failed: ValidationException: The provided model doesn't support counting tokens.
[DEBUG] countTokensWithFallback: [REDACTED] returned null, trying haiku fallback (0 tools)

getContextUsage reported 9 categories while issuing 26 calls, consistent with one call per item across categories[], mcp_tools[], memory_files[], agents[] and skills[]. The count scales with loaded MCP servers and plugins: a session with more MCP servers produced 40 calls / ~170k tokens for a single turn.

Expected

A context-usage breakdown should not cost billed inference calls, and certainly not one per item.

Suggested fixes, smallest first

Enable memoisation of token counts. The cache wrapper around the count helper appears to be gated on a function that returns false unconditionally, so nothing is reused between turns. Most items are byte-identical turn to turn (system prompt, tool schemas, memory files, agent and skill definitions) — a content-hash cache would take the steady-state cost close to zero with no behavioural change. This alone fixes the per-turn repetition.
Don't use billed inference as the fallback. When CountTokens is unavailable, a local tokenizer estimate is adequate for a UI breakdown; a max_tokens: 1 round trip is not.
Batch the fan-out. Count once and apportion, or issue one request covering all items, rather than N concurrent requests.

CountTokens not accepting application inference profiles looks like an AWS-side gap and we're raising it with them separately. But the fallback amplifying one logical operation into 26–40 billed calls is a Claude Code issue independent of that, and fix (1) would make the AWS behaviour largely harmless.

View original on GitHub ↗

3 Comments

dafzthomas · 1 month ago

This reproduces in the plain CLI with no SDK involved, which localises it to the CLI rather than to getContextUsage().

CLI 2.1.246, same Bedrock application-inference-profile setup as above:

claude -p "/context" --debug-file /tmp/ctx.txt # 37 × source=count_tokens
claude -p "Reply with exactly: OK" --debug-file /tmp/ctrl.txt # 0 × source=count_tokens

grep -c 'CountTokens failed' /tmp/ctx.txt # 37
grep -c 'CountTokens failed' /tmp/ctrl.txt # 0

Each of the 37 is preceded by the same pair:

[ERROR] Bedrock CountTokens failed: ValidationException: The provided model doesn't support counting tokens.
[DEBUG] countTokensWithFallback: [REDACTED] returned null, trying haiku fallback (0 tools)

So /context and getContextUsage() share one breakdown path. The 26 → 37 difference between the two sessions is just more plugins and MCP servers loaded — the call count tracks item count, as described.

Related: #80477 reports the same billed-Haiku fallback on Bedrock application inference profiles from a different surface (Desktop sandbox blocking the local tokenizer), filed independently. Both are defused by fix (1) — enabling the memoisation gate. The fan-out only bills because nothing is reused between items or between turns.

radelqui · 1 month ago

The quickest way to stop the SDK from paying a separate Haiku inference for every context item is to avoid using an application‑inference‑profile ARN when the SDK needs to call CountTokens. The CountTokens endpoint only works with a foundation‑model ID (or a system‑wide cross‑region profile), so give it a plain model ARN and keep your custom profile only for the actual messages.create calls.

In your AWS console create or locate the standard Claude‑3‑Haiku model ARN, e.g. arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-haiku-20240307-v1:0.
Set the environment variables that the SDK reads to this ARN instead of your custom profile:

export ANTHROPIC_MODEL=arn:aws:bedrock:…/foundation-model/anthropic.claude-3-haiku-20240307-v1:0
export ANTHROPIC_DEFAULT_HAIKU_MODEL=$ANTHROPIC_MODEL.

Keep your application‑inference‑profile ARN only in the variable that the CLI uses for generation, e.g. export CLAUDE_CODE_PROFILE_ARN=arn:aws:bedrock:…/inference-profile/your‑profile. The SDK will still send the full request to that profile but will call CountTokens against the foundation model, which is free on Bedrock.
Restart the Claude Code CLI or your node process so the new env vars are read.

After this change getContextUsage() will hit CountTokens once per turn, not once per context item, and you’ll stop paying 26–40 extra Haiku inferences.

Full disclosure: I'm the developer of Multi Instance for Claude Desktop, which runs several isolated Claude Desktop instances at once. The app launches separate child processes for each instance and routes their STDIO independently.
https://apps.microsoft.com/detail/9NG247TJ47P0?cid=gh-multi

dimamix · 1 month ago

Summary: Second trigger for the same fallback, on a system-defined cross-region profile rather than an application inference profile, and one correction to the "haiku fallback" description: on Bedrock the fallback bills whatever the small-fast-model resolves to, which is the session's main model unless a Haiku model is explicitly configured. When the model is one bedrock-runtime CountTokens doesn't support (Sonnet 5 today), the profile type doesn't matter: CountTokens fails, the counter falls back to billed 1-token inferences on the main model, and nothing in the SDK's usage reporting shows it. We ran this in production for a month before finding it by reconciling CloudTrail against the AWS bill.

Setup: Agent SDK (TypeScript) 0.3.246 / CLI 2.1.246 (same CLI as the comment above), CLAUDE_CODE_USE_BEDROCK=1, us-east-1, global.anthropic.claude-sonnet-5 and global.anthropic.claude-opus-5 (system-defined, no application profiles), no ANTHROPIC_DEFAULT_HAIKU_MODEL or ANTHROPIC_SMALL_FAST_MODEL set. Our service called query.getContextUsage() after every assistant message to log a context breakdown.

What's new relative to the report above:

Unsupported model, not profile type. aws bedrock-runtime count-tokens returns ValidationException: The provided model doesn't support counting tokens for anthropic.claude-sonnet-5 and for both its us. and global. profiles. Sonnet 5's Bedrock model card lists Count tokens as Not Supported on bedrock-runtime and Supported on bedrock-mantle. Haiku 4.5's foundation-model id does count. So for Sonnet 5 the fallback is unconditional, and the workaround above (pointing the model variables at a foundation model) cannot apply.

The "haiku fallback" bills the main model unless Haiku is configured. Reading countTokensWithFallback in the 2.1.246 binary: the fallback request's model comes from the small-fast-model getter, which returns Haiku only when ANTHROPIC_SMALL_FAST_MODEL or ANTHROPIC_DEFAULT_HAIKU_MODEL is set; otherwise, on Bedrock with a --model passed (the SDK always passes one), it returns the main-loop model. Our CloudTrail for one day: 136 non-streaming InvokeModel calls on Sonnet 5, 114 on Opus 5, zero on Haiku, all claude-cli/2.1.246, each max_tokens: 1 with tool schemas attached and no cache_control. Opus sessions paid Opus input rates for their token counts.

An IAM prerequisite before CountTokens is even tried. The counter first resolves the profile with GetInferenceProfile. Our task role granted only InvokeModel*, so that step was AccessDenied and the counter went to the fallback without ever calling CountTokens. Granting it would not have changed the outcome because of point 1, but it is a second silent exit on the same path.

Production scale, and invisible to usage. Over one month, CloudTrail shows about 25,000 of these calls from our production role, roughly 12 per streaming turn. CloudWatch shows 403M uncached input tokens against 143M cache reads account-wide; on a production-only day, cache read and write matched our streaming turns exactly and every uncached token sat on the non-streaming calls. The uncached input was roughly half of our Sonnet 5 input spend. Through all of it the SDK's streaming usage reported 2 uncached input tokens per turn and total_cost_usd never moved. Daily uncached input went from ~283K tokens to 62.6M the day the getContextUsage() call shipped, and back to zero the day we turned it off.

bedrock-mantle can count Sonnet 5. A SigV4-signed POST https://bedrock-mantle.<region>.api.aws/anthropic/v1/messages/count_tokens with model: anthropic.claude-sonnet-5 returns HTTP 200. Profile ids 404 there. IAM action bedrock-mantle:CountTokens. The CLI's counter only tries bedrock-runtime today.

Questions:

Was getContextUsage({ detail: 'summary' }) in Agent SDK 0.3.257 intended as the fix for this? The changelog entry doesn't link an issue.
Would you consider making the billed fallback opt-in, or at least surfacing it without debug (a stderr warning, and/or counting these calls in usage)? On Bedrock it can cost more than the turn it is measuring, at main-model rates, and today the only trace is the debug log.
Is routing the counter to bedrock-mantle's count_tokens, when CLAUDE_CODE_USE_MANTLE is set or the model lacks bedrock-runtime CountTokens support, on the roadmap? For Sonnet 5 it is the only free path.