[BUG] getContextUsage fans out one billed Haiku inference per context item on Bedrock application inference profiles

Summary

On Bedrock, CountTokens rejects application-inference-profile/* ARNs. Claude Code's token counter falls back to a real messages.create call with max_tokens: 1 purely to read usage.input_tokens back — and the context-usage breakdown does this once per context item (each category, plus each MCP tool, memory file, agent and skill), with the memoisation wrapper disabled.

One SDK getContextUsage() call therefore costs 26–40 billed Haiku inferences and ~110k–170k uncached input tokens. SDK clients that render a context bar call it on every turn, so this repeats per turn for the whole session.

This is invisible on the Anthropic API, where count_tokens is free — it only bills on Bedrock with application inference profiles.

Environment

• Claude Code CLI 2.1.232 (native installer)
• @anthropic-ai/claude-agent-sdk 0.3.232
• Node v24.14.1, macOS (darwin 25.6.0)
• CLAUDE_CODE_USE_BEDROCK=1, AWS_REGION=<REGION>
• ANTHROPIC_MODEL and ANTHROPIC_DEFAULT_HAIKU_MODEL both set to application inference profile ARNs

Reproduction

Setup — any Bedrock account reproduces this

The trigger is purely that the configured models are application inference profiles rather than foundation-model IDs or system-defined (cross-region) profiles. Create two throwaway profiles in your own account:

# one for the main model, one for the small/fast model
aws bedrock create-inference-profile \
  --inference-profile-name counttokens-repro-main \
  --model-source copyFrom=<ARN of a system-defined inference profile or foundation model, e.g. a Sonnet/Opus one>

aws bedrock create-inference-profile \
  --inference-profile-name counttokens-repro-haiku \
  --model-source copyFrom=<ARN of the Haiku foundation model or its cross-region profile>

Each returns an inferenceProfileArn of the form
arn:aws:bedrock:<REGION>:<ACCOUNT_ID>:application-inference-profile/<PROFILE_ID>. Point Claude Code at them:

export CLAUDE_CODE_USE_BEDROCK=1
export AWS_REGION=<REGION>
export ANTHROPIC_MODEL=<main profile ARN>
export ANTHROPIC_DEFAULT_HAIKU_MODEL=<haiku profile ARN>

Substituting a foundation-model ID or a system-defined inference profile ARN for ANTHROPIC_MODEL makes the bug disappear — CountTokens succeeds and no fallback inference is issued. That contrast is the cleanest confirmation of the cause.

The script

repro.mjs runs one trivial turn and calls getContextUsage() when the result message arrives — what an SDK client does to render a context bar. Pass no as the second argument to skip that call as a negative control.

import { query } from "@anthropic-ai/claude-agent-sdk";

let release;
const keepOpen = new Promise((r) => (release = r));

async function* prompts() {
  yield {
    type: "user",
    parent_tool_use_id: null,
    message: { role: "user", content: "Reply with exactly: OK. Do not use any tools." },
  };
  await keepOpen; // hold the session open so the control request can be answered
}

const q = query({
  prompt: prompts(),
  options: {
    permissionMode: "bypassPermissions",
    extraArgs: { "debug-file": process.argv[2] },
  },
});

for await (const msg of q) {
  if (msg.type === "result") {
    if (process.argv[3] !== "no") {
      const u = await q.getContextUsage();
      console.log(`getContextUsage -> ${u.totalTokens} tokens across ${u.categories.length} categories`);
    }
    release();
    break;
  }
}
await q.return?.();

node repro.mjs "$PWD/with-usage.txt"        # calls getContextUsage
node repro.mjs "$PWD/without-usage.txt" no  # negative control

for f in with-usage.txt without-usage.txt; do
  echo "== $f"; grep -o 'source=[a-z_0-9]*' $f | sort | uniq -c
done

Results

| Run | getContextUsage() | source=count_tokens requests |
|---|---|---|
| with-usage.txt | called | 26 |
| without-usage.txt | not called | 0 |

The two runs are otherwise identical — same prompt, same settings, same session shape. Every one of the 26 is a separate POST /model/<haiku-profile-arn>/invoke, fired as one concurrent burst at turn end, each preceded by:

[ERROR] Bedrock CountTokens failed: ValidationException: The provided model doesn't support counting tokens.
[DEBUG] countTokensWithFallback: [REDACTED] returned null, trying haiku fallback (0 tools)

getContextUsage reported 9 categories while issuing 26 calls, consistent with one call per item across categories[], mcp_tools[], memory_files[], agents[] and skills[]. The count scales with loaded MCP servers and plugins: a session with more MCP servers produced 40 calls / ~170k tokens for a single turn.

Expected

A context-usage breakdown should not cost billed inference calls, and certainly not one per item.

Suggested fixes, smallest first

1. Enable memoisation of token counts. The cache wrapper around the count helper appears to be gated on a function that returns false unconditionally, so nothing is reused between turns. Most items are byte-identical turn to turn (system prompt, tool schemas, memory files, agent and skill definitions) — a content-hash cache would take the steady-state cost close to zero with no behavioural change. This alone fixes the per-turn repetition.
2. Don't use billed inference as the fallback. When CountTokens is unavailable, a local tokenizer estimate is adequate for a UI breakdown; a max_tokens: 1 round trip is not.
3. Batch the fan-out. Count once and apportion, or issue one request covering all items, rather than N concurrent requests.

Related

CountTokens not accepting application inference profiles looks like an AWS-side gap and we're raising it with them separately. But the fallback amplifying one logical operation into 26–40 billed calls is a Claude Code issue independent of that, and fix (1) would make the AWS behaviour largely harmless.