claude-api bundled skill inlines ~227K tokens on invocation — progressive disclosure defeated by bundled packaging

Status Open
Reported on v2.1.220
Maintainer reply None cached
Activity 5 comments · opened Jul 26, 2026

Note on venue: verified before filing — the skill source at anthropics/skills uses an on-demand file-read model with no inlined reference docs, while the bundled variant delivers those files pre-rendered. The regression is in the bundled-skill packaging/render step, so filing here rather than anthropics/skills. Distinct from #14882 / #14834 / #16616 / #39686, which cover startup loading and /context display — this is invocation-time payload size.

Environment: Claude Code 2.1.220 (Linux), Opus 5 (1M context), bundled
skill at bundled-skills/2.1.220/<hash>/claude-api. Repo: Go + Rust + TS
monorepo, go.mod at root, no Anthropic SDK imports anywhere (project targets
Ollama / OpenAI-compatible endpoints).

What happened: One invocation of the claude-api skill (Skill tool, args:
"Opus 5 model ids, 1M context pricing tier, effort levels, when to use opus vs
sonnet vs haiku for subagents") added +227,284 tokens to the context in a
single step — 66% of the session's total context at that point. Measured from
the session transcript's per-request usage fields: prompt total jumped
53,590 → 280,874 on the request following the skill turn. The skill content
arrived as one user turn containing a single 598,008-character block.

What the docs say should happen:
https://platform.claude.com/docs/en/agents-and-tools/agent-skills/claude-api-skill
claims: "Claude loads only the documentation relevant to your project's
language, surface (Messages API or Managed Agents), and the specific task at
hand (tool use, streaming, batches, and so on), rather than loading everything
at once."

Observed vs claimed, per axis:

  • Language: WORKED — only go/ docs were included (not the other 7 languages).
  • Surface: FAILED — both Messages API docs AND the full Managed Agents doc set

(~14 shared/managed-agents-*.md files: core, environments, events, memory,
multiagent, outcomes, overview, scheduled-deployments, self-hosted-sandboxes,
tools, webhooks, client-patterns, api-reference, onboarding) were inlined.
The task mentioned nothing about Managed Agents.

  • Task: FAILED — the full model-migration guide (largest single doc),

prompt-caching, error-codes, anthropic-cli, platform-availability,
token-counting, tool-use-concepts, agent-design all arrived regardless of the
task, which needed roughly 2K tokens of model-table + effort-ladder content.

Mechanism (verified, not hypothesized): the source SKILL.md at
anthropics/skills instructs on-demand reads of separate files and embeds no
reference docs. The bundled payload received in Claude Code 2.1.220 contained
those same files pre-rendered inline as <doc path="..."> blocks under an
"## Included Documentation" section. The inlining therefore happens in Claude
Code's bundled-skill packaging/render step, which forecloses the on-demand
disclosure the source implements and the docs page advertises. Selection
survives only at the language level (Go-only was correctly chosen).

Measurement caveat (for precision): the 598,008-char block against the
+227,284-token delta implies ~2.6 chars/token — dense but plausible for
table/code-heavy markdown; the delta window includes a small amount of other
turn content. Order of magnitude is not in question; the exact figure is
±~10%.

Impact: in a repo with no Anthropic SDK code, any prompt that merely names
a Claude model can trigger the skill and consume a quarter of a 1M window (or
exceed a 200K window outright) for a lookup-sized question. The skill's SKIP
heuristic (grep for other providers) does not protect repos that use neither
the Anthropic SDK nor a competitor's (e.g. local-model tooling).

Suggested directions (either would resolve it):

  1. Render only the sections matching the detected surface + a task keyword

match, keeping the rest as on-demand file reads (the model the docs already
describe); or

  1. Split the payload: always-inline the quick-reference tables (~2-5K), and

gate the migration guide + managed-agents corpus behind explicit
subcommands (migrate, managed-agents-onboard) that already exist.

Repro: in any Go repo without Anthropic SDK imports, invoke
/claude-api <any model question> and inspect the next request's
cache_creation_input_tokens / prompt-size delta in the session transcript.

View original on GitHub ↗

3 Comments

Anderson-RC · 1 month ago

Independent corroboration on the same version (2.1.220), macOS, with the opposite detection outcome — and it confirms your packaging thesis from the filesystem side.

You had language detection succeed (Go repo → only go/). I hit the fallback path, where it fails, and that's the worst case: all eight language variants ship.

The bundled tree has no SKILL.md

This is the part that most directly supports "progressive disclosure defeated by bundled packaging":

| | Source (anthropic-agent-skills marketplace copy) | Bundled (bundled-skills/2.1.220/<hash>/claude-api) |
|---|---|---|
| SKILL.md router | 71,590 chars | absent — 0 found |
| Reference tree | 779,501 chars across 64 files | 779,501 chars across 64 files |
| File count | 66 | 64 |

The router that's supposed to decide what to read isn't in the bundle. The references are there pre-rendered instead. Whatever the render step does, it isn't shipping the on-demand-read model the source describes. (Two hash directories exist under 2.1.220/; diff -rq says they're byte-identical duplicates.)

Full inventory of what's on disk

shared/      494,077 chars  (25 files)   ← loads unconditionally, per #80190
python        65,659
typescript    54,833
java          39,450
go            36,916
php           29,118
csharp        26,355
curl          16,854
ruby          16,239
             ─────────
languages    285,424 chars  (8 variants)
TOTAL        779,501 chars  (64 files)

Largest single file: shared/model-migration.md at 174,614 chars — 22% of the entire bundle, in the unconditional shared/ set, loaded for every invocation regardless of whether the session has anything to do with migrating models.

Two floors, both high

  • Language detected (say Python): shared/ + python/ = 559,736 chars
  • Language not detected: everything = 779,501 chars

So successful detection saves 219,765 chars — real, but it's a discount off an already-enormous baseline, not progressive disclosure. shared/ alone is 63% of the worst case.

Chars → tokens

Worth calibrating, because the intuitive ~3.5 chars/token badly understates this content. Two independently measured reports:

  • #74473: 570,113 chars → 210,658 tokens (cache_creation_input_tokens) = 2.71
  • this issue: 598,008 chars → 227,284 tokens (per-request usage delta) = 2.63

2.67 chars/token. Applying it:

| Case | Chars | Est. tokens |
|---|---:|---:|
| Language detected (Python) | 559,736 | ~210k |
| Language not detected | 779,501 | ~292k |

A second user reproduced the fallback path in a clean session and measured context going 47k → 377k (+330k tokens) on a single by-name invocation — above the static estimate, since it also carries the tool_use block, thinking, and reply. The two figures agree.

This is a regression against a report that was closed as a duplicate

#50079 filed this exact fallback in April on 2.1.112: same repro (directory with no auto-detectable source files), same quoted string in the expansion —

No project language was auto-detected. Ask the user which language they are using, then refer to the matching docs below.

— followed by all the docs anyway, so the "ask the user" step can only narrow what gets quoted, never unload what's already cached. It measured 133,510 tokens, twice, agreeing within 7 tokens across sessions.

It was auto-closed as a duplicate of #14882 after 3 days and auto-locked 7 days later. Nobody triaged it.

The same bug is now ~292k tokens. It more than doubled while sitting closed.

It exceeds a 200k context window outright

Not an arithmetic point — it's documented twice:

  • #74473 (comment): Haiku 4.5, docs-only repo so detection had nothing to find, throwaway model-comparison question → single message of 788,744 bytes → session dies on the spot.
  • anthropics/skills#1487: 624,401 chars → Prompt is too long on the next call. "One tool call from a fresh session to a dead session."

At ~292k tokens the fallback cannot fit a 200k window under any circumstances, so on those models the failure isn't degraded quality or cost — it's an unrecoverable session, triggered by a skill the user may never have asked for.

---

Corroborating measurements by Victor, who independently ran the clean-session experiment (47k → 377k) and identified the empty-working-tree fallback as the trigger. Happy to add their GitHub handle here on request.

Related: #14882 (canonical), #79457 (cost impact, triaged), #80190 (shared/ inlines unconditionally), #81059 (trigger breadth), #50079 (closed/locked, same fallback at half the size), anthropics/skills#1487 and #1486.

victor-bajanov · 1 month ago

I've also observed this. Further, if you try to invoke the skill from a weaker model with 200k context (e.g. haiku 4.5) it immediately triggers compaction.

<img width="862" height="190" alt="Image" src="https://github.com/user-attachments/assets/803f049a-f99f-42dd-b737-aa804c5214a9" />

hangox · 27 days ago

Independent third corroboration — Claude Code 2.1.220, macOS. Measured the bundled tree at bundled-skills/2.1.220/<hash>/claude-api directly (byte counts via wc -c), no target file in the working directory, language fell back to TypeScript:

shared/                                494,077 chars  (25 files)
  shared/model-migration.md            174,614 chars  (largest single file)
typescript/claude-api + managed-agents  54,833 chars
SKILL entry + shared + typescript      ~549,000 chars total for this invocation

Every figure matches the inventory already posted here exactly (494,077 / 174,614 / 54,833), confirming the bundle is deterministic across machines/sessions and the root cause is packaging, not per-session variance.

I filed this independently as #83532 before finding this thread — closing mine as a duplicate of this one, since this issue already has the fuller root-cause writeup (source-vs-bundle diff, regression history via #50079, chars/token calibration).

Showing cached comments. Read the full discussion on GitHub ↗