/skill-doctor's per-skill "7d tokens" / "uses" numbers don't reflect a skill's actual cost or invocation count
Title
/skill-doctor's per-skill "7d tokens" / "uses" numbers don't reflect a skill's actual cost or invocation count
Body
Summary
/skill-doctor's per-skill token/usage table is misleading enough to actively mislead pruning decisions. I audited ~20 of the highest-ranked skills in my own /skill-doctor output and found two distinct inflation mechanisms, neither of which reflects the skill's own marginal cost.
**Mechanism 1 — a session's full token total is attributed to every skill it touched, not divided**
When one session invokes N different skills over its lifetime, that session's cache_read_input_tokens total (which compounds every turn — into the hundreds of millions, sometimes over a billion, for a long multi-hour session) appears to get attributed in full under each of the N skills in the per-skill table, rather than split or weighted.
Concretely: I found 3 marathon sessions (74.8MB/7418 lines, 53.4MB/5640 lines, 32MB/3003 lines) that each showed up as the single largest token contributor for several different, unrelated skills at once (e.g. one 75MB session was the top contributor for both a Figma-analysis skill and an iOS-simulator skill that has nothing to do with Figma). One skill invoked exactly once inside a 75MB/7418-line session — which did ~732 unrelated Bash calls, 272 simulator ops, and 42 agent dispatches, with no calls to the skill's own actual data source at all — was still reported as consuming ~21.8M tokens for that single use.
Mechanism 2 — "uses" count inflated by incidental name mentions, not actual invocations
Many skill names are also referenced inside other subagents' descriptions (e.g. a build-agent's description mentioning "used by skill X's Phase 2.7"). That descriptive text is embedded in the system prompt of nearly every session (in my case, 117–118 out of 118 recent sessions contained the literal skill-name string as plain text, regardless of whether the skill was ever invoked).
Cross-checking real Skill(skill=X) tool invocations in my own session transcripts against /skill-doctor's reported "uses" found real counts consistently an order of magnitude lower:
| skill | reported uses | real invocations (grepped from transcripts) |
|---|---|---|
| (skill A) | 175 | 8 |
| (skill B) | 131 | 6 |
| (skill C) | 235 | 14 |
| (skill D) | 158 | 3 |
Why this matters
/skill-doctor's own output nudges the user toward disabling "expensive" or "unused" skills based on these numbers. Right now the leaderboard actively points at the wrong targets — the skills at the top of the token/uses ranking are not necessarily doing expensive work themselves; they're just skills that happened to get invoked once inside a long, unrelated marathon session, or whose name happens to appear in other agents' descriptions.
Suggestion
- Attribute a session's token cost proportionally (e.g. by turns-since-invocation, or by tokens actually consumed by tool calls made in service of that skill) instead of the full session total to every skill touched.
- Count "uses" from actual
Skilltool invocations only, not from string occurrence of the skill name anywhere in context.
Happy to share the (locally-run) audit script/methodology if useful — didn't want to paste raw session-transcript content here since it may contain private project details.