[Bug] skill-creator run_loop.py burns ~50% of a usage window in minutes: full claude -p agent per query x run x iteration, no cost preview or cap

skill-creator description optimization (run_loop.py): the evaluation algorithm is fundamentally too expensive for its benefit, needs a redesign, not just guardrails

Repo: anthropics/claude-code (skill-creator plugin, claude-plugins-official)

Core problem (the algorithm)

To tune a skill's ~100-word description, run_loop.py measures triggering by booting a full claude -p agent per query, per run, per iteration. Each boot loads the entire system prompt + full skill/command/tool catalog before evaluating one binary signal ("would this description make the model consult the skill?"). It repeats each query N times to denoise, and hill-climbs the description string against a tiny hand-written train set for up to max-iterations.

Total cost = queries x runs_per_query x iterations full agent boots. With the skill's own recommended ~20-query set and defaults (runs_per_query 3, max_iterations 5), that is ~300 full claude -p sessions (plus ~4 description-rewrite calls). Because eval runs in separate parallel processes (--num-workers 10), there is no prompt-cache reuse: every one of the ~300 boots re-pays the full fixed context.

The benefit is a marginally reworded description, overfit to ~12 train queries. The cost/benefit is structurally inverted. In practice this consumed ~50% of a 5-hour usage window in ~10 minutes.

Why guardrails are not enough

A cost preview, a token/cost cap, and per-iteration checkpointing are all worth adding (and should exist for any script that loops over model calls), but they do not fix this: even with all of them, the method still needs ~hundreds of full agent boots to do its job. A cap just aborts mid-run with partial/garbage output.

Suggested redesign

1. Batch the evaluation: one (or a few) model call(s) that score all queries against a description at once, instead of one full claude -p per query x run. ~100x fewer calls.
2. Batch candidate generation + scoring: propose K description variants in one call, score them in one batched call, pick the best. The whole loop becomes a handful of calls.
3. Reconsider whether an LLM-in-the-loop hill-climb is warranted at all for ~100 words of prose against ~12 queries.
4. Independently: add a pre-run cost estimate + confirmation, a token/cost budget cap, and checkpointing.

Environment

Claude Code, skill-creator plugin (claude-plugins-official, scripts/run_loop.py + run_eval.py). macOS.

• Platform: darwin
• Terminal: ghostty
• Version: 2.1.193
• Feedback ID: 27e24cf2-f81b-4f99-a321-fb6ff8f72b06

Errors

[]