skill-creator plugin: trigger-eval harness false-negatives (shared probe-root contamination, first-tool-only detection, 30s timeout)

Component

The skill-creator skill shipped in the anthropic-skills plugin (creatorType: anthropic, Claude Desktop local agent mode) — specifically its description-triggering eval harness, scripts/run_eval.py.

Summary

Evaluating a previously optimized skill description returned trigger_rate ~0.0 on every should-trigger case. Digging in, three compounding harness issues produce false negatives; the first is decisive.

1. Cross-probe contamination (decisive)

All parallel workers (default --num-workers 10) write their uuid-suffixed synthetic command file into the SAME project root's .claude/commands. Each probe's claude -p therefore sees up to N near-identical clones of the skill and invokes an arbitrary one; the detector greps for its own uuid, so almost every probe reads as non-triggered. A single serial probe with one command file triggers instantly.

Fix that worked: give each probe an isolated temp project root (copy the fixture in, write exactly one command file), run claude -p with cwd there, rmtree in finally.

2. First-tool-only detection

run_single_query returns False the moment the first tool_use block is anything other than Skill/Read. Models that explore the repo before consulting a skill (an ls, a Read package.json) are scored as non-triggered even when they invoke the skill on the second tool call. Newer thinking models exhibit exactly this explore-first behavior.

Fix: watch the whole stream until the result event; conclude on match, result, or timeout.

3. Default 30s timeout vs thinking models

The 30s timeout straddles time-to-first-tool for thinking models (measured 18.3s on one run of the same query that timed out on others). With fix 2, probes need a larger window anyway; 90s worked well.

Result

With all three applied, the same description scored 31/34 (all 16 should-not-trigger cases clean at 0.00). Happy to share the patched run_single_query.

Environment: Claude Desktop local agent mode, claude CLI 2.1.207, model claude-fable-5, macOS.