skill-creator: trigger eval framework uses slash-command proxy — 0% recall on path-triggered skills

Skill-creator framework cannot test path-triggered skills (uses slash-command proxy)

priority : low
found-during : D-031 — skill-creator official run on header-pattern skill (2026-05-27)
status : documented limitation, no workaround at framework level
target : Anthropic claude-plugins-official marketplace (upstream repo URL TBD via research)

Context

The skill-creator's run_eval.py measures skill triggering accuracy by :

1. Writing a temp .claude/commands/<unique-name>.md slash command file with the skill's description field
2. Spawning claude -p <query> subprocess
3. Watching stream events for Skill or Read tool invocations matching the unique name

Problem : slash commands are not the same as skills. Slash commands are user-invoked via /<name> typed in the prompt — they do NOT auto-trigger based on the user's natural language query. Real skills auto-trigger based on (a) description match, (b) paths: matchers, (c) other mechanisms.

For path-triggered skills (skill has paths: ["**/*.md"] etc.), the framework cannot model the real triggering mechanism at all. Tests will return 0% recall regardless of description quality — because the slash-command proxy never gets activated by the natural language query.

Evidence

header-pattern skill has paths: ["**/*.md", "**/*.log"]. Full skill-creator run (3 iterations, all 3 description proposals) returned identical 0% recall on the test framework. Train: 12/24 (50%), Test: 8/16 (50%), Precision: 100% (zero false positives), Recall: 0% (zero true positives) — on every iteration.

However, the subprocess Claude DID apply the skill's instructions during one of the test runs : docs/architecture/AUTH_ROLES.md (1018 lines) received a complete D-028 4-block header (TLDR + Loading + Search-by) added by the test subprocess. The skill worked in reality — the test just couldn't measure it because of the slash-command-proxy approach.

Proposed change

Three possible improvements to the framework :

Option A : Support paths: matchers in trigger tests

Write actual skill files (not slash commands) during the test setup. Activate the skill via the same path-matching mechanism Claude Code uses in real sessions. Most accurate but requires deeper integration with skill loader.

Option B : Document the limitation explicitly in skill-creator README

Add a section "Limitations" explaining that path-triggered skills cannot be reliably tested via the description trigger eval. Recommend manual real-world testing as the primary validation. Minimal effort, maximum clarity.

Option C : Run actual skills + grep transcripts for evidence

Spawn claude -p with the skill installed normally, then parse the full output transcript for evidence of the skill's instructions being followed (not just the Skill tool invocation). More flexible but harder to grade reliably.

Recommendation : Start with Option B (docs only). Option A is the real fix but is invasive.

Files

• scripts/run_eval.py (lines 35-180 approx) — run_single_query function
• SKILL.md README — needs limitation note
• Possibly scripts/run_loop.py — needs to be aware path-triggered skills cannot be optimized via description loop alone

Effort

Option B : ~15 min (README update only).
Option A : ~4-8 hours (deeper skill-loader integration).

Filing target

Confirmed via research 2026-05-27 : anthropics/claude-code GitHub issues.

gh issue create --repo anthropics/claude-code --title "skill-creator: trigger eval framework uses slash-command proxy — fails for path-triggered skills" --body-file <this-backlog-file>