skill-creator's trigger-eval harness measures slash-command registration, not skill auto-triggering

Summary

The skill-creator plugin's trigger-optimization tooling (scripts/run_eval.py, driven by scripts/run_loop.py) tests candidate skill descriptions by writing a synthetic file to .claude/commands/<skill_name>-skill-<uuid>.md and checking whether claude -p <query> invokes it via the Skill or Read tool. This methodology is broken in current Claude Code (tested against 2.1.210): files under .claude/commands/*.md register only in the slash_commands array of the CLI's stream-json init event, not in the skills array that governs auto-triggering. Because the eval's pass/fail check hinges entirely on the model choosing to invoke something that was never a candidate for auto-triggering in the first place, the harness's trigger-recall metric measures approximately nothing — it reads at or near 0% regardless of description quality. Observed in practice: running the full run_loop.py optimization for 5 iterations on a real skill produced 0-11% recall throughout, including on a query with an essentially perfect keyword match against the description.

Root cause

• run_single_query() in run_eval.py (lines 51-68) builds command_file = project_root/.claude/commands/<clean_name>.md and writes YAML frontmatter with a description: field, mimicking the shape of a real SKILL.md.
• The docstring at lines 45-46 states this is done "so it appears in Claude's available_skills list" — this is the load-bearing but incorrect assumption. improve_description.py (line 81) and the skill's own SKILL.md (line 398) repeat the same "available_skills list" framing.
• In the current CLI, the mechanism that actually populates the auto-triggering skill list is discovery of .claude/skills/<name>/SKILL.md (and equivalent plugin-provided skills) — a separate registration path from .claude/commands/<name>.md (slash commands, invoked only via an explicit /name typed by the user).
• The stream-json system/init event exposes both lists directly ("skills": [...] and "slash_commands": [...]), making the mismatch directly observable per-run.

Repro (performed twice independently, by two separate agents)

From a directory containing .claude/:

UNIQ="harnessrepro7e2b1c"
cat > ".claude/commands/${UNIQ}.md" <<'MD'
---
description: |
  This skill handles synthetic trigger-testing repro queries about harness repro verification for a suspected bug in skill-creator eval tooling.
---

# harnessrepro7e2b1c

This skill handles: synthetic trigger-testing repro queries.
MD

env -u CLAUDECODE claude -p "test query for harnessrepro7e2b1c synthetic skill" \
  --output-format stream-json --verbose --include-partial-messages \
  --model claude-sonnet-5 > /tmp/repro_init_event.json

rm -f ".claude/commands/${UNIQ}.md"

Expected (per the harness's design intent): harnessrepro7e2b1c appears in the "skills" array of the first system/init event, making it eligible for auto-triggering via the Skill tool.

Actual: The first stream event (type: system, subtype: init) contained a "skills" array with the session's real installed skills, and harnessrepro7e2b1c was absent. It appeared only in "slash_commands", alongside those same real skill names (real skills appear in both lists; the synthetic file appeared only in the second).

Contrast with a real installed skill

The same init event's "skills" array included a real skill installed at .claude/skills/<name>/SKILL.md in the same working directory. Real skills registered under .claude/skills/ appear in both "skills" and "slash_commands" (invocable both by auto-trigger and by explicit /name), whereas the harness's synthetic .claude/commands/*.md file appeared only in "slash_commands". This is the clean contrast establishing the bug: the registration path the harness uses (commands/) is structurally incapable of reaching the array that drives auto-triggering; only skills/<name>/SKILL.md reaches it.

As a further check, we also manually spot-checked a real installed skill (not the synthetic harness probe) against 4 realistic prompts that clearly matched its description, run as fresh claude -p sessions with the skill genuinely discoverable. The Skill tool was not invoked in any of the 4 runs — the model went straight to ad hoc investigation (gh/git/Explore subagent) instead, in one case manually Read-ing the SKILL.md file directly rather than triggering it as a skill. This suggests the real-world auto-triggering rate for "review this PR"-shaped requests may be low even independent of the harness bug, though that's a separate, less rigorously isolated observation.

Impact

• run_eval.py's trigger/no-trigger classification for every query is effectively noise dominated by whatever else the model chooses to do when it can't find the described capability, since the synthetic entity never appears as a candidate skill.
• run_loop.py's iterative description-improvement loop optimizes against this broken signal — improve_description() receives eval results that don't reflect real triggering behavior, so proposed description edits are scored on an unusable metric.
• Any decision made using this tool (e.g. "this description doesn't trigger reliably enough, keep iterating") is not evidence-based under the current CLI version.
• quick_validate.py, aggregate_benchmark.py, and generate_report.py were not inspected in this pass but likely consume run_eval.py's output format and would inherit the same invalidity.

Suggested fix

Change run_single_query() (and any other place that builds the synthetic probe) to register the candidate as a real skill rather than a slash command:

• Write to <project_root>/.claude/skills/<clean_name>/SKILL.md instead of <project_root>/.claude/commands/<clean_name>.md.
• Ensure the frontmatter matches the real SKILL.md contract (name: + description: at minimum — verify against parse_skill_md() in utils.py and against a real installed SKILL.md for exact required fields).
• Update cleanup logic to remove the created directory (e.g. shutil.rmtree) rather than unlink() a single file, since SKILL.md lives inside a per-skill directory.
• Re-validate that the new synthetic entry appears in the init event's "skills" array (not just "slash_commands") before trusting eval output again — worth adding as a one-time sanity check inside run_single_query() or a standalone smoke test, so a future CLI change to skill discovery is caught immediately rather than silently producing flat near-zero recall again.
• Audit improve_description.py and skill-creator's own SKILL.md documentation for the same stale "available_skills" terminology while making this fix.

Environment

• Claude Code CLI version: 2.1.210
• Affected component: bundled skill-creator skill, scripts/run_eval.py / scripts/run_loop.py / scripts/improve_description.py