skill-creator run_eval.py: false negative when model calls non-Skill tool before Skill tool
Bug
In run_single_query in scripts/run_eval.py, the stream event handler for content_block_start contains:
if tool_name in ("Skill", "Read"):
pending_tool_name = tool_name
accumulated_json = ""
else:
return False # BUG
If Claude calls any other tool first (e.g. AskUserQuestion, TodoWrite, Bash) before calling Skill, the function immediately returns False — a false negative. The skill was triggered correctly but gets scored as a miss.
Impact
Skills that prompt for clarification or perform other actions before reading the skill (a common pattern) always score 0% trigger rate even when working correctly. This makes the eval tool unreliable for any skill that uses multi-step tool sequences.
Reproduction
- Create a skill whose description causes Claude to first call
AskUserQuestionbeforeSkill - Run
run_eval.pywith a query that triggers this skill - Observe trigger_rate = 0/3 despite the skill being invoked
Fix
Replace return False with pending_tool_name = None to reset state and keep scanning the stream for a subsequent Skill call:
if tool_name in ("Skill", "Read"):
pending_tool_name = tool_name
accumulated_json = ""
else:
pending_tool_name = None # reset, keep scanning for Skill call
Environment
- Claude Code CLI, skill-creator plugin
- Observed on Windows 11, likely affects all platforms
This issue has 1 comment on GitHub. Read the full discussion on GitHub ↗