skill-creator run_eval.py: false negative when model calls non-Skill tool before Skill tool

Resolved 💬 1 comment Opened May 20, 2026 by guidohajanssen Closed Jun 19, 2026

Bug

In run_single_query in scripts/run_eval.py, the stream event handler for content_block_start contains:

if tool_name in ("Skill", "Read"):
    pending_tool_name = tool_name
    accumulated_json = ""
else:
    return False  # BUG

If Claude calls any other tool first (e.g. AskUserQuestion, TodoWrite, Bash) before calling Skill, the function immediately returns False — a false negative. The skill was triggered correctly but gets scored as a miss.

Impact

Skills that prompt for clarification or perform other actions before reading the skill (a common pattern) always score 0% trigger rate even when working correctly. This makes the eval tool unreliable for any skill that uses multi-step tool sequences.

Reproduction

  1. Create a skill whose description causes Claude to first call AskUserQuestion before Skill
  2. Run run_eval.py with a query that triggers this skill
  3. Observe trigger_rate = 0/3 despite the skill being invoked

Fix

Replace return False with pending_tool_name = None to reset state and keep scanning the stream for a subsequent Skill call:

if tool_name in ("Skill", "Read"):
    pending_tool_name = tool_name
    accumulated_json = ""
else:
    pending_tool_name = None  # reset, keep scanning for Skill call

Environment

  • Claude Code CLI, skill-creator plugin
  • Observed on Windows 11, likely affects all platforms

View original on GitHub ↗

This issue has 1 comment on GitHub. Read the full discussion on GitHub ↗