skill-creator: run_eval.py uses select() on a pipe, fails on all Windows runs (WinError 10038) and reports a misleading score

Summary

skill-creator's trigger-evaluation harness (scripts/run_eval.py) is unusable on Windows. It polls the claude -p child process's stdout pipe with select.select(), which on Windows accepts sockets only. Every query in the eval set fails with WinError 10038, and the failure is silent in a way that produces a plausible-looking but meaningless score.

Environment

• Windows 11 Pro 26200
• Python 3.12
• Plugin: skill-creator from claude-plugins-official
• Path: ~/.claude/plugins/cache/claude-plugins-official/skill-creator/unknown/skills/skill-creator/scripts/run_eval.py

The bug

run_eval.py line 108:

ready, _, _ = select.select([process.stdout], [], [], 1.0)

process.stdout here is a pipe from subprocess.Popen. On POSIX select() accepts any file descriptor; on Windows it accepts socket handles only, so this raises immediately. There is no platform guard anywhere in the file.

Minimal repro, no plugin needed:

import subprocess, select, sys
p = subprocess.Popen([sys.executable, "-c", "print(1)"], stdout=subprocess.PIPE)
select.select([p.stdout], [], [], 1.0)
# Windows -> OSError: [WinError 10038] An operation was attempted on something that is not a socket

In practice:

$ python -m scripts.run_eval --eval-set evals.json --skill-path <skill> --runs-per-query 1
Warning: query failed: [WinError 10038] An operation was attempted on something that is not a socket
... (once per query)

Why this is worse than a plain crash

The exception is caught per-query and the run still reports a score. A failed query is recorded as "did not trigger", so:

• every should_trigger: true case fails
• every should_trigger: false case passes trivially

With a balanced eval set you get a clean-looking "10/20 passed" that is entirely an artifact. Nothing in the summary output indicates the queries never ran. I initially read it as a genuine description-quality problem and started rewriting a correct description.

Suggested fix

A reader thread feeding a queue preserves the incremental, non-blocking reads the early trigger detection depends on, and works on both platforms without branching:

import queue, threading   # replaces `import select`

chunks: "queue.Queue[bytes | None]" = queue.Queue()

def _pump(fd: int, out: "queue.Queue[bytes | None]") -> None:
    try:
        while True:
            data = os.read(fd, 8192)
            if not data:
                break
            out.put(data)
    except OSError:
        pass
    finally:
        out.put(None)

threading.Thread(target=_pump, args=(process.stdout.fileno(), chunks), daemon=True).start()

try:
    while time.time() - start_time < timeout:
        try:
            chunk = chunks.get(timeout=1.0)
        except queue.Empty:
            if process.poll() is not None:
                break
            continue
        if chunk is None:
            break
        buffer += chunk.decode("utf-8", errors="replace")
        # ... existing line-splitting loop unchanged

os.read on the raw fd (rather than stream.read(8192)) keeps reads returning as soon as data is available, so content_block_start is still seen before the run completes. Verified locally against a real skill — the socket errors disappear and streaming detection still fires early.

Separately, it would be worth failing loudly rather than scoring a run where any query errored, since a partial run is not comparable to a complete one.

Secondary observation: harness under-reports when the skill is installed

Not a crash, but it produced a second wrong conclusion for me, so flagging it.

run_eval.py tests the description by writing a throwaway slash command named {skill_name}-skill-{uuid} into <project_root>/.claude/commands/, then counting a trigger only when that unique name appears:

if tool_name == "Skill" and clean_name in tool_input.get("skill", ""):

If the skill under test is also installed normally — which it is, for anyone iterating on a skill they already deployed — Claude invokes the real skill (skill: "metlife-das"), which does not contain metlife-das-skill-<uuid>. The harness scores that as a non-trigger.

After fixing the select() bug, the harness still reported a miss on a query where the skill demonstrably did fire; a direct test checking for the real skill name scored the same eval set 20/20. Worth either matching the bare skill name as well, or documenting that the skill should be uninstalled before evaluating.