[BUG] Skills System: Natural Language Intent Matching Completely Broken

Preflight Checklist

• [x] I have searched existing issues and this hasn't been reported yet
• [x] This is a single bug report (please file separate reports for different bugs)
• [x] I am using the latest version of Claude Code

What's Wrong?

Component: Skills System / Intent Matching
Version: Claude Code v2.0.31
Severity: Critical - Skills unusable for natural language interaction

Summary

Skills cannot be invoked via natural language despite having properly configured trigger phrases in skill descriptions. Even explicit invocation ("use pr-summary") fails initially. Skills appear to require matching slash commands, creating an unacceptable double-entry maintenance burden.

What Should Happen?

Expected Behavior

According to Skills Documentation:

1. Create .claude/skills/my-skill/skill.md with description containing trigger phrases
2. User types natural language matching those phrases
3. CC matches intent to skill and invokes it automatically

Actual Behavior

Natural language intent matching does not work at all. Skills are never invoked naturally, requiring explicit "use skill-name" invocation, which itself has issues.

Error Messages/Logs

included in repro steps

Steps to Reproduce

Reproduction Steps

Test Case: pr-summary skill

Setup: Skill properly configured with trigger phrases in description:

description: "Use for 'summarize this pr', 'summarize the pr', 'generate PR summary', 'create PR description', 'write PR summary'. Generates formatted PR summary with objective, changes, test plan, ticket references."

Test 1: Natural language - "summarize this pr"

Result: ❌ CC uses manual git/gh commands, never invokes skill

Test 2: Natural language - "summarize pr"

Result: ❌ CC uses manual git commands, never invokes skill

Test 3: Natural language - "summarize my pr"

Result: ❌ CC reads README, uses manual git commands, never invokes skill

Test 4: Explicit invocation - "use pr-summary"

Result: ⚠️  CC first tries slash command /flow:pr:summary
        Error: "Unknown slash command: flow:pr:summary"
        Then successfully invokes skill (after failure)

Test 5: Natural language from repo root - "summarize this pr"

⏺ I'll help you summarize this PR...
⏺ Bash(git rev-parse --abbrev-ref HEAD)
⏺ Bash(gh pr view --json number,title,url...)
⏺ I found PR #14133. Let me use the pr-summary skill to generate a comprehensive summary.
  ⎿ Error: Interrupted by user

Result: ⚠️  CC correctly identifies it should use the skill
        ❌ CC fails to actually invoke it

Claude Model

Sonnet (default)

Is this a regression?

Yes, this worked in a previous version

Last Working Version

N/A

Claude Code Version

v2.0.31

Platform

Anthropic API

Operating System

macOS

Terminal/Shell

iTerm2

Additional Information

Root Causes Identified

1. Skills Not Checked for Intent Matching

Slash commands have explicit priority in CC's system prompt:

**IMPORTANT - Intent Matching:**
Before starting any task, CHECK if the user's request matches one of the slash commands listed below.

Skills only have weak suggestion:

When users ask you to perform tasks, check if any of the available skills below can help...

Impact: CC interprets slash commands as MUST-CHECK but skills as NICE-TO-HAVE.

2. Skill Descriptions Truncated in System Prompt

What skill.md contains (204 chars):

"Use for 'summarize this pr', 'summarize the pr', 'generate PR summary', 'create PR description', 'write PR summary'. Generates formatted PR summary with objective, changes, test plan, ticket references."

What CC's system prompt shows (~50 chars):

Formatted PR summary ready for GitHub (project)

Impact: Trigger phrases are stripped before intent matching, making natural language matching impossible.

3. Skills Require Matching Slash Commands (Critical Architecture Flaw)

When explicitly invoking "use pr-summary", CC first tries to find /flow:pr:summary slash command. Evidence:

• Deleted .claude/commands/flow/pr/summary.md to migrate to pure skill approach
• CC error: "Unknown slash command: flow:pr:summary" even in a fresh CC session
• Skill only works after slash command lookup fails

Why This Is Unacceptable:

Skills were introduced as a NEW, more powerful system (3-stage pipelines, structured I/O, testability). Requiring BOTH creates:

1. Double maintenance: Every skill needs a slash command wrapper
2. No migration path: Can't replace slash commands with skills
3. Documentation lie: Docs imply skills are standalone via trigger phrase descriptions
4. Architecture contradiction: Why have two systems if one requires the other?

10 Critical Design Flaws Documented

Full analysis with evidence is available: (for me: /tmp/skill_invocation_bugs.md)

1. ❌ Asymmetric intent matching (slash commands prioritized over skills)
2. ❌ Skill descriptions truncated in system prompt
3. ❌ No debugging tools for skill discovery
4. ❌ Documentation doesn't match implementation
5. ❌ Hardcoded implementation assumptions
6. ❌ No clear priority documentation
7. ❌ Skills cannot enforce tool restrictions
8. ❌ No validation that service functions are CLI-accessible
9. ❌ Skill invocation interrupted/fails after recognition
10. ❌ Skills require slash commands (unacceptable double-entry)

Required Fixes

High Priority (Blockers)

1. Skills must work standalone without requiring slash command wrappers
2. Intent matching must check skills with same priority as slash commands
3. Full descriptions must be included in system prompt for intent matching
4. Skills must be scanned at startup and registered in <available_skills>

Medium Priority

1. Add debugging tools: /skills:list, /skills:debug <name>, /skills:test <name> <query>
2. Validate skill dependencies at load time (referenced service functions exist as CLI commands)
3. Add allowed-tools enforcement to prevent skills from being ignored

Impact

This makes the skills system fundamentally unusable for its documented purpose. Developers must either:

• Maintain double-entry (skill + slash command) for every automation
• Force users to memorize skill names instead of using natural language
• Abandon skills entirely and use only slash commands

Environment

• Claude Code Version: v2.0.31
• OS: macOS (Darwin 24.6.0)
• Repo: Project with .claude/skills/ directory containing 20 properly configured skills
• Testing: Multiple fresh CC sessions from repo root directory

Additional Context

This issue has been blocking effective skill development for weeks. Every skill works perfectly when invoked manually (via Skill tool or explicit name), but natural language invocation never works despite following all documentation guidelines.

🆕 UPDATE: Testing Results with Shortened Description (CC v2.0.31)

Test Setup

Modified pr-summary description to put trigger phrases FIRST:

description: "Use for 'summarize this pr', 'summarize the pr', 'generate PR summary', 'create PR description', 'write PR summary'. Generates formatted PR summary with objective, changes, test plan, ticket references."

Fresh CC Session Test - 2025-10-31

User input: summarize this pr

CC behavior (transcript):

⏺ I'll help you generate a summary for this PR. Let me gather the necessary information about the current PR.

⏺ Bash(node /Users/jkaster/.claude/skills/shared/dist/github.js get-pr-context)
  ⎿  Error: Exit code 1
     Error: Cannot find module '/Users/jkaster/.claude/skills/shared/dist/github.js'

⏺ Let me use the slash command for PR summary:
  ⎿  Error: Unknown slash command: flow:pr:summary

⏺ Let me use the skill directly:

> The "pr-summary" skill is running

Analysis of This Behavior

What CC Did (in order):

1. ❌ Tried manual tool approach FIRST - Called Bash with a hardcoded path to a non-existent script
2. ❌ Tried slash command SECOND - Attempted /flow:pr:summary which doesn't exist
3. ✅ Used skill THIRD - Finally invoked the pr-summary skill as last resort

Critical Findings

Finding #1: Front-Loading Trigger Phrases DID Help

Evidence: CC eventually recognized the skill and invoked it. Previously it would have continued with manual tools.

However: Skill invocation happened LAST, after two failed attempts.

Finding #2: CC Has Hardcoded Fallback Logic

Evidence: CC tried to execute:

node /Users/jkaster/.claude/skills/shared/dist/github.js get-pr-context

This suggests CC has internal knowledge or cached patterns about PR summarization that override skill discovery. It's reaching for a specific script path instead of checking available skills first.

This is a MAJOR design flaw: CC shouldn't have hardcoded assumptions about skill implementations!

Finding #3: Slash Command Takes Priority Over Skills

Evidence: After the manual tool failed, CC tried the slash command (/flow:pr:summary) before trying the skill.

This confirms Design Flaw #1 - skills have the LOWEST priority in CC's decision tree:

1. Manual tools / hardcoded patterns (FIRST)
2. Slash commands (SECOND)
3. Skills (LAST RESORT)

Finding #4: CC's Internal State May Be Stale

The error path /Users/jkaster/.claude/skills/shared/dist/github.js suggests:

• CC is looking in personal skills directory (~/.claude/skills/)
• But the pr-summary skill is in project directory (.claude/skills/)
• CC may have cached information from a previous skill implementation

This reveals: CC doesn't cleanly separate personal vs project skills in its internal routing logic.

Design Flaw #5: Hardcoded Implementation Assumptions

Problem: CC appears to have internal knowledge about specific skill implementations (like the github.js script path). When these assumptions become stale, CC:

• Tries non-existent paths
• Fails multiple times
• Only falls back to actual skill discovery as last resort

Impact: Even with perfect skill.md descriptions, CC will try its hardcoded assumptions first, causing:

• Error messages shown to users
• Slower execution (multiple failed attempts)
• Confusion about which skill actually runs

Fix Required: CC should NOT have hardcoded knowledge of skill implementations. It should ALWAYS:

1. Check available skills FIRST (based on description matching)
2. Invoke matched skill using the stageMap from skill.md
3. Never cache or hardcode paths to skill internals

Design Flaw #6: No Clear Priority Documentation

Problem: The execution order (manual tools → slash commands → skills) is not documented anywhere. Developers building skills have no way to know their skill will be the LAST thing tried.

Fix Required: Document the decision tree explicitly:

When user makes a request:
1. Check for exact slash command match (/foo)
2. Check for slash command intent match (documented triggers)
3. Check for skill intent match (description field)
4. Fall back to manual tool usage

Skills are LAST RESORT only.

Or better yet: Make skills EQUAL priority with slash commands so they're checked together in step 2.

---

Historical Context

This bug has been preventing effective skill usage throughout the Flow project. We keep building skills that work perfectly when invoked manually but CC never recognizes when to use them naturally. The investigation reveals this is due to fundamental design flaws in CC's skill system, not errors in our skill implementation.

Latest Update: Even with optimized descriptions, CC tries multiple failed approaches before invoking skills, revealing deeper architectural issues with hardcoded assumptions and priority ordering.

---

🆕 UPDATE 3: Skill Invocation Completely Broken (CC v2.0.31)

Test Results - Multiple Phrasings

After adding all missing CLI commands and updating skill.md, tested in fresh CC session with multiple trigger phrases:

Attempt 1: summary pr

• ❌ Used manual git branch --show-current
• ❌ Used manual git log development..HEAD --oneline
• ❌ Used manual git diff development...HEAD --stat
• ❌ Never invoked skill

Attempt 2: summarize pr

• ❌ Used manual git diff development...HEAD --numstat | head -50
• ❌ Never invoked skill

Attempt 3: summarize my pr

• ❌ Read README.md
• ❌ Used manual git log development..HEAD --format="%s" | head -20
• ❌ Never invoked skill

Analysis

Even with:

• ✅ Skill description starting with trigger phrases
• ✅ All service functions exposed as CLI commands
• ✅ Skill.md properly structured
• ✅ Multiple variations tested ("summary pr", "summarize pr", "summarize my pr")

CC NEVER checks the pr-summary skill. It goes straight to manual tool usage.

Critical Finding: Skill Descriptions May Not Be Checked At All

The skill description in skill.md says:

description: "Use for 'summarize this pr', 'summarize the pr', 'generate PR summary', 'create PR description', 'write PR summary'. Generates formatted PR summary with objective, changes, test plan, ticket references."

The user tried:

• "summary pr" ❌
• "summarize pr" ❌
• "summarize my pr" ❌

These are VERY close to the documented triggers! Yet CC never even attempted to invoke the skill.

Hypothesis: CC's intent matching may be:

1. Checking slash commands ONLY (working)
2. Completely skipping skill descriptions (broken)
3. Only using manual tools as fallback (broken priority)

The Workaround That Doesn't Work

According to earlier testing, explicitly saying the skill name worked:

> use pr-summary skill
✅ Skill invoked

But requiring users to know skill names defeats the entire purpose of natural language intent matching!

Recommendation for Users

Until this is fixed, the ONLY reliable way to use skills is:

1. Explicitly invoke by name: "use pr-summary skill"
2. Or use a slash command wrapper (if it exists)

Natural language intent matching is fundamentally broken and cannot be relied upon.

---

🆕 UPDATE 4: Natural Language Shows Skill Recognition, Then Fails (CC v2.0.31)

Test Sequence From Repo Root

Attempt 1: summarize this pr

CC behavior:

⏺ I'll help you summarize this PR. Let me gather the necessary information about the current PR.

⏺ Bash(git rev-parse --abbrev-ref HEAD)
  ⎿ jk.flow-18930-make-flow-skills

⏺ Bash(gh pr view --json number,title,url 2>/dev/null || echo "No PR found")
  ⎿ {
      "number": 14133,
      "title": " [FLOW-18930] [Flowmingo] Make skills for component creating and updating",
      ...
  }

⏺ I found PR #14133. Let me use the pr-summary skill to generate a comprehensive summary.
  ⎿ Error: Interrupted by user

Critical Finding: CC DOES Recognize The Skill Should Be Used!

Key observation: After gathering basic info, CC explicitly said:

"Let me use the pr-summary skill to generate a comprehensive summary."

This means:

• ✅ CC KNOWS the pr-summary skill exists
• ✅ CC KNOWS it should use it for this task
• ❌ But then immediately errors/stops before actually invoking it

Then user tried explicit invocation:

Attempt 2: use pr-summary

CC behavior:

⏺ Error: Unknown slash command: flow:pr:summary

> The "pr-summary" skill is running

> The "pr-summary" skill is running

Analysis

From Attempt 1 (natural language):

1. ✅ CC correctly identified the task (summarize PR)
2. ✅ CC stated it would use pr-summary skill
3. ❌ CC errored/interrupted before invoking the Skill tool
4. ❌ User interrupted to prevent more manual tool attempts

From Attempt 2 (explicit):

1. ❌ CC first tried slash command /flow:pr:summary (note the namespace prefix!)
2. ✅ Then successfully invoked the skill (message appeared twice - possible UI glitch)

Design Flaw #9: Skill Invocation Interrupted/Fails After Recognition

Problem: CC can recognize when a skill should be used (evidenced by the explicit message "Let me use the pr-summary skill"), but the actual invocation fails or is interrupted.

Hypothesis: Between recognizing the skill and invoking it, CC:

• May try to gather additional context first
• Gets interrupted by permission prompts or errors
• Loses track of the skill invocation intention
• Falls back to manual tools

Additional finding: When trying explicit invocation with "use pr-summary", CC tries /flow:pr:summary first (with namespace prefix), suggesting:

• CC may be looking for a namespaced slash command flow:pr:summary
• Skills might need both .claude/skills/pr-summary/skill.md AND .claude/commands/flow/pr/summary.md to work reliably

Design Flaw #10: Skills Require Matching Slash Commands (Unacceptable Double-Entry System)

Problem: Skills appear to expect a corresponding slash command to exist. When user says "use pr-summary", CC tries to invoke /flow:pr:summary slash command before falling back to the skill.

Evidence:

• .claude/commands/flow/pr/summary.md was deleted in favor of pure skill-based approach
• CC error: "Unknown slash command: flow:pr:summary"
• This suggests skills and slash commands are meant to work together, not independently

Why This Is Unacceptable:

Skills were introduced as the NEW, more powerful automation system with 3-stage pipelines, structured I/O, and testability. Slash commands are simple markdown files that expand to prompts. Requiring BOTH creates:

1. Double maintenance burden: Every skill needs a corresponding slash command wrapper
2. Inconsistent behavior: Skills work differently if slash command exists vs doesn't
3. No migration path: Can't replace slash commands with skills without breaking invocation
4. Documentation lie: Docs say skills have "descriptions with trigger phrases" for intent matching, implying they're standalone
5. Architecture contradiction: Why have two separate systems if one requires the other?

User's Goal (Reasonable):

• Delete .claude/commands/flow/pr/summary.md (old system)
• Keep .claude/skills/pr-summary/skill.md (new system)
• Have natural language "summarize this pr" invoke the skill directly
• Eliminate double-entry maintenance

Current Reality (Broken):

• Deleting slash command breaks skill invocation
• Skills cannot work standalone
• Must maintain both files with duplicate trigger phrase documentation

Fix Required:

Skills MUST work standalone without requiring slash command wrappers. This is fundamental architecture, not a feature request:

1. CC must scan .claude/skills/*/skill.md at startup
2. Build registry with full descriptions for intent matching
3. Invoke skills directly via Skill tool when intent matches
4. Slash commands should be OPTIONAL compatibility layer, not required dependency

If skills require slash commands by design, this is a critical architecture flaw that makes skills unusable for their intended purpose.

Natural language intent matching is fundamentally broken and cannot be relied upon.