›_ Claude Code Issues
Claude Code Issues / #23333

Golden Dataset: qcounsel_intent_classification Prompt Evaluation

Resolved 💬 1 comment Opened Feb 5, 2026 by KennyRich Closed Feb 5, 2026

Context

The qcounsel_intent_classification prompt is a core component of QCounsel that classifies user queries into intent categories and assesses complexity. It determines the downstream workflow path - whether to search legal databases, emails, documents, or act on the current context.

Langfuse Prompt: https://cloud.langfuse.com/project/cmfzvibe005k3ad0764crvuuw/prompts/qcounsel_intent_classification

Objective

Create a golden dataset (40-50 examples) for evaluating the intent classification prompt across all categories, with balanced single-turn and multi-turn conversations.

Critical Issue: Email/Professional Communication Scope

Problem Identified

Professional email drafting is incorrectly classified as NON_LEGAL_TASKS when it should be CURRENT_CONTEXT.

Example Query:

"Please can you reply to Neil, sorry for the delay been a crazy week. Due to us deploying resources in our UK hosting our lead engineer hasn't had any bandwidth. I hope to speak with him before the end of the week but will be able to come back at the start of next week."

Current (Incorrect) Classification:

  • Intent: NON_LEGAL_TASKS
  • Reasoning: "This is a professional email reply but not a legal research/drafting/review task. It is general professional communication, so it falls under NON_LEGAL_TASKS per the policy."

Expected Classification:

  • Intent: CURRENT_CONTEXT
  • Rationale: The prompt states "drafting/replying to email → CURRENT_CONTEXT" but the scope definition creates ambiguity about professional (non-legal) emails.

Recommended Prompt Clarification

Update Step 3 scope definition:

Current:

WITHIN SCOPE (LEGAL_TASK): Research, drafting, reviewing, summarizing, analyzing, or operational/document-management.

Proposed:

WITHIN SCOPE (PROFESSIONAL_TASK): Legal research, drafting (including professional email correspondence), reviewing, summarizing, analyzing, or operational/document-management. Email drafting and replies are ALWAYS in scope, regardless of whether the content is strictly legal.

Golden Dataset Structure

Input Schema

{
  "messages": [
    {"role": "user", "content": "User query..."},
    {"role": "assistant", "content": "Assistant response..."}  // For multi-turn
  ],
  "context": "Open document/email content if any",
  "attachments": ["Extracted text from attachments"]
}

Expected Output Schema

{
  "reasoning": "Step-by-step logic following 5-step hierarchy",
  "intent": "CURRENT_CONTEXT | SEARCH_LEGAL_CONTENT | SEARCH_EMAIL_OR_ATTACHMENT | SEARCH_DOCUMENT | MULTI_SOURCE_SEARCH | NON_LEGAL_TASKS",
  "complexity": "low | medium | high",
  "last_user_message_only": true | false,
  "involves_email_generation": true | false
}

Evaluation Criteria

1. Intent Accuracy (Primary)

REQUIRED: intent matches expected category
CONDITIONAL: IF complexity == "high" → intent MUST be "MULTI_SOURCE_SEARCH" (validator forces this)

2. Complexity Assessment

REQUIRED: complexity is appropriate for query complexity
  - "low": Simple, single-step tasks (summarize this, translate this)
  - "medium": Standard tasks requiring some reasoning
  - "high": Multi-source analysis, cross-referencing, complex legal research

3. Reasoning Quality

REQUIRED: reasoning follows 5-step hierarchy (Steps 1-5)
REQUIRED: reasoning addresses HITL reconstruction if applicable
FORBIDDEN: reasoning skips steps or uses circular logic

4. Email Generation Flag

CONDITIONAL: IF query mentions "draft", "reply", "compose", "write email" → involves_email_generation MUST be true
CONDITIONAL: IF involves_email_generation == true AND intent == "CURRENT_CONTEXT" → valid email drafting flow

5. Topic Change Detection

CONDITIONAL: IF latest message is unrelated to previous context → last_user_message_only SHOULD be true

Test Case Categories

Category A: CURRENT_CONTEXT (8-10 cases)

Single-turn and multi-turn examples for:

  • [ ] Document summarization
  • [ ] Email drafting/replies (CRITICAL - include professional non-legal emails)
  • [ ] Translation tasks
  • [ ] Audio transcription (always CURRENT_CONTEXT)
  • [ ] Q&A on attached documents
  • [ ] Clause drafting

Category B: SEARCH_LEGAL_CONTENT (6-8 cases)

  • [ ] Statute/law lookups
  • [ ] Regulatory requirements
  • [ ] Legal definitions
  • [ ] Jurisdiction-specific legal queries
  • [ ] Case law research

Category C: SEARCH_EMAIL_OR_ATTACHMENT (6-8 cases)

  • [ ] Email thread searches
  • [ ] Attachment retrieval from emails
  • [ ] Named entity email searches (e.g., "emails from Sarah")
  • [ ] Date-filtered email searches

Category D: SEARCH_DOCUMENT (5-6 cases)

  • [ ] SharePoint document searches
  • [ ] OneDrive file retrieval
  • [ ] Named repository searches
  • [ ] Folder-specific searches

Category E: MULTI_SOURCE_SEARCH (6-8 cases)

  • [ ] Cross-referencing documents with laws
  • [ ] Compliance checks (document + regulation)
  • [ ] Combined email + document analysis
  • [ ] Contract comparison with legal standards

Category F: NON_LEGAL_TASKS (4-5 cases)

  • [ ] General knowledge questions (weather, sports)
  • [ ] File format conversions
  • [ ] Recipes, jokes, off-topic requests
  • [ ] Pure chit-chat

Category G: HITL Reconstruction (4-5 cases)

Multi-turn conversations where:

  • [ ] User provides clarification to assistant question
  • [ ] Brief confirmations ("Yes", "Correct", "UK")
  • [ ] Parameter additions after prompting

Category H: Intent Transitions (4-6 cases)

Multi-turn conversations where intent changes:

  • [ ] SEARCH_LEGAL_CONTENT → CURRENT_CONTEXT (found law, now summarize)
  • [ ] SEARCH_EMAIL_OR_ATTACHMENT → MULTI_SOURCE_SEARCH (found email, compare with law)
  • [ ] NON_LEGAL_TASKS → SEARCH_LEGAL_CONTENT (rejected, then valid query)

Special Test Cases: Email Drafting Edge Cases

These MUST be classified as CURRENT_CONTEXT with involves_email_generation: true:

  1. Professional apology email (the reported issue):

``
"Please can you reply to Neil, sorry for the delay been a crazy week..."
``
Expected: CURRENT_CONTEXT, involves_email_generation: true

  1. Meeting follow-up email:

``
"Draft a follow-up email to the team about today's meeting outcomes"
``
Expected: CURRENT_CONTEXT, involves_email_generation: true

  1. Status update email:

``
"Compose an email to the client updating them on project progress"
``
Expected: CURRENT_CONTEXT, involves_email_generation: true

  1. Scheduling email:

``
"Write a reply accepting the meeting invitation for next Tuesday"
``
Expected: CURRENT_CONTEXT, involves_email_generation: true

Acceptance Criteria

  • [ ] 40-50 input/expected-output pairs
  • [ ] Balanced single-turn (~50%) and multi-turn (~50%) conversations
  • [ ] All 6 intent categories represented with adequate coverage
  • [ ] At least 5 email drafting edge cases (professional, non-legal)
  • [ ] HITL reconstruction scenarios included
  • [ ] Intent transition scenarios included
  • [ ] All assertions are machine-checkable

Technical Notes

Files to Reference:

  • [planning.py:1411-1755](verdict/app/qcounsel/langgraph_workflows/nodes/planning.py#L1411-L1755) - intent_node function
  • [models.py:89-120](verdict/app/qcounsel/langgraph_workflows/models/models.py#L89-L120) - EnhancedIntentClassification model
  • [test_enhanced_intent_classification.py](verdict/tests/qcounsel/test_enhanced_intent_classification.py) - Existing test cases (can be used as reference)

Model Configuration:

  • Model: azure/gpt-5.1-2025-11-13
  • Temperature: 0.2
  • Response format: Structured output via Instructor

🤖 Generated with Claude Code

View original on GitHub ↗

This issue has 1 comment on GitHub. Read the full discussion on GitHub ↗