[Technical Debt] Extract hardcoded confidence threshold to configuration

Resolved 💬 3 comments Opened Nov 17, 2025 by roblafave Closed Jan 18, 2026

Summary

The 0.8 confidence threshold for auto-execution in intent inference is hardcoded in the system prompt, making it impossible to tune without code changes.

Current State

In app/llm/intent_inference.py:282, the system prompt contains:

- 0.8: Clear intent, some details need inference (THRESHOLD for auto-execution)

This threshold is critical for HITL escalation decisions but cannot be adjusted per business or workflow type.

Problems

  1. No runtime configuration: Cannot A/B test different thresholds (0.7 vs 0.8 vs 0.9)
  2. No per-tenant customization: High-risk businesses may want higher threshold (0.9)
  3. Hard to tune: Cannot optimize threshold based on accuracy metrics
  4. Inconsistent: Threshold appears in prompt but not as a code constant

Proposed Solution

  1. Extract to module-level constant:

``python
# app/llm/intent_inference.py
MODIFICATION_INTENT_CONFIDENCE_THRESHOLD = 0.8
``

  1. Reference in prompt template:

``python
f"- {MODIFICATION_INTENT_CONFIDENCE_THRESHOLD}: Clear intent (THRESHOLD for auto-execution)"
``

  1. (Optional) Allow per-business configuration:

``python
threshold = deps.business_config.get("intent_confidence_threshold", 0.8)
``

Acceptance Criteria

  • [ ] Confidence threshold extracted to named constant
  • [ ] Constant referenced in prompt template (DRY principle)
  • [ ] Documentation explains how threshold affects HITL escalation
  • [ ] Tests verify threshold is applied correctly
  • [ ] (Stretch) Support per-business threshold override

Priority

Low - Nice to have for tuning but not blocking

Estimated Effort

1-2 hours (basic) or 3-4 hours (with per-business config)

Related

  • Agent 7 (LLM Modules)
  • app/llm/intent_inference.py
  • HITL escalation logic

View original on GitHub ↗

This issue has 3 comments on GitHub. Read the full discussion on GitHub ↗