[Technical Debt] Extract hardcoded confidence threshold to configuration
Summary
The 0.8 confidence threshold for auto-execution in intent inference is hardcoded in the system prompt, making it impossible to tune without code changes.
Current State
In app/llm/intent_inference.py:282, the system prompt contains:
- 0.8: Clear intent, some details need inference (THRESHOLD for auto-execution)
This threshold is critical for HITL escalation decisions but cannot be adjusted per business or workflow type.
Problems
- No runtime configuration: Cannot A/B test different thresholds (0.7 vs 0.8 vs 0.9)
- No per-tenant customization: High-risk businesses may want higher threshold (0.9)
- Hard to tune: Cannot optimize threshold based on accuracy metrics
- Inconsistent: Threshold appears in prompt but not as a code constant
Proposed Solution
- Extract to module-level constant:
``python``
# app/llm/intent_inference.py
MODIFICATION_INTENT_CONFIDENCE_THRESHOLD = 0.8
- Reference in prompt template:
``python``
f"- {MODIFICATION_INTENT_CONFIDENCE_THRESHOLD}: Clear intent (THRESHOLD for auto-execution)"
- (Optional) Allow per-business configuration:
``python``
threshold = deps.business_config.get("intent_confidence_threshold", 0.8)
Acceptance Criteria
- [ ] Confidence threshold extracted to named constant
- [ ] Constant referenced in prompt template (DRY principle)
- [ ] Documentation explains how threshold affects HITL escalation
- [ ] Tests verify threshold is applied correctly
- [ ] (Stretch) Support per-business threshold override
Priority
Low - Nice to have for tuning but not blocking
Estimated Effort
1-2 hours (basic) or 3-4 hours (with per-business config)
Related
- Agent 7 (LLM Modules)
app/llm/intent_inference.py- HITL escalation logic
This issue has 3 comments on GitHub. Read the full discussion on GitHub ↗