[Bug] Agent hallucinated parameter enum values; incomplete validation left unvalidated gap despite completeness claim
Bug Description
Title: Agent mode (Claude Haiku) hallucinated a parameter's valid values; claimed-complete validation left the exact gap unvalidated
Body:
A GitHub Copilot agent-mode session (model: Claude Haiku) authored commit 3cc158a in devops_os_mcp, adding input validation and docstrings across 8 MCP tool wrappers. One docstring claimed a parameter's valid values were 'basic' or 'complete'. The actual enum it wrapped (WORKFLOW_TYPES in the same repo, same diff's neighborhood) never contained 'basic' — confirmed via full git history search (287 commits). The commit message claimed "Integrate validators into all 8 tools," but this specific parameter was left unvalidated, and none of the "107 tests" covered the claimed value. The unvalidated bad value reached a sys.exit(1) in a wrapped library function; called from a live server's async worker, that silently hung the server for 120s+ per call and crashed the whole connection on cancellation.
Notably, a same-day prior commit (64af968) from the same agent session had already produced a baseline audit explicitly listing "No timeout enforcement (potential hang)" as a known gap — the very next commit closed that audit item on paper without actually checking the new work against it.
Why this is worth flagging to agent-mode specifically: the failure isn't just a wrong value — it's a pattern: (1) describing a parameter from plausible inference instead of reading the enum it constrains, (2) an unverified-per-item completeness claim ("all 8 tools"), (3) a prior self-authored risk audit that wasn't cross-checked against the fix meant to close it. Agent-mode's task-completion review step could plausibly catch all three by requiring claims to cite the exact source line and by diffing new work against the session's own earlier audit findin
Repro context / evidence: commit 3cc158a vs. devops_os/core/scaffold_gha.py's WORKFLOW_TYPES; commit 64af968's audit doc.
Environment Info
- Platform: darwin
- Terminal: Apple_Terminal
- Version: 2.1.285
- Feedback ID: 23e39976-3cac-49e5-ad2f-6df0ef88851b
Errors
[]This issue has 1 comment on GitHub. Read the full discussion on GitHub ↗