Orphaned Claude Code shell process consumed 2.88M tokens over 2+ days undetected

Summary

An orphaned Claude Code shell process ran an infinite loop for 2+ days, consuming approximately 2,880,000 tokens (~$50-100 in API costs) before being manually discovered. The issue went undetected despite monitoring infrastructure being in place.

Root Cause

while true; do
  curl -s http://127.0.0.1:8012/api/chat \
    -H 'Content-Type: application/json' \
    -d '{"question": "test", "repo": "agro", "final_k": 5}' > /dev/null
  sleep 2
done

Each API call:

• Retrieved 100-200 documents from RAG index
• Triggered Cohere reranking API on ALL documents (no limit enforced)
• Each document ≈ 175 tokens → 3,500+ tokens per call
• Loop executed every 2 seconds → 2,000+ tokens/minute

Impact

• Token consumption: 2.88M tokens over 48+ hours
• Cost impact: ~$50-100 (Cohere reranking rates)
• Detection time: 2+ days (discovered via manual Grafana observation)
• Root detection: Grep through process list and query logs

Why It Wasn't Caught

1. No rate limiting - API endpoints had no per-client limits
2. No process watchdog - Orphaned processes run indefinitely
3. No endpoint frequency tracking - Single client making 30+ calls/min went undetected
4. No alerting - Metrics were collected but no alerts fired on anomalies
5. No API call categorization - Couldn't detect pattern of repeated calls

Why This Matters for Claude Code

When users run multiple Claude Code agents or sessions, orphaned processes can silently consume tokens/credits. This is especially problematic because:

• Multiple concurrent Claude Code shells are common (agent orchestration, experimentation)
• Orphaned processes are hard to detect (long-running shells in background)
• Token consumption is invisible until bill arrives
• No built-in safeguards or alerts

What Should Have Prevented This

1. ✅ Rate limiting middleware: Reject client making >N requests/min
2. ✅ Process watchdog: Alert if same process makes 1000+ calls without human interaction
3. ✅ Endpoint frequency tracking: Alert on "orphaned loop" patterns
4. ✅ API cost anomaly detection: Alert on 10x cost spike
5. ✅ Alerting system: Route alerts to user (Slack, email, etc)

Potential Fixes

1. Add per-client rate limiting (e.g., max 10 req/min from same IP/process)
2. Implement request deduplication (reject identical queries in short time window)
3. Add process lifetime tracking (warn if process makes >100 calls)
4. Integrate with shell history to detect loops in ~/.bash_history or similar
5. Add cost/token warnings to CLI output

Related Context

This incident occurred in an AGRO (RAG orchestration) environment with comprehensive monitoring (Prometheus, Grafana) but zero alerting configured. All infrastructure for detection existed; only alerting was missing.

Affected component: Potential issue in how Claude Code processes (particularly long-running agent shells) are managed and monitored.