[BUG] CLI hangs indefinitely when SSE stream dies silently — no inactivity watchdog
Summary
When a streaming SSE connection dies silently (no TCP RST packet), the CLI hangs indefinitely. There is no watchdog timer monitoring the SSE stream for event gaps. Common triggers: NAT timeout during extended thinking, silent Wi-Fi disconnect, load balancer idle timeout. Users must kill the process manually.
Root Cause Analysis
The streaming API call iterates over SSE events using an async generator:
for await (const event of stream) {
// process event
yield event;
}
When the TCP connection dies silently (e.g., NAT drops the mapping), the read() call on the response body stream blocks forever — there's no FIN or RST to signal the connection is dead.
Why silent deaths happen
- NAT gateway timeout: Corporate/home NATs expire idle TCP mappings after 5-30 min. If the API is "thinking" (extended thinking for Opus can take 60s+), no events flow. The NAT drops the mapping. Server sends events into the void. Client waits forever.
- Network transition: Wi-Fi drops briefly without sending RST. Old TCP session is gone.
- Load balancer timeout: CDN/API gateway closes idle connections. No RST reaches the client through the proxy chain.
Current timeout constants (from minified cli.js)
uF9 = 1800000 // 30 min — max operation timeout
BF9 = 20000 // 20 sec — retry delay cap
mF9 = 600000 // 10 min — fallback timeout
None of these is a stream inactivity watchdog. They appear to be operation-level timeouts, not event-gap detectors. The Anthropic API sends periodic SSE ping events during extended thinking, but without a watchdog to detect when pings stop arriving, they provide no protection.
Proposed Fix
class StreamWatchdog {
private timer: ReturnType<typeof setTimeout> | null = null;
constructor(
private readonly timeoutMs: number,
private readonly onTimeout: () => void,
) {}
/** Call on every SSE event (including ping/heartbeat) */
kick() {
if (this.timer) clearTimeout(this.timer);
this.timer = setTimeout(() => this.onTimeout(), this.timeoutMs);
}
stop() {
if (this.timer) {
clearTimeout(this.timer);
this.timer = null;
}
}
}
// Integration into streaming API call:
async function* streamApiResponse(request: ApiRequest, options: StreamOptions) {
const timeoutMs = parseInt(
process.env.CLAUDE_CODE_STREAM_TIMEOUT || '45000', 10
);
const abortController = new AbortController();
const watchdog = new StreamWatchdog(timeoutMs, () => {
debug(`Stream watchdog: no events for ${timeoutMs}ms, aborting`);
abortController.abort(new Error('Stream inactivity timeout'));
});
try {
const stream = await client.messages.create({
...request,
stream: true,
signal: AbortSignal.any([options.signal, abortController.signal]),
});
watchdog.kick(); // Start watchdog
for await (const event of stream) {
watchdog.kick(); // Reset on every event
yield event;
}
} catch (error) {
if ((error as Error).message === 'Stream inactivity timeout') {
// Convert to retryable connection error
const err = new Error('Connection lost during streaming');
(err as any).code = 'ESTREAM_TIMEOUT';
throw err;
}
throw error;
} finally {
watchdog.stop();
}
}
Design Decisions
- 45-second default: Anthropic API sends SSE pings. Extended thinking can have long pauses, but pings should still arrive. 45s is generous. Configurable via
CLAUDE_CODE_STREAM_TIMEOUT - Reset on ALL events:
ping,content_block_start,content_block_delta, etc. Any event proves liveness - Custom error code
ESTREAM_TIMEOUT: Integrates with the proposed connection retry logic — stream timeouts should be retried AbortSignal.any(): Composes user abort (Escape) with watchdog abort — whichever fires first wins- No partial response recovery: When watchdog fires, the entire response is aborted and retried from scratch. Partial recovery is a separate feature (#26729 Layer 3)
Testing Plan
Unit Tests
| Test | Input | Expected |
|------|-------|----------|
| Fires after timeout with no kicks | Watchdog(1000ms), no kick | onTimeout at ~1000ms |
| Resets on kick | Watchdog(1000ms), kick at 800ms | onTimeout at ~1800ms |
| No fire with frequent kicks | Kick every 500ms for 5s | Never fires |
| Stop cancels timer | Stop at 500ms | Never fires |
| Multiple rapid kicks | 100 kicks in 10ms | Only 1 active timer |
| Custom timeout from env var | CLAUDE_CODE_STREAM_TIMEOUT=10000 | 10s timeout |
| Default is 45s | No env var | 45000ms |
Integration Tests
| Test | Setup | Expected |
|------|-------|----------|
| Normal stream completes | Events every 100ms for 5s | Completes, watchdog never fires |
| Extended thinking with pings | No content 30s, pings every 10s | Watchdog reset by pings |
| Dead connection detected | 3 events then silence 46s | ESTREAM_TIMEOUT at 45s |
| Watchdog + user Escape | Dead connection + Escape | First signal wins, no crash |
| Timeout triggers retry | Dead connection | ESTREAM_TIMEOUT retried (with connection retry fix) |
| Cleanup on normal completion | Stream completes | Timer cleared, no leaks |
| Cleanup on error | API returns 500 | Timer cleared |
E2E Tests
| Test | Steps | Expected |
|------|-------|----------|
| Firewall drop mid-stream | Start prompt → iptables DROP port 443 → wait 50s | "Connection lost during streaming. Retrying..." at ~45s |
| Extended thinking works | Complex Opus prompt | No false timeout; pings keep watchdog alive |
| Escape still works | Dead connection → Escape before 45s | Immediate abort by user |
| CLI doesn't hang | Kill network during streaming | Error message within 45s, prompt returns |
Impact
Prevents the worst possible UX: infinite hang requiring kill -9. The fix is ~40 lines with clear boundaries. Pairs naturally with the connection retry fix (issue #37077) for automatic recovery.
Related Issues
- #26729 — Streaming Resilience: Detect network loss, save state, auto-resume
- #5674 — Persistent ECONNRESET (often manifests as hangs during streaming)
This issue has 3 comments on GitHub. Read the full discussion on GitHub ↗