[BUG] OTEL telemetry ECONNRESET propagates to main event loop, making CLI unusable
Summary
When the OpenTelemetry exporter encounters ECONNRESET, the error propagates to the main event loop and makes the entire CLI unresponsive. The process stays alive but no input is processed. This is a critical violation of the principle that observability infrastructure must never affect application availability.
Duplicate/related to #36294 — filing separately with full root cause analysis and proposed implementation.
Root Cause Analysis
From the minified bundle (cli.js), the telemetry export path:
- The
doExportfunction sends telemetry events viafetch()to the OTEL endpoint - On ECONNRESET,
fetch()throws anErrorwithcode: 'ECONNRESET' - This error is caught by
queueFailedEventswhich re-queues the failed events for retry - The retry creates more ECONNRESET errors (endpoint is still unreachable)
- Error logging at
ERRORlevel triggers the CLI's error display - The exponential retry storm + error logging overwhelms the event loop, blocking user input
From debug logs (issue #36294):
[ERROR] [3P telemetry] OTEL diag error: {
"message": "Failed to export 23 events (code=ECONNRESET, read ECONNRESET)"
}
From issue #28557 debug logs:
Error: 1P event logging: 14 events failed to export (code=ECONNRESET)
Key problems
- No error boundary: Telemetry export errors propagate up instead of being swallowed
- Failed events re-queued: Creates a retry storm that generates more errors
- ERROR-level logging: Clutters the terminal with non-actionable messages
- No circuit breaker: Exporter keeps trying forever even when the endpoint is unreachable
Proposed Fix
class ResilientTelemetryExporter {
private consecutiveFailures = 0;
private static readonly MAX_FAILURES = 5;
private disabled = false;
async export(events: TelemetryEvent[]): Promise<void> {
if (this.disabled) return; // Circuit breaker open
try {
await this.doExport(events);
this.consecutiveFailures = 0;
} catch (error) {
this.consecutiveFailures++;
// Debug level ONLY — never error
debug(`Telemetry export failed (${this.consecutiveFailures}/${ResilientTelemetryExporter.MAX_FAILURES}): ${(error as Error).message}`);
if (this.consecutiveFailures >= ResilientTelemetryExporter.MAX_FAILURES) {
this.disabled = true;
debug('Telemetry disabled for this session due to repeated failures');
}
// NEVER re-throw — telemetry failures are always swallowed
// NEVER re-queue failed events — drop them
}
}
}
// Fire-and-forget wrapper — never awaited, never rejects
function emitTelemetry(event: TelemetryEvent): void {
exporter.export([event]).catch(() => {});
}
// Process-level safety net for any escaped rejections
process.on('unhandledRejection', (reason) => {
if (isTelemetryError(reason)) {
debug('Swallowed telemetry unhandled rejection');
return; // Don't propagate
}
});
function isTelemetryError(error: unknown): boolean {
const msg = (error as Error)?.message || '';
return msg.includes('events failed to export') ||
msg.includes('Failed to export') ||
msg.includes('telemetry') ||
msg.includes('OTEL');
}
Design Decisions
- Circuit breaker (5 failures): After 5 consecutive failures, disable telemetry for the session. Persistent network issues won't improve during a single session
- Debug-level only: Telemetry failures should never appear unless
--verboseis used - Drop failed events: Don't re-queue. Telemetry data loss is acceptable; CLI availability is not
- Fire-and-forget:
emitTelemetry()returns void, never awaits, never rejects - Process-level safety: Catch telemetry-related
unhandledRejectionas defense-in-depth
Testing Plan
Unit Tests
| Test | Input | Expected |
|------|-------|----------|
| Success resets counter | 2 failures → 1 success | consecutiveFailures = 0 |
| Circuit breaker at 5 | 5 consecutive failures | disabled = true |
| Disabled exporter skips export | disabled = true | No network call made |
| Export error never propagates | doExport() throws | export() resolves, no throw |
| emitTelemetry never rejects | Exporter throws | No unhandled rejection |
| Failed events dropped | Export fails | Queue size unchanged |
| Debug-level logging only | Export fails | No ERROR level logs |
| Non-telemetry rejections propagate | Non-telemetry error | Handler passes through |
Integration Tests
| Test | Setup | Expected |
|------|-------|----------|
| CLI responsive during telemetry failure | Mock OTEL endpoint → ECONNRESET | User can type prompts, receive responses |
| Circuit breaker triggers | All exports fail | After 5, no more export attempts |
| API works while telemetry disabled | Circuit breaker open | API calls succeed normally |
| Multiple concurrent failures | 3 exports fail simultaneously | No race condition, counter correct |
E2E Tests
| Test | Steps | Expected |
|------|-------|----------|
| Block OTEL endpoint | Firewall block OTEL host → use Claude normally | Zero error messages, CLI fully functional |
| Debug mode shows failures | Block OTEL + --verbose | Debug messages visible, CLI still works |
| 20 prompts with OTEL down | Block OTEL → send 20 prompts | All 20 succeed, no visible errors |
Impact
This is a P0 fix — it converts a CLI-killing failure into a silent, graceful degradation. The fix is ~30 lines and carries zero risk to the main API workflow.
Related Issues
- #36294 — OTEL telemetry ECONNRESET makes CLI unusable (primary duplicate)
- #28557 — ECONNRESET issues (includes telemetry error in logs)
This issue has 3 comments on GitHub. Read the full discussion on GitHub ↗