[BUG] OTEL telemetry ECONNRESET propagates to main event loop, making CLI unusable

Resolved 💬 3 comments Opened Mar 21, 2026 by WYSIATI Closed Apr 16, 2026

Summary

When the OpenTelemetry exporter encounters ECONNRESET, the error propagates to the main event loop and makes the entire CLI unresponsive. The process stays alive but no input is processed. This is a critical violation of the principle that observability infrastructure must never affect application availability.

Duplicate/related to #36294 — filing separately with full root cause analysis and proposed implementation.

Root Cause Analysis

From the minified bundle (cli.js), the telemetry export path:

  1. The doExport function sends telemetry events via fetch() to the OTEL endpoint
  2. On ECONNRESET, fetch() throws an Error with code: 'ECONNRESET'
  3. This error is caught by queueFailedEvents which re-queues the failed events for retry
  4. The retry creates more ECONNRESET errors (endpoint is still unreachable)
  5. Error logging at ERROR level triggers the CLI's error display
  6. The exponential retry storm + error logging overwhelms the event loop, blocking user input

From debug logs (issue #36294):

[ERROR] [3P telemetry] OTEL diag error: {
  "message": "Failed to export 23 events (code=ECONNRESET, read ECONNRESET)"
}

From issue #28557 debug logs:

Error: 1P event logging: 14 events failed to export (code=ECONNRESET)

Key problems

  1. No error boundary: Telemetry export errors propagate up instead of being swallowed
  2. Failed events re-queued: Creates a retry storm that generates more errors
  3. ERROR-level logging: Clutters the terminal with non-actionable messages
  4. No circuit breaker: Exporter keeps trying forever even when the endpoint is unreachable

Proposed Fix

class ResilientTelemetryExporter {
  private consecutiveFailures = 0;
  private static readonly MAX_FAILURES = 5;
  private disabled = false;

  async export(events: TelemetryEvent[]): Promise<void> {
    if (this.disabled) return; // Circuit breaker open

    try {
      await this.doExport(events);
      this.consecutiveFailures = 0;
    } catch (error) {
      this.consecutiveFailures++;
      // Debug level ONLY — never error
      debug(`Telemetry export failed (${this.consecutiveFailures}/${ResilientTelemetryExporter.MAX_FAILURES}): ${(error as Error).message}`);

      if (this.consecutiveFailures >= ResilientTelemetryExporter.MAX_FAILURES) {
        this.disabled = true;
        debug('Telemetry disabled for this session due to repeated failures');
      }
      // NEVER re-throw — telemetry failures are always swallowed
      // NEVER re-queue failed events — drop them
    }
  }
}

// Fire-and-forget wrapper — never awaited, never rejects
function emitTelemetry(event: TelemetryEvent): void {
  exporter.export([event]).catch(() => {});
}

// Process-level safety net for any escaped rejections
process.on('unhandledRejection', (reason) => {
  if (isTelemetryError(reason)) {
    debug('Swallowed telemetry unhandled rejection');
    return; // Don't propagate
  }
});

function isTelemetryError(error: unknown): boolean {
  const msg = (error as Error)?.message || '';
  return msg.includes('events failed to export') ||
         msg.includes('Failed to export') ||
         msg.includes('telemetry') ||
         msg.includes('OTEL');
}

Design Decisions

  1. Circuit breaker (5 failures): After 5 consecutive failures, disable telemetry for the session. Persistent network issues won't improve during a single session
  2. Debug-level only: Telemetry failures should never appear unless --verbose is used
  3. Drop failed events: Don't re-queue. Telemetry data loss is acceptable; CLI availability is not
  4. Fire-and-forget: emitTelemetry() returns void, never awaits, never rejects
  5. Process-level safety: Catch telemetry-related unhandledRejection as defense-in-depth

Testing Plan

Unit Tests

| Test | Input | Expected |
|------|-------|----------|
| Success resets counter | 2 failures → 1 success | consecutiveFailures = 0 |
| Circuit breaker at 5 | 5 consecutive failures | disabled = true |
| Disabled exporter skips export | disabled = true | No network call made |
| Export error never propagates | doExport() throws | export() resolves, no throw |
| emitTelemetry never rejects | Exporter throws | No unhandled rejection |
| Failed events dropped | Export fails | Queue size unchanged |
| Debug-level logging only | Export fails | No ERROR level logs |
| Non-telemetry rejections propagate | Non-telemetry error | Handler passes through |

Integration Tests

| Test | Setup | Expected |
|------|-------|----------|
| CLI responsive during telemetry failure | Mock OTEL endpoint → ECONNRESET | User can type prompts, receive responses |
| Circuit breaker triggers | All exports fail | After 5, no more export attempts |
| API works while telemetry disabled | Circuit breaker open | API calls succeed normally |
| Multiple concurrent failures | 3 exports fail simultaneously | No race condition, counter correct |

E2E Tests

| Test | Steps | Expected |
|------|-------|----------|
| Block OTEL endpoint | Firewall block OTEL host → use Claude normally | Zero error messages, CLI fully functional |
| Debug mode shows failures | Block OTEL + --verbose | Debug messages visible, CLI still works |
| 20 prompts with OTEL down | Block OTEL → send 20 prompts | All 20 succeed, no visible errors |

Impact

This is a P0 fix — it converts a CLI-killing failure into a silent, graceful degradation. The fix is ~30 lines and carries zero risk to the main API workflow.

Related Issues

  • #36294 — OTEL telemetry ECONNRESET makes CLI unusable (primary duplicate)
  • #28557 — ECONNRESET issues (includes telemetry error in logs)

View original on GitHub ↗

This issue has 3 comments on GitHub. Read the full discussion on GitHub ↗