Remote Control: reconnect burst wedges hosts in a no-backoff CPU spin (A); a busy turn is misread as "stopped responding mid-turn" (B)
Version: claude-code 2.1.199, Linux, Remote Control host mode. Setup: several always-on claude-rc@* hosts, each spawning worktree sub-sessions, driven from the Desktop app.
Two related but distinct RC-transport robustness issues; splitting them out below.
Issue A — reconnect spin-loop with no backoff (the serious one)
Summary
When many RC sessions reconnect in a short burst (observed: ~10 new sessions started in quick succession), the RC handshake fails for most of them, and each affected host TUI drops into a busy repaint/retry loop on the dead socket with no backoff — the "Remote Control disconnected" banner re-renders at ~1 Hz. It does not self-heal; each session stays stuck until the operator clicks Retry.
Impact
Self-inflicted, host-wide. Measured during the incident (16-core box):
vmstat:us=2 sy=42 id=56— ~zero userspace CPU, ~42% of the box in the kernel; ~20k interrupts/s, ~7k context-switches/s.- ~11
.claude-unwrappedprocesses each pegging 66–98% of a core in R state, spinning on poll/read/write of the dead socket. Per spinning process ≈ 640 nonvoluntary context-switches/s (a preempted busy loop making no forward progress). - Not resource starvation (56% idle CPU, RAM free, swap idle). Purely the reconnect state machine spinning.
The kernel-CPU storm then makes the box slower for every session, which appears to further delay the reconnect handshakes — a feedback loop that keeps the fleet wedged.
Likely root cause (inferred)
The post-disconnect reconnect path retries (and repaints) immediately in a tight loop rather than backing off, and treats a burst-induced handshake failure as retry-forever instead of a recoverable transient. No exponential backoff and no jitter, so N simultaneous reconnects also thundering-herd the relay.
Repro
- Point several RC host sessions at the relay.
- Start ~10 new sessions within a few seconds (a "burst").
- Observe multiple hosts stick on "Remote Control disconnected";
vmstatshows system CPU dominating with near-zero userspace; the banner repaints ~1 Hz; only a manual Retry per session clears it.
Suggested fixes
- Exponential backoff + jitter on RC reconnect (e.g. 0.5s → cap ~30s), so a burst doesn't thundering-herd and a wedged session isn't a ~1 Hz spin.
- Yield between attempts — a failed poll/read on a dead socket should not busy-spin a core; sleep to the next backoff tick.
- Self-heal — after the backoff cycle, re-establish automatically instead of requiring a manual Retry.
- Decouple repaint from retry — the banner should not re-render every frame while disconnected.
Issue B — a busy turn is misread as "stopped responding mid-turn"
Summary
While a session is busy handling a turn (running tools / long-running Bash), the RC transport declares it dead and shows: "The bridged Claude Code process stopped responding mid-turn. Check your terminal for errors (you may need to run /login), then resend your message." It then reconnects on its own, typically within ~10 s. So it is a false positive — the process was alive and working, just not servicing the RC heartbeat while occupied in the turn.
Impact
Cosmetic-to-disruptive: frequent spurious disconnect banners on any actively-working session, worse under load and with many concurrent sessions. The suggested remedy ("resend your message") is misleading — the turn was not lost. The trigger correlates with the session being busy in a turn, and the timeout appears server/relay-side (not client-tunable).
Likely root cause (inferred)
The liveness heartbeat shares the thread / event loop that a turn occupies, so a long turn starves the heartbeat past the relay's "stopped responding" timeout, even though the process is healthy.
Suggested fixes
- Heartbeat independent of turn progress — emit liveness from a path that a busy turn cannot starve, so "working" is distinguishable from "dead."
- Distinguish busy from unresponsive in the relay, or make the mid-turn timeout longer / configurable.
- Fix the guidance — if it was just a heartbeat gap, don't tell the user to
/loginand resend (the turn is intact).
Workaround (for others hitting this)
- A (stuck): stagger session starts (a few at a time, not a burst) so reconnects don't handshake at once; when wedged, click Retry per host (do not restart the host process — that kills its spawned sub-sessions). Detectable with a per-host cgroup
cpu.stattripwire: a kernel-dominated CPU delta + ≥100 nonvoluntary ctxt-sw/s + two-run persistence separates a real spin from ordinary busy work. - B (transient): benign — let it recover / resend; it self-heals in ~10 s.