Background daemon restart drops in-flight agent workers without respawn (bg adopt: respawned=0 dead=N)

Status Open
Reported on v2.1.214
Maintainer reply None cached
Activity 5 comments · opened Jul 20, 2026

Summary

When the background-agent daemon churns (transient daemons losing lock races and exiting), in-flight background agent workers are dropped and never respawned. On every daemon (re)start, bg adopt logs adopted=0 respawned=0 dead=N — the workers are recognized as dead, but nothing restarts them and no failure notification reaches the session that dispatched them. The result is silent agent death: no final report, no error, work just vanishes.

Environment / scale

  • Claude Code 2.1.214–2.1.215, Linux (WSL2)
  • Heavy multi-agent usage: ~30–50 background agent jobs/day. Symptom onset correlates directly with scaling up from a handful of jobs/day (race odds multiply with volume).

Evidence

  • bg adopt: adopted=0 respawned=0 dead=N on every daemon (re)start (observed N=3,5,2,7,8,12…).
  • Frequent another daemon won the lock race (pid=…) — exiting / another daemon is already running. On 2026-07-18 the daemon restarted 5 times in ~35 minutes (03:49, 03:52, 03:54:04, 03:54:55, 04:25), each restart dropping workers (dead=3/5/1/2).
  • Every claude invocation can spawn a competing transient daemon; the loser exits and its adopted workers are at risk.

Expected

On daemon adopt, live-but-orphaned or recently-dead in-flight workers should be respawned/resumed — or at minimum a worker-lost event should be delivered to the owning session so the dispatcher can react, instead of waiting forever.

View original on GitHub ↗

4 Comments

IgorGanapolsky · 1 month ago

The adopt log is the whole bug: dead=N with respawned=0 means the dispatcher is waiting forever on workers the daemon already classified as gone. A lock-race restart that drops in-flight jobs is silent death — no final report, no error to the session that launched them. At minimum, deliver a worker-lost event so the owner can react.

If those workers have to outlive the local daemon (and the machine sleeping under WSL), they cannot live in that process. I run that as hosted Hermes on a fenced VPS, $10/mo, 14-day trial: https://thumbgate.app — money/customer/prod pauses in the browser.

jan-suchan · 1 month ago

Another data point from a different trigger, with what looks like a clean version boundary.

Same end state, but adoption appears to succeed first. On this box every restart logs a successful adopt and then loses the workers ~19s later:

+0..3s    [bg] bg adopt: adopted=N respawned=0 dead=0     <- looks like a clean handover
+18..23s  [bg] bg adopt: adopted=0 respawned=0 dead=N     <- gone

Anything reading the first line treats it as a survival signal, which is how this went unnoticed here for a week — a local watchdog reported "nothing lost" on an event that lost everything.

Version boundary. I scanned every binary at … changed event in ~/.claude/daemon.log back to 2026-06-30 and counted workers lost in the 120s after each:

| Range | Binary changes | Workers lost |
|---|---|---|
| 2.1.197 → 2.1.239 (30 Jun – 21 Aug) | 39 | 0 (two isolated crashed settles, 14 + 15 Jul) |
| 2.1.239 → 2.1.241 onward (23 Aug – 30 Aug) | 8 | all of them |

2026-08-23  2.1.239 → 2.1.241   +19s dead=1
2026-08-24  2.1.241 → 2.1.243   +23s dead=1
2026-08-25  2.1.243 → 2.1.245   +18s dead=2
2026-08-26  2.1.245 → 2.1.246   +19s dead=1
2026-08-27  2.1.246 → 2.1.247   +23s dead=1
2026-08-27  2.1.247 → 2.1.248   +19s dead=1
2026-08-28  2.1.248 → 2.1.250   +19s dead=1
2026-08-28  2.1.250 → 2.1.251   +19s dead=1
2026-08-30  2.1.251 → 2.1.236   +14s (lost via `bg settled <id> (done)` rather than `dead=`)

Before 2.1.241 a restart adopted up to ten workers with dead=0 and they kept running. Eight for eight after it. If that holds elsewhere, 2.1.239→2.1.241 is where to bisect.

Differences from the original report that may narrow it:

  • Not volume-dependent. 1–3 concurrent workers here, not 30–50/day.
  • Not lock-race churn. These are single deliberate restarts triggered by the binary path changing, not transient daemons losing a lock race.
  • Idle sessions are killed too. The workers lost on 30 Aug had been idle 7h, 7h and 16h — sessions stopped waiting for a human answer, with no request in flight. So this is not only about in-flight work; a session parked awaiting input is also lost.
  • Direction of the version change does not matter. The 30 Aug event was a downgrade (2.1.251 → 2.1.236, stable channel trailing) and lost workers just the same.
  • Linux, native install, daemon running as a persistent systemd user unit (claude daemon run, origin=foreground), so no idle-exit involved.

Last observation, on the loss being counted. Seven of the eight recorded it as dead=N; the 30 Aug one recorded it as bg settled <id> (done) instead — the same loss, a different line, and (done) reads as a normal completion. Any tooling that counts only dead= will under-report.

Practical impact: on a box where sessions are reviewed from a phone, a killed session stays listed in claude agents with no pid, so the session opens but nothing answers.

jan-suchan · 1 month ago

Ran the controlled measurement this comment was waiting on: 2.1.236 → 2.1.259, one throwaway
claude --bg probe per phase, survival judged by kill -0 on its pid, not by the adopt line.

Both phases lost the worker — including the control, no version change at all:

| Phase | Action | Probe survived? |
|---|---|---|
| control | systemctl --user restart claude-daemon@<acct>, binary unchanged (2.1.236) | No — dead immediately, first adopt line already read dead=8 |
| upgrade | claude update, 2.1.236 → 2.1.259 | No — alive at +30s, dead by +50s |

The control result updates my own "before 2.1.241, 0 of 39" table above — a same-binary restart
on 2.1.236 lost every accumulated worker outright this time. Different daemon uptime (4+ days,
9 accumulated workers) than most of that sample, so I'm not sure yet whether that's the same
regression under a bigger backlog or a separate thing.

Found the actual mechanism for the upgrade case, and it isn't in the daemon at all — it's the
systemd unit fighting the daemon's own self-restart.
Reading ~/.claude/daemon.log timestamps
directly (not the adopt-line summary) for the upgrade phase:

T+0.00s   binary changed 2.1.236 → 2.1.259 — self-restarting for upgrade
T+0.00s   supervisor shutting down (cause=upgrade, live_workers=1)
T+0.28s   daemon start version=2.1.259
T+0.31s   bg adopt: adopted=1 respawned=0 dead=0      <- clean handover, worker preserved
T+0.42s   bg orphan-spare reap: 1
T+0.70s   supervisor shutting down (cause=signal, uptime=0s, live_workers=1)   <- second restart, ~0.4s later
T+17.8s   daemon start version=2.1.259
T+18.3s   bg adopt: adopted=0 respawned=0 dead=1      <- the same worker, now dead

The daemon's own self-managed restart-on-binary-change works exactly as intended — the first
bg adopt genuinely succeeds. But claude-daemon@.service runs under Restart=always /
RestartSec=10. When the daemon's self-managed handoff exits the pid systemd was tracking as
MainPID, systemd can't tell that apart from a crash, and ~10-17s later (RestartSec=10 plus a
few seconds of our own ExecStartPre) it fires a second, independent restart — visible in
systemctl --user status as Scheduled restart job, restart counter is at 1, i.e. systemd's own
doing, not the daemon's. That second restart is what actually kills the worker the first one
had just preserved.

This fits the ~19s delayed-death shape from my earlier comment better than either the lock-race
or the retire-sweep theory: the first adopt line isn't lying or racing anything — it's reporting
a real, successful handover — a second restart just undoes it a few seconds later. If this
generalizes, the fix isn't in the adopt/handover path at all, it's in not letting an external
supervisor's blind Restart=always treat the daemon's own graceful self-restart as a crash to
recover from — a way for the daemon to signal "I already handled this, don't restart me again"
(sentinel exit code + RestartPreventExitStatus=, or Type=notify with the new pid handed over
via sd_notify(MAINPID=…)) would stop the second restart from happening at all. I can't tell
from the outside whether the daemon already exits with a distinguishable code on a self-managed
handoff — happy to test a specific one if useful.

Environment (unchanged from my last comment)

  • Linux x64, native install (~/.local/share/claude/versions/<v>, launcher symlink at ~/.local/bin/claude)
  • Supervisor is a systemd user unit, claude daemon run in its own cgroup, Type=simple,

Restart=always, RestartSec=10, DISABLE_AUTOUPDATER=1

  • Updates are deliberate only: a guarded job runs claude update at 06:00 and only when no

worker carries a pid

  • 1-3 workers typical, not the 30-50/day churn in the original report

Smaller, separate papercut (still stands)

claude update printed Successfully updated from 2.1.251 to version 2.1.236 when
autoUpdatesChannel: "stable" resolved backwards. It's an update in the sense of "moved to
the channel head," but "Successfully updated" for a downgrade cost me a while to notice.
Switched to stable channel: 2.1.251 -> 2.1.236 (downgrade) would have been unambiguous.

jan-suchan · 1 month ago

Correcting my own comment above: its log excerpt is right, its conclusion is not. On this box the cause turned out to be local — my systemd unit, not Claude Code.

I blamed the Restart=always / RestartSec=10 restart firing ~10-17s later. Re-reading the same daemon.log to the millisecond, the worker is already dead long before that:

+0.28s  daemon start version=2.1.259 pid=4129926     <- successor is a CHILD pid, not an exec
+0.31s  bg adopt: adopted=1 respawned=0 dead=0       <- handover genuinely works
+0.70s  shutting down (cause=signal, uptime=0s, live_workers=1)   <- killed HERE
+17.8s  daemon start version=2.1.259 pid=4130025
+18.3s  bg adopt: adopted=0 respawned=0 dead=1       <- only discovers the corpse

The +17.8s restart is the recovery, not the kill. What kills the worker is systemd's stop sequence under the default KillMode=control-group: the daemon self-restarts by spawning a child, so the successor lands in the unit's cgroup while systemd still tracks the original pid as MainPID; when that original exits, systemd SIGTERMs everything left in the cgroup — the fresh successor at uptime=0s, and every worker with it.

Verified directly — cgroup.procs for the unit lists the supervisor beside every worker:

4130025 claude daemon run
4130045 claude bg-pty-host --bg-pty-host /tmp/cc-daemon-1001/.../spare/...
4130050 claude bg-spare    --bg-spare    /tmp/cc-daemon-1001/.../spare/...

So my suggested fix was wrong too. RestartPreventExitStatus, or Type=notify with sd_notify(MAINPID=…), cannot help on their own: the teardown happens in the stop sequence, before any restart decision is taken. For anyone running claude daemon run under systemd, the lever is KillMode=process so the stop sequence signals only the tracked MainPID and leaves the successor and its workers alone. I have not shipped that yet — when I have measured it I will report back either way.

What this means for this issue. The "supervisor restart drops in-flight workers" symptom, on a systemd-supervised Linux box, can be entirely local configuration and not the daemon's doing. Anyone reporting this shape should check systemctl show <unit> -p KillMode before attributing it upstream. The daemon's own binary-change handover is clean here on 2.1.259 — adopted=1 dead=0, and the worker survives it. That also retracts the retire-sweep theory I floated from the 2.1.257 changelog note.

Two things that would still help, and they are the reason I am not just closing this out:

  1. A documented way for an external supervisor to tell a self-managed handoff from a crash. Right now the daemon's graceful self-restart and a segfault look identical from outside: MainPID exits. sd_notify(MAINPID=<new pid>) under Type=notify would let systemd follow the handover instead of tearing it down.
  2. A line in the docs for supervised deployments — if you run the daemon under systemd, KillMode=process. The default is the footgun, and nothing in claude daemon --help hints at it.

One correction to my previous comment's other claim as well: I said a same-binary systemctl restart losing 8 of 8 workers contradicted my earlier "0 of 39 before 2.1.241" table. It does not — that table counted binary-change self-restarts, whereas a systemctl restart SIGTERMs the cgroup outright and was already known to be fatal. Two different event classes; I should not have pooled them.

Showing cached comments. Read the full discussion on GitHub ↗