OOM crash under multi-subagent load: subagent CLI processes grow to ~6.5 GB RSS each; CliGovernor logs critical memory pressure for 8 min but evicts nothing (Linux, Desktop 1.17377.0 / CLI 2.1.187)

Status Open
Reported on v2.1.187
Maintainer reply None cached
Activity 6 comments · opened Jul 3, 2026

Summary

Under a multi-subagent workload (5–7 background agents over ~35 min), individual Claude Code CLI child processes grew to ~6.5 GB RSS each (plus 3–7.6 GB swapped, each). Four concurrent processes exhausted a 31 GB RAM + 20 GB swap Linux machine. The kernel OOM-killer selected the desktop app's NetworkService utility process (oom_score_adj 300), which fatally broke the Electron main process ("networking is broken until relaunch") — the user experiences this as the desktop app crashing. The app's own memory governor logged pressure at critical repeatedly for ~8 minutes before the kill and took no action each time.

Two distinct defects:

  1. CLI subagent memory ballooning — CC CLI subagent/session processes reaching ~6.2–6.7 GB RSS each (17 GB total-vm on the largest). Four concurrent = ~25.6 GB RSS + ~18.6 GB swap.
  2. CliGovernor takes no action at critical pressure — from 15:50:48 to 15:57 it logged a warning/critical pair every ~40–60 s, each time reporting "would evict 0 idle session(s), 1 effective", and never evicted, throttled, or surfaced anything to the user before the kernel intervened.

Environment

  • Claude Code Desktop 1.17377.0, Electron 42.5.1, bundled CC CLI 2.1.187
  • Pop!_OS (Linux 7.0.11-76070011-generic), 31 GB RAM + 20 GB swap, COSMIC desktop
  • Workload: interactive session spawning 5–7 background subagents (Task tool fan-out) doing HTTP QA + file writes

Timeline (2026-07-03, CDT)

  • 15:23 — app launch; heavy subagent fan-out session begins
  • 15:50:48 — first [CliGovernor] memory pressure (warning): would evict 0 idle session(s), 1 effective
  • 15:51:28 → 15:57 — memory pressure (critical) + same "would evict 0" line repeating every ~40–60 s
  • 15:58:21 — kernel OOM (free swap 180 kB of 20 GB); kills claude-desktop NetworkService (oom_score_adj 300)
  • 15:58:22 — app log: Network Service process gone (reason=killed, exitCode=15) outside app quit — main-process networking is broken until relaunch; Sentry event emitted (should exist server-side)
  • 15:58:42–15:59:09 — app auto-relaunches
  • 15:58:51 — kernel kills the largest CLI process (6.8 GB anon-rss)

Key evidence (verbatim)

Kernel journal:

Jul 03 15:58:21 pop-os kernel: tokio-rt-worker invoked oom-killer: gfp_mask=0x140cca(...), order=0, oom_score_adj=0
Jul 03 15:58:21 pop-os kernel: Free swap  = 180kB
Jul 03 15:58:21 pop-os kernel: Total swap = 20970996kB
Jul 03 15:58:21 pop-os kernel: [  13123]  1000 13123  4253598  1663156  1662700  456  0 29573120  1987257  0 claude
Jul 03 15:58:21 pop-os kernel: [  14644]  1000 14644  3204926  1679146  1678681  465  0 23130112  1166198  0 claude
Jul 03 15:58:21 pop-os kernel: [  14852]  1000 14852  2680590  1625919  1625454  465  0 19800064   810017  0 claude
Jul 03 15:58:21 pop-os kernel: [  14974]  1000 14974  3204910  1749861  1749398  463  0 21512192   908963  0 claude
Jul 03 15:58:21 pop-os kernel: Out of memory: Killed process 8799 (claude-desktop) total-vm:1530612248kB, anon-rss:318980kB, ... oom_score_adj:300
Jul 03 15:58:51 pop-os kernel: Out of memory: Killed process 13123 (claude) total-vm:17014392kB, anon-rss:6806964kB, ...

(rss column is pages: 1,663,156 pages ≈ 6.35 GB RSS; swapents 1,987,257 ≈ 7.6 GB swapped — per process.)

App log (~/.config/Claude/logs/main.log):

2026-07-03 15:50:48 [warn] [CliGovernor] memory pressure (warning): would evict 0 idle session(s), 1 effective
2026-07-03 15:51:28 [warn] [CliGovernor] memory pressure (critical): would evict 0 idle session(s), 1 effective
   (warning/critical pair repeats every ~40–60s, 15:50 → 15:57)
2026-07-03 15:58:22 [error] Network Service process gone (reason=killed, exitCode=15) outside app quit — main-process networking is broken until relaunch
2026-07-03 15:58:53 [info] Starting app {

systemd user journal:

Jul 03 15:58:21 pop-os systemd[1531]: app-cosmic-claude-desktop-8422.scope: Failed with result 'oom-kill'.

Notes for triage

  • No crashpad minidump (all Crashpad dirs empty) — expected, SIGKILL is uncatchable. A Sentry event fired at 15:58:22 with the NetworkService error; should be retrievable server-side.
  • The 2.1.199 "background-agent daemon self-kill after unclean shutdown" bug is NOT implicated (daemon.log shows clean idle exit the prior evening, no respawn churn).
  • Suggested fixes: (a) memory ceiling / recycling for CLI subagent processes; (b) CliGovernor should act at critical (evict, pause spawns, or at minimum surface a user-visible warning) rather than logging "would evict 0" — the governor had 8 minutes of runway; (c) consider oom_score_adj on CLI children so the kernel prefers them over the app's NetworkService (killing networking takes down the whole app; killing one subagent is recoverable).

Repro guidance

Long interactive session on Linux; fan out 4+ background subagents each doing sustained tool work (HTTP calls, file reads/writes) for 30+ min; watch ps -o pid,rss,etime,args -C claude — RSS climbs into multiple GB per process without release.

View original on GitHub ↗

5 Comments

IgorGanapolsky · 1 month ago

Really detailed writeup. The "CliGovernor logged critical pressure but evicts nothing" behavior is particularly nasty — the governor has the telemetry to know it should act, but the eviction policy isn't wired up to actually kill child processes before the kernel OOM-killer steps in.

For the multi-subagent reliability angle: if you're running 5-7 background agents, you might benefit from an external coordination layer that can enforce resource limits and kill/respawn agents before they hit 6.5 GB RSS. I've been working on ThumbGate — it's primarily a pre-action check layer (prevents bad tool calls), but it also includes a governance/rate-limiting layer that wraps agent processes and can enforce per-agent resource ceilings.

The core idea: instead of relying on Claude's internal governor (which clearly isn't evicting aggressively enough here), you put an external watchdog that has hard kill authority. When an agent process exceeds a memory ceiling, the watchdog kills it and logs the failure pattern so the same workload can be prevented from spinning up in the same configuration next time.

Free tier covers basic usage. Might be worth testing alongside your multi-agent setup to see if the external governance prevents the OOM cascade. https://thumbgate.ai

yurukusa · 1 month ago

The nastiest part of this is the victim selection: the balloon is in the CLI subagents, but the OOM-killer took the desktop's NetworkService (oom_score_adj 300), so the crash lands on the Electron app instead of the process that actually grew. Two real fixes are vendor-side — the CliGovernor logging critical for 8 minutes while evicting 0 is a governor that has the telemetry but no wired-up eviction, and ~6.5 GB RSS per subagent is a leak, not a working set. Nothing below fixes either; it just gets you ahead of the kernel.
Cap the fan-out to what your RAM can hold. At ~6.5 GB resident per subagent, 5–7 concurrent is ~33–45 GB before swap — a 31 GB box is guaranteed to OOM. Until the per-process footprint is fixed, floor((RAM_GB - headroom) / 6.5) is the real concurrency ceiling (≈3 on your machine, not 7). This is the single most effective mitigation.
Watch RSS externally so you're warned before the kernel picks a victim. A tiny external poller sums the heavy node/claude processes and warns at a threshold you set below your total. Tested it on Linux — reports ok at idle, escalates to critical (exit 2) past the ceiling, and on a synthetic 4×6.5 GB load it sums to 26.0 GiB and trips on a 31 GB box:

WARN_GIB="${WARN_GIB:-16}"; CRIT_GIB="${CRIT_GIB:-22}"; sum=0; n=0
while read -r rss comm; do
  case "$comm" in *node*|*claude*) sum=$((sum+rss)); n=$((n+1));; esac
done < <(ps -eo rss=,comm= 2>/dev/null | awk '$1>200000')  # only >200 MiB procs
g=$(awk -v k="$sum" 'BEGIN{printf "%.1f",k/1048576}')
if awk -v g="$g" -v c="$CRIT_GIB" 'BEGIN{exit !(g>=c)}'; then
  echo "CRITICAL: $n heavy node/claude procs = ${g}GiB — cut subagent concurrency / stop spawning before the OOM-killer hits the desktop app" >&2; exit 2
elif awk -v g="$g" -v w="$WARN_GIB" 'BEGIN{exit !(g>=w)}'; then
  echo "WARNING: ${g}GiB resident — consider narrowing fan-out" >&2; exit 1
else echo "ok: ${g}GiB ($n heavy procs)"; fi

If you must run near the edge, stop the desktop app from being the victim. The kill hit NetworkService because its oom_score_adj was 300 (more killable than the runaway CLI). Raising the subagents' oom_score_adj (so the kernel sacrifices a subagent instead of the app) keeps the desktop alive when pressure does spike — but this only changes who dies, not that something dies, so it's a fallback behind the concurrency cap.
Honest limits: none of this fixes the leak or makes CliGovernor evict — it's early warning + a concurrency ceiling so a runaway fan-out degrades into "a subagent gets killed" instead of "the desktop app's networking breaks until relaunch." The would evict 0 idle session(s), 1 effective line is the key vendor-side thread: at critical pressure the governor should be able to suspend/evict the effective session's subagents, not just idle ones.

joshuarocksolid · 1 month ago

Same failure on the current build, different distro and desktop. Logs below in case they help. A couple of the lines look different from what's already in the thread, but I don't know the internals well enough to say what they mean, so I'll just put them here.

Setup:

  • Claude Desktop 1.20186.1 (latest in the apt repo as of today), bundled CLI 2.1.205, node 24.17.0
  • Debian 12 (bookworm), kernel 6.12.95+deb12-amd64, GNOME on X11
  • 30.9 GiB RAM + 8 GiB swap

So whatever this is, it isn't confined to Pop!_OS/COSMIC or to 1.17377.0.

CliGovernor reporting a non-zero evict count

In the original report the governor always says would evict 0, which I suppose could just mean there was nothing idle to evict at the time. On my machine it reported a non-zero count while at critical pressure, and as far as I can tell from the log it still didn't evict anything:

2026-07-13 14:14:01 [warn] [CliGovernor] memory pressure (critical): would evict 4 idle session(s), 12 effective
2026-07-13 14:22:41 [warn] [CliGovernor] memory pressure (critical): would evict 5 idle session(s), 13 effective
2026-07-13 14:48:02 [warn] [CliGovernor] memory pressure (critical): would evict 5 idle session(s), 13 effective
2026-07-13 15:00:12 [warn] [CliGovernor] memory pressure (critical): would evict 5 idle session(s), 13 effective
2026-07-13 15:05:22 [warn] [CliGovernor] memory pressure (critical): would evict 5 idle session(s), 13 effective

The count stays at 5 from 14:22 to 15:05. I'd have expected it to drop if an eviction had actually run, but I don't know what would evict is meant to do at this point in the code, so I could easily be misreading it.

at cap=10 lines

I also saw these, which I didn't see mentioned in the issue:

2026-07-13 14:13:58 [info] [CliGovernor] at cap=10; would evict local_4f2625ea… (idle 2s)  for user spawn
2026-07-13 14:14:00 [info] [CliGovernor] at cap=10; would evict local_4f2625ea… (idle 3s)  for user spawn
2026-07-13 14:14:57 [info] [CliGovernor] at cap=10; would evict local_4f2625ea… (idle 61s) for user spawn

Right after those, effective goes 10 → 12 → 13, and sits at 13 for the next ~50 minutes. Same session ID named all three times, idle time climbing 2s → 61s, and the spawn appears to go through regardless. Could be the same unwired path as above, could be a separate thing. Posting it in case it's the latter.

What happened

Jul 13 15:27:11 debian systemd[1365]: app-com.anthropic.Claude-6668.scope: A process of this unit has been killed by the OOM killer.
Jul 13 15:27:38 debian systemd[1365]: app-com.anthropic.Claude-6668.scope: Failed with result 'oom-kill'.

That's the second of two today. The first one, during a session that peaked at 13 effective sessions, locked the machine up badly enough that I power-cycled it (last shows the unclean reboot at 15:10). Kernel PSI recorded 25.3 seconds of full memory stall inside a 21-minute uptime window. Honestly the freeze is worse than the crash from a user point of view — the machine is unusable for a long stretch before anything gets killed.

After the reboot the app came back, restored its sessions, and was OOM-killed again 16 minutes later.

On counts: the app writes every log line twice, so a raw grep -c doubles everything. Deduped, that's 218 distinct memory-pressure events today, 73 of them critical.

Things I checked that don't seem to be involved

  • The GPU busy-loop from #76307 — isHardwareAccelerationDisabled isn't in my claude_desktop_config.json and there are no GPU crash events in main.log.
  • Cowork's VM — never started here: [startVM] VM not supported (linux/x64), skipping and [Bundle:status] rootfs.img missing.
  • MCP sidecars — [LocalMcpServerManager] Closing all (0 servers), none configured.

Which would seem to leave the local agent session process trees, though I didn't measure their RSS directly.

What I can't tell you

kernel.dmesg_restrict=1 on this box and I didn't catch it live, so I have no kernel OOM report. No per-process RSS numbers, and I can't tell you which process the kernel actually picked. So I can neither confirm nor contradict the NetworkService victim-selection finding.

One difference worth noting: my main process logged a clean shutdown (all the onQuitCleanup handlers ran, then beforeQuit/willQuit) rather than the Network Service process gone … networking is broken until relaunch error. So the kill landed on a child process here and the app then exited gracefully. Whether that's a different victim or just a different path, I don't know.

Workarounds I'm using

earlyoom with -m 10 -s 10 and an --avoid regex covering Xorg/gnome-shell/sshd. Fixes nothing, but it means one process dies instead of the desktop seizing up.

A soft cap via user systemd. No root needed since the app runs under the user manager. The scope name has the PID in it, so it needs the trailing-dash drop-in directory, and both scope names:

~/.config/systemd/user/app-com.anthropic.Claude-.scope.d/memory.conf
~/.config/systemd/user/app-gnome-com.anthropic.Claude-.scope.d/memory.conf

[Scope]
MemoryHigh=16G

MemoryHigh rather than MemoryMax on purpose — throttle and reclaim, instead of trading a system-wide OOM for a cgroup OOM.

Neither of those touches the actual problem. Keeping the session count down is the only real lever, and by @yurukusa's rule of thumb that's around 4 on a 31 GB box. The app let me get to 13 without any pushback.

yurukusa · 1 month ago

@joshuarocksolid Useful data point, thanks for the logs. Worth underlining that your repro is on CLI 2.1.205 — roughly 18 releases after the original 2.1.187 — so neither root cause has been touched: the ~6.5 GB RSS per subagent (a leak, not a working set) and CliGovernor logging critical for 8 min while evicting 0.

One genuinely new thing landed after this thread, though it's a partial mitigation, not a fix. 2.1.212 added a per-session cap on subagent spawns (and a couple of related ceilings):

# 2.1.212+ only — bounds how many subagent processes a session can accumulate
export CLAUDE_CODE_MAX_SUBAGENTS_PER_SESSION=4      # default is 200
export CLAUDE_CODE_MAX_WEB_SEARCHES_PER_SESSION=50  # default is 200
# MCP calls >2 min now auto-background; tune/disable with CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS

The caveat matters here: this caps the count of spawns, not per-process memory. The default 200 is uselessly high for this failure mode — if each subagent balloons to ~6.5 GB, a 31 GB box is gone at ~4 concurrent, so the built-in ceiling only helps if you push it into single digits as a hard blast-radius limit. It does nothing for the leak itself, the ungated CliGovernor, or the OOM victim-selection (the killer still lands on the desktop's NetworkService via oom_score_adj 300, not the CLI process that actually grew).

To be precise about what's mine and what isn't: I run multi-subagent background loads continuously and I'm on 2.1.202 (pre-cap), so I can confirm the ceiling isn't present on either version in this thread and that setting CLAUDE_CODE_MAX_SUBAGENTS_PER_SESSION has no effect before 2.1.212. I have not reproduced the exact ~6.5 GB RSS figures myself — treat those as your and OP's measurement, not a claim of mine.

PiotrBlachnio · 1 month ago

Additional datapoint confirming this is still present in 2.1.219, on the same host profile as the original report (Linux, 31 GB RAM), with per-process RSS roughly 2.5x worse than the ~6.5 GB described here.

Two independent lockups ten days apart, 13 kernel OOM kills total. Both required a hard reset from the hypervisor console.

Environment

  • Claude Code CLI 2.1.219 (second event) and 2.1.209 (first event), Desktop remote build (~/.claude/remote/ccd-cli/<version>)
  • Ubuntu 24.04.4 LTS, kernel 6.8.0-136-generic, x86_64, QEMU/KVM guest
  • 31 GB RAM (32088 MiB), 8 GB swapfile, vm.swappiness=10
  • No cgroup memory limits, no systemd-oomd, no earlyoom at the time
  • Sessions driven over SSH, running in a user-<uid>.slice/session-N.scope

Observed

Unlike the fan-out-of-many-lean-processes pattern in #70523, here a small number of individual CLI processes each reach double-digit GB.

Top RSS at the moment of the final kill (2026-07-27 15:12:52), from the kernel task dump:

RSS_MB   PID     NAME
16917    28381   2.1.219
14061    27475   2.1.219
192      20618   2.1.219
181      19119   2.1.219
50       19250   2.1.219
26       507     multipathd
20       31162   node
17       1122    dockerd

Two processes accounted for ~31 GB of 31 GB. Everything else on the host combined was under 400 MB.

Kill lines across both events:

2026-07-17 08:18:58  Killed process 1422513 (2.1.209) total-vm:19136472kB, anon-rss:18144512kB
2026-07-17 08:20:11  Killed process 1423458 (2.1.209) total-vm:17039064kB, anon-rss:15838592kB
2026-07-17 09:14:06  Killed process 1451692 (2.1.209) total-vm:29623256kB, anon-rss:26527488kB
2026-07-17 09:17:47  Killed process 1452531 (2.1.209) total-vm:33818136kB, anon-rss:30842752kB
2026-07-17 09:24:30  Killed process 1454959 (2.1.209) total-vm:33818136kB, anon-rss:31019520kB
2026-07-17 15:15:35  Killed process 1594103 (2.1.209) total-vm:33817944kB, anon-rss:29573120kB
2026-07-27 11:45:11  Killed process  100746 (2.1.219) total-vm:21256712kB, anon-rss:13386592kB
2026-07-27 11:46:34  Killed process  103577 (2.1.219) total-vm:19159432kB, anon-rss:14618240kB
2026-07-27 11:47:46  Killed process  103831 (2.1.219) total-vm:23353992kB, anon-rss:11701332kB
2026-07-27 11:53:59  Killed process  113690 (2.1.219) total-vm:23354056kB, anon-rss:14530304kB
2026-07-27 15:09:36  Killed process   26091 (2.1.219) total-vm:19159176kB, anon-rss:11566848kB
2026-07-27 15:10:39  Killed process   26503 (2.1.219) total-vm:17061960kB, anon-rss:10538360kB
2026-07-27 15:12:52  Killed process   28381 (2.1.219) total-vm:21256840kB, anon-rss:17320576kB

Peak single-process anon-RSS was 31.0 GB on 2.1.209 and 17.3 GB on 2.1.219.

Notes that may help triage

1. It is not transcript or workload size. The largest session transcript on the day of the second event was 12 MB, and the entire ~/.claude/projects tree is 1.2 GB. 17 GB resident against 12 MB of session data is roughly a 1400x blowup.

2. Correlates with subagent fan-out. The session active during the second event had 24 subagent transcripts under <session-id>/subagents/. Consistent with #74035 and #74001, though here the growth lands on the session processes rather than staying distributed across lean children.

3. Kills repeat in bursts. Four kills in nine minutes (11:45, 11:46, 11:47, 11:53). The process is killed, restarts, and re-balloons within about a minute, so a single kill does not settle the host.

4. Swap exhaustion is what makes it a lockup rather than a crash. Free swap = 0kB in every one of the 13 dumps. The 8 GB swapfile gets churned for minutes before the kernel OOM killer fires, and the box is unreachable over SSH well before then. The last journal line before the reset was Under memory pressure, flushing caches.

5. Log-grepping gotcha worth documenting. Because the Desktop remote build is a version-named binary, the kernel records comm as the bare version string (2.1.219, 2.1.209). Grepping kernel logs for claude finds nothing, and a case-insensitive grep for oom collides with unrelated words. Anyone triaging this on a Desktop-remote host should grep for the version string or for Out of memory: Killed.

Mitigation that worked

Since the leak is upstream, we bounded the blast radius at the OS level. Posting in case it helps others whose hosts are getting hard-reset:

# /etc/systemd/system/user-.slice.d/50-memory-guard.conf
[Slice]
MemoryHigh=20G
MemoryMax=24G
MemorySwapMax=2G

MemorySwapMax is the important one. Capping slice swap prevents the multi-minute thrash spiral, so the cgroup OOM killer takes out the single offending process quickly while the rest of the host stays responsive. Verified against a throwaway systemd-run --scope -p MemoryMax=512M: the allocator was SIGKILLed at exactly the cap with Memory cgroup out of memory, and nothing else on the machine was affected.

earlyoom added as a machine-wide backstop (-m 6,4 -s 6,4).

Happy to supply full unabridged OOM dumps if useful.

Showing cached comments. Read the full discussion on GitHub ↗