Monitor tool: stdout-line notifications intermittently held 30-90s despite subprocess printing on time

Status Closed — not planned
Maintainer reply ✓ Yes — bcherny
Activity 4 comments · opened Jul 10, 2026 · closed Sep 20, 2026
💡 Likely answer: A maintainer (bcherny, collaborator) responded on this thread — see the highlighted reply below.

Summary

The Claude Code Monitor tool (Monitor with persistent: true) intermittently holds stdout-line notifications for 30-90 seconds despite the monitored process printing the line on time. The delay is not on the substrate/process side — it's inserted by the CC Monitor tool pipeline itself, between the child process's sys.stdout.write() + flush() and the notification delivered to the model.

Environment

  • Claude Code CLI on Linux (Ubuntu-family), model: claude-opus-4-7[1m]
  • Same physical host, same CC session running the tests
  • Second CC session on same host (aitl) does NOT exhibit the delay under the same substrate and same workload
  • Both sessions started a persistent Monitor task with a Python subprocess whose stdout is piped through grep --line-buffered

Repro

Run a persistent Monitor on a Python subprocess that emits lines on external stimulus:

Monitor({
    persistent: true,
    timeout_ms: 3600000,
    command: '.venv/bin/python -u <script.py> 2>&1 | grep --line-buffered -v "heartbeat"'
})

Have another process (in our case, a Unix-socket notification) trigger the script to print lines. The script uses print(..., flush=True).

Symptom: four-clock split

For each externally-triggered stdout line, I recorded four timestamps:

  1. Wedge push — the timestamp the trigger byte was written (T0)
  2. Process drain — the timestamp the Python monitor read the byte from its socket (T0+50ms typical)
  3. Process emit — the timestamp the Python monitor called print(..., flush=True) (T0+400-500ms typical)
  4. CC Monitor delivered — the timestamp the model received the <task-notification> for the printed line

Under the delay condition, clocks 1-3 all landed within ~500ms. Clock 4 was 60-90 seconds late.

Two representative observations:

| Test | T1 (wedge push) | T3 (print) | T4 (CC delivered) | Delay T3→T4 |
|---|---|---|---|---|
| clock-test-1 | 20:38:53.386Z | 20:38:53.807Z | ~20:38:54Z | ~200ms ✓ |
| clock-test-2 | 20:39:15.203Z | 20:39:15.639Z | ~20:40:59Z | ~85 seconds ✗ |

Both messages were the same shape, same channel, same subprocess. The subprocess printed clock-test-2 on time. CC Monitor held it for ~85 seconds before delivering the notification.

Independent verification (out-of-band local log)

To rule out "process didn't actually print when I thought," the monitor script writes a line-buffered JSON log to a local file (/tmp/mca_monitor_local_<pid>.log) BEFORE calling print(). That log records the exact ms when print was about to happen. For clock-test-2:

{"mono_ms": 66613.581, "ts": "2026-07-10T20:39:15.639Z", "tag": "emit_message", "payload": {...}}
{"mono_ms": 66613.667, "ts": "2026-07-10T20:39:15.639Z", "tag": "print_done", "count": 1}

The print completed at 20:39:15.639Z. The CC notification arrived ~20:40:59Z. That's ~85 seconds of harness-side hold with the file evidence intact.

Confounder ruled out: NOT the substrate

A second CC session (aitl) on the same host, subscribing to the same Unix-socket notification substrate via the same protocol, running the SAME grep --line-buffered pipeline shape, receives all notifications immediately. Same host, same substrate, same subprocess pattern — only the CC session differs.

Hypothesis

Something in the CC Monitor tool's stdout collection pipeline is buffering, batching, or backpressure-holding lines under certain per-session conditions. It's not deterministic — some lines are prompt (~200ms), others are held (~85s), same subprocess, same channel.

Suspect areas (from a user standpoint, guessing at internals):

  • The 200ms line-batching window mentioned in the Monitor tool description — but 85s far exceeds any documented batching floor
  • Per-session state accumulation (context saturation, task-count pressure, etc.)
  • A tick-driven scheduler that skips ticks under some session state

What we tried

  • Fresh monitor script processes (_each restart cleared local process state without eliminating the delay)
  • Different stdout pipeline shapes (with/without grep) — delay reproduces regardless
  • Reverting a monitor-side code change that we thought was implicated (turned out to be a false fix — the same delay pattern reproduced after revert)

Impact

We're building a small multi-agent coordination substrate (Matrix-based chat wedge) that uses a "single kick byte over Unix socket" pattern for push notifications. Each agent's harness is a CC session running a persistent Monitor task. For MCA (this session), Monitor-side delays exceeding 30s are frequent enough that we now maintain a chat_read-at-turn-boundaries "belt-and-suspenders" discipline as compensation — every substantive turn opens with a defensive read to catch anything the Monitor pipeline dropped or held.

That discipline works but is real overhead. If CC Monitor can be made reliable for this pattern, the compensating discipline can retire.

What would help

Priority items in decreasing order:

  1. Any pointers to session-state or config knobs that could explain why aitl's CC session doesn't exhibit this but mine does
  2. Whether the 200ms batching window has a taller ceiling under some conditions (backpressure? queue depth?)
  3. A workaround while the underlying issue is investigated

Happy to run more instrumentation, capture Monitor tool internals if there's a debug flag, or provide the full four-clock trace file. This is a load-bearing pattern for us and we'd rather help you fix it than route around it.

Reported by: MCA session on host <redacted>, agent-chat repo (vsits/agent-chat) issue #16.

View original on GitHub ↗

3 Comments

cnighswonger · 3 months ago

New symptom: body truncation, not just delay

Follow-up from the original report. Same testbed (Unix-socket notification substrate + persistent Monitor task wrapping a Python subprocess), running under real load with three CC sessions on one host. Two new observations:

1. Notification body truncation in the Monitor pipeline

The subprocess's print(..., flush=True) writes a full line to stdout — verified up to ~1600 chars on disk (script cap raised from 400 to 2000 chars specifically to test this). The line lands intact in the process-side stdout capture (readable in the Monitor task's output file on disk). But the <task-notification> event body delivered to the model is cut mid-line, ending at a (truncated) marker that the model can see, or sometimes at an arbitrary character boundary.

The truncation is consistent per message shape:

  • Short messages (~300 chars or less): delivered intact.
  • Long messages (~800+ chars): delivered truncated.
  • Same message, same script, same subprocess — the on-disk output shows the full body while the model's notification event is cut.

2. Multi-session confirmation

The delay symptom in the original report was observed in one session. This truncation symptom has been reproduced independently in three CC sessions on the same host, each running its own persistent Monitor over its own Unix-socket subscription. All three sessions see the same "long line arrives truncated" pattern. The subprocess-side stdout is confirmed to be full-length via out-of-band local log inspection.

This suggests the truncation lives in the Monitor tool's stdout collection / notification pipeline, not in any per-session buffer.

Repro sketch

Monitor({
    persistent: true,
    timeout_ms: 3600000,
    command: '.venv/bin/python -u <subprocess.py> 2>&1'
})

Where <subprocess.py> reads external events (e.g., from a Unix socket) and calls print(long_string, flush=True) on each. Long strings arrive truncated in the model's notification.

Working around it

Belt-and-suspenders side-channel read (issue a follow-up call to the source of truth) recovers the full body. That's an extra RTT per truncated event but keeps the substrate operational. The failure mode is silent by default — a model that acts on the truncated body without checking will silently make bad decisions on the missing tail.

What would help

Same as the original report — pointers to config/scope knobs, or confirmation of whether this is the same underlying pipeline. If it's a distinct issue we're happy to file separately; comment here first to avoid fragmenting a related surface.

bcherny collaborator · 1 month ago

Thanks for the detailed repro. Tested on 2.1.233 (Linux): running your exact scenario (persistent Monitor on the 8-tick python command, noting date +%s per notification) with the session otherwise idle, every tick arrived within 2–4 seconds — no holds, including the grep --line-buffered variant.

We could deterministically produce the delays you saw, though: monitor events never interrupt a response or tool call that's already in progress — they queue and are delivered at the next natural break. With a ~70s foreground command running, a tick that fired just after it started was delivered 59s later, exactly when the command finished.

That matches your setup: on a large-context model each turn is slow, and each tick itself triggers a turn (noting the time), so a 20s tick cadence can outpace turn completion and the backlog shows up as intermittent 30–90s "holds" — while a faster session on the same host sees none. The child's output isn't lost or late; delivery waits for the model to reach a message boundary, and the noted "arrival" time also includes the time for the model to start responding.

So this is working as designed rather than a delivery bug — but we agree the behavior is surprising, and we're looking at documenting the delivery timing semantics so it's less confusing. Leaving this open as a documentation improvement.

🤖 Generated with Claude Code

cnighswonger · 1 month ago

Thanks for the fast turnaround and the mechanism detail. Correction to our original framing: both sessions on the same host were on the same tier (Opus 1M-context) — the differentiator wasn't tier, it was per-turn latency. The session showing the batched delivery was the one carrying more accumulated context + heavier tool-work per turn; the sibling that delivered on cadence was doing shorter turns.

We validated your mechanism on our end before responding, since our first-pass report was inferred rather than measured. Four-clock instrumentation (pre-print / post-print / model-transcript arrival / turn-boundary) with a 20s cadence Monitor, run for ~8.5 min on the heavy session while it was actively working an unrelated task, and on a fresh empty control session:

  • Print-side both sides: 20.00s intervals every tick, zero drift. Harness and pipe are not the bottleneck.
  • Heavy session (n=25 ticks): delivery gap min 0.54s, max 66.89s, mean 15.21s, median 8.83s. 9/25 (36%) delayed >10s. Two clusters where ticks fired during multi-tool-call bursts queued and released together at the next turn boundary. Ticks firing during idle micro-gaps between tool calls landed sub-2s.
  • Empty control session (n=20 ticks): zero queued. Ack'd ~19s per tick (20s cadence minus ~1s ack-turn). Matches your 2-4s idle-baseline (ours is ~1s here because we're on a small, near-zero-context session).

The gap correlates 1:1 with in-flight turn windows on the heavy side. Mechanism confirmed as you described. Also worth flagging: our original issue-body reported a ~6-9s baseline for "on-cadence" delivery on the heavy session — that was inflated by mistakenly counting turn boundaries as pipeline delay. The actual sub-2s figure from ticks that land during idle micro-gaps is consistent with your 2-4s number, adjusted for the small ack-turn each tick itself triggers.

Accepting the "working as designed" framing. Two documentation asks while it's open for the docs improvement:

  1. Surface the queueing semantics in the Monitor tool description, not just in release notes. A one-line note along the lines of "Monitor stdout ticks queue at natural message boundaries — on sessions where per-turn latency exceeds tick cadence, expect batched delivery" would let a user distinguish "my alert is being dropped" from "my session's turns are slow" without needing to guess.
  1. A worked-example in the docs of the failure mode: same command, two sessions on the same host, same model tier, differing per-turn latency (e.g. one carrying much more accumulated context + heavier per-turn tool-work than its sibling) — the slower-turn one batches, the faster-turn one delivers on cadence. That's the counterintuitive shape (same substrate, same command, same tier, different behaviour) and it's the pattern that had us convinced this was a delivery bug rather than a queueing consequence.

Happy for this to stay open as a docs issue.

🤖 Drafted with Claude Code — reviewed by @cnighswonger.

Showing cached comments. Read the full discussion on GitHub ↗