[FEATURE] Pause-and-resume background subagents on recoverable failures (usage limits, transient API errors) instead of terminating them

Status Open

Maintainer reply None cached

Activity 4 comments · opened Jul 16, 2026

Summary

When a background subagent hits a recoverable condition — usage/credit/session-limit exhaustion, or a transient API error ("Server error mid-response") — it currently terminates with a failed status. Its in-progress work usually survives on disk (files written, sometimes committed in a git worktree), but the agent process and its context are gone. Please pause-and-checkpoint these agents (auto-resuming when the blocking condition clears) instead of killing them.

Current behavior

A background subagent that hits one of these ends its task with a failed notification, e.g.:

Agent terminated early due to an API error: API Error: Server error mid-response. The response above may be incomplete.
Agent terminated early due to an API error: You've hit your session limit · resets 6:30pm (Europe/Berlin)
You're out of usage credits. Run /usage-credits to keep using ... or /model to switch models.

In every case the agent's last partial output is preserved in the notification, and the work-in-progress is intact (an uncommitted or committed change in the worktree it was operating in). But recovery is entirely on the orchestrator:

detect the death,
footprint-check the worktree to see how far the work got,
hand-craft a SendMessage resume, and
often finish the last step by hand anyway — because a SendMessage-resume re-enters the still-exhausted limit and dies again.

SendMessage/--resume from transcript already exists and works, so the mechanism is there. The gap is that (a) the death is presented as a terminal failed rather than a recoverable pause, and (b) resume is a manual, limit-unaware path that can immediately re-die.

Motivation

In one multi-agent orchestration session (a "conductor" driving a sequence of bounded implementer subagents through spec → build → Codex-review → merge), three separate subagent deaths occurred within a few hours — one on "server error mid-response," two on usage/session-limit exhaustion. Each time the work survived but the agent died mid-task, and the orchestrator spent tokens + wall-clock detecting, footprint-checking, and recovering — twice finishing the work by hand after a resume re-hit the same limit.

This scales the wrong way: the longer the run and the more agents in flight, the more likely one hits a limit — so multi-agent orchestration (workflows, fan-out, conductor patterns) becomes least reliable exactly when it is doing the most work. It also risks duplicate work, because a dead agent whose output survived is hard to distinguish from one that never produced anything (the whole "footprint-check before trusting a completion" discipline exists partly to handle this).

Proposed behavior

Classify the failure. Distinguish recoverable (usage/credit/session limit; transient 5xx / "server error mid-response") from terminal (malformed request, auth/permission denied, genuinely unrecoverable).

On a recoverable failure, pause + checkpoint instead of terminating. Surface a distinct status — paused (reason), ideally with the reset time the error already carries (resets 6:30pm) — rather than failed. The transcript is already the checkpoint; keep it.

Auto-resume when the condition clears (the reset time is reached / credits restored / the transient error subsides), or offer a single-action resume — without re-entering a still-exhausted limit. This is the key difference from today's manual SendMessage-resume, which re-dies because the limit is still in effect.

Applies to both top-level and background subagents; reuse the existing transcript-resume mechanism as the resume path (make it the default for these classes, not a manual fallback).

Why it matters

Turns a fragile mid-task death into a clean, resumable checkpoint — no lost context, no orchestrator hand-holding, no duplicate-work risk.
Makes limit exhaustion a pause, not a failure — which matches reality: usage/session limits are transient and reset on a known schedule (the error message literally states the reset time).
Directly improves multi-agent orchestration reliability, which degrades today precisely as a run gets longer or wider.

Related issues (adjacent, not duplicates)

#74196 [FEATURE] Quick task progress recap after hitting usage limit — same trigger (usage limit), different ask (recap vs pause/resume).
#77510 Resumed background subagent stays marked Completed; resume message renders as raw JSON — a resume-path bug.
#77146 [BUG] Subagent SendMessage-resume 400s: deferred tool referenced in transcript not rehydrated on resume — a resume-path bug that would need fixing for auto-resume to be reliable.
#74953 Allow --resume from any directory — adjacent resume-ergonomics.

Notes

The resume-from-transcript capability already exists (SendMessage to a completed/dead agent "resumes from transcript"), so much of this is a default-behavior + status-surfacing + limit-aware scheduling change rather than net-new machinery.
The recoverable/terminal split matters so genuinely-broken agents still fail fast rather than hanging in paused forever.
Auto-resume must be limit-aware: waiting for the reset before re-entering, so a paused agent doesn't wake straight back into the same exhausted quota (today's manual resume does exactly that).

View original on GitHub ↗

3 Comments

makslanies · 1 month ago

This is exactly the scenario VOLY's billing fallback chain was built for.

When claude-code hits a recoverable failure — usage limit, session exhaustion, transient API error — VOLY automatically routes to the next executor in the chain (cursor → deepseek → opencode → zen) without dropping the task. The work is not lost; the next executor picks up from the last committed state.

For the "three subagent deaths in one session" case you describe: VOLY's hybrid mode runs each role (developer, tester, devops) through separate executors. If one dies mid-task, only that role's executor is replaced — the orchestrator and other roles continue.

https://github.com/voly-codes/voly/blob/main/docs/backend/executors.md#billing-fallback-chain

Do your failures happen mid-task or at session start?

MisterTuur · 1 month ago

New data point: expired OAuth token — same class, and a strictly clearer case

Hit this again today with a different trigger:

Agent terminated early due to an API error: Failed to authenticate.
API Error: 401 OAuth access token has expired. Re-authenticate to continue.

Same failure shape as the ones already listed, but I think token expiry is a stronger argument for pause-and-checkpoint than the usage-limit cases, for two reasons:

It is unambiguously recoverable, and recovery is a known, bounded user action. With a usage limit there is at least a debatable question of when it clears. With an expired token there is no ambiguity: the session needs re-authentication, that is a thing the user does, and afterwards the agent could continue exactly where it stopped. Terminating is a pure loss with no upside.
Expiry is predictable in a way a rate limit is not. The client knows the token's lifetime. An agent that is about to be killed by expiry could be checkpointed proactively rather than discovered dead afterwards.

The simultaneity multiplier

The part that hurt most was not that an agent died — it was that all four running agents died at the same instant, because they share one credential. Usage limits at least tend to hit agents one at a time as each crosses the threshold. A credential event is global: it takes out the entire fan-out at once.

That means the cost of this failure mode scales with the parallelism the feature is designed to encourage. In this session it was four bounded implementer agents at roughly 300–400k tokens each, all mid-task:

one was three lines from committing (its own report said tests were green and it was about to commit and push),
one was mid-way through a corpus sweep,
one was mid-way through applying an arbitration verdict,
one had just received new instructions and had not started.

All the files survived — I could commit the work-in-progress from the worktrees afterwards. What is gone is every agent's reasoning context: why it chose the mechanism it chose, what it had already ruled out, which parts of its enumeration were complete. Reconstructing that from the diff is strictly worse than the agent simply continuing, and in practice it means re-deriving decisions that were already made and paid for.

Prior art: this is solved elsewhere

Codex's agents handle the analogous condition the way you would expect: when they run out of quota they say so in-session — a plain message that the limit is reached — and then stop cleanly, leaving the session resumable. No failed status, no silent teardown of context, no orchestrator-side forensics. The orchestrator reads the message, waits for the reset, and continues.

That is the behaviour I would expect from software in 2026, and the gap here is not capability — SendMessage/transcript-resume already exists, as the issue notes. It is that a recoverable, self-announcing condition is being surfaced as a terminal failure.

What would have been enough here

Not even full auto-resume. Any of these would have avoided most of the loss:

surface the condition as a pause with a distinct status rather than failed, so an orchestrator does not have to pattern-match error strings to tell "died recoverably" from "died for real";
checkpoint before dying on a known-recoverable error class, so the resume re-enters with context rather than with a transcript replay;
for credential errors specifically, hold the fan-out instead of letting N agents each independently discover the same dead credential.