Workflow tool keeps dispatching queued agents after a rate-limit stop (no circuit breaker)
Summary
A Workflow tool phase that enqueues many agents at once has no rate-limit circuit breaker. Once the account/session hits its rate limit, the dispatcher keeps draining the rest of the phase's queue instead of pausing — every remaining queued agent is dispatched, instantly fails on the same limit, and is recorded as error rather than skipped. One exhausted limit produces one StopFailure per remaining queued agent instead of one signal for the whole phase.
Environment
- Claude Code 2.1.268, macOS
- Workflow tool, a phase that fans out one agent per item over a list (~650 agents queued in the phase in the observed run)
What happened
- A workflow phase enqueued a large batch of agents (all sharing the same
queuedAttimestamp in the workflow's persisted state — a single simultaneous enqueue, not a trickle). - Partway through dispatching that batch, the account hit its session rate limit (
StopFailurewitherror: "rate_limit",last_assistant_messagecontaining the reset time). - The dispatcher did not stop. It continued pulling the next queued agent, dispatching it, and immediately failing it on the same exhausted limit — repeated for 160 consecutive agents inside a single minute (near-zero gaps between consecutive failures; the harness-level notification log shows ~144 of 160 gaps under 1 second).
- Of those 160 failed agents: 148 had consumed 0 tokens (never got far enough to make a model call before being marked failed) and 12 had already been mid-flight doing real tool work (multiple
Bashtool calls each) and were cut off, discarding that partial work. - ~95% of the resulting failure records carried an
agent_id/agent_typepair (i.e., these are per-agentStopFailures, not one top-level session failure), confirming this is N independent dispatch-and-fail cycles rather than a single reported error. - Each remaining agent's terminal state was recorded as
error(with the literal rate-limit message as itserrorfield), notskipped— indistinguishable in the workflow's own state from a real per-agent failure.
Where the state lives
The per-workflow-run JSON at ~/.claude/projects/<project>/<session>/workflows/wf_*.json records this exactly: a workflowProgress array of per-agent records with label (e.g. verify:<finding>), state, queuedAt, startedAt, tokens, toolCalls, error. In the observed run, ~650 entries for one phase share an identical queuedAt, and 160 of them transition straight to state: "error" with the rate-limit message, back-to-back, within about a minute of the first one.
Why this matters
- No circuit breaker: after the first
StopFailure{error:"rate_limit"}in a phase, the dispatcher should stop issuing new dispatches from that phase's queue rather than draining it into guaranteed-instant failures. - Wrong terminal state: an agent that never got to run because the account was already rate-limited is not the same outcome as an agent whose own work failed. Recording it as
skipped(vserror) would let any consumer of workflow state (retry logic, summaries, dashboards) tell "didn't run" from "ran and failed" apart. - Redundant signaling: the reset time is identical across all N failures (it's the same limit); surfacing it once per phase (or once per workflow run) instead of once per queued agent would make the failure legible instead of a wall of near-duplicate lines.
- Not visible to
PreToolUseguards: fan-out guards that gate on top-levelAgent/Tasktool calls cannot see this, because the additional dispatches happen inside the Workflow tool's own internal queue — they are not separate top-level tool invocations the host application's hooks observe.
Requested behavior
- On the first
StopFailure{error:"rate_limit"}(or any account-level rate-limit signal) observed while dispatching a phase's queue, stop dispatching new agents from that queue immediately. - Mark the remaining not-yet-started queued agents
skipped(with a reference to the triggering limit), noterror. - Surface the rate-limit / reset-time information once per phase (or once per run) rather than once per skipped agent.
Related
#91419 covers a related but distinct issue — StopFailure misattribution for internal helper forks and background-agent-death cascades re-waking a parked main loop. This report is about the Workflow tool's own phase-queue dispatcher continuing to drain and fail queued (not-yet-started) agents after the same limit is already known to be exhausted.