Workflow tool: parallel() retry storm on external API rate-limit hits 1000-agent cap, burns 8.6M tokens with zero output

Summary

A Workflow tool script using parallel() to fan out ~14 concurrent agent() calls (7 items each, verifying citation metadata against arXiv/Crossref APIs) triggered a runaway retry storm. The workflow ran until it hit the hard 1000-agent cap, burned 8,641,786 subagent tokens over 226 seconds, and returned zero usable output — the run status was failed.

What happened

• Script had one parallel(chunks.map(...)) call (14 chunks, 7 agent() calls each = ~98 planned agent calls) plus a couple of single agent() calls in a later "Audit" phase.
• Each chunked agent was instructed to call fetch_json/arXiv MCP tools against external APIs (arXiv export.arxiv.org/api/query, Crossref api.crossref.org/works/<doi>) to verify ~94 citations.
• Many of these agent() calls failed with API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited.
• Instead of surfacing these as bounded per-item failures (as documented: "A thunk that throws ... resolves to null in the result array"), the task-notification shows the failure list growing to indices like parallel[1] through parallel[2760] — i.e. far beyond the ~14-98 calls the script actually issued.
• Execution continued until the global agent-call cap (1000) was hit:

``
Workflow agent() call cap reached (1000). This usually means a loop using
budget.remaining() never terminates because no token budget was set —
remaining() returns Infinity when budget.total is null. Add a hard
iteration cap to the loop, or pass a token budget.
`
Note: the script contained **no loop** and **no use of budget.remaining()`** at all — this generic diagnostic message does not match the actual script structure, which makes root-causing harder for the caller.

• The terminal error surfaced to the caller was:

``
Error: items.map is not a function. (In 'items.map(i => ({ id: i.id,
authors: i.authors, title: i.title, rel: i.rel }))', 'items.map' is
undefined)
at <anonymous> (workflow.js:83:43)
`
items was const items = args at the top of the script (the args value passed into the Workflow tool call, a 94-element JSON array). This should not have been able to become non-array given const binding, which suggests either (a) the whole top-level script was silently re-executed on retry with args` not correctly threaded through, or (b) some other internal state corruption during the retry storm.

Impact

• 8.64M tokens consumed for zero output — no partial results were salvageable from the failed run (task notification only contains a giant failures list + usage stats, no partial return value).
• No visible rate-limit backoff/circuit-breaker: once external API calls started failing with 429-style responses, the system kept retrying at what looks like the same concurrency rather than backing off or aborting after a small number of consecutive failures.
• The diagnostic message shown to the user/caller (about budget.remaining() loops) is misleading when the actual script has no such loop, making it hard for a caller to self-diagnose.

Expected behavior

1. parallel()/agent() retries on transient tool/API errors (rate limiting) should have an exponential backoff and a small bounded retry count per logical call site, not an unbounded retry that can spiral into hundreds/thousands of synthetic call indices.
2. If a parallel() call's item count is known (here: 14, from chunks.map), the failure/retry indices surfaced to the caller should not exceed that count by orders of magnitude (parallel[2760] for a 14-item parallel() call is a strong signal something is re-invoking the whole call repeatedly rather than retrying individual items).
3. There should be a global "abort workflow" circuit breaker when a high fraction of calls are failing with the same rate-limit error, well before the 1000-agent hard cap, to avoid burning tokens on a run that cannot succeed.
4. The cap-reached diagnostic message should reflect what's actually observable from the script (e.g., "N parallel/agent calls retried at site X due to repeated rate-limit errors") rather than a generic budget-loop message that doesn't apply when no budget.remaining() usage exists in the script.
5. Ideally, a failed Workflow run should still return whatever partial/successful agent() results it did manage to collect before the abort, rather than nothing.

Environment

• Claude Code (harness) with the Workflow tool, agent() calls with schema option, MCP tools mcp__arxiv__get_paper / mcp__fetch__fetch_json used inside subagents.
• Observed usage on the failed task notification: agent_count: 1000, subagent_tokens: 8641786, tool_uses: 216, duration_ms: 226572.

Happy to share the exact workflow script text if useful for repro (it's ~100 lines, verifying a 94-item citation list against arXiv/Crossref).