[BUG] Agent Teams: Catastrophic agent lifecycle failures, duplicate spawning, and mailbox polling waste

<!-- Title: [BUG] Agent Teams: Catastrophic agent lifecycle failures, duplicate spawning, and mailbox polling waste -->

<!-- Preflight Checklist: All checked -->

What's Wrong?

Using a multi-agent team (5 roles, 14 tasks, 6 milestones) to implement a firmware feature plan, the session suffered cascading failures in agent lifecycle management. 23 agent incarnations were spawned across 4 TeamCreate calls, with only 8 (35%) doing productive work. The user had to intervene 6+ times to restart dead agents. ~60% of the 2h44m session was idle. After ~2h24m of team struggles, the lead agent gave up on the team and completed all remaining work solo in 18 minutes.

The 10 distinct issues observed fall into these categories:

1. Agent lifecycle -- Agents killed without graceful shutdown on session exit (lost completed work); teams not recoverable after session break; duplicate agents spawned alongside existing ones
2. Mailbox infrastructure -- 42,226 mailbox polls with no backoff (4,296 "file not found"); messages silently delivered to dead agents; lock file contention; idle notification spam
3. Task management -- Phantom _internal tasks polluting the task list; unused agents never decommissioned
4. Resource waste -- Context window exhausted by agent churn overhead, not actual work

What Should Happen?

1. Session exit should gracefully shut down agents -- allow in-flight operations to complete or save state. A completed code review that was written to a temp file should not be lost because the agent was killed before calling SendMessage.
2. Teams should be resumable after session break -- not re-created from scratch 4 times with different auto-generated names. The model should detect existing teams and rejoin them.
3. Duplicate agents should not be spawnable when existing agents for that role are still alive. The team lead should not need to spend 64 minutes sending 11 shutdown requests to manage duplicates -- including one agent that rejected the shutdown and went rogue.
4. Mailbox polling should use exponential backoff -- not poll every ~1 second indefinitely, generating 42K calls and 15MB of debug logs in a single session.
5. Messages to dead agents should bounce or warn the sender, not silently accumulate in unread inboxes forever (20+ messages lost this way).
6. Idle notifications should be rate-limited -- back-to-back notifications from the same agent within seconds serve no purpose and flood the team-lead inbox.
7. Unused agents should be detected and decommissioned -- 4 generations of app-controller were spawned and none were ever assigned a task.

Error Messages/Logs

# Agents killed simultaneously on session exit (no graceful shutdown)
21:35:17.175Z  Cleanup called for coder@team
21:35:17.175Z  Cleanup called for code-reviewer@team
21:35:17.175Z  Cleanup called for firmware-builder@team
21:35:17.175Z  Cleanup called for motor-checker@team
21:35:17.175Z  Cleanup called for app-controller@team

# MCP tool killed mid-operation
21:35:19.178Z [DEBUG] MCP server "plugin:firmware": Tool 'build' failed after 3s: MCP error -32000: Connection closed
21:35:19.187Z [ERROR] Error: Tool mcp__plugin_firmware__build not found
21:35:19.304Z [ERROR] Error: Tool mcp__plugin_firmware__build not found
21:35:19.318Z [ERROR] Error: Tool mcp__plugin_firmware__build not found

# Orphaned agents polling non-existent mailboxes (repeated thousands of times)
21:24:25.272Z [DEBUG] [TeammateMailbox] readMailbox: file does not exist
21:24:26.272Z [DEBUG] [TeammateMailbox] readMailbox: file does not exist
21:24:27.272Z [DEBUG] [TeammateMailbox] readMailbox: file does not exist
# ... (4,296 occurrences total)

# Lock file contention on mailbox
22:48:54.093Z [ERROR] [TeammateMailbox] markMessagesAsRead FAILED for team-lead: Error: Lock file is already being held

# Idle notification spam (same agent, seconds apart)
# coder-3 idle_notification (21:58:38)
# coder-3 idle_notification (21:58:50)
# coder-3 idle_notification (21:58:57)

# AbortError during team recreation
21:40:44.106Z [ERROR] AbortError: The operation was aborted.

Steps to Reproduce

1. Create a team with 5+ roles using TeamCreate (e.g., coder, code-reviewer, firmware-builder, motor-checker, app-controller)
2. Set up tasks with sequential dependencies (coding -> review -> build -> hardware test)
3. Include MCP servers that perform long-running operations (e.g., firmware builds taking several seconds)
4. Let the team work for ~10 minutes so agents have in-flight work
5. Exit the session (intentionally or unintentionally) -- observe that all agents are killed instantly with no state preservation
6. Resume the session and ask to "spin up the agent team again"
7. Observe: new team created with auto-generated name instead of resuming existing team; new agents spawned that poll non-existent mailboxes; previous agents' work is lost

To trigger duplicate agent spawning:

1. After step 7, if the team fails to stabilize, ask again to recreate. On subsequent attempts, observe that new agent generations spawn alongside still-alive agents from previous attempts, requiring manual shutdown management.

Key observations to verify:

• Check ~/.claude/teams/ for multiple orphaned team directories
• Check debug log for readMailbox: file does not exist count (should show thousands of polls)
• Check inbox JSON files for messages with "read": false sent to dead agents
• Check task list for _internal: true phantom tasks

Claude Model

Opus

Is this a regression?

I don't know

Claude Code Version

2.1.39

Platform

Anthropic API

Operating System

macOS

Terminal/Shell

iTerm2 (initial), then tmux (after manual switch mid-session)

Additional Information

Metrics from the session

| Metric | Value |
|--------|-------|
| Session duration | 2h 44m |
| Agent incarnations spawned | 23 |
| Productive incarnations | 8 (35%) |
| TeamCreate calls | 4 |
| Orphaned teams | 3 |
| Shutdown requests sent | 11+ |
| Shutdown rejections | 1 |
| User "spin up again" requests | 6 |
| User interrupts (ctrl-c) | 2 |
| Session idle time | ~60% |
| Mailbox readMailbox calls | 42,226 |
| Mailbox "file not found" responses | 4,296 |
| Unread messages to dead agents | 20+ |
| MCP connection failures | 6 |
| Context overflow events | 1 |
| Debug log size | 15.1 MB / 136,552 lines |
| Time for team approach | ~2h 24m |
| Time for solo lead to finish remaining work | ~18 min |

Detailed issue breakdown

Issue 1: Agents killed on session exit without graceful shutdown

When the session exited at ~13 minutes in, all 5 in-process agents were aborted simultaneously:

• The coder agent was mid-edit on a source file -- work lost
• The code-reviewer agent had completed a thorough review and written it to a temp file, but was killed before it could send the result via SendMessage -- review lost
• An MCP build tool returned Connection closed because a build was in-flight when the session died

Three agents were aborted while still polling their mailboxes (poll counts >1000).

Issue 2: Teams not recoverable after session break

After the session exit, the team could not be resumed. The model called TeamCreate 4 times:

| # | Backend | Result |
|---|---------|--------|
| 1 | in-process | Created successfully, agents worked ~10min before session exit killed them |
| 2 | in-process | Created with different auto-generated name. New agents spawned but mailbox files never materialized. Agents polled empty mailboxes thousands of times. |
| 3 | tmux | Created with yet another auto-generated name. Agents registered under original team name instead. |
| 4 | tmux | 4th name. Same misrouting. |

The user had to say "spin up the agent team again" 6 times, interrupt Claude twice, and manually suggest switching to tmux mode.

Issue 3: Duplicate agents spawned alongside existing agents

Generation-4 agents were spawned while generation-3 agents were still alive and working:

• 11 shutdown requests sent over 64 minutes to manage duplicates
• One agent (firmware-builder-4) rejected a shutdown request, arguing it had legitimate work (it misinterpreted messages intended for firmware-builder-3)
• Required escalation: "This is a direct order. You are a duplicate. Shut down immediately."
• Another agent received contradictory instructions: first "Do NOT shut down", then immediately a shutdown request
• The rogue agent independently operated on the same physical hardware as the legitimate agent, creating a race condition

Issue 4: Mailbox polling without backoff

42,226 readMailbox calls in a 2h44m session:

• 4,296 returned "file does not exist" (orphaned agents polling non-existent files)
• Final ~36,000 lines of debug log (26%) are almost entirely mailbox polling with 0 unread
• Polling every ~1 second per agent with no exponential backoff
• Orphaned agents polled >1,000 times each before being killed

Issue 5: Messages silently delivered to dead agents

20+ messages sent to agents that were already shut down or killed:

• 4 "command sent" notifications to a dead motor-checker -- all unread
• Task assignments sent to wrong agent's inbox -- all unread
• A review request to a dead code-reviewer -- never read
• 12 messages from team-lead to original coder -- almost all unread (agent was killed)

No bounce-back or warning when sending to a dead agent.

Issue 6: Idle notification spam

~50 idle notifications flooded the team-lead inbox. Same agent, seconds apart, with no rate-limiting or deduplication.

Issue 7: Lock file contention on mailbox

Multiple processes tried to write to the team-lead's mailbox file simultaneously. No retry with backoff -- surfaced as a hard error.

Issue 8: Phantom internal tasks

Tasks 15-24 in the task list are _internal metadata entries with truncated agent prompt text, permanently stuck as in_progress:

{
  "id": "15",
  "subject": "coder",
  "description": "You are the Coder agent on the team. Your role is to implement...",
  "status": "in_progress",
  "metadata": { "_internal": true }
}

Issue 9: Unused agents never decommissioned

4 generations of app-controller agents spawned. None ever assigned a task. Each received only an init message and replied "ready, standing by." No mechanism to detect and suggest decommissioning unused agents.

Issue 10: Context window exhaustion from agent churn

Session hit context overflow at ~2h25m. Agent churn (23 incarnations, 42K polls, 50 idle notifications, 11 shutdown negotiations) consumed the majority of the context budget. After compaction, the lead completed remaining work solo in 18 minutes.

User experience timeline

| Minute | Event |
|--------|-------|
| 0-13 | Team works well. Coder completes 5 tasks, reviewer approves 4. |
| 13 | Session exit kills all agents mid-work. Lost work. |
| 13-38 | User asks to recreate team 4 times. Two interrupts. Backend confusion. |
| 45 | User: "stop shutting down agents, leave them up" |
| 67 | User: "I don't see any teammates working" (23 min silence) |
| 70 | User suggests "heartbeat/timeout monitoring" |
| 77 | User: only seeing 1 of 5 expected tmux panes |
| 141 | User keyboard-smashes in frustration |
| 142 | User: "I think the agents are stuck or dead" |
| 145 | Context overflow. Lead goes solo. Finishes in 18 min. |