Feature Request: Background Agent Execution (Task tool async support)

Feature Request: Background Agent Execution

Problem Statement

The Task tool currently executes agents synchronously, blocking the orchestrator until all spawned agents complete their work. This creates significant limitations when working with:

• Large codebase analysis: 10k+ file repositories requiring 1-2 hours of comprehensive scanning
• Security audits: Deep vulnerability analysis across dependencies and code patterns (30-60 minutes)
• Performance profiling: Memory leak detection, bottleneck analysis, and optimization recommendations (45-90 minutes)
• Parallel refactoring: Multiple independent worktrees requiring simultaneous agent coordination

While the Bash tool provides run_in_background=true for async shell execution with BashOutput and KillShell for monitoring/control, no equivalent exists for agent tasks. This forces users to either:

1. Wait synchronously for all agents to complete before continuing work
2. Use workarounds like claude -p in separate terminals (losing integration with orchestrator)
3. Sacrifice parallelism by running agents sequentially to maintain responsiveness

This is particularly problematic for orchestration workflows where Claude spawns multiple specialized agents and needs to:

• Monitor progress across parallel workstreams
• Continue other work while agents execute
• Gracefully handle long-running analysis tasks
• Coordinate results as agents complete incrementally

Proposed Solution

Extend the Task tool with background execution capabilities, mirroring the existing Bash tool pattern:

API Design

1. Background Agent Spawning

Task({
  subagent_type: "security-analyzer",
  prompt: "Comprehensive security audit of authentication system",
  run_in_background: true  // New parameter
})

Returns:

{
  "agent_id": "agent_abc123",
  "status": "running",
  "started_at": "2025-10-19T14:30:00Z",
  "message": "Agent running in background. Use AgentOutput(agent_id='agent_abc123') to check progress."
}

2. AgentOutput Tool (New)

AgentOutput({
  agent_id: "agent_abc123",
  filter?: "regex_pattern"  // Optional: filter output like BashOutput
})

Returns:

{
  "agent_id": "agent_abc123",
  "status": "running|completed|failed",
  "output": "New output since last check...",
  "progress": {
    "completed_tasks": 3,
    "total_tasks": 10,
    "current_task": "Analyzing dependency vulnerabilities"
  },
  "started_at": "2025-10-19T14:30:00Z",
  "completed_at": null,
  "exit_status": null
}

3. KillAgent Tool (New)

KillAgent({
  agent_id: "agent_abc123"
})

Returns:

{
  "agent_id": "agent_abc123",
  "status": "terminated",
  "terminated_at": "2025-10-19T15:45:00Z"
}

Consistency with Existing Bash Tool

This proposal directly mirrors the existing Bash tool implementation:

| Feature | Bash Tool | Proposed Agent Tool |
|---------|-----------|---------------------|
| Background execution | run_in_background: true | run_in_background: true |
| Output monitoring | BashOutput(bash_id) | AgentOutput(agent_id) |
| Process termination | KillShell(shell_id) | KillAgent(agent_id) |
| Output filtering | filter: "regex" | filter: "regex" |
| Status tracking | status: "running|completed" | status: "running|completed|failed" |

Use Cases with Metrics

1. Large Codebase Analysis

Scenario: Security audit of 15k-file enterprise monorepo

// Spawn multiple background agents
Task({
  subagent_type: "security-analyzer",
  prompt: "Analyze authentication and authorization patterns for vulnerabilities",
  run_in_background: true
}) // Returns agent_id: "security_001"

Task({
  subagent_type: "performance-optimizer",
  prompt: "Identify memory leaks and performance bottlenecks in core services",
  run_in_background: true
}) // Returns agent_id: "perf_001"

Task({
  subagent_type: "code-quality-reviewer",
  prompt: "Comprehensive code quality analysis with architectural recommendations",
  run_in_background: true
}) // Returns agent_id: "quality_001"

// Continue orchestration work while agents execute
// ... (work on other tasks)

// Check progress periodically
AgentOutput({ agent_id: "security_001" })
AgentOutput({ agent_id: "perf_001" })
AgentOutput({ agent_id: "quality_001" })

Metrics:

• Current approach: 90-120 minutes blocking time (sequential execution)
• With background execution: 30-45 minutes parallel execution, orchestrator remains responsive
• Productivity gain: 2-3x faster time-to-results, orchestrator can coordinate other work

2. Parallel Worktree Development

Scenario: Multi-feature development across 4 worktrees

// Spawn agents in different worktrees
Task({
  subagent_type: "general-purpose",
  prompt: "Implement user authentication backend in auth-backend worktree",
  working_directory: "/repo/.worktrees/auth-backend-a1b2c3d4",
  run_in_background: true
}) // agent_id: "auth_backend"

Task({
  subagent_type: "general-purpose",
  prompt: "Build authentication UI components in auth-frontend worktree",
  working_directory: "/repo/.worktrees/auth-frontend-e5f6g7h8",
  run_in_background: true
}) // agent_id: "auth_frontend"

Task({
  subagent_type: "security-analyzer",
  prompt: "Security review of authentication implementation",
  working_directory: "/repo/.worktrees/auth-security-i9j0k1l2",
  run_in_background: true
}) // agent_id: "auth_security"

// Orchestrator coordinates integration as agents complete
while (any_agent_running()) {
  let results = [
    AgentOutput({ agent_id: "auth_backend" }),
    AgentOutput({ agent_id: "auth_frontend" }),
    AgentOutput({ agent_id: "auth_security" })
  ]

  // Handle completed agents, coordinate integration
  // Continue monitoring until all complete
}

Metrics:

• Current approach: Sequential agent execution, 3-4 hours total
• With background execution: True parallelism, 1-1.5 hours wall clock time
• Productivity gain: 3x faster feature delivery

3. Continuous Background Monitoring

Scenario: Long-running test suite with incremental feedback

Task({
  subagent_type: "general-purpose",
  prompt: "Run comprehensive test suite and report failures incrementally",
  run_in_background: true
}) // agent_id: "test_runner"

// Poll for output while working on fixes
setInterval(() => {
  const output = AgentOutput({
    agent_id: "test_runner",
    filter: "FAILED|ERROR"  // Only show failures
  })

  if (output.contains_failures) {
    // Start fixing failures while tests continue
  }
}, 60000) // Check every minute

Metrics:

• Current approach: Wait 20-30 minutes for full suite, then start debugging
• With background execution: Start debugging within 2-3 minutes as first failures appear
• Productivity gain: 10x faster feedback loop

4. Resource-Intensive Analysis

Scenario: Memory profiling across entire application

Task({
  subagent_type: "performance-optimizer",
  prompt: "Profile memory usage across all services, identify leak patterns, generate optimization recommendations",
  run_in_background: true
}) // agent_id: "memory_profiler"

// Continue other work
// ... (implement other features, review code, etc.)

// Check if analysis complete
const status = AgentOutput({ agent_id: "memory_profiler" })
if (status.status === "completed") {
  // Review comprehensive analysis results
}

Metrics:

• Current approach: 60-90 minutes blocking orchestrator
• With background execution: Zero blocking time, orchestrator productive throughout
• Productivity gain: 100% orchestrator utilization vs 0% during wait

Optional Enhancements

Streaming Progress Updates

Leverage existing stream-json capability for real-time progress:

Task({
  subagent_type: "security-analyzer",
  prompt: "Security audit of authentication system",
  run_in_background: true,
  stream_progress: true  // Enable progress streaming
})

Stream format:

{"type": "progress", "agent_id": "agent_abc123", "completed": 3, "total": 10, "current_task": "Analyzing JWT implementation"}
{"type": "progress", "agent_id": "agent_abc123", "completed": 4, "total": 10, "current_task": "Checking CSRF protection"}
{"type": "completed", "agent_id": "agent_abc123", "output": "...full results..."}

Agent Lifecycle Management

// List all running agents
ListAgents()
// Returns: [{ agent_id: "...", status: "running", started_at: "..." }, ...]

// Get agent metadata
GetAgentInfo({ agent_id: "agent_abc123" })
// Returns: { agent_id, subagent_type, prompt, status, started_at, working_directory, ... }

Alternative Workarounds

Current Manual Approach

Users can spawn background agents using shell workarounds:

# In terminal outside Claude Code
claude -p "Comprehensive security audit of authentication system" &
AGENT_PID=$!

# Monitor manually
tail -f ~/.claude/agent_output.log

# Kill if needed
kill $AGENT_PID

Limitations:

• No integration with orchestrator
• Manual process management
• No structured output retrieval
• Cannot coordinate with other Task tool agents
• Loses context sharing between orchestrator and agent
• No progress tracking or incremental results

Polling with Short-Lived Agents

// Inefficient workaround: repeatedly spawn agents to check progress
setInterval(() => {
  Task({
    subagent_type: "general-purpose",
    prompt: "Check if security analysis in /tmp/security_results.json is complete"
  })
  // Spawns entirely new agent context each time
}, 60000)

Limitations:

• Massive overhead (new agent context every poll)
• No true background execution
• Expensive API calls for simple status checks
• Cannot interrupt or control actual analysis agent

Benefits for Claude Code Users

1. True Orchestration: Claude can spawn multiple specialized agents and coordinate results without blocking
2. Responsive UX: Long-running analysis doesn't freeze the orchestrator's ability to respond to user queries
3. Parallel Workflows: Multiple worktrees can have active agents simultaneously
4. Incremental Results: Early findings can be acted upon while analysis continues
5. Resource Efficiency: Background agents can utilize idle time during user thinking/review
6. Consistency: Same mental model as existing Bash background execution
7. Production Readiness: Essential for enterprise-scale codebases requiring comprehensive analysis

Implementation Notes

Security Considerations

• Agent lifecycle tied to parent orchestrator session (agents terminate when orchestrator ends)
• Resource limits per background agent (memory, CPU, wall-clock time)
• Maximum concurrent background agents (default: 5-10)
• Audit logging for background agent spawning and termination

Error Handling

• Failed agent status with error details in AgentOutput
• Automatic cleanup of zombie agents after timeout
• Graceful degradation if background execution unavailable (fall back to synchronous)

Compatibility

• run_in_background defaults to false (backward compatible)
• Existing Task tool calls continue working unchanged
• New tools (AgentOutput, KillAgent) only available when background execution enabled

Related References

• Existing Bash tool with run_in_background parameter
• Existing BashOutput and KillShell tools for async bash monitoring
• /parallel-review slash command (currently synchronous, would benefit from this feature)
• /agent-spawn slash command (multiple agents, currently blocking)

Conclusion

Background agent execution is a natural extension of Claude Code's existing async capabilities (Bash tool) to the Task tool. This enables true orchestration workflows where Claude can spawn multiple specialized agents, coordinate their results, and remain responsive to user needs—essential for enterprise-scale development workflows with large codebases and long-running analysis tasks.

The proposed API design maintains consistency with existing patterns, provides comprehensive lifecycle management, and delivers 2-10x productivity improvements for common orchestration scenarios.