stream-json: background sub-agent events missing usage data (run_in_background asymmetry)

Summary

When using claude --output-format stream-json programmatically, foreground sub-agents (run_in_background: false) emit their internal assistant events with full usage data in the parent's stdout stream — but background sub-agents (run_in_background: true) emit nothing. This creates a significant observability gap: you get per-tool-call token tracking for foreground agents but lose it entirely for background agents, even though both execute the same tools and consume the same resources.

This is a production issue for us — we run a Slack bot powered by Claude Code that tracks per-tool-call token usage for dashboarding, cost attribution, and session analytics. The moment Claude decides to use run_in_background: true (or a skill/prompt instructs it to), we lose all granular token data for those sub-agents.

The Asymmetry

Foreground sub-agent (run_in_background: false)

The parent's stream-json stdout includes the sub-agent's internal events:

{"type":"assistant","message":{"content":[{"type":"tool_use","id":"toolu_sub1","name":"Read","input":{...}}],"usage":{"input_tokens":8200,"cache_read_input_tokens":5100,"cache_creation_input_tokens":490,"output_tokens":350}}}
{"type":"user","message":{"content":[{"type":"tool_result","tool_use_id":"toolu_sub1","content":"..."}]}}

Result: We can parse usage from each assistant event and attribute context_tokens (input + cache_read + cache_creation) to individual tool calls. ✅

Background sub-agent (run_in_background: true)

The parent's stream-json stdout shows only the top-level Agent tool_use and its eventual tool_result. None of the sub-agent's internal tool events appear in the stream.

Result: We have no usage data for any of the sub-agent's tool calls. The PostToolUse hook still fires (so we know the tools ran), but hooks don't include token/usage data. ❌

Real-World Impact (Production Data)

From a production session (48d67f5c-5336-4d39-a957-fba7699ed118) processing an RFP questionnaire:

| Agent | run_in_background | Tool Calls | With context_tokens | Without |
|-------|---------------------|------------|-----------------------|---------|
| Batch 1 | false | ~40 | 40 (100%) | 0 |
| Batch 2 | false | ~40 | 40 (100%) | 0 |
| Batch 3 | false | ~39 | 39 (100%) | 0 |
| Batch 3-retry | true | ~98 | 0 (0%) | 98 |
| Batch 4 | true | ~98 | 0 (0%) | 98 |
| Batch 5 | true | ~98 | 0 (0%) | 98 |
| Batch 4-retry (new request) | true | ~95 | 0 (0%) | 95 |

Total: 163 tool calls with token data, 294 without. The cutoff is perfectly correlated with the run_in_background flag — not timing, not context size, not compaction.

What We've Tried / Ruled Out

1. --verbose flag: Does not cause background agent events to appear in stream
2. --include-partial-messages: Only affects streaming deltas, not background agent visibility
3. --include-hook-events: Includes hook lifecycle events but not background agent tool events
4. PostToolUse hook: Fires for all tool calls (foreground and background) but does NOT include usage / token data in its payload
5. SubagentStop hook: Fires when background agent completes, provides agent_transcript_path but does NOT include usage data. Parsing the transcript .jsonl file is a workaround but the format is undocumented and fragile
6. Claude Agent SDK (Python/TypeScript): Evaluated as an alternative — actually provides less per-tool-call token granularity than the CLI approach. SDK gives aggregate per-session tokens only, and has no documented run_in_background equivalent

Proposed Solutions (any of these would help)

Option A: Include background agent events in stream-json (preferred)

Emit background sub-agent assistant/user events in the parent's stdout stream, the same way foreground sub-agents already work. Add a field like "agent_id": "abc123" to distinguish them from the parent's events. This would give full parity between foreground and background agents.

Option B: Add usage data to SubagentStop hook payload

When the SubagentStop hook fires, include aggregate usage for the sub-agent's execution:

{
  "hook_event_name": "SubagentStop",
  "agent_id": "abc123",
  "agent_type": "general-purpose",
  "agent_transcript_path": "...",
  "last_assistant_message": "...",
  "usage": {
    "input_tokens": 45000,
    "output_tokens": 12000,
    "cache_read_input_tokens": 38000,
    "cache_creation_input_tokens": 2000
  }
}

Option C: Add usage data to PostToolUse hook payload

Include the current context token count in every PostToolUse event:

{
  "hook_event_name": "PostToolUse",
  "tool_name": "Read",
  "tool_use_id": "toolu_xxx",
  "session_id": "...",
  "agent_id": "abc123",
  "usage": {
    "input_tokens": 13790,
    "cache_read_input_tokens": 10200,
    "cache_creation_input_tokens": 1100,
    "output_tokens": 490
  }
}

This would solve both the background agent problem AND give better observability for all tool calls regardless of agent type.

Option D: Document the transcript JSONL format

If none of the above are feasible short-term, officially documenting the .jsonl transcript format (especially the usage fields in assistant messages) would let us reliably parse agent_transcript_path from SubagentStop hooks without worrying about format changes.

Environment

• Claude Code CLI (latest, auto-updated daily)
• --output-format stream-json
• Running programmatically via Node.js child_process.spawn
• macOS / Linux (Docker)

Related Issues

• #10164 — Show sub-agent token usage in /context and Task tool output
• #10388 — Agent Token Usage API — Critical Infrastructure
• #11320 — Task Tool Execution Metadata for Subagent Monitoring (canonical)
• #16424 — Expose Agent Context in Hook Event Payloads
• #22625 — Per-Subagent Token Usage Tracking

All five of these were closed as stale or duplicated, but the underlying problem remains unsolved. This issue adds concrete production data showing the exact asymmetry and its impact on programmatic usage.