JSONL session logs record placeholder output_tokens (always 1-2) instead of real values

Summary

Claude Code session JSONL logs (~/.claude/projects/<project>/<session>.jsonl) record a placeholder value (typically 1-2) for output_tokens in the message.usage block on assistant records. The real output token count — which can be 10-1000x larger — is only available in the streaming result event, which is never persisted to the JSONL session file.

All other usage fields (input_tokens, cache_creation_input_tokens, cache_read_input_tokens) are accurate. Only output_tokens is wrong.

Evidence

Controlled Experiment

Three controlled prompts via claude -p, comparing --output-format json (correct) vs the JSONL session log on disk:

| Prompt | JSON output output_tokens | JSONL log output_tokens | Visible response chars |
|---|---|---|---|
| "What is 2+2? Reply with just the number." | 5 | 1 | 1 |
| "Explain why the sky is blue in 3 sentences." | 113 | 1 | 509 |
| "Write a binary search function with edge cases." | 294 | 2 | 832 |

Input-side fields match exactly between JSON output and JSONL — only output_tokens differs:

| Field | JSON output | JSONL log | Match? |
|---|---|---|---|
| input_tokens | 3 | 3 | ✅ |
| cache_creation_input_tokens | 4,678 | 4,678 | ✅ |
| cache_read_input_tokens | 17,997 | 17,997 | ✅ |
| output_tokens | 113 | 1 | ❌ |

Root Cause: Stream Event Timing

Capturing --output-format stream-json reveals two events with usage data:

1. assistant event (start of stream): output_tokens: 1 — this is what gets written to the JSONL
2. result event (end of stream): output_tokens: 118 — correct value, never persisted

The JSONL session file contains only the assistant-type record. The result event (which carries the final accumulated output_tokens) is used for --output-format json output but is never written to the session log. We confirmed zero result-type records exist across 50+ session JSONL files.

Cache Delta Validation

To confirm the JSON output values are correct, we ran a 3-turn conversation and checked whether the previous turn's real output appears in the next turn's cache accounting:

| Turn | Real output_tokens (from JSON) | Next turn cache delta | Residual |
|---|---|---|---|
| 0 | 891 | +909 | 18 (≈ user prompt tokens) |
| 1 | 191 | +207 (net) | 16 (≈ user prompt tokens) |

Cache growth matches real output tokens with ~16-18 token residuals (the new user prompt). This confirms the JSON output values are correct and the JSONL values are placeholders.

Impact

• Any tool reading Claude Code session logs will dramatically understate output token usage
• total_tokens computed from JSONL is missing the vast majority of output tokens
• There is no way to recover real output_tokens from the static JSONL logs for agentic sessions (tool results dominate cache deltas, making reverse-engineering impossible)

Environment

• Claude Code version: 2.1.42
• Model: claude-opus-4-6
• Platform: macOS (Darwin 25.2.0)

Reproduction

# Run a prompt and get the real output_tokens from JSON output
claude -p "Explain why the sky is blue in 3 sentences." --output-format json 2>/dev/null \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print(f'JSON output_tokens: {d[\"usage\"][\"output_tokens\"]}'); print(f'Session: {d[\"session_id\"]}')"

# Then check the JSONL log for the same session (replace SESSION_ID):
# python3 -c "
# import json, glob
# for line in open(glob.glob(f'$HOME/.claude/projects/*/SESSION_ID.jsonl')[0]):
#     rec = json.loads(line)
#     if rec.get('type') == 'assistant':
#         print(f'JSONL output_tokens: {rec[\"message\"][\"usage\"][\"output_tokens\"]}')
# "

Suggested Fix

Write the final output_tokens from the message_delta or result streaming event back to the assistant record in the JSONL, or append a result-type record with the final usage data. All other usage fields are already accurate — only output_tokens needs the final-value writeback.