[FEATURE] JSONL usage.input_tokens are placeholder values; need complete token accounting

Preflight Checklist

• [x] I have searched existing requests and this feature hasn't been requested yet
• [x] This is a single feature request (not multiple features)

Problem Statement

Claude Code's JSONL conversation logs record usage.input_tokens as a streaming placeholder (typically 1) rather than the final value. The remaining output gap (10-17x) is consistent with thinking tokens being excluded from usage.output_tokens. Together, this makes JSONL unreliable for token accounting.

I discovered this while building an energy monitor that reads the statusbar's context_window totals. When I independently summed JSONL tokens (deduplicated by requestId), the results diverged significantly from statusbar cumulative totals — but in a revealing pattern:

Feb 20 (heavy Opus 4.6 day, 20 sessions, 1,365 unique requests):

| Metric | JSONL (dedup) | Statusbar | Ratio |
|---|---|---|---|
| Input tokens | 41,444 | 7,199,162 | 174x |
| Output tokens | 183,829 | 3,208,365 | 17x |
| Cache read | 104,353,324 | 114,798,863 | 1.1x |
| Cache creation | 3,170,696 | 2,717,775 | 0.9x |

Feb 24 (12 sessions, 1,228 unique requests):

| Metric | JSONL (dedup) | Statusbar | Ratio |
|---|---|---|---|
| Input tokens | 11,758 | 1,193,366 | 102x |
| Output tokens | 69,449 | 748,337 | 11x |
| Cache read | 74,254,777 | 67,710,877 | 0.9x |
| Cache creation | 2,817,739 | 2,003,545 | 0.7x |

Cache metrics match closely (~1x), confirming both sources track the same API calls. But input tokens diverge by 100-174x, and output by 10-17x.

Root cause: 75% of JSONL entries have usage.input_tokens of 1 or 0 — these are streaming placeholder values that never get updated to the final count. JSONL also logs streaming duplicates (same requestId appears 2-10 times with identical placeholder values).

Proposed Solution

Two improvements would make JSONL useful for token accounting:

1. Write final usage values. After a request completes, update the JSONL entry (or append a final entry) with the real input_tokens and output_tokens including thinking tokens.

1. Deduplicate streaming entries. Either log only the final state per request, or add a "final": true marker so consumers know which entry to use.

Alternatively, a separate append-only usage log would work:

{"ts":1740000000,"requestId":"req_...","model":"opus","input_tokens":1234,"output_tokens":567,"cache_read":50000,"cache_creation":800}

Alternative Solutions

I read the statusbar's context_window.total_input_tokens / total_output_tokens via a custom statusline script, which gives accurate cumulative totals. But this only provides session-level aggregates, not per-call breakdowns.

I also verified the discrepancy using an independent JSONL parser (no third-party dependencies), reproducing the same pattern across two different days.

Priority

Medium - Would be very helpful

Feature Category

API and model interactions

Use Case Example

1. I use Claude Code daily and want to track my compute footprint
2. I sum token usage from JSONL logs — the only per-call data source available
3. The result is 100-174x too low on input tokens because usage.input_tokens is a placeholder
4. Tools like ccusage that read JSONL are affected by the same issue
5. With accurate final values in JSONL, any tool could provide reliable usage tracking

Additional Context

• Independent JSONL parser with deduplication: sum_jsonl.py
• Statusbar token semantics validated via analyze_tokens.py: total_input_tokens = fresh input only (excludes cache), total_output_tokens includes thinking
• Full investigation: FINDINGS.md
• Cache metrics matching at ~1x across both data sources confirms they're tracking the same API calls — the discrepancy is in the recorded values, not missing entries
• The output gap (10-17x) is partially explained by thinking tokens being included in statusbar but not JSONL, consistent with ~60-70% thinking overhead on Opus