[FEATURE] `compact_boundary`: include the full `post_tokens` in `compact_metadata`, not just conversation tokens (`claude -p` mode)

Preflight Checklist

• [x] I have searched existing requests and this feature hasn't been requested yet
• [x] This is a single feature request (not multiple features)

Problem Statement

Currently, compact_boundary reports only the size of the post-compaction conversation
(i.e. post_tokens only represents the messages contribution) as part of the compact_metadata stats - both in -p and interactive mode. It omits the fixed request overhead (baseline token usage from system prompt, tool schemas, memory
files, custom agents, skills etc.). Thus, it is not possible to compute the total post-compaction context usage without manually running another /context request.

See here a concrete example (v2.1.233, claude-haiku-4-5, 200k window) of a manual /compact followed 114s later by
/context, with nothing in between:

{
  "type": "system",
  "subtype": "compact_boundary",
  ...
  "compact_metadata": {
    "trigger": "manual",
    "pre_tokens": 61895,
    "post_tokens": 5706,
    "cumulative_dropped_tokens": 113485,
    "duration_ms": 54116,
    "preserved_segment": {
      ...
    },

compact_boundary   post_tokens = 5,706     (it seems that this is emitted as estimation chars / 4)
/context           total_tokens = 37,142   (Messages: 5,790, i.e. ≈ post_tokens)
                   System prompt 8,739 + System tools 12,689 + Custom agents 2,232
                   + Memory files 5,699 + Skills 1,993  = 31,352 never reported

Proposed Solution

Add a baseline-inclusive total to compact_metadata, emitted on the -p
--output-format stream-json event, not only written to the session transcript:

{
  "type": "system",
  "subtype": "compact_boundary",
  "compact_metadata": {
    "trigger": "[manual|auto]",
    "pre_tokens": 61895,
    "post_tokens": 5706,
    "post_total_tokens": 37142,
    "context_window": 200000
  }
}

post_total_tokens would be what /context reports immediately after the boundary, so a
client can render post_total_tokens / context_window directly. context_window is
included as well here for convenience, although it could be obtained from the result following compaction as well.

The data already exists in this exact shape. /context produces a context_usage object
(total_tokens, raw_max_tokens, percentage, per-category breakdown) that ships on the
synthetic assistant event in stream-json, and is persisted as contextUsage on a
system/local_command entry in the transcript. It is simply never produced at a
compaction boundary. Reusing that object wholesale would be even better than a single
total, and more future-proof.

 "context_usage": {
    "model": "claude-haiku-4-5",
    "total_tokens": 37142,
    "raw_max_tokens": 200000,
    "percentage": 19,
    "categories": [
      {
        "name": "System prompt",
        "tokens": 8739,
        "kind": "used"
      },
      {
        "name": "System tools",
        "tokens": 12689,
        "kind": "used"
      },
     ...

Alternative Solutions

For regular Assistant turns, the context usage can be computed in the following way that aligns with the /context output:
(inputTokens + cacheReadInputTokens + cacheCreationInputTokens) / contextWindow.
Here, the numerator can be derived from the last Assistant turn whereas the context window is included in the Result that follows at the end of the agent run. Note that in this case, there is not necessarily any need from client-perspective to distinguish between messages and baseline context usage.

However in the case of /compact, there currently does not appear to be any reliable source in the run that could supply the missing baseline token usage (total beyond the Messages component) following a compact_boundary:

1. From pre_tokens (full usage including output_tokens in the last API call prior to compaction) and cumulative_dropped_tokens, we cannot infer the post-compaction baseline usage beyond the messages-only post_tokens

1. the result event's modelUsage carries contextWindow, but its token counts seem to represent the accumulated usage for that given model id - not the current context filled up post-compaction

  "modelUsage": {
    "claude-haiku-4-5": {
      "inputTokens": 1529,
      "outputTokens": 4983,
      "cacheReadInputTokens": 23024,
      "cacheCreationInputTokens": 74715,
      "webSearchRequests": 0,
      "costUSD": 0.1781764,
      "contextWindow": 200000,
      "maxOutputTokens": 32000,
      "canonicalModel": "claude-haiku-4-5",
      "provider": "firstParty"
    }
  }

1. No assistant event is emitted following a compaction (see num_turns: 0 in the result), meaning above computation cannot be carried out post-compaction. Nor can we rely on earlier assistant turns prior to compaction, as these would represent stale state.

1. Relying on totalTokensReminder: "countdown" would not work either, as is not necessarily triggered right after compaction.

Moreover, this reminder is only included in transcripts and thus not accessible by an external harness driving -p (same limitation reported in #75203).

1. Parsing transcript JSONL would not be a solution either, as it does not provide any additional information that is lacking as described above. Furthermore, it would require file I/O, complex parsing, and only works post-session.

Priority

Medium - Would be very helpful

Feature Category

Developer tools/SDK

Use Case Example

The use case here is for a client consuming -p --output-format stream-json to keep a context
indicator up-to-date across a compaction, ideally from the compact_boundary metadata exclusively - for both manual and auto type compactions.

Currently, the scenario can be as follows:

1. Context indicator reads 30% (60,000 / 200,000 * 100%)
2. A compaction runs, and a compact_boundary is received with post_tokens: 5706
3. The indicator should now read 19% (37,000 / 200,000 * 100%). With only post_tokens it can render 3% (incorrect, as it omits the baseline usage) or keep the stale pre-compaction 30% (also incorrect)
4. The value cannot be reliably updated until the next real turn produces an assistant event

Additional Context

• Observed on v2.1.233 (macOS), with various models including claude-haiku-4-5 and

claude-opus-5[1m]

• conversation_reset (following a /clear) has the same gap and carries no token usage fields at all nor context window. The expected post-clear value would be the baseline token usage (all categories except Messages)
• Related but different contexts: #75203, #11008