[Bug] Transcript corruption from unsafe tool_result/thinking block separation during compaction

# Persisted append-streams must enforce record atomicity: transcript corruption as the case for a ring + Avro + DuckLake tier

Component (trigger): Claude Code CLI 2.1.152
Platform: Linux 6.12.86+deb13-amd64
Severity: High — permanently bricks affected sessions; auto-resume turns it into an infinite crash loop with no in-app recovery
Date: 2026-05-27

---

1. The defect (motivating case study)

When a session uses server-side tools (advisor, via "advisorModel": "opus") and/or
extended thinking, /compact rewrites the on-disk JSONL transcript and can:

• (a) separate a server_tool_use block from its paired advisor_tool_result across

the message boundary, orphaning the result; and/or

• (b) modify a thinking / redacted_thinking block in the latest assistant message.

Both violate API invariants. The corrupted transcript is persisted to disk, so every
subsequent resume / scheduled-task replay deterministically fails:

400 messages.N.content.M: unexpected `tool_use_id` found in `advisor_tool_result` blocks: <id>.
    Each `advisor_tool_result` block must have a corresponding `server_tool_use` block before it.

400 messages.N.content.M: `thinking` or `redacted_thinking` blocks in the latest assistant
    message cannot be modified ...

Evidence

In 0b3913a2-49be-4c5a-8e86-a5de12b0df81, 3 of 4 advisor pairs were intact
(server_tool_use immediately followed by its advisor_tool_result, parent-linked). The
4th pair was split:

• server_tool_use (uuid 380112d5, id srvtoolu_01MmmEEPKP6xakr5onpwjQjE) at line 3474
• its advisor_tool_result orphaned 35 lines later at line 3509, parentUuid = 55f97e3e

— linked to an unrelated message, not to the server_tool_use's uuid.

That orphan reconstructs at messages.9.content.0 with no preceding server_tool_use. The
session had been repaired earlier the same day; a compact-while-live re-introduced the split
with a different id and different messages.N position — proving the corruption is
regenerated by compaction, not a one-off.

2. Root fragility (the generalizable lesson)

The transcript is a structured, invariant-bearing event stream treated as a naive
line-appended log:

• No atomic record unit. server_tool_use+advisor_tool_result is a logically

indivisible pair, but the log addresses lines/blocks, so a rewrite (compaction) can cut
between them. Same for "thinking block must remain byte-identical."

• One serialization across all access patterns. The same JSON-per-line is the hot

replay format and the durable store and the thing compaction mutates in place. There
is no boundary at which atomicity or immutability is enforced.

• No self-healing on load. A poisoned store hard-fails forever instead of repairing

API-optional history (drop historical thinking; repair/drop orphaned pairs).

This is the same class of problem any persisted append-stream faces — including the
telephone wire (OcfAppender → logs/hooks/*.avro → read_avro glob). The durable fix
is architectural: make the record frame the atomic, immutable, schema-evolving unit, and
separate transport / storage / query formats.

3. Long-term remediation — ring + Avro + DuckLake

A three-layer persistence tier where atomicity is structural: the addressable element is
a whole frame, so nothing downstream can split an indivisible unit.

3.1 Ring (hot path) — atomic, zero-copy framing

• mmap'd circular log with sequence cursors (Disruptor-in-a-file): singl

readers each tracking a seq; restart-survivable; back-pressure = reader-la

• The unit on the ring is a whole frame (Arrow RecordBatch or FlatBuff

for zero-copy reads). A pair like server_tool_use+result, or a thinking block, is
written as one indivisible frame — there is no line boundary to cut be

• For the telephone wire: backs the Switchboard ring with :atomics cursors + a Rust NIF

buffer (or Aeron if you'd rather not build the log-buffer/flow-control yourself).

3.2 Avro — the append contract (schema evolution, never the mutation s

• Avro stays the wire/append contract (you already do Avro-RPC over UDS): compact,

row-oriented, real reader/writer schema resolution, aliases, defaults, uni

• Flexible schema via envelope + variant, so new fields never re-register a schema:

typed common columns (ts, scope, event, dunbar_layer, alarm_immediate, correlation_id,
handle, actor_kind) + one flexible map<string,string> / union payload for the
(--detail KV). Strictly better than the current NullPayload trick — comm
typed and prunable.

• Immutability rule: records are append-only frames. "Compaction" never edits

place; it produces a new snapshot referencing whole frames (see 3.3). This is
guarantee Claude Code's transcript layer lacks.

3.3 DuckLake — query + archive (collapses L1+L2+L3)

DuckLake (DuckDB Labs, 2025) keeps all table metadata — schema, snapshots,
column stats — in a SQL catalog DB, data in Parquet. It replaces the read_avro('*.avro')
glob and the separate Parquet archive pass with one managed format:

• Data inlining = the ring→lake flush, native. Small streaming inserts l

catalog DB (hot, transactional, queryable); a scheduled merge flushes them to
This is your OcfAppender→telephone.archive pass and your cheese_batch "o
per N records" — but ACID, with hot+cold read on the same query path. Implements
"L0 if hot, L2 otherwise" read semantics without branching in `cheese_call

• Snapshots / time-travel = safe compaction. Rewrites produce a new snapshot o

frames; you can never split an atomic unit because compaction operates on immuta
catalog pointers, not in-place edits. FROM hooks AT (TIMESTAMP => …) for repla

• Pruning + small-file compaction. Min/max stats → predicate pushdown (t

merge_adjacent_files() solves the small-file problem; expire_snapshots() for retention.

INSTALL ducklake; ATTACH 'ducklake:logs/wire.ducklake' AS wire (DATA_PATH 'l
CALL wire.merge_adjacent_files();
CALL wire.expire_snapshots(older_than => now() - INTERVAL 7 DAY);

3.4 Tier map (telephone wire)

| Tier | Today | Target | Format | Atomicity guarantee |
|---|---|---|---|---|
| L0 hot | Switchboard RAM ring + UDS | mmap circular log + seq cursors | Arrow / me write; no line to split |
| transport | UDS Avro-RPC | keep (contract only) | Avro | record = indivisible message |
| L1 commit | OcfAppender → *.avro | DuckLake data inlining | catalog roo in-place edit |
| L2 query | read_avro('*.avro') glob | query DuckLake table | Parquet + stats | reads immutable snapshots |
| L3 archive | mix …telephone.archive | merge_adjacent_files + snapshot expiryapshot over whole frames |

3.5 Cautions

• Keep Avro strictly as the transport contract — don't conflate wire serializa

storage format (that conflation is the present cost).

• DuckLake's catalog is a single-writer serialization point (writers coordinate through

the SQL DB's transactions). Ideal for one-host/one-daemon; micro-batch thr
Switchboard so there's effectively one writer.

• Zero-copy formats (FlatBuffers/Cap'n Proto) cost codegen/evolution ergonom

middle ground (columnar, zero-copy, batch-oriented — micro-batch the ring to amo

4. Immediate mitigation (until the tier lands)

• Point-in-time: strip thinking / redacted_thinking / `advisor_tool_

server_tool_use(name=="advisor") blocks from the JSONL, preserving
text / tool_use / tool_result. Recurs on the next compact-with-advis

• Durable stop-gap: disable the advisor tool (remove "advisorModel" from

~/.claude/settings.json) so the advisor block class can no longer be created.

5. What Claude Code itself should do (upstream ask)

1. Treat server_tool_use+advisor_tool_result as an atomic unit — never spl

re-parent the result.

1. Never mutate thinking / redacted_thinking; drop API-optional historical

instead of editing it.

1. Self-heal on load — repair/drop orphaned pairs and historical thinkin

hard-400-ing forever.