Transcript JSONL corruption; records spliced by unsynchronized writers
Preflight Checklist
- [x] I have searched existing issues and this hasn't been reported yet
- [x] This is a single bug report (please file separate reports for different bugs)
- [x] I am using the latest version of Claude Code
What's Wrong?
Claude Code's transcript writer occasionally corrupts a session's JSONL mid-session during routine operation — no crash, no kill: the same session keeps appending valid records right after the corrupt line.
Corpus audit of one machine (~10,949 transcript *.jsonl under ~/.claude/projects, incl. */subagents/, ~6 months of use): 23 files contain 30 invalid lines. 24/30 show a record-splice signature:
| Signature | Count | Detail |
| --- | --- | --- |
| Truncated splice | 21x | a record is cut off mid-content and the NEXT complete record begins immediately on the same physical line — no separator, no junk bytes |
| Missing newline | 3x | two fully-valid records back-to-back on one line (}{ junction) |
(The remaining 6/30 are 0x00 runs then one valid record — the known #49876 signature, not this report.)
Key facts
- Dates: 2026-01-14 → 2026-07-19, 17 distinct days — steady recurrence, not one incident.
- Corrupt lines written by 16 distinct CC versions (2.1.7 … 2.1.215; 2.1.220 too recent to have samples).
- All 30 lines: both halves belong to the SAME
sessionId— NOT two sessions sharing a file; the session's own writers racing. - Sessions continue normally after the bad line — distinct from #52387 (SIGKILL mid-write). Related but distinct: #52387, #49876.
Code audit of the installed 2.1.220 bundle indicates the race is still present
- Four independent writer domains append to the same session
.jsonlwith no shared lock:
- the 100ms batched drain queue;
appendEntryToFileAsync(separate per-path promise chain: custom-title/pr-link/...);reAppendSessionMetadataAsync(callable outside the drain guard);- a sync
appendFileSyncexit handler bypassing both queues.
performRemoveByUuiddoesopen("r+")→truncate(A)→ positional write: an append landing between those awaits is clobbered mid-record at offsetA— matching the dominant truncated-splice signature exactly, on a routine streaming path (tombstone/replacesUuidsremoval).performCompactTranscript's concurrency guard is a 3x4KiB sampling heuristic, not a lock.- Un-queued full-line appenders interleaving with the queues account for the pure missing-newline cases.
- Differential: the append-path region of 2.1.217 vs 2.1.220 has identical logic (identifier-stripped minified skeletons match) — no fix landed; 2.1.217's transcript changelog entry is observability only.
- Upstream telemetry
tengu_transcript_compact_failed{reason:"snapshot_mid_line"}shows the mid-line state is already detected in the wild (compaction aborts on it) but writers are not serialized.
Impact: strict NDJSON consumers (jq, json.loads) abort at the bad line; anything streaming the transcript silently loses everything after it without per-line error recovery.
What Should Happen?
Every record append is atomic and all writers to a session file are serialized through one queue/lock — one write of json + "\n" per record — so the transcript is valid NDJSON at every instant; in-place mutations (removeByUuid, compaction) exclude appenders while they run.
Error Messages/Logs
$ jq -c .type 071f66b9-4802-47b7-9031-bd074c435770.jsonl
jq: parse error: Invalid numeric literal at line 794, column 1116truncated splice — record cut mid-value, next record starts with no separator:
..."sessionId":"915a2{"parentUuid":"..."pure missing-newline — two complete records on one line:
...,"gitBranch":"main"}{"type":"file-history-snapshot",...Steps to Reproduce
Not deterministically reproducible — statistical: 30 occurrences over ~6 months, ~0.2% of session files.
Common conditions: long sessions with subagents/background tasks; Linux, ext4, local NVMe.
Scanner:
python3 - <<'EOF'
import json, glob, os
for f in glob.glob(os.path.expanduser('~/.claude/projects/**/*.jsonl'), recursive=True):
for i, line in enumerate(open(f, encoding='utf-8', errors='replace'), 1):
try: json.loads(line)
except Exception as e: print(f, i, e)
EOFSplice lines: two "parentUuid" on one physical line; json.JSONDecoder().raw_decode at the failure offset recovers a complete second record.
Claude Model
_No response_
Is this a regression?
No, this never worked
Last Working Version
_No response_
Claude Code Version
2.1.220
Platform
Anthropic API
Operating System
Ubuntu/Debian Linux
Terminal/Shell
Other
Additional Information
Full audit artifacts (scanner, per-line results, raw_decode reconstruction, bundle-audit notes) available on request. The 6 NUL-padded lines matching #49876's signature were excluded to keep this a single bug report.
Secondary observation from the same audit (not implicated in the corruption above): the drain queue's MAX_CHUNK_BYTES is 104857600, so a single batched transcript write is effectively unbounded — a memory-spike/robustness concern worth a look while in this code.
3 Comments
Thanks for the detailed audit. Tried to reproduce on the released 2.1.233 build (macOS): scanned a local corpus of ~5,500 transcript
.jsonlfiles (many months of use, including subagent transcripts) and found 0 invalid lines, and a fresh session with two parallel subagents produced fully valid main and subagent transcripts. So we couldn't reproduce the splice locally in the time budget, but your data (17 distinct days, 16 versions, same-session halves on one line) is convincing.Classification: real bug, not a regression. We confirmed from the current code that several independent writers still append to the same session file without a single shared lock, and that the in-place tombstone-removal path (truncate + positional rewrite of the tail) can overlap with one of the out-of-band metadata appenders. That is a plausible source of spliced/partial lines, and we agree the intended contract is "one atomic write of
json + \nper record, all writers serialized per file, in-place rewrites exclude appenders." We're tracking a fix along those lines (and will look at the unbounded batch size you flagged).If you can, one sanitized example would help pin the mechanism: for a truncated-splice line, the
typeof the cut-off record and of the record that follows it, plus whether a}fragment line appears anywhere later in the same file. Also OS/filesystem details for any non-ext4 occurrences.🤖 Generated with Claude Code
Follow-up: mechanism details for the truncated splices
@bcherny
Ran the follow-up on the same corpus, now 13,013 files (~2,064 added since the report). Zero new occurrences in the added files — nothing newer than 2.1.215 (installed is 2.1.234). Answers to your three questions first, then a new structural finding that surfaced while extracting them.
1. Record types at each truncated splice
All 21 instances (letters are anonymized per-file IDs):
| line | cut-off
type| followingtype(role / block) | version ||---|---|---|---|
| A:948 | message | assistant (text) | 2.1.113 |
| A:1029 | message | assistant (text) | 2.1.113 |
| B:2532 | user | file-history-snapshot | — |
| D:1250 | user | user (tool_result) | 2.1.81 |
| E:258 | message | progress | 2.1.81 |
| F:7138 | user | assistant (thinking) | 2.1.38 |
| G:596 | message | assistant (text) | 2.1.91 |
| I:62 | progress | user (tool_result, isSidechain) | 2.1.50 |
| K:1457 | message | file-history-snapshot | — |
| L:62 | message | assistant (thinking) | 2.1.215 |
| M:315 | user | assistant (tool_use) | 2.1.185 |
| M:319 | message | system | 2.1.185 |
| N:794 | user | assistant (thinking) | 2.1.215 |
| O:663 | user | assistant (text) | 2.1.91 |
| T:980 | message | user (tool_result) | 2.1.74 |
| U:318 † | message | file-history-snapshot | 2.1.50 |
| U:470 † | message | file-history-snapshot | 2.1.50 |
| V:365 | (cut before
type) | progress | 2.1.71 || V:376 | (cut before
type) | file-history-snapshot | — || AA:238 | user | user (tool_result) | 2.1.7 |
| AA:345 | (cut before
type) | assistant (thinking) | 2.1.7 |† the only two page-unaligned cut points (see the finding below).
Roll-up:
message×9,user×7,progress×1, cut beforetype×3assistant×9,file-history-snapshot×5,user×3,progress×2,system×12. Later
}fragment linesNo, in all 21 cases. The dropped tail bytes never reappear anywhere later in the file.
3. Filesystem
Every occurrence is on a single local ext4 NVMe volume — verified via identical
st_devacross all 13,013 files, so there are no non-ext4 occurrences to report.New finding: cut points sit on 4 KiB page boundaries
19 of the 21 splice cut points are at an exact multiple of 4096 bytes from the start of the file (only U:318 and U:470 are unaligned). Higher alignments do not hold — 8192: 12/19, 16384: 7/19, i.e. chance level — so the boundary is specifically the 4 KiB page.
Two related observations:
file-history-snapshot, 1×custom-title). 4 of the 6 NUL runs end on an exact 4096 multiple.Both damage modes match a page-granular partial write whose remainder is never retried: a short
write()/flush that isn't looped on, with the file offset advancing anyway. The splice case is the buffer's first N pages landing and the tail being dropped; the NUL case is the mirror image — the offset advancing over a region whose data never reached the page cache, leaving a materialized hole ending on a page boundary. This doesn't require interleaving between writers — a single writer mishandling a short write produces exactly this signature.Follow-up: we audited the deployed 2.1.234 binary (beautified extract of the transcript-store region) to test the short-write hypothesis, and the page-alignment evidence points at a specific function — not the append path.
The append hot path is safe
All four writer domains funnel into one primitive built on
fs/promises.appendFile, which loops on partial writes. Verified empirically (RLIMIT_FSIZE probe, Bun and Node agree):appendFile/appendFileSyncthrow rather than short-write; bareFileHandle.writesilently resolves short (e.g.bytesWritten3096 of 6000).But three
FileHandle.writecall sites on the transcript path discardbytesWrittenperformCompactTranscript): the chunked flushes to the.compact.tmp.<hex>file, the final flush, and the concurrent-tail catch-up write;performRemoveByUuid): the positional tail rewrite after the truncate.The compaction rewriter matches the corpus signature point for point
FileHandle.writeto a file seeded with a deliberately unaligned 1234-byte prefix returnedbytesWritten = 64302with no error, leaving the file at exactly 65536 bytes (size % 4096 === 0): the first partial page was filled from 1234 to 4095, then 15 full pages, then it stopped — the cut lands on a page boundary of the file offset regardless of where the write began. Identical on Bun and Node.rename()d over the live session file. Every pre-rename guard (statino/size re-samples,h.sync()) validates the source transcript; nothing validates the bytes just written —h.sync()sits exactly where abytesWrittencheck would have caught it.Triggers, and the fix shape already in the tree
On triggers: ENOSPC is the classic real-world partial-write cause, and 2.1.217's changelog already added disk-full warnings for transcript writes, so this condition is known to occur in the field on this path. Notably, the codebase already contains the correct pattern — one auxiliary transcript-append helper hand-rolls a
bytesWrittenretry loop ("tryAppend: short write … stalled") — while these three sites rely on unchecked writes. Fix shape: that same loop at the three call sites, plus a written-bytes/final-statassertion before the rename.What the original race still explains
The exit-handler race and the un-serialized out-of-band metadata re-append from the original report are still present in 2.1.234, but they can only account for the 3 unaligned missing-newline junctions — they cannot cut a record mid-string, since every append emits whole
json + "\n"strings through the loopingappendFile.---
Probe scripts: https://gist.github.com/NubeBuster/1a03409a3e95c322607f01d972048522