[BUG] Duplicate entries in session .jsonl files when using stream-json input format

[BUG] Duplicate entries in session .jsonl files when using stream-json input format

Environment

• Platform: Claude Code
• Claude CLI version: 1.0.67 (Claude Code)
• Operating System: macOS 15.0
• Terminal: Terminal App, JetBrains IDE Terminal

Bug Description

When using --input-format stream-json for multi-turn conversations, Claude Code writes duplicate entries to session .jsonl files. Each subsequent message in a conversation causes the previous conversation history to be duplicated in the session file.

Steps to Reproduce

1. Start Claude Code with streaming JSON input:

``bash
claude --output-format stream-json --input-format stream-json --verbose
``

1. Send first message via stdin:

``json
{"type":"user","message":{"role":"user","content":"What's the capital of France?"},"session_id":"default","parent_tool_use_id":null}
``

1. Send second message via stdin:

``json
{"type":"user","message":{"role":"user","content":"What's its population?"},"session_id":"<captured_session_id>","parent_tool_use_id":null}
``

1. Check the session file at ~/.claude/projects/<project>/<session_id>.jsonl

Expected Behavior

The session file should contain exactly 4 entries:

• User message 1 + Assistant response 1
• User message 2 + Assistant response 2

Actual Behavior

The session file contains 6+ entries with duplicates:

• User message 1 + Assistant response 1 (original)
• User message 1 + Assistant response 1 (duplicate)
• User message 2 + Assistant response 2

Each new message causes all previous entries to be duplicated, leading to exponential growth of session file size.

Impact

• Session files grow much larger than expected
• Parsing session history becomes complex due to duplicates
• Storage usage increases unnecessarily
• Makes session analysis/debugging difficult

Reproducible Examples

Python SDK (also affected)

The issue occurs even when using the official Python SDK:

from claude_code_sdk import ClaudeSDKClient

async with ClaudeSDKClient() as client:
    await client.query("First message")
    async for msg in client.receive_response():
        if isinstance(msg, ResultMessage):
            break
    
    await client.query("Second message")  # This triggers duplication
    async for msg in client.receive_response():
        if isinstance(msg, ResultMessage):
            break

Direct CLI usage

echo '{"type":"user","message":{"role":"user","content":"Message 1"},"session_id":"default","parent_tool_use_id":null}' | claude --output-format stream-json --input-format stream-json --verbose
# Get session ID from output, then:
echo '{"type":"user","message":{"role":"user","content":"Message 2"},"session_id":"<session_id>","parent_tool_use_id":null}' | claude --resume <session_id> --output-format stream-json --input-format stream-json --verbose

Additional Context

• The issue does NOT occur with regular interactive mode (without --input-format stream-json)
• Resume functionality works correctly despite the duplicates
• Context is maintained properly - this is purely a logging/persistence issue
• Related to issues #3187 (stream-json hang) and #1920 (missing final result event)

Analysis

This appears to be caused by Claude Code rewriting/appending the entire conversation history to the session file each time a new message is processed, rather than appending only the new entries.

Workaround

Parse session files by deduplicating entries based on message IDs and UUIDs when reading conversation history.

---

Would appreciate guidance on whether this is expected behavior or a bug, and if there are plans to fix the session file format consistency.