[BUG] U+FFFD replacement characters in Agent SDK streaming output for CJK text

Description

When using Claude Code CLI via Agent SDK (Python) with --output-format stream-json, Japanese (CJK) text in TextBlock.text occasionally contains U+FFFD replacement characters (�), indicating UTF-8 multibyte sequences are being corrupted during streaming output.

This also occurs in Claude Code CLI interactive mode (non-SDK), suggesting the issue is in the CLI's stdout streaming layer.

Steps to Reproduce

1. Use Claude Agent SDK (Python) with Japanese prompts
2. Have the model generate Japanese text responses
3. Read TextBlock.text from streamed AssistantMessage

Expected Behavior

All CJK characters should be intact in the streamed text output.

Actual Behavior

Some multibyte characters are replaced with U+FFFD (�).

Example:

• Expected: 承知しました。溝内さんのファイルを作成します。作成しました。
• Actual: 承知しました。溝内さんのファイルを作成します。作成し���した。

(ました → ���した — 2 characters corrupted into 3 replacement characters)

Environment

• Claude Code CLI: bundled via claude-agent-sdk (Python)
• Python: 3.13.9
• OS: macOS (Darwin 23.6.0, Apple Silicon)
• Transport: SubprocessCLITransport → TextReceiveStream(process.stdout)
• CLI flags: --output-format stream-json --input-format stream-json

Analysis

The CLI streams JSON lines to stdout. CJK characters are 3 bytes in UTF-8. When the Node.js CLI flushes stdout, a multibyte sequence can be split across two write() calls. If the CLI decodes/re-encodes with replacement semantics at any point, U+FFFD is emitted.

The Python-side anyio.TextReceiveStream uses an incremental UTF-8 decoder that handles chunk boundaries correctly, so the corruption likely occurs before the bytes reach stdout — inside the CLI's JSON serialization or stream flushing logic.

Related Issues

• #14310 — Panic when processing UTF-8 characters (byte boundary error)
• #14946 — Panic: UTF-8 byte boundary errors when rendering CJK text
• #13080 — Write tool corrupts UTF-8 multi-byte characters
• #2154 — MultiEdit UTF-8 encoding bug

Workaround

Detecting U+FFFD in the consumer and logging for diagnostics:

if "\ufffd" in response_text:
    log.warning(f"U+FFFD detected in response")