[DOCS] Hooks reference: say that hook stdin is UTF-8, because Python hooks on Windows decode it with the ANSI code page and silently corrupt non-ASCII input

The same applies in the other direction. A SessionStart hook printing additionalContext that contains Japanese, measured with Claude Code 2.1.280 on the same machine (Python 3.11, no PYTHONUTF8), asking the model to quote the sentence back:

| Hook prints | Model received |
|---|---|
| print(json.dumps(obj, ensure_ascii=False)) | mojibake: 16 × U+FFFD, starting �����t�́u |
| print(json.dumps(obj)) (non-ASCII as \u escapes) | 合言葉は「さくら並木」です |
| sys.stdout.reconfigure(encoding="utf-8"), then ensure_ascii=False | 合言葉は「さくら並木」です |

So on Windows, hook stdout also needs to be UTF-8 (or ASCII-only JSON). A line in the docs covering both directions would cover it, e.g. "Hook input and output are UTF-8. On Windows, Python hooks should read sys.stdin.buffer and write UTF-8 (sys.stdout.reconfigure(encoding="utf-8")), or run with PYTHONUTF8=1."

I checked the public Hookify implementation for the same stdin boundary and reproduced a related failure in its actual hook executors, independently of the rule-file encoding issue.

At main 56f36532530f88b572854538d685fcf781141e8c, all four Hookify entry points call json.load(sys.stdin). With the same raw UTF-8 JSON and an ASCII-only rule containing a Unicode regex escape, PreToolUse returns permissionDecision: deny under UTF-8, but a cp932 text stream reports a decoding error and exits 0; cp1252 and cp932 with surrogateescape silently return {} for that same matching command.

This was tested on macOS by setting PYTHONIOENCODING on real Python hook subprocesses, not on a native Windows machine. No model request or command from a hook payload was executed.

The minimal patch changes the four input reads to:

input_data = json.loads(sys.stdin.buffer.read().decode('utf-8'))

It leaves ASCII-escaped JSON output and the existing systemMessage / exit-0 error handling in place. I have a patch with regression tests covering the four entry points, non-ASCII transcript paths, a non-ASCII PreToolUse command that must be denied, ASCII matches, non-matches, malformed JSON, and invalid UTF-8. The original implementation fails 15 valid UTF-8 regression scenarios; a further 12 invalid-UTF-8 cases fail the desired diagnostic check. All 84 scenarios pass after the patch.

I checked the full diffs of #77289 and #79645: those fix rule/transcript file decoding, not stdin. #23967 repairs unescaped backslashes but still reads the locale-decoded text stream. This patch would complement those changes; it does not fix the separate prompt-field issue or claim that native Windows / Claude Code integration has been verified.

For maintainers: PR creation is currently restricted to collaborators. What is the preferred channel for submitting this small public Hookify patch and its tests? I can provide the patch through an accepted channel or work with a collaborator. The broader UTF-8 input/output documentation request here still stands.

This patch and the local checks were prepared with AI assistance.

Measured on Windows 11, Python 3.12.8, system ANSI code page cp1251 — three separate things happen on the way to a hook, and only one of them is fixable inside the hook, which is why the docs sentence matters:

1. Python decodes hook stdin with the ANSI code page, not UTF-8.

$ python -c "import sys; print(sys.stdin.encoding)"
cp1251

So json.loads(sys.stdin.read()) decodes UTF-8 JSON as cp1251 (cp1252, cp932, cp936 elsewhere). Non-ASCII file paths and prompt text arrive as mojibake, and a byte that is undefined in the page raises UnicodeDecodeError instead. sys.stdin.buffer.read().decode("utf-8") is the fix on the hook side, and it works on every Python version.

2. If the hook is invoked through PowerShell 5.1, the text can be destroyed before the hook sees it. $OutputEncoding defaults to ASCIIEncoding, and the pipeline encodes with it:

PS> '{"t":"café"}' | python read_stdin.py
raw: b'{"t":"caf?"}\r\n'          # é is already gone

No hook-side decoding can recover that; it has to be set upstream.

3. The usual upstream fix adds a BOM. Setting $OutputEncoding = [System.Text.Encoding]::UTF8 delivers correct UTF-8 — and a \xef\xbb\xbf preamble with it, because that .NET encoding emits one:

PS> $OutputEncoding = [System.Text.Encoding]::UTF8; '{"t":"café"}' | python read_stdin.py
raw: b'\xef\xbb\xbf{"t":"caf\xc3\xa9"}\r\n'

json.loads on that raises Expecting value: line 1 column 1 (char 0), which reads like malformed JSON rather than an encoding problem — and [System.Text.UTF8Encoding]::new($false) is the variant without the preamble.

So the documentation sentence people need is arguably two: the JSON on stdin is UTF-8 and a hook must decode it as UTF-8 explicitly rather than rely on the platform default, with utf-8-sig if it may be invoked through a PowerShell pipeline. A one-line example in the hooks reference would prevent all three.

(I package this handling — UTF-8 with a tolerated BOM on the way in, no BOM on the way out — in https://github.com/AlKor13/winseam, but the three measurements above are the point; the fix is two lines of stdlib either way.)