Edit tool silently corrupts pre-existing non-UTF-8 bytes anywhere in a file, even when the edit target is elsewhere

Bug: Edit tool silently corrupts pre-existing non-UTF-8 bytes anywhere in a file, even when the edit target is elsewhere

Environment

• Claude Code version: 2.1.227
• OS: Windows 11 (MINGW64/Git Bash shell), also observed to affect files opened from a native Windows path
• File under test: a plain text file encoded as Windows-1252 (cp1252), not UTF-8

Summary

When the Edit tool modifies any part of a file, the entire file appears to be decoded as UTF-8 on read and re-encoded as UTF-8 on write. If the file contains a legacy single-byte character that is not valid standalone UTF-8 (e.g. byte 0x99, the Windows-1252 code point for the trademark symbol ™), that byte is silently replaced with the UTF-8 encoding of the Unicode replacement character (U+FFFD, i.e. bytes EF BF BD) -- even when the edit itself targets a completely different, unrelated line in the file.

This is silent, permanent, byte-level data corruption of source files that are not UTF-8, and it happens regardless of where the edit is targeted -- editing line 1 of a 1000-line file corrupts a legacy byte on line 999.

Steps to reproduce

1. Create a text file encoded as Windows-1252 containing ordinary ASCII content plus one legacy non-ASCII byte, e.g. (pseudocode, using Python to control the exact encoding):

``python
data = "Example\x99 end\n".encode("cp1252") # 0x99 = (TM) in cp1252
open("repro.txt", "wb").write(data)
``

1. Confirm the byte on disk: Example followed by raw byte 0x99 followed by end.
2. Use Claude Code's Edit tool to change an unrelated line elsewhere in the same file (no edit anywhere near the 0x99 byte).
3. Re-inspect the raw bytes of the file around the original 0x99 byte.

Expected behavior

The byte 0x99 is untouched, since the edit did not target that region of the file, and the file's original encoding should be preserved on files Claude Code did not create.

Actual behavior

The byte 0x99 becomes the 3-byte sequence EF BF BD (UTF-8 for U+FFFD, the replacement character). The original character is unrecoverably lost. This happens on every Edit-tool write to the file, regardless of edit location.

Impact

Any Windows-1252 (or other non-UTF-8) source file that legitimately contains non-ASCII bytes (smart quotes, trademark/degree/accented characters, etc. -- common in legacy enterprise codebases with pre-Unicode file encodings) will have those bytes silently and permanently corrupted the first time Claude Code's Edit or Write tool touches the file for any reason, even a change unrelated to that content. This can go unnoticed without an external diff/encoding check, since Claude Code gives no indication that a byte outside the edited region changed.

Suggested fix

• Preserve the original byte-level encoding of a file when writing back changes that don't touch the affected bytes (e.g., detect encoding on read, or perform edits as byte-level patches rather than full decode/re-encode round-trips), or
• At minimum, detect when a file's on-disk bytes are not valid UTF-8 before editing, and warn/refuse rather than silently substituting replacement characters, so users aren't surprised by data loss.

Workaround found

Comparing the file's post-edit bytes against its last-committed (git HEAD) bytes via a PostToolUse hook on Write|Edit can detect the corruption after the fact (diffing byte histograms), but this only surfaces the problem after it has already happened on disk -- it does not prevent it.