Python file I/O snippets should always include encoding='utf-8' to prevent silent data corruption on Windows (follow-up to #62828)

Open 💬 0 comments Opened Jun 29, 2026 by evansmith-everag

Summary

This is a follow-up to #62828, which was auto-closed as stale on 2026-06-27. The bot asked that a new issue be opened if it's still relevant — it is. I re-verified the root cause on a current Windows + Python 3.14 setup today.

When Claude Code generates Python scripts that read/write text files using open(), it often omits the encoding parameter. On Windows, Python's text-mode default is the locale encoding (cp1252), not UTF-8. For files containing non-ASCII characters this produces silent data corruption (mojibake) or, depending on the bytes, a hard UnicodeDecodeError.

Re-verification (today, Windows 11, Python 3.14)

>>> import locale; locale.getpreferredencoding(False)
'cp1252'

Failure mode 1 — silent mojibake (accented Latin-1 text). A UTF-8 JSON file containing "Café déjà vu résumé" round-tripped through the encoding-less pattern below becomes garbled, with no error and valid JSON output:

with open(p, "r") as f:          # reads UTF-8 bytes as cp1252
    data = json.load(f)
with open(p, "w") as f:          # writes garbled strings back as cp1252/UTF-8
    json.dump(data, f, ensure_ascii=False, indent=2)
# -> "Caf? d?j? vu r?sum?"  (corrupted, no exception)

Failure mode 2 — hard crash. The same pattern on text containing curly quotes / em-dashes (bytes undefined in cp1252) raises:

UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 49: character maps to <undefined>

Why it matters

  • Affects any Windows user asking Claude Code to generate Python that touches text files.
  • Especially harmful for i18n/locale files, configs, and source with non-ASCII content.
  • The silent variant passes linters, formatters, JSON validators, and tests — it only surfaces when a human reads the file.

Expected behavior

Generated Python text-mode file I/O should always pass encoding="utf-8" explicitly:

with open("messages/fr/parameters.json", "r", encoding="utf-8") as f:
    data = json.load(f)
with open("messages/fr/parameters.json", "w", encoding="utf-8") as f:
    json.dump(data, f, ensure_ascii=False, indent=2)

This is already Python community best practice — PEP 597 added EncodingWarning (Python 3.10+) to surface exactly this bug class.

Suggested fix

A system-level / guideline instruction that when generating Python open() calls on text files, encoding="utf-8" is always included as a keyword argument.

Environment

  • OS: Windows 11
  • Python: 3.14 (default cp1252 locale)
  • Re-verified: 2026-06-29
  • Original report: #62828

View original on GitHub ↗