Unicode/UTF-8 Character Corruption in Hook System

Claude Code Bug Report: Unicode/UTF-8 Character Corruption in Hook System

Summary

Claude Code incorrectly encodes non-ASCII characters (Turkish, Chinese, Arabic, etc.) when passing user input to hooks, causing severe character corruption that affects session logging and other hook-based functionality.

Environment

• Claude Code Version: Latest (as of August 21, 2025)
• Platform: Windows 10/11
• Language: Turkish characters tested (çÇ şŞ ğĞ üÜ ıİ öÖ)
• Hook System: user_prompt_submit.py hook receiving corrupted data

Issue Description

Expected Behavior

When a user types Turkish text like "Ağustos" (August in Turkish), hooks should receive properly encoded UTF-8 characters.

Actual Behavior

Claude Code corrupts Turkish characters by incorrectly interpreting UTF-8 bytes as individual Latin-1 characters, then JSON-escaping them.

Character Corruption Examples

| Input Character | Correct UTF-8 | Corrupted in JSON | Result After JSON Decode |
|-----------------|---------------|-------------------|-------------------------|
| ğ | \u011f | \u00c4\u0178 | Displays as "Ä" + control char |
| ş | \u015f | \u00c5\u0178 | Displays as "Å" + control char |
| ı | \u0131 | \u00c4\u00b1 | Displays as "ı" |
| ç | \u00e7 | \u00c3\u00a7 | Displays as "ç" |
| ü | \u00fc | \u00c3\u00bc | Displays as "ü" |
| ö | \u00f6 | \u00c3\u00b6 | Displays as "ö" |

Technical Analysis

Root Cause

Claude Code's input processing pipeline incorrectly:

1. Takes proper UTF-8 input from user
2. Misinterprets UTF-8 byte sequences as individual Latin-1 characters
3. JSON-encodes these misinterpreted characters as Unicode escape sequences
4. Sends corrupted data to hooks

Example Data Flow

User Input: "Türkçe" (Turkish language)
What hooks receive: "T\u00c3\u00bcrk\u00c3\u00a7e"
After JSON decode: "Türkçe" (corrupted)
Should be: "Türkçe"

Steps to Reproduce

1. Set up Claude Code with hook system in a project directory
2. Configure a user_prompt_submit.py hook to log user input to JSON
3. Type any message containing Turkish characters: "Merhaba dünya"
4. Check the JSON log file created by the hook
5. Observe corrupted Unicode escape sequences instead of proper Turkish characters

Impact

Affected Systems

• Hook-based session logging - All non-ASCII characters corrupted
• International users - Turkish, Chinese, Arabic, Russian, and other non-Latin script users affected
• Custom workflow automation - Any hook processing user text with international characters

Severity

• High - Affects all international users using hooks
• Data integrity - User input is permanently corrupted in hook-generated logs
• Accessibility - Makes Claude Code unusable for non-English workflows involving hooks

Important Note

The main Claude Code conversation transcripts (.jsonl files) store characters correctly. This bug specifically affects the hook system's data pipeline, not the core conversation storage.

Location of correct transcripts: ~/.claude/projects/[project-hash]/[session-id].jsonl

Workaround Implemented

We developed a character mapping fix that reconstructs corrupted Turkish characters:

replacements = {
    '\u00c5\u0178': 'ş',  # ş
    '\u00c4\u0178': 'ğ',  # ğ  
    '\u00c4\u00b1': 'ı',  # ı
    '\u00c3\u00a7': 'ç',  # ç
    '\u00c3\u00bc': 'ü',  # ü
    '\u00c3\u00b6': 'ö',  # ö
}

However, this is a band-aid solution. The core issue needs to be fixed in Claude Code's text processing pipeline.

Recommended Fix

1. Fix input encoding: Ensure user input maintains proper UTF-8 encoding throughout the pipeline
2. JSON serialization: Use proper UTF-8 aware JSON serialization when sending data to hooks
3. Testing: Add comprehensive Unicode tests for all hook types
4. Documentation: Update hook documentation to specify UTF-8 handling

Test Cases for Validation

Turkish Characters

• Input: "Türkçe çalışma şekli için ğöz üıl"
• Expected in hooks: Exact same characters, properly encoded

Other Languages

• Chinese: "你好世界"
• Arabic: "مرحبا بالعالم"
• Russian: "Привет мир"

All should pass through to hooks without corruption.

Additional Context

This bug significantly impacts the international usability of Claude Code's hook ecosystem. Users from non-English speaking countries cannot reliably use hook-based features, which limits Claude Code's global adoption for development workflows.

The fix should be straightforward - ensuring proper UTF-8 encoding is maintained from user input through to hook delivery, rather than the current Latin-1 misinterpretation approach.