## Environment - Platform (select one): - [ ] Anthropic API - [ ] AWS Bedrock - [ ] Google Vertex AI - [X] Other: Anthropic Max subscription - Claude CLI version: 1.0.16 (Claude Code) - Operating System: Archlinux Linux 6.14.7-arch2-1 x86_64 - Terminal: Konsole - Python: 3.13.3 - Locale: en_US.UTF-8 (properly configured) - Tool: Claude Code Write tool ## Bug Description The Claude Code Write tool that causes Unicode corruption, resulting in data loss for non-English content. This i

[BUG] UTF-8 Corruption Bug

Open 💬 16 comments Opened Jun 6, 2025 by sumlin

Environment

Platform (select one):
[ ] Anthropic API
[ ] AWS Bedrock
[ ] Google Vertex AI
[X] Other: Anthropic Max subscription
Claude CLI version: 1.0.16 (Claude Code)
Operating System: Archlinux Linux 6.14.7-arch2-1 x86_64
Terminal: Konsole
Python: 3.13.3
Locale: en_US.UTF-8 (properly configured)
Tool: Claude Code Write tool

Bug Description

The Claude Code Write tool that causes Unicode corruption, resulting in data loss for non-English content. This issue affects file operations and impacts user workflows sometimes.

Problem Summary

The Write tool occasionally corrupts Unicode characters when creating files, specifically:

Russian Cyrillic text becomes unreadable gibberish (Dialog-First ?>4E>4** - 15+ ?0BB5@=>2 B@8335@>2 4;O 02B><0B8G5A:>3> >1=0@C65=8O 206=>9 8=D>@<0F88)
Emoji are replaced with control characters
English text remains unaffected
Issue occurs "time to time" (intermittent but recurring)

Impact

This bug causes actual data corruption requiring manual file recovery. For users working with non-English content, this represents a significant reliability issue that affects productivity and trust in the tool.

Technical Details

I have prepared a comprehensive bug report with:

Complete system environment details
Technical analysis of the corruption pattern
File examples (corrupted vs corrected)
Hex dumps and encoding analysis
Reproduction information
Root cause hypothesis

The evidence suggests UTF-8 bytes are being incorrectly interpreted as CP1251/Windows-1251 encoding during file write operations, likely due to a race condition or intermittent encoding handling bug.

Steps to Reproduce

Using Write() utility when writing non-Cyrillic characters (very rare)

Expected Behavior

🔍 Dialog-First подход - 15+ паттернов триггеров для автоматического обнаружения важной информации
🔄 Smart Update логика - интеллектуальная обработка изменений (ADD, UPDATE, DELETE, MERGE)
🗣️ Natural Language навигация - естественное взаимодействие ("покажи вчерашнее обсуждение")

Actual Behavior

Dialog-First ?>4E>4** - 15+ ?0BB5@=>2 B@8335@>2 4;O 02B><0B8G5A:>3> >1=0@C65=8O 206=>9 8=D>@<0F88

=^D Smart Update ;>38:0 - 8=B5;;5:BC0;L=0O >1@01>B:0 87<5=5=89 (ADD, UPDATE, DELETE, MERGE)
=ã^O Natural Language =02830F8O - 5AB5AB25==>5 2708<>459AB285 ("?>:068 2G5@0H=55 >1AC645=85")

Additional Context

Important! Claude Code is able to fix this error on his own if he sees that something is written incorrectly in the file. But once he suspended the work saying that I was violating the rules of using the application, stating as a reason that I was allegedly trying to exploit vulnerabilities (lol)

report_unicode.md - detailed description
README.md
README_corrupted.md

View original on GitHub ↗

16 Comments

volkv · 1 year ago

Yeah. This also happens to me when generating .md files

## "5E=>;>38G5A:89 AB5:

### Core Technologies
- **Blade Templates** - Laravel templating engine 4;O server-side rendering
- **LESS** - CSS ?@5?@>F5AA>@ 4;O AB@C:BC@8@>20==>3> =0?8A0=8O AB8;59
- **Vite** - !>2@5<5==K9 A1>@I8: 0AA5B>2 8 dev-A5@25@
- **PurgeCSS** - #40;5=85 =58A?>;L7C5<KE AB8;59 2 ?@>40:H5=5
- **Inter Font** - ^^A=>2=>9 A8AB5<=K9 H@8DB (Regular, Medium, SemiBold, Bold)

### Asset Pipeline
- **Development**: Live reload G5@57 Vite dev server
- **Production**: ^\8=8D8:0F8O 8 >?B8<870F8O 0AA5B>2
- **CSS Inlining**: ^Z@8B8G5A:85 AB8;8 8=;09=OBAO 2 `<head>` 4;O ?@>872>48B5;L=>AB8
- **Image Optimization**: SVG 8:>=:8 8 >?B8<878@>20==K5 87>1@065=8O

## ^P@E8B5:BC@0 AB8;59 (LESS)

### !B@C:BC@0 D09;>2

cq-esports · 1 year ago

ping

!Image

volkv · 1 year ago

🗿

volkv · 1 year ago

IAkumaI · 9 months ago

Same problem:

Even more, in the same file might be half OK, half BAD.

Only one solution is to ask CC to fix it. And it sees the wrong symbols.
Happens mostly in VSCode extension rather than in cli

mikehdt · 9 months ago

I've noticed this in just normal claude.ai web chat with Sonnet 4.5 - It misinterprets an é for Ã© for example.

skierpage · 8 months ago

I've noticed this in just normal claude.ai web chat with Sonnet 4.5 - It misinterprets an é for Ã© for example.

I uploaded a valid UTF-8 file to claude.ai (Sonnet 4.5) and asked it to check grammar. Instead Claude complained about gibberish due to it bungling the encoding. E.g. it reported '"lotâ€™s" - Should be "lot's" (incorrect apostrophe encoding)'.

ckrailo · 8 months ago

Claude webapp wrote a bunch of emojis in a file and then can't even read them back in. Is Claude unable to consume Claude's content? Is UTF-8 not the default everywhere?

leandrodsilvamg-code · 8 months ago

The Claude assistant integrated into Cursor is corrupting files with ANSI encoding (Windows-1252 or similar) during edits. Even when the original file is in ANSI, Claude forces saving in UTF-8, resulting in:

Replacement of accented characters (e.g., ç, ã, õ, á, é) with mojibake or black boxes (�).
Irreversible data loss in legacy files (e.g., old systems, .bat, .vbs, .ini scripts).
Inconsistent and non-configurable behavior — it does not respect the original encoding.

sumlin · 8 months ago

Claude Code 2.0.36
It corrupts the data and then look at this as an "appears to violate our Usage Policy". LOL


∴ Thinking…

  Отлично! Я вижу проблему. После строки 148 должна быть }, (с запятой), но вместо этого там } (без запятой). Это означает, что между 7-м и 8-м объектом отсутствует 
  запятая-разделитель.

  Строка 148:   } - это конец 7-го объекта, но должно быть   },

  Исправляю файл.

● Update(/home/kladmv/ClaudePrompts/yadro/tatlin-object-docs-training/results/generation/day_3_setup_ops_codex_087.json)
  ⎿  Error: No changes to make: old_string and new_string are exactly the same.

∴ Thinking…

  Ой, я скопировал одинаковый текст. Мне нужно правильно посмотреть на строку 148. Давайте еще раз прочитаем эту область.

● Bash(sed -n '146,151p' results/generation/day_3_setup_ops_codex_087.json | cat -A)
  ⎿        "domains": ["security", "user-management", "UX"]$                                                                                                    
         }$
       },$
       {$
         "question": "M-PM-^ZM-PM-0M-PM-:M-PM-8M-PM-5 endpoints M-QM-^AM-PM-2M-QM-^OM-PM-7M-PM-0M-PM-=M-QM-^K M-QM-^A auto-lock
     M-PM-:M-PM->M-PM-=M-QM-^DM-PM-8M-PM-3M-QM-^CM-QM-^@M-PM-0M-QM-^FM-PM-8M-PM-5M-PM-9 M-PM-4M-PM-;M-QM-^O users, M-PM-8 M-PM-:M-PM-0M-PM-: M-PM->M-PM-=M-PM-8
     M-QM-^@M-PM-0M-PM-7M-PM-;M-PM-8M-QM-^GM-PM-0M-QM-^NM-QM-^BM-QM-^AM-QM-^O M-PM->M-QM-^B manual lock endpoints M-PM-8M-PM-7
     M-PM-?M-QM-^@M-PM-5M-PM-4M-QM-^KM-PM-4M-QM-^CM-QM-^IM-PM-8M-QM-^E M-QM-^DM-QM-^@M-PM-0M-PM-3M-PM-<M-PM-5M-PM-=M-QM-^BM-PM->M-PM-2
     M-PM-<M-PM-0M-QM-^BM-QM-^@M-PM-8M-QM-^FM-QM-^K?",$
         "question_type": "factual",$
  ⎿ API Error: Claude Code is unable to respond to this request, which appears to violate our Usage Policy (https://www.anthropic.com/legal/aup). Please double press esc to edit 
    your last message or start a new session for Claude Code to assist with a different task. If you are seeing this refusal repeatedly, try running /model 
    claude-sonnet-4-20250514 to switch models.

∴ Thinking…

  Отлично! Теперь я вижу. На строке 148 есть   }, (с запятой!). Но проблема в кодировке - cat

> А что с агентами? 
  ⎿ API Error: Claude Code is unable to respond to this request, which appears to violate our Usage Policy (https://www.anthropic.com/legal/aup). Please double press esc to edit 
    your last message or start a new session for Claude Code to assist with a different task. If you are seeing this refusal repeatedly, try running /model 
    claude-sonnet-4-20250514 to switch models.

∴ Thinking…

  Пользователь спрашивает о статусе агентов. Давайте посмотрим, что произошло:

  1. Я запустил 3 параллельных агента для обработки документов 85

> А? 
  ⎿ API Error: Claude Code is unable to respond to this request, which appears to violate our Usage Policy (https://www.anthropic.com/legal/aup). Please double press esc to edit 
    your last message or start a new session for Claude Code to assist with a different task. If you are seeing this refusal repeatedly, try running /model 
    claude-sonnet-4-20250514 to switch models.

∴ Thinking…

  Пользователь спрашивает "А?" после того, как я получил ошибку Usage Policy violation. Давайте разберемся что произошло:

  1. Я запустил 3 параллельных агента для обработки документов 85-100
  2. Все 3 агента успешно завершили работу и вернули отчеты
  3. Я нач

reidab · 8 months ago

Encountered a variant of this bug where the Write tool injected 26 null bytes when writing Unicode arrow characters (→,
U+2192) to a new README.md file. The null bytes were scattered throughout the file, not just near the corrupted characters.

Result:

Arrow chars corrupted to � (U+FFFD)
File classified as binary data instead of UTF-8 text
Git treated it as binary: Binary files differ

Verification:

file README.md # data
python3 -c "print(open('README.md','rb').read().count(b'\x00'))"  # 26

joomlafun · 7 months ago

Same happens when you upload a file in an online project through the web GUI.

shortpoet · 7 months ago

Bug Report: Box-Drawing Character Corruption in Markdown Output

## Summary
Claude Code occasionally outputs ASCII box-drawing diagrams that get corrupted into null byte sequences, rendering them unreadable in the final markdown file.

## Environment

Claude Code Version: [Your version here]
OS: macOS (Darwin 25.1.0)
File Type: Markdown (.md)
Context: AI-generated documentation/planning files

## Reproduction
When Claude Code generates ASCII box diagrams (like architectural flowcharts or data structure visualizations), the box-drawing characters get replaced with null bytes and
sentinel characters.

Expected Output:
┌───────────────────────────────────────┐
│ BOOTSTRAP (runs once at startup) │
├───────────────────────────────────────┤
│ 1. Pre-parse globals → Configure svcs │
└───────────────────────────────────────┘

Actual Output (hex view):
0c 00 00 00 00 00 00 ... 10 0a
02 20 42 4f 4f 54 53 54 52 41 50 ...
1c 00 00 00 00 00 00 ... 00 24
02 20 31 2e 20 50 72 65 2d 70 61 72 73 65 ...

## Character Mapping Discovered

| Hex Bytes | Intended Character | Description |
|-----------|-------------------|-------------|
| 0x02 0x0a | │\n | Vertical line + newline |
| 0x02 0x20 | │ | Vertical line + space |
| 0x00 (repeated) | ─ | Horizontal line (variable length) |
| 0x0c 0x00 0x00 | ┌ | Top-left corner |
| 0x10 0x0a | ┐\n | Top-right corner + newline |
| 0x1c 0x00 0x00 | ├ | Left T-junction |
| 0x00 0x24 | ┤ | Right T-junction (null + dollar sign) |
| 0x14 0x00 | └ | Bottom-left corner |
| 0x18 0x0a | ┘\n | Bottom-right corner + newline |

## Additional Issues

Inconsistent horizontal line lengths: Variable-length sequences of 0x00 bytes result in misaligned right edges across different boxes
Dollar sign quirk: The byte 0x24 (ASCII `/feedback Regression in plan/markdown artifact output. Recently this has been happening consistently enough that we made a

quick fix script.
) appears as a sentinel marker for right-edge characters, not as an actual dollar sign

## Impact

Documentation files become unreadable
Box diagrams render as binary garbage in editors/viewers
Manual hex editing required to fix corrupted files

## Workaround
Created a repair script that maps null byte patterns back to proper UTF-8 box-drawing characters with width normalization: /scripts/workspace/fix-markdown-nulls.ts

#!/usr/bin/env bun

/**
 * Fix Markdown Null Bytes - Replace null byte sequences with proper box-drawing characters
 *
 * Claude Code occasionally outputs box-drawing characters that get corrupted into null byte
 * sequences during markdown generation. This script maps those sequences back to proper
 * UTF-8 box-drawing characters.
 *
 * Usage:
 *   bun scripts/workspace/fix-markdown-nulls.ts <input-file>
 *   bun scripts/workspace/fix-markdown-nulls.ts <input-file> --dry-run
 *
 * Options:
 *   --dry-run    Show what would be fixed without modifying the file
 *
 * Exit codes:
 *   0: Success (fixes applied or dry-run completed)
 *   1: Error (file not found, read/write failure)
 *   2: Usage error
 *
 * Architecture:
 *   - KISS-extensible: Easy to add new fix modes (tables, links, etc.)
 *   - Pure functions for testability
 *   - Explicit byte pattern mappings with comments
 */

import { readFileSync, writeFileSync } from "fs"
import { relative } from "path"

// ─────────────────────────────────────────────────────────────────────────────
// Box Character Mappings (Single-Line Style)
// ─────────────────────────────────────────────────────────────────────────────

/**
 * Null byte patterns that Claude Code outputs when rendering box-drawing characters.
 * Each entry maps a hex byte sequence to its UTF-8 replacement.
 *
 * Pattern format:
 *   - Hex bytes are space-separated (e.g., "02 0a")
 *   - Multi-byte sequences are represented as arrays
 *   - The replacement is the actual UTF-8 character
 */
interface BytePattern {
  readonly bytes: readonly number[]
  readonly replacement: string
  readonly description: string
}

const BOX_CHAR_PATTERNS: readonly BytePattern[] = [
  // Vertical lines
  {
    bytes: [0x02, 0x0a], // │\n
    replacement: "│\n",
    description: "Vertical line + newline",
  },
  {
    bytes: [0x02, 0x20], // │ (space)
    replacement: "│ ",
    description: "Vertical line + space",
  },

  // Horizontal lines (often repeated for width)
  {
    bytes: [0x00], // ───
    replacement: "─",
    description: "Horizontal line (single instance, may repeat)",
  },

  // Corners and junctions
  {
    bytes: [0x0c, 0x00, 0x00], // ┌
    replacement: "┌",
    description: "Top-left corner",
  },
  {
    bytes: [0x18, 0x0a], // ┘\n
    replacement: "┘\n",
    description: "Bottom-right corner + newline",
  },
  {
    bytes: [0x18], // ┘
    replacement: "┘",
    description: "Bottom-right corner",
  },
  {
    bytes: [0x1c, 0x00, 0x00], // ├
    replacement: "├",
    description: "Left T-junction",
  },
  {
    bytes: [0x00, 0x24], // ┤ (null + dollar)
    replacement: "┤",
    description: "Right T-junction",
  },
  {
    bytes: [0x10, 0x0a], // ┴\n
    replacement: "┐\n",
    description: "Top-right corner + newline",
  },
  {
    bytes: [0x10], // ┴
    replacement: "┐",
    description: "Top-right corner",
  },
  {
    bytes: [0x14, 0x00], // └
    replacement: "└",
    description: "Bottom-left corner",
  },
  {
    bytes: [0x14], // ─
    replacement: "─",
    description: "Em-dash",
  },
] as const


// ─────────────────────────────────────────────────────────────────────────────
// Core Fix Functions (Pure, Testable)
// ─────────────────────────────────────────────────────────────────────────────

/**
 * Convert byte sequence to Buffer-compatible format for matching
 */
function bytesToBuffer(bytes: readonly number[]): Buffer {
  return Buffer.from(bytes)
}

/**
 * Replace null byte sequences with proper box-drawing characters.
 * Pure function - returns new buffer, doesn't modify input.
 */
function fixNullBytes(content: Buffer): Buffer {
  let result = content

  // Sort patterns by length (longest first) to avoid partial matches
  const sortedPatterns = [...BOX_CHAR_PATTERNS].sort((a, b) => b.bytes.length - a.bytes.length)

  for (const pattern of sortedPatterns) {
    const needle = bytesToBuffer(pattern.bytes)
    const replacement = Buffer.from(pattern.replacement, "utf-8")

    // Replace all occurrences
    let position = 0
    while ((position = result.indexOf(needle, position)) !== -1) {
      result = Buffer.concat([
        result.subarray(0, position),
        replacement,
        result.subarray(position + needle.length),
      ])
      position += replacement.length
    }
  }

  return result
}

/**
 * Normalize box widths - each box's horizontal lines match its longest content line.
 * Pure function - returns new string with normalized boxes.
 */
function normalizeBoxWidths(content: string): string {
  const lines = content.split("\n")
  const result: string[] = []
  let boxStart = -1
  let maxWidth = 0

  for (let i = 0; i < lines.length; i++) {
    const line = lines[i]
    const hasBoxChars = /[│├┤┌┐└┘─]/.test(line)

    // Start of a box
    if (hasBoxChars && boxStart === -1) {
      boxStart = i
      maxWidth = line.length
    }
    // Inside a box
    else if (hasBoxChars && boxStart !== -1) {
      maxWidth = Math.max(maxWidth, line.length)
    }
    // End of a box (non-box line after box content)
    else if (!hasBoxChars && boxStart !== -1) {
      // Normalize the box we just finished
      for (let j = boxStart; j < i; j++) {
        result.push(normalizeBoxLine(lines[j], maxWidth))
      }
      boxStart = -1
      maxWidth = 0
      result.push(line) // Add the non-box line
    }
    // Outside any box
    else {
      result.push(line)
    }
  }

  // Handle box that extends to end of file
  if (boxStart !== -1) {
    for (let j = boxStart; j < lines.length; j++) {
      result.push(normalizeBoxLine(lines[j], maxWidth))
    }
  }

  return result.join("\n")
}

/**
 * Normalize a single box line to the target width.
 * Extends horizontal lines (─) and adjusts right edge characters.
 */
function normalizeBoxLine(line: string, targetWidth: number): string {
  // Lines with horizontal bars that need extending
  if (/^[┌├└]─+[┐┤┘]?$/.test(line)) {
    const leftChar = line[0]
    const rightChar = line[line.length - 1]
    const isRightEdge = /[┐┤┘]/.test(rightChar)

    if (isRightEdge) {
      // Left char + horizontal bars + right char
      const barCount = targetWidth - 2
      return leftChar + "─".repeat(barCount) + rightChar
    } else {
      // Left char + horizontal bars (no right edge yet)
      const barCount = targetWidth - 1
      return leftChar + "─".repeat(barCount)
    }
  }

  // Content lines with │ edges - pad the content
  if (/^│.*│?$/.test(line)) {
    const hasRightEdge = line.endsWith("│")
    if (hasRightEdge) {
      const content = line.slice(1, -1) // Remove both │
      const paddedContent = content.padEnd(targetWidth - 2, " ")
      return "│" + paddedContent + "│"
    } else {
      const content = line.slice(1) // Remove left │
      const paddedContent = content.padEnd(targetWidth - 1, " ")
      return "│" + paddedContent
    }
  }

  // Return line as-is if it doesn't match known box patterns
  return line
}

/**
 * Analyze content and report what would be fixed (for dry-run mode).
 * Returns human-readable statistics.
 */
interface FixStats {
  readonly patternCounts: ReadonlyMap<string, number>
  readonly totalReplacements: number
}

function analyzeNullBytes(content: Buffer): FixStats {
  const counts = new Map<string, number>()
  let total = 0

  // Sort patterns by length (longest first) to avoid partial matches
  const sortedPatterns = [...BOX_CHAR_PATTERNS].sort((a, b) => b.bytes.length - a.bytes.length)

  for (const pattern of sortedPatterns) {
    const needle = bytesToBuffer(pattern.bytes)
    let position = 0
    let count = 0

    while ((position = content.indexOf(needle, position)) !== -1) {
      count++
      position += needle.length
    }

    if (count > 0) {
      counts.set(pattern.description, count)
      total += count
    }
  }

  return {
    patternCounts: counts,
    totalReplacements: total,
  }
}

// ─────────────────────────────────────────────────────────────────────────────
// CLI Formatting
// ─────────────────────────────────────────────────────────────────────────────

const COLORS = {
  reset: "\x1b[0m",
  bold: "\x1b[1m",
  dim: "\x1b[2m",
  green: "\x1b[32m",
  yellow: "\x1b[33m",
  cyan: "\x1b[36m",
  red: "\x1b[31m",
} as const

function formatStats(stats: FixStats, fileRelative: string): string {
  if (stats.totalReplacements === 0) {
    return `${COLORS.green}✓ ${fileRelative}${COLORS.reset} - No null bytes found\n`
  }

  const lines = [
    `${COLORS.cyan}◆ ${fileRelative}${COLORS.reset}`,
    `  ${COLORS.bold}Total replacements:${COLORS.reset} ${stats.totalReplacements}`,
    "",
    `  ${COLORS.dim}Pattern breakdown:${COLORS.reset}`,
  ]

  for (const [description, count] of stats.patternCounts) {
    lines.push(`    ${description}: ${count}`)
  }

  lines.push("")

  return lines.join("\n")
}

function printUsage() {
  console.error(`
Fix Markdown Null Bytes - Replace null byte sequences with box-drawing characters

Usage:
  bun scripts/workspace/fix-markdown-nulls.ts <input-file>
  bun scripts/workspace/fix-markdown-nulls.ts <input-file> --dry-run

Options:
  --dry-run    Show what would be fixed without modifying the file

Exit codes:
  0: Success (fixes applied or dry-run completed)
  1: Error (file not found, read/write failure)
  2: Usage error
`)
}

// ─────────────────────────────────────────────────────────────────────────────
// Main
// ─────────────────────────────────────────────────────────────────────────────

async function main() {
  const args = process.argv.slice(2)

  if (args.length === 0) {
    console.error("Error: No input file specified\n")
    printUsage()
    process.exit(2)
  }

  const filePath = args[0]
  const isDryRun = args.includes("--dry-run")
  const cwd = process.cwd()
  const fileRelative = relative(cwd, filePath)

  // Read file as binary buffer to preserve exact bytes
  let content: Buffer
  try {
    content = readFileSync(filePath)
  } catch (err) {
    const message = err instanceof Error ? err.message : String(err)
    console.error(`${COLORS.red}✗ Error reading file:${COLORS.reset} ${message}`)
    process.exit(1)
  }

  // Analyze content
  const stats = analyzeNullBytes(content)

  if (isDryRun) {
    console.log(`${COLORS.bold}DRY RUN MODE${COLORS.reset} - No changes will be made\n`)
    console.log(formatStats(stats, fileRelative))

    if (stats.totalReplacements > 0) {
      console.log(
        `${COLORS.yellow}⚠ Run without --dry-run to apply fixes${COLORS.reset}`
      )
    }

    process.exit(0)
  }

  // Apply fixes
  if (stats.totalReplacements === 0) {
    console.log(formatStats(stats, fileRelative))
    process.exit(0)
  }

  // Step 1: Replace null bytes with box characters
  const fixedBytes = fixNullBytes(content)

  // Step 2: Normalize box widths
  const fixedString = fixedBytes.toString("utf-8")
  const normalized = normalizeBoxWidths(fixedString)

  // Write fixed content
  try {
    writeFileSync(filePath, normalized, "utf-8")
  } catch (err) {
    const message = err instanceof Error ? err.message : String(err)
    console.error(`${COLORS.red}✗ Error writing file:${COLORS.reset} ${message}`)
    process.exit(1)
  }

  console.log(formatStats(stats, fileRelative))
  console.log(`${COLORS.green}✓ Fixes applied (null bytes replaced + boxes normalized)${COLORS.reset}`)

  process.exit(0)
}

main().catch((err: unknown) => {
  const message = err instanceof Error ? err.message : String(err)
  console.error(`${COLORS.red}Fatal error:${COLORS.reset} ${message}`)
  process.exit(1)
})

github-actions[bot] · 6 months ago

This issue has been inactive for 30 days. If the issue is still occurring, please comment to let us know. Otherwise, this issue will be automatically closed in 30 days for housekeeping purposes.

dimitar-grigorov · 5 months ago

I hit this constantly with CP1251 and CP1252 files. Built an MCP server in Go as a workaround. it auto-detects encoding and preserves it on writes instead of forcing UTF-8. Also adds a proper tree tool since the built-in one is missing on Windows.
https://github.com/dimitar-grigorov/mcp-file-tools
Won't fix the root cause, but keeps non-ASCII files intact right now.

ArtKoKo · 4 months ago

The Unicode corruption you describe (Russian Cyrillic, among others) shares the same root cause as the curly quote normalization in Write/Edit. While the Cyrillic issue may have additional factors, the curly→straight normalization is a documented, intentional transformation that corrupts non-ASCII textual content.

A /curly option has been proposed to address this class of problems with context-aware verification instead of blanket normalization.

→ [FEATURE] /curly option: context-aware curly/straight character handling in Write/Edit — #31482