Unicode/UTF-8 Character Corruption in Hook System

This issue has been inactive for 30 days. If the issue is still occurring, please comment to let us know. Otherwise, this issue will be automatically closed in 30 days for housekeeping purposes.

Yes its still a problem

This issue has been inactive for 30 days. If the issue is still occurring, please comment to let us know. Otherwise, this issue will be automatically closed in 30 days for housekeeping purposes.

Bug Report: Box-Drawing Character Corruption in Markdown Output

## Summary
Claude Code occasionally outputs ASCII box-drawing diagrams that get corrupted into null byte sequences, rendering them unreadable in the final markdown file.

## Environment

• Claude Code Version: [Your version here]
• OS: macOS (Darwin 25.1.0)
• File Type: Markdown (.md)
• Context: AI-generated documentation/planning files

## Reproduction
When Claude Code generates ASCII box diagrams (like architectural flowcharts or data structure visualizations), the box-drawing characters get replaced with null bytes and
sentinel characters.

Expected Output:
┌───────────────────────────────────────┐
│ BOOTSTRAP (runs once at startup) │
├───────────────────────────────────────┤
│ 1. Pre-parse globals → Configure svcs │
└───────────────────────────────────────┘

Actual Output (hex view):
0c 00 00 00 00 00 00 ... 10 0a
02 20 42 4f 4f 54 53 54 52 41 50 ...
1c 00 00 00 00 00 00 ... 00 24
02 20 31 2e 20 50 72 65 2d 70 61 72 73 65 ...

## Character Mapping Discovered

| Hex Bytes | Intended Character | Description |
|-----------|-------------------|-------------|
| 0x02 0x0a | │\n | Vertical line + newline |
| 0x02 0x20 | │ | Vertical line + space |
| 0x00 (repeated) | ─ | Horizontal line (variable length) |
| 0x0c 0x00 0x00 | ┌ | Top-left corner |
| 0x10 0x0a | ┐\n | Top-right corner + newline |
| 0x1c 0x00 0x00 | ├ | Left T-junction |
| 0x00 0x24 | ┤ | Right T-junction (null + dollar sign) |
| 0x14 0x00 | └ | Bottom-left corner |
| 0x18 0x0a | ┘\n | Bottom-right corner + newline |

## Additional Issues

1. Inconsistent horizontal line lengths: Variable-length sequences of 0x00 bytes result in misaligned right edges across different boxes
2. Dollar sign quirk: The byte 0x24 (ASCII `/feedback Regression in plan/markdown artifact output. Recently this has been happening consistently enough that we made a

quick fix script.
) appears as a sentinel marker for right-edge characters, not as an actual dollar sign

## Impact

• Documentation files become unreadable
• Box diagrams render as binary garbage in editors/viewers
• Manual hex editing required to fix corrupted files

## Workaround
Created a repair script that maps null byte patterns back to proper UTF-8 box-drawing characters with width normalization: /scripts/workspace/fix-markdown-nulls.ts

#!/usr/bin/env bun

/**
 * Fix Markdown Null Bytes - Replace null byte sequences with proper box-drawing characters
 *
 * Claude Code occasionally outputs box-drawing characters that get corrupted into null byte
 * sequences during markdown generation. This script maps those sequences back to proper
 * UTF-8 box-drawing characters.
 *
 * Usage:
 *   bun scripts/workspace/fix-markdown-nulls.ts <input-file>
 *   bun scripts/workspace/fix-markdown-nulls.ts <input-file> --dry-run
 *
 * Options:
 *   --dry-run    Show what would be fixed without modifying the file
 *
 * Exit codes:
 *   0: Success (fixes applied or dry-run completed)
 *   1: Error (file not found, read/write failure)
 *   2: Usage error
 *
 * Architecture:
 *   - KISS-extensible: Easy to add new fix modes (tables, links, etc.)
 *   - Pure functions for testability
 *   - Explicit byte pattern mappings with comments
 */

import { readFileSync, writeFileSync } from "fs"
import { relative } from "path"

// ─────────────────────────────────────────────────────────────────────────────
// Box Character Mappings (Single-Line Style)
// ─────────────────────────────────────────────────────────────────────────────

/**
 * Null byte patterns that Claude Code outputs when rendering box-drawing characters.
 * Each entry maps a hex byte sequence to its UTF-8 replacement.
 *
 * Pattern format:
 *   - Hex bytes are space-separated (e.g., "02 0a")
 *   - Multi-byte sequences are represented as arrays
 *   - The replacement is the actual UTF-8 character
 */
interface BytePattern {
  readonly bytes: readonly number[]
  readonly replacement: string
  readonly description: string
}

const BOX_CHAR_PATTERNS: readonly BytePattern[] = [
  // Vertical lines
  {
    bytes: [0x02, 0x0a], // │\n
    replacement: "│\n",
    description: "Vertical line + newline",
  },
  {
    bytes: [0x02, 0x20], // │ (space)
    replacement: "│ ",
    description: "Vertical line + space",
  },

  // Horizontal lines (often repeated for width)
  {
    bytes: [0x00], // ───
    replacement: "─",
    description: "Horizontal line (single instance, may repeat)",
  },

  // Corners and junctions
  {
    bytes: [0x0c, 0x00, 0x00], // ┌
    replacement: "┌",
    description: "Top-left corner",
  },
  {
    bytes: [0x18, 0x0a], // ┘\n
    replacement: "┘\n",
    description: "Bottom-right corner + newline",
  },
  {
    bytes: [0x18], // ┘
    replacement: "┘",
    description: "Bottom-right corner",
  },
  {
    bytes: [0x1c, 0x00, 0x00], // ├
    replacement: "├",
    description: "Left T-junction",
  },
  {
    bytes: [0x00, 0x24], // ┤ (null + dollar)
    replacement: "┤",
    description: "Right T-junction",
  },
  {
    bytes: [0x10, 0x0a], // ┴\n
    replacement: "┐\n",
    description: "Top-right corner + newline",
  },
  {
    bytes: [0x10], // ┴
    replacement: "┐",
    description: "Top-right corner",
  },
  {
    bytes: [0x14, 0x00], // └
    replacement: "└",
    description: "Bottom-left corner",
  },
  {
    bytes: [0x14], // ─
    replacement: "─",
    description: "Em-dash",
  },
] as const


// ─────────────────────────────────────────────────────────────────────────────
// Core Fix Functions (Pure, Testable)
// ─────────────────────────────────────────────────────────────────────────────

/**
 * Convert byte sequence to Buffer-compatible format for matching
 */
function bytesToBuffer(bytes: readonly number[]): Buffer {
  return Buffer.from(bytes)
}

/**
 * Replace null byte sequences with proper box-drawing characters.
 * Pure function - returns new buffer, doesn't modify input.
 */
function fixNullBytes(content: Buffer): Buffer {
  let result = content

  // Sort patterns by length (longest first) to avoid partial matches
  const sortedPatterns = [...BOX_CHAR_PATTERNS].sort((a, b) => b.bytes.length - a.bytes.length)

  for (const pattern of sortedPatterns) {
    const needle = bytesToBuffer(pattern.bytes)
    const replacement = Buffer.from(pattern.replacement, "utf-8")

    // Replace all occurrences
    let position = 0
    while ((position = result.indexOf(needle, position)) !== -1) {
      result = Buffer.concat([
        result.subarray(0, position),
        replacement,
        result.subarray(position + needle.length),
      ])
      position += replacement.length
    }
  }

  return result
}

/**
 * Normalize box widths - each box's horizontal lines match its longest content line.
 * Pure function - returns new string with normalized boxes.
 */
function normalizeBoxWidths(content: string): string {
  const lines = content.split("\n")
  const result: string[] = []
  let boxStart = -1
  let maxWidth = 0

  for (let i = 0; i < lines.length; i++) {
    const line = lines[i]
    const hasBoxChars = /[│├┤┌┐└┘─]/.test(line)

    // Start of a box
    if (hasBoxChars && boxStart === -1) {
      boxStart = i
      maxWidth = line.length
    }
    // Inside a box
    else if (hasBoxChars && boxStart !== -1) {
      maxWidth = Math.max(maxWidth, line.length)
    }
    // End of a box (non-box line after box content)
    else if (!hasBoxChars && boxStart !== -1) {
      // Normalize the box we just finished
      for (let j = boxStart; j < i; j++) {
        result.push(normalizeBoxLine(lines[j], maxWidth))
      }
      boxStart = -1
      maxWidth = 0
      result.push(line) // Add the non-box line
    }
    // Outside any box
    else {
      result.push(line)
    }
  }

  // Handle box that extends to end of file
  if (boxStart !== -1) {
    for (let j = boxStart; j < lines.length; j++) {
      result.push(normalizeBoxLine(lines[j], maxWidth))
    }
  }

  return result.join("\n")
}

/**
 * Normalize a single box line to the target width.
 * Extends horizontal lines (─) and adjusts right edge characters.
 */
function normalizeBoxLine(line: string, targetWidth: number): string {
  // Lines with horizontal bars that need extending
  if (/^[┌├└]─+[┐┤┘]?$/.test(line)) {
    const leftChar = line[0]
    const rightChar = line[line.length - 1]
    const isRightEdge = /[┐┤┘]/.test(rightChar)

    if (isRightEdge) {
      // Left char + horizontal bars + right char
      const barCount = targetWidth - 2
      return leftChar + "─".repeat(barCount) + rightChar
    } else {
      // Left char + horizontal bars (no right edge yet)
      const barCount = targetWidth - 1
      return leftChar + "─".repeat(barCount)
    }
  }

  // Content lines with │ edges - pad the content
  if (/^│.*│?$/.test(line)) {
    const hasRightEdge = line.endsWith("│")
    if (hasRightEdge) {
      const content = line.slice(1, -1) // Remove both │
      const paddedContent = content.padEnd(targetWidth - 2, " ")
      return "│" + paddedContent + "│"
    } else {
      const content = line.slice(1) // Remove left │
      const paddedContent = content.padEnd(targetWidth - 1, " ")
      return "│" + paddedContent
    }
  }

  // Return line as-is if it doesn't match known box patterns
  return line
}

/**
 * Analyze content and report what would be fixed (for dry-run mode).
 * Returns human-readable statistics.
 */
interface FixStats {
  readonly patternCounts: ReadonlyMap<string, number>
  readonly totalReplacements: number
}

function analyzeNullBytes(content: Buffer): FixStats {
  const counts = new Map<string, number>()
  let total = 0

  // Sort patterns by length (longest first) to avoid partial matches
  const sortedPatterns = [...BOX_CHAR_PATTERNS].sort((a, b) => b.bytes.length - a.bytes.length)

  for (const pattern of sortedPatterns) {
    const needle = bytesToBuffer(pattern.bytes)
    let position = 0
    let count = 0

    while ((position = content.indexOf(needle, position)) !== -1) {
      count++
      position += needle.length
    }

    if (count > 0) {
      counts.set(pattern.description, count)
      total += count
    }
  }

  return {
    patternCounts: counts,
    totalReplacements: total,
  }
}

// ─────────────────────────────────────────────────────────────────────────────
// CLI Formatting
// ─────────────────────────────────────────────────────────────────────────────

const COLORS = {
  reset: "\x1b[0m",
  bold: "\x1b[1m",
  dim: "\x1b[2m",
  green: "\x1b[32m",
  yellow: "\x1b[33m",
  cyan: "\x1b[36m",
  red: "\x1b[31m",
} as const

function formatStats(stats: FixStats, fileRelative: string): string {
  if (stats.totalReplacements === 0) {
    return `${COLORS.green}✓ ${fileRelative}${COLORS.reset} - No null bytes found\n`
  }

  const lines = [
    `${COLORS.cyan}◆ ${fileRelative}${COLORS.reset}`,
    `  ${COLORS.bold}Total replacements:${COLORS.reset} ${stats.totalReplacements}`,
    "",
    `  ${COLORS.dim}Pattern breakdown:${COLORS.reset}`,
  ]

  for (const [description, count] of stats.patternCounts) {
    lines.push(`    ${description}: ${count}`)
  }

  lines.push("")

  return lines.join("\n")
}

function printUsage() {
  console.error(`
Fix Markdown Null Bytes - Replace null byte sequences with box-drawing characters

Usage:
  bun scripts/workspace/fix-markdown-nulls.ts <input-file>
  bun scripts/workspace/fix-markdown-nulls.ts <input-file> --dry-run

Options:
  --dry-run    Show what would be fixed without modifying the file

Exit codes:
  0: Success (fixes applied or dry-run completed)
  1: Error (file not found, read/write failure)
  2: Usage error
`)
}

// ─────────────────────────────────────────────────────────────────────────────
// Main
// ─────────────────────────────────────────────────────────────────────────────

async function main() {
  const args = process.argv.slice(2)

  if (args.length === 0) {
    console.error("Error: No input file specified\n")
    printUsage()
    process.exit(2)
  }

  const filePath = args[0]
  const isDryRun = args.includes("--dry-run")
  const cwd = process.cwd()
  const fileRelative = relative(cwd, filePath)

  // Read file as binary buffer to preserve exact bytes
  let content: Buffer
  try {
    content = readFileSync(filePath)
  } catch (err) {
    const message = err instanceof Error ? err.message : String(err)
    console.error(`${COLORS.red}✗ Error reading file:${COLORS.reset} ${message}`)
    process.exit(1)
  }

  // Analyze content
  const stats = analyzeNullBytes(content)

  if (isDryRun) {
    console.log(`${COLORS.bold}DRY RUN MODE${COLORS.reset} - No changes will be made\n`)
    console.log(formatStats(stats, fileRelative))

    if (stats.totalReplacements > 0) {
      console.log(
        `${COLORS.yellow}⚠ Run without --dry-run to apply fixes${COLORS.reset}`
      )
    }

    process.exit(0)
  }

  // Apply fixes
  if (stats.totalReplacements === 0) {
    console.log(formatStats(stats, fileRelative))
    process.exit(0)
  }

  // Step 1: Replace null bytes with box characters
  const fixedBytes = fixNullBytes(content)

  // Step 2: Normalize box widths
  const fixedString = fixedBytes.toString("utf-8")
  const normalized = normalizeBoxWidths(fixedString)

  // Write fixed content
  try {
    writeFileSync(filePath, normalized, "utf-8")
  } catch (err) {
    const message = err instanceof Error ? err.message : String(err)
    console.error(`${COLORS.red}✗ Error writing file:${COLORS.reset} ${message}`)
    process.exit(1)
  }

  console.log(formatStats(stats, fileRelative))
  console.log(`${COLORS.green}✓ Fixes applied (null bytes replaced + boxes normalized)${COLORS.reset}`)

  process.exit(0)
}

main().catch((err: unknown) => {
  const message = err instanceof Error ? err.message : String(err)
  console.error(`${COLORS.red}Fatal error:${COLORS.reset} ${message}`)
  process.exit(1)
})

This issue has been automatically closed due to 60 days of inactivity. If you're still experiencing this issue, please open a new issue with updated information.

This issue was closed incorrectly despite recent human comments. This behavior of the bot is reported at https://github.com/anthropics/claude-code/issues/16497. Upvote or comment on that issue, so maybe it gets noticed.

This issue has been automatically locked since it was closed and has not had any activity for 7 days. If you're experiencing a similar issue, please file a new issue and reference this one if it's relevant.

This issue has been automatically locked since it was closed and has not had any activity for 7 days. If you're experiencing a similar issue, please file a new issue and reference this one if it's relevant.