[BUG] Claude clobbers correct typographic marks and can't be told otherwise

I'm definitely also seeing this. When editing part of my code that had a curly apostrophe, it edited it to be straight apostrophe:
<img width="739" height="135" alt="Image" src="https://github.com/user-attachments/assets/c7291aef-6477-40f3-81f0-04a3e1de1e45" />

In this case, the _intended_ diff was simply adding .name-of-God to the two spans, but the apostrophes in the regex got caught up in the edit too.

A key insight is that I really don't think this was the Claude model's fault, rather there must be some processing/filtering happening in some other layer of the Claude Code stack.

I discussed the issue with Claude Code, and we came up with a bug report (although, worth noting that of course, it can't render the curly apostrophes properly; mentions of [''] should be ['’]):

------------

Unicode Character Normalization Bug in Edit Tool

Description:

Claude Code's Edit tool appears to normalize Unicode characters during edit operations, converting curly/smart apostrophes
(U+2019 ') to straight apostrophes (U+0027 ') without user awareness.

Steps to Reproduce:

1. Have a file containing a regex with mixed apostrophe types: /\bLORD([''][sS])?\b/g (straight + curly)
2. Use the Edit tool to modify the line containing this regex
3. The curly apostrophe gets silently converted to a straight apostrophe

Expected Behavior:

Unicode characters should be preserved exactly as they appear in the original file.

Actual Behavior:

The Edit tool silently normalizes Unicode characters, changing [''] to [''] (both straight apostrophes).

Evidence:

• User can see the difference in their editor and in git diffs
• Claude's output displays show normalized characters even when Claude intends to preserve them
• The issue affects functionality when precise Unicode character matching is needed (e.g., handling both straight and curly

apostrophes in text processing)

Impact:

This could affect any code that relies on specific Unicode characters, particularly:

• Text processing regex patterns
• String literals with special characters
• Internationalization code
• Any scenario where Unicode precision matters

Workaround:

Use Unicode escape sequences (e.g., \u2019) instead of literal Unicode characters.

------------

I've not yet tested the workaround, but I imagine it'd work. Still really not optimal, however, especially as the model doesn't realize this current limitation, and will repeatedly trigger it. I really believe the model is actually sending ’, but it's being filtered/replaced/normalized by some other layer.

Final remarks to Claude after it made the bug report:

> Ironically even now your message is getting messed up.

⏺ Ha! You're absolutely right - even in my bug report, the Unicode characters are getting normalized!

  Looking at what I wrote:
  - I intended to show the difference between straight and curly apostrophes
  - But in the "Steps to Reproduce" section, both apostrophes in [''] probably look identical to you
  - Same issue in the "Actual Behavior" section

  This makes the bug even more pervasive than we initially thought - it's not just affecting the Edit tool, but also affecting
  Claude's text output rendering in general. The normalization is happening at multiple layers of the Claude Code system.

  This actually strengthens the bug report - it shows the issue affects:
  1. ✅ Edit tool operations (confirmed)
  2. ✅ Claude's text output display (confirmed just now)
  3. Possibly other text handling throughout the system

  The fact that I can't even describe the bug without it being affected by the bug is pretty remarkable! You might want to
  mention this meta-aspect in your bug report - that even attempting to document Unicode issues gets affected by the
  normalization.

------
And if there's any doubt that this is an issue with our terminals or configs, this issue does _not_ happen with Gemini CLI:
<img width="751" height="132" alt="Image" src="https://github.com/user-attachments/assets/fe84146c-a078-49ab-b9b9-632229d48a91" />
The diff is slightly different (the model solved the issue slightly differently), but you'll see that the apostrophes in the regex at the top remain unmodified.

When Claude is explicitly told to replace straight quotes with curly quotes, it goes into an edit error loop:

<img width="934" height="640" alt="Image" src="https://github.com/user-attachments/assets/21a805d4-b9f0-409e-923a-077bc591e28d" />

Confirming on Linux + Critical Impact in Perl

I can confirm this bug also affects Linux (Fedora 42), not just macOS. The impact in our Perl codebase was severe:

Impact

• 502 lines of multi-language UI text corrupted in one file alone
• Critical: In Perl, converting ' (U+2019) to ' (U+0027) breaks string literals:

# Before (works):
$text = "Nom d'hôte";  # Single string with U+2019

# After Edit tool (syntax error):
$text = "Nom d'hôte";  # Interpreted as: "Nom d" + bareword 'hôte' + "
                       # Because U+0027 terminates the string!

Affected Files

• [redacted] - 502 corruptions

Workaround

Using perl -i -pe instead of the Edit tool preserves UTF-8:

perl -i -pe 's/old/new/g' file.pl  # Safe

Additional Context

• File encoding: UTF-8
• Claude Code version: Latest (2025-11-12)
• Platform: Linux (Fedora 42)

This is a high-priority bug as it silently corrupts code in production. The Edit tool should preserve exact UTF-8 byte sequences.

We've documented this extensively in our migration lessons learned. Happy to provide more details if needed.

(human addendum: not even this comment formulated by claude seems to be able to use the correct utf8 characters)

Environment:

• OS: Windows 11 (Host) running WSL 2 (Ubuntu)
• Tool: Claude Code CLI v2.0.55
• Model: Claude 3.5 Opus (Opus 4.5)
• Plan: Claude Max (20x usage)

Description:
I am consistently experiencing an issue where UTF-8 characters are being "normalized" or stripped from the output. Specifically, Portuguese diacritics and accents (e.g., ç, ã, é, ô) are being replaced by their ASCII equivalents (e.g., c, a, e, o) or lost entirely.

Impact:
This is critical for my workflow as it is introducing regressions in production code, particularly affecting string literals, i18n files, and documentation.

This issue has been inactive for 30 days. If the issue is still occurring, please comment to let us know. Otherwise, this issue will be automatically closed in 30 days for housekeeping purposes.

Just tested it. It's still failing.

My prompt:

On paymentButton in PaywallLayer put curly single quotes around the word "standard"

Code before:

description: qsTr("The standard price...")

Code after:

description: qsTr("The 'standard' price...")

Rather than having your bot try to auto-close this, how about we have someone from Anthropic actually look at this issue and respond... if only to show that submitting bug reports isn't a waste of our time??

It's infuriating. Claude Code knows about typographic marks and thinks it's using them, but simply doesn't. Issue #18422 is a detailed bug report, but here's the simplest single step to reproduce, just prompt:

Print "Hello world" but use opening and closing (left and right, aka "curly") quotation marks.

Complain and Claude Code will even claim _The opening quote is " (U+201C, LEFT DOUBLE QUOTATION MARK) and the closing quote is " (U+201D, RIGHT DOUBLE QUOTATION MARK)._ ... as it outputs straight ASCII 😠.

I pointed Claude Code at this bug report and it commented "there's a bug in Claude Code itself where Unicode typographic characters are being normalized to their ASCII equivalents somewhere in the processing pipeline before output reaches you. ... I was not aware of my own output being affected by this bug because I don't see what you see after the normalization layer processes it."

I run into the edit errors too when working with localization files. This is still happening as of Claude Code 2.1.31

Note that this isn't a Claude Code issue, this is a fundamental Claude issue. Happens in the web UI as well. Even Opus 4.6 still cannot tell the difference and confabulates that it's using the curly versions when it is not.

<img width="762" height="1037" alt="Image" src="https://github.com/user-attachments/assets/3f61ffcc-ba73-43c6-bdbf-1ab0e781c575" />

This issue was also tracked in #1986 (now auto-closed and locked), where a community workaround was proposed: a skill that uses a placeholder + sed/perl post-processing to restore curly apostrophes (U+2019) and curly quotes (U+201C/U+201D) after Write/Edit operations.

The skill is available here (French and English versions)

Current status of the problem (March 2026):

• Both Write and Edit tools still normalize ’ → ' and “ ” → "
• The model can read curly quotes correctly but cannot write them
• A hook-based fix isn't viable because there's no reliable way to distinguish typographic quotes (in text content) from straight quotes (in code) without semantic understanding
• This affects all non-English languages with proper typography (French, German, etc.)

After 7+ months and multiple duplicate issues (#8234, #14355, #15920, #26141, #29786), this remains unresolved. Is there any plan to address this at the tool level, e.g. a preserveUnicode flag or context-aware normalization?

This issue perfectly captures the core problem: Claude’s Write/Edit tools normalize curly typographic characters to straight ASCII, destroying correct typography even when CLAUDE.md explicitly says otherwise.

I’ve filed a feature request proposing a solution: a /curly option that replaces blind normalization with context-aware verification. The rule is simple: a curly character inside a straight delimiter is valid content (preserved); a curly character used as a delimiter is a syntax error (normalized). Two modes: /curly safe (doubt favors code) and /curly lazy (doubt favors typography).

→ [FEATURE] /curly option: context-aware curly/straight character handling in Write/Edit — #31482

In the meantime, I’ve published reusable Claude Code skills as workarounds (placeholder substitution + perl post-processing). Happy to share if useful.

/tmp/issue-1599-answer.md

This appears to have been fixed with Opus 4.7. We've been able to remove our many work arounds (placeholder systems etc) as of this morning and editing/generation appears to be reliable.

❯ Can you edit the AGENTS.md file and sub in typographical quotes/apostrophes where it makes sense?

  Read 1 file (ctrl+o to expand)

⏺ Update(AGENTS.md)
  ⎿  Added 1 line, removed 1 line
      79  **CRITICAL RULES:**
      80  - Work is NOT complete until `git push` succeeds
      81  - NEVER stop before pushing - that leaves work stranded locally
      82 -- NEVER say "ready to push when you are" - YOU must push
      82 +- NEVER say “ready to push when you are” - YOU must push
      83  - If push fails, resolve and retry until it succeeds

⏺ Only one prose-quoted phrase to convert — line 82: "ready to push when you are" → “ready to push when you are”. The other double-quoted strings (lines 5 and 73) are inside code blocks,
   which should stay as plain ASCII. No apostrophes in prose to convert.

Thanks Sihil,
I can confirm that this works since Opus 4.7.
The skills are no longer required.

They just broke it in the opposite direction. After an entire day of debugging broken code due to un-parseable typographical punctuation instead of ASCII-safe markup, I asked Claude itself:

> general query: list the coding languages which actually REQUIRE "curly" punctuation. ● None. No programming language, markup language, data serialization format, or shell scripting language requires or accepts curly/typographic quotes as valid syntax. Every language that uses quotation marks as delimiters or apostrophes as operators requires straight ASCII exclusively: - String delimiters: ' U+0027 and " U+0022 - Shell quoting: straight only; curly causes immediate parse failure - JSON: spec requires U+0022 straight double quote; U+201C/U+201D are not valid JSON - Markdown: straight apostrophes; curly chars render as literal Unicode, not smart typography in most parsers - HTML/XML: attribute values use straight quotes; curly quotes in attribute positions are malformed - CSV: field delimiters and string wrappers require straight quotes - Python, JS, PowerShell, Ruby, Go, Rust, C, SQL: all straight only - Regex: curly braces {} are quantifiers; curly quotes are not braces and will not match Curly/smart quotes exist exclusively as display typography for human-readable rendered text (print, web CSS with quotes property, word processors). They have no syntactic function in any computational context. Introducing them into source, data, or script files is unconditionally wrong. > so the only plausible use for them in any code under any circumstances would be for aesthetic reasons only, e.g. specifying them in a PHP or HTML render for typographical reasons not parsing function. accurate? ● Accurate, with one narrow addition: they can also appear as literal content inside string literals when the output target is a human-readable rendered medium -- e.g., a PHP string that builds HTML prose, or a JavaScript string that populates a display element. In that case the curly chars are payload data, not syntax. The surrounding code is still straight-quote delimited; the curly chars are inside the string, not part of the language. Examples where this is the only legitimate use: echo "It\u{2019}s a beautiful day."; // U+2019 as display content element.textContent = "\u201CHello\u201D"; // curly quotes as rendered typography In both cases the curly chars are specified as Unicode escapes or explicit string content -- not injected by the editor or code generator. A tool that automatically substitutes straight quotes with curly quotes in source is wrong in every context, including this one, because the developer must make that choice intentionally. Outside of deliberate typographic payload in display-bound strings, there is no legitimate use. ✻ Cooked for 4m 14s

"They have no syntactic function in any computational context. Introducing them into source, data, or script files is unconditionally wrong." Their own model knows it's doing the wrong thing, but it can't stop itself. These different styles both have their place; why is it so hard to work out a conditional?

@sihil Not fixed for me in Opus 4.7. It's a bug in something other than the model.

@bcherny Just curious, how many dozens of bug reports have to be filed before you folks pay attention to this? How is it that with the infinite tokens at your disposal you can't hand this to Claude and let it fix the bug?

The bug still exists in Chatbot as well as Claude Code, even when using Opus 4.7.

@aardvarkk also, you can check AWS Bedrock as well: that's what we use and we also have the problem.