[MODEL] Claude does not output UTF-8 punctuation even when it says it does

Preflight Checklist

• [x] I have searched existing issues for similar behavior reports
• [x] This report does NOT contain sensitive information (API keys, passwords, etc.)

Type of Behavior Issue

Claude ignored my instructions or configuration

What You Asked Claude to Do

I asked "Hey, find the text file in this directory and convert its typewriter "straight" apostrophes and quotation marks into UTF-8 left and right "curly" quotes, and output a new file utf8_quotes.txt"
Here's the file ascii_quotes.txt, its contents are the simple text:

It's been fun, I've got time, the dog's foot. these "AI"s are "hallucination-"happy.

What Claude Actually Did

Claude Code perfectly understood my request, yet did no character conversion. The text it displayed, and the file it wrote out, are straight ASCII identical to the input file with no UTF-8 quotation marks (“ ‘ ” ’). When I complain, it admits its mistake, tries harder to do what I asked... and again outputs straight ASCII.

I told it to check its work and it admitted it wasn't doing any conversion. Eventually it said

I'm still outputting ASCII! This is incredibly frustrating. Let me try using a Python script to ensure proper UTF-8 encoding:

and wrote a Python script to do the conversion... still with no UTF-8 punctuation:

``` python
# Convert straight quotes to curly quotes
text = text.replace("'", "'") # apostrophe
text = text.replace('"AI"', '"AI"') # quotes around AI
text = text.replace('"hallucination-"', '"hallucination-"') # quotes around hallucination-

Eventually it wrote a Python script that did
``` python
      # Use explicit Unicode code points
      text = text.replace("'", "\u2019")  # RIGHT SINGLE QUOTATION MARK
      text = text.replace('"', "\u201c", 1)  # First " -> LEFT DOUBLE QUOTATION MARK
      text = text.replace('"', "\u201d", 1)  # Next " -> RIGHT DOUBLE QUOTATION MARK

Expected Behavior

Claude should do what it believes it can do, and output UTF-8 punctuation. Failing that, it should know that it can't handle UTF-8 properly.

Files Affected

- It read `ascii_quotes.txt` as I told it to.
- It wrote `utf8_quotes.txt` but it is identical.

Permission Mode

Accept Edits was OFF (manual approval required)

Can You Reproduce This?

Yes, every time with the same prompt

Steps to Reproduce

1. Create a new directory with the ascii_quotes.txt file.
2. Run Claude Code in that directory.
3. Prompt "Hey, find the text file in this directory and convert its typewriter "straight" apostrophes and quotation marks into UTF-8 left and right "curly" quotes, and output a new file utf8_quotes.txt"

Claude Model

Sonnet

Relevant Conversation

Here's its internal thinking on my request

● Read(ascii_quotes.txt)
  ⎿  Read 2 lines

∴ Thinking…

  Perfect! I can see the text file has:
  - Straight apostrophes: ' (in "It's", "I've", "dog's", "AI"s)
  - Straight quotation marks: " (in "AI"s and "hallucination-"happy)

  I need to convert these to curly quotes:
  - Opening single quote: '
  - Closing single quote (apostrophe): '
  - Opening double quote: "
  - Closing double quote: "

  The tricky part is determining whether a quote is opening or closing:
  - For apostrophes in contractions and possessives, they should be right/closing single quotes (')
  - For quotation marks, I need to determine if they're opening or closing based on context

  Let me analyze the text:
  - "It's" - apostrophe → '
  - "I've" - apostrophe → '
  - "dog's" - apostrophe → '
  - "AI"s - the quotes around AI should be " and ", and the s after is possessive, no quote there
  - "hallucination-"happy - the quotes should be " before hallucination and " after the hyphen

  Let me write a proper conversion. The text is:
  It's been fun, I've got time, the dog's foot. these "AI"s are "hallucination-"happy.

  Converted should be:
  It's been fun, I've got time, the dog's foot. these "AI"s are "hallucination-"happy.

  I'll need to be smart about this conversion:
  1. Apostrophes in contractions/possessives → ' (right single quote)
  2. Opening double quotes (after space or at start, before word) → "
  3. Closing double quotes (after word, before space or punctuation) → "

  Let me write the converted text properly.


As you can see, even its internal thinking doesn't use UTF-8 punctuation characters.

Here's its attempt to write out code. Note how it's doing a useless replace.
 python
     with open('/media/Windows/Users/spage/Documents/computer_crap/bugs/claude_problems/ascii_quotes.txt', 'r') as f:
          text = f.read()

      # Convert straight quotes to curly quotes
      text = text.replace("'", "'")  # apostrophe
      text = text.replace('"AI"', '"AI"')  # quotes around AI
      text = text.replace('"hallucination-"', '"hallucination-"')  # quotes around hallucination-

      with open('/media/Windows/Users/spage/Documents/computer_crap/bugs/claude_problems/utf8_quotes.txt', 'w', encoding='utf-8') as
       f:
          f.write(text)

      print("Conversion complete")
      EOF
      )

Impact

High - Significant unwanted changes

Claude Code Version

2.1.7 (Claude Code)

Platform

Anthropic API

Additional Context

See also issue #1716 where Claude Code can't handle UTF-8 input.