JSON Serialization Failure: Unicode Surrogate Pair Error in API Requests

Bug Description

Bug Report: Persistent JSON Serialization Error - "no low surrogate in string"

Issue Summary

Claude Code repeatedly fails with API Error 400 due to invalid JSON serialization, making the tool unusable for extended sessions. The error persists across multiple sessions and reinstalls.

Error Message

⎿ API Error: 400 
{"type":"error","error":{"type":"invalid_request_error","message":"The 
request body is not valid JSON: no low surrogate in string: line 1 
column 158990 (char 158989)"}}

Environment

• Claude Code Version: 1.0.71 and 1.0.72 (both affected)
• OS: macOS Darwin 24.5.0 (Apple Silicon)
• Node Version: (run node --version to check)
• Installation Path: /Users/Viren/.npm-global
• Locale: en_IN.UTF-8
• Date First Observed: August 8, 2025
• Frequency: Constant - occurs in every session after ~5-10 minutes of use

Reproduction Steps

1. Start Claude Code: claude
2. Login successfully: /login
3. Work normally for 5-10 minutes
4. Attempt to send any message (even simple text without special characters)
5. Receive the "no low surrogate" error
6. Session becomes completely unusable
7. Error persists even after:

• Restarting Claude Code
• Running /login again
• Clearing cache with rm -rf ~/.claude/
• Reinstalling Claude Code

What I've Tried (All Failed)

1. ✗ Removed emojis from all project files
2. ✗ Checked file encodings (all UTF-8)
3. ✗ Cleared Claude cache: rm -rf ~/.claude/
4. ✗ Reinstalled Claude Code multiple times
5. ✗ Downgraded from 1.0.72 to 1.0.70
6. ✗ Set LC_ALL environment variable
7. ✗ Removed corrupted __store.db file
8. ✗ Cleared npm cache

Additional Context

• The error occurs even with plain text messages (no emojis or special characters)
• API key authentication initially works but fails to persist
• The __store.db file wasn't updating (stuck at May 9th timestamp)
• Error has caused 5-6 session failures in 24 hours
• Working directories include: /Users/Viren/Desktop/CRM, /Users/Viren/Desktop/VMS, /Users/Viren/Desktop/CodeWing

Impact

CRITICAL - Tool is completely unusable. Cannot maintain any productive session longer than 5-10 minutes. This is blocking all development work.

Expected Behavior

Claude Code should properly serialize all JSON requests without Unicode/surrogate pair errors, especially for plain ASCII text.

Actual Behavior

Any message after ~5-10 minutes of session time triggers the JSON serialization error, regardless of content.

Diagnostic Information

# Claude doctor output shows:
ERROR Raw mode is not supported on the current process.stdin

# Locale settings:
LANG=en_IN.UTF-8
LC_ALL=(not set)

# Installation check:
/Users/Viren/.npm-global/lib
└── @anthropic-ai/claude-code@1.0.72

Suggested Fix

The issue appears to be in the JSON serialization layer where the conversation context accumulates malformed Unicode characters over time. Possible solutions:

1. Sanitize all strings before JSON serialization
2. Implement proper Unicode surrogate pair handling
3. Add context size limits or periodic context cleanup
4. Fix the persistent storage mechanism for API keys

Related Issues

This may be related to:

• Session state corruption
• API key persistence failures
• Context accumulation without proper cleanup

---
Please investigate urgently as this makes Claude Code completely unusable for any serious development work.

Environment Info

• Platform: darwin
• Terminal: Apple_Terminal
• Version: 1.0.70
• Feedback ID: 5fd72b69-40f3-441e-bdf4-ea78d3e0441a

Errors

[{"error":"Error: ENOENT: no such file or directory, open '/Users/Viren/.claude/statsig/statsig.session_id.2656274335'\n    at writeFileSync (node:fs:2409:20)\n    at qi1.setItem (file:///Users/Viren/.npm-global/lib/node_modules/@anthropic-ai/claude-code/cli.js:686:6696)\n    at Object.setItem (file:///Users/Viren/.npm-global/lib/node_modules/@anthropic-ai/claude-code/cli.js:61:18795)\n    at Object.zy9 [as _setObjectInStorage] (file:///Users/Viren/.npm-global/lib/node_modules/@anthropic-ai/claude-code/cli.js:61:19102)\n    at Lk9 (file:///Users/Viren/.npm-global/lib/node_modules/@anthropic-ai/claude-code/cli.js:61:43604)\n    at $k9 (file:///Users/Viren/.npm-global/lib/node_modules/@anthropic-ai/claude-code/cli.js:61:43061)\n    at Object.get (file:///Users/Viren/.npm-global/lib/node_modules/@anthropic-ai/claude-code/cli.js:61:42666)\n    at Object.get (file:///Users/Viren/.npm-global/lib/node_modules/@anthropic-ai/claude-code/cli.js:61:42562)\n    at Ho0.<anonymous> (file:///Users/Viren/.npm-global/lib/node_modules/@anthropic-ai/claude-code/cli.js:61:48464)\n    at Generator.next (<anonymous>)","timestamp":"2025-08-09T09:36:24.737Z"},{"error":"Error: ENOENT: no such file or directory, open '/Users/Viren/.claude/statsig/statsig.session_id.2656274335'\n    at writeFileSync (node:fs:2409:20)\n    at qi1.setItem (file:///Users/V

Note: Error logs were truncated.