[Bug] File loading adds 70% token overhead due to line number formatting

Bug Description

Claude Code Context Overhead Issue Report

Date: 2026-01-22
Claude Code Version: VSCode Extension + CLI
Model: claude-sonnet-4-5-20250929

Summary

Claude Code's file loading mechanism (both @ syntax and Read tool) adds approximately 70% token overhead due to line number formatting. Combined with an undocumented 2000-line truncation limit, this significantly impacts context efficiency and causes unexpected behavior.

Problem Statement

Issue 1: Excessive Token Overhead (70% waste)

When files are loaded into context via @filename syntax or the Read tool, Claude Code adds line numbers and formatting that consumes 1.7x the raw token count.

Impact:

• A 31,000 token documentation set consumes 52,700 tokens (21,700 tokens wasted on formatting)
• Users hit context limits much faster than expected
• Context cost is significantly higher than necessary

Issue 2: Undocumented 2000-Line Truncation

Files loaded via @ syntax are silently truncated at 2000 lines without clear warning to the user.

Impact:

• Users believe they've loaded complete files when they haven't
• Critical information beyond line 2000 is silently omitted
• No error message or warning is displayed
• This matches the Read tool's default limit but is not documented for @ syntax

Reproduction & Evidence

Testing Environment:

• ✅ Claude Code CLI (Terminal)
• ✅ Claude Code VSCode Extension
• Both environments exhibited identical behavior - this is not VSCode-specific

Test Setup

We created controlled test files with known token counts and measured the actual context consumption.

Test files created:

# File with 1000 lines of numbers (1\n2\n3\n...)
test_1000_numbers.txt: 2,001 tokens (raw)

# File with 5000 lines of code (const value1 = 1;)
test_5000_short.js: 43,008 tokens (raw)

Test 1: Medium File (1000 lines)

Command: @ claude-token-test/test_1000_numbers.txt

Results:

• Raw tokens: 2,001
• Messages category: 6,400 tokens
• Multiplier: 3.2x ⚠️

Note: This 3.2x includes both formatting overhead AND conversation overhead (Claude's response). The pure file overhead is ~1.7x.

Test 2: Large File (5000 lines)

Command: @ claude-token-test/test_5000_short.js

Results:

• Full file: 43,008 tokens (5000 lines)
• Only first 2000 lines loaded: 16,002 tokens (raw)
• Messages category: 27,300 tokens
• Multiplier: 1.7x ✓
• 63% of file was silently truncated ⚠️

Claude's response confirmed: "I'm seeing the first 2000 lines"

Test 3: Real-World Documentation

6 documentation files loaded via @ syntax:

| File | Lines | Raw Tokens |
|------|-------|------------|
| docs/01-overview/01-project-overview.md | 191 | 1,871 |
| docs/00-current-state/project-status.md | 578 | 5,142 |
| docs/00-current-state/developer-onboarding.md | 718 | 4,683 |
| docs/00-current-state/technical-inventory.md | 572 | 4,350 |
| tasks/conversation-layer--overview.md | 638 | 4,501 |
| tasks/conversation-layer-1--cli.md | 1232 | 10,543 |
| Total | 3,929 | 31,090 |

Results:

• Expected context usage (raw): 31,090 tokens
• Actual Messages category: 54,400 tokens
• Overhead: 23,310 tokens (75% waste)
• Multiplier: 1.75x

Test 4: Read Tool vs @ Syntax Comparison

Both mechanisms exhibit similar overhead:

| Method | File | Raw Tokens | Context Tokens | Multiplier |
|--------|------|------------|----------------|------------|
| Read tool | test_1000_numbers.txt | 2,001 | ~6,300 | 3.15x |
| @ syntax | test_1000_numbers.txt | 2,001 | ~6,400 | 3.2x |
| @ syntax | test_5000_short.js (truncated) | 16,002 | 27,300 | 1.7x |

Note: The 3.x multipliers include conversation overhead. Pure file formatting overhead is consistently ~1.7x.

Root Cause Analysis

Line Number Formatting

The Read tool output includes line numbers in this format:

     1→1
     2→2
     3→3
   ...
  1000→1000

Each line contains:

• Padding spaces (6 characters for alignment)
• Line number (1-4+ digits)
• Arrow separator →
• Original content
• Newline character

This formatting adds approximately 70% overhead to the raw file content.

Evidence from GitHub Issues

A reported bug documented similar findings:

"A file with 20,010 lines containing just line numbers consumed 59,038 tokens"

This represents a 2.95x multiplier, consistent with our findings.

Impact on Users

1. Context Exhaustion

Users hit the 200k token limit much faster than expected:

• 6 documentation files (31k raw) consume 52k tokens with overhead
• Reduces effective context window by ~40%
• Auto-compact triggers earlier and more frequently

2. Unexpected Costs

For API usage:

• Input tokens are 1.7…

Note: Content was truncated.