[FEATURE] /tokens command - expose free count_tokens API for pre-flight file size checks

Preflight Checklist

• [x] I have searched existing requests and this feature hasn't been requested yet
• [x] This is a single feature request (not multiple features)

Note: #2529 requested similar functionality but was closed before the free /v1/messages/count_tokens API endpoint existed. The landscape has changed.

Problem Statement

Claude Code has hard limits that users cannot predict in advance:

• Read tool: 25,000 token limit
• Bash output: ~30KB truncation

Users discover these limits only after hitting them:

File content (47382 tokens) exceeds maximum allowed tokens (25000).

There is no way to check file sizes in advance, leading to failed operations and wasted context.

Why External Estimation Does Not Work

Anthropic does not publish a tokenizer library (unlike OpenAI's tiktoken). Third-party estimates are unreliable. In testing across multiple files, the ratio between common estimators and Claude's actual token count varied significantly:

| Estimator | Claude Actual | Ratio |
|-----------|---------------|-------|
| ~23,000 (tiktoken) | 40,047 | 1.77x |
| ~22,000 (tiktoken) | 34,956 | 1.57x |
| ~23,000 (tiktoken) | 49,720 | 2.08x |

The variance (1.57x - 2.08x) makes pre-flight estimation unreliable.

Proposed Solution

Add a /tokens slash command:

/tokens myfile.json

Output:

myfile.json: 47,382 tokens
[!] Exceeds Read tool limit (25,000 tokens)

For multiple files:

/tokens src/*.rs

Output:

src/main.rs: 1,203 tokens [OK]
src/lib.rs: 8,847 tokens [OK]
src/parser.rs: 31,205 tokens [!] Exceeds limit

Implementation would call the existing /v1/messages/count_tokens API endpoint, which is free (rate-limited only, no token cost).

Alternative Solutions / Current Workarounds

The only accurate workaround today requires users to:

1. Create a separate Anthropic API account
2. Add a payment method to that account
3. Call the free /v1/messages/count_tokens endpoint externally
4. Use those counts to inform Claude Code usage

This is unreasonable because:

• Max subscribers already have a payment method on file with Anthropic
• The endpoint is free (no token cost)
• Claude Code already uses the same underlying tokenizer internally

Requiring a separate API account with separate billing setup to access a free endpoint - when users are already paying $20-200/month for Max - should not be necessary.

Priority

High - This blocks reliable use of Claude Code for any workflow involving files that may exceed limits.

Feature Category

Core CLI functionality / Developer experience

Use Case Example

1. User has a JSON file they need Claude to process
2. User runs /tokens data.json
3. Output shows data.json: 31,205 tokens [!] Exceeds Read limit (25,000)
4. User splits the file or adjusts approach before wasting context on a failed operation
5. User runs /tokens data-part1.json data-part2.json
6. Output shows both files under limit
7. User proceeds with confidence

Additional Context

The /v1/messages/count_tokens API endpoint:

• Already exists and is production-ready
• Is free to call (rate-limited only)
• Returns exact counts using the same tokenizer Claude Code uses internally
• Requires only model name and content (minimal payload)

This is not a new capability request. It is exposing existing free infrastructure that Claude Code already has access to.

Related Issues

• #2529 (closed as not planned, predates free token counting API)
• #10593, #13891, #10388, #15931, #1109 (various token visibility requests)