[Bug] Anthropic API token counting performance degrades linearly with conversation size

Bug Description

Progressive performance degradation - API token counting scales poorly with conversation size

What's Wrong?
Token counting API (/v1/messages/count_tokens) performance degrades dramatically as conversation grows, causing 14.7x slowdown (103ms → 1514ms) over a single session with 73 messages.

Root Cause:

• Full conversation history sent with each count_tokens request
• Server-side processing time scales linearly (or worse) with message count
• No caching or incremental token counting

Evidence:
Request# | Server Time | Total Time | Message Count (est)
---------|-------------|------------|--------------------
5 | 103ms | 260ms | ~10 messages
8 | 566ms | 773ms | ~40 messages
11 | 1391ms | 1622ms | ~70 messages
12 | 1514ms | 1711ms | ~73 messages

Degradation: 14.7x slower server processing

Proposed Solutions:

1. Cache token counts - Only recount new messages since last check
2. Client-side estimation - Use approximate tokenization for UI display
3. Incremental counting - Send only new messages + delta
4. Throttle count_tokens calls - Reduce frequency of token checking

Impact:

• Severity: HIGH
• Users experience cumulative slowdown in long sessions
• 2-second delays make tool feel unresponsive
• No workaround except restarting session

Environment Info

• Platform: linux
• Terminal: gnome-terminal
• Version: 2.0.50
• Feedback ID: abdf92bf-68a2-4333-9844-a335d5bf8916

Log Excerpts

Request #5 (early, fast):
[log_*, request-id: "req_011CVeKQ5QhucXs6noZ63Zqg"]
post https://api.anthropic.com/v1/messages/count_tokens?beta=true
succeeded with status 200 in 260ms
x-envoy-upstream-service-time: 103ms

Request #11 (later, slow):
[log_*, request-id: "req_011CVeKQCgaTLVSRmQgq8D6R"]
post https://api.anthropic.com/v1/messages/count_tokens?beta=true
succeeded with status 200 in 1622ms
x-envoy-upstream-service-time: 1391ms

Request #12 (latest, slowest):
[log_a98904, request-id: "req_011CVeKQCnDmjUrUUohkEKgm"]
post https://api.anthropic.com/v1/messages/count_tokens?beta=true
succeeded with status 200 in 1711ms
x-envoy-upstream-service-time: 1514ms

Analysis

The x-envoy-upstream-service-time header shows server-side processing time.
This eliminates network latency and client processing as factors.

The degradation is clearly server-side, and correlates with conversation size growth.

Possible fix

since conversation is progressing incrementally, token counting should also take an identifier of previous request and continue from there.