claude_code.llm_request spans emit non-standard token attrs (input_tokens / output_tokens) instead of gen_ai.usage.*

Summary

Claude Code 2.1.114 emits token counts on claude_code.llm_request spans as flat attributes (input_tokens, output_tokens, cache_read_tokens, cache_creation_tokens) instead of the names required by the OpenTelemetry GenAI Semantic Conventions (gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cached_tokens). Downstream OTel-native observability tools — Langfuse, Datadog, Honeycomb — read the standard names, so they render zero tokens for every Claude Code interaction even though the span does carry the counts under non-standard keys.

This makes Claude Code spans look empty in any dashboard that follows the OTel spec. Model name is visible (Langfuse picks up gen_ai.request.model), tokens are not.

Reproduction

Run a minimal Claude Code session against an OTel collector using the officially supported env vars:

CLAUDE_CODE_ENABLE_TELEMETRY=1 \
CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1 \
OTEL_TRACES_EXPORTER=otlp \
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf \
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://127.0.0.1:4318/v1/traces \
OTEL_LOG_TOOL_DETAILS=1 \
OTEL_RESOURCE_ATTRIBUTES=service.name=claude-smoke \
  claude "Read /etc/hostname"

Collector configured with a file exporter. Inspect the emitted span; the claude_code.llm_request attributes section contains:

{
  "scope": { "name": "com.anthropic.claude_code.tracing" },
  "attributes": {
    "gen_ai.system": "anthropic",
    "gen_ai.request.model": "claude-opus-4-7",
    "gen_ai.response.id": "req_...",
    "model": "claude-opus-4-7",
    "input_tokens": 6,
    "output_tokens": 137,
    "cache_read_tokens": 23251,
    "cache_creation_tokens": 23056
  }
}

Note: no gen_ai.usage.input_tokens / output_tokens / cached_tokens.

Expected behaviour

Per the OpenTelemetry GenAI Semantic Conventions and the Anthropic-specific page, the span should carry:

• gen_ai.usage.input_tokens = input_tokens + cache_read_tokens + cache_creation_tokens (the spec defines this as the total input count, cache breakdown reported separately)
• gen_ai.usage.output_tokens = output_tokens
• gen_ai.usage.cached_tokens (or the split cache_read.input_tokens / cache_creation.input_tokens per open-telemetry/semantic-conventions-genai#23)

Keeping the existing input_tokens / output_tokens / cache_*_tokens attributes in parallel is acceptable and non-breaking — this issue asks only that the standard names be added alongside.

Observed impact

Shared across the OTel-LLM ecosystem:

• Langfuse (self-hosted v3.169.0 OSS): model table shows claude-opus-4-7 / claude-sonnet-4-6 / claude-haiku-4-5 / claude-opus-4-7[1m] with 0 tokens, 0 cost. Verified the ingester reads gen_ai.usage.* per their docs and langfuse-js/packages/core/src/constants.ts; the flat names are not in the accepted-attr list.
• Any OTel pipeline consuming Claude Code spans is in the same position.

Requested change

Add the standard gen_ai.usage.* attributes when the span records token usage, in addition to the current flat attributes. Non-breaking, add-only.

Happy to open a PR if the maintainers confirm this is the desired direction.

Context

Filed as a joint investigation between two factory agents (Atenea + Artemisa) working with @avaj1007. We have a local SpanProcessor patch registered under slug claude-code-gen-ai-usage-normalization to normalise these attributes at export time, with an explicit policy that the patch is retired as soon as a Claude Code release ships the standard names. This issue is to close the loop upstream so the patch can be removed.