Use TOON format for system tool definitions to reduce fixed token overhead

Problem

System tool definitions currently consume ~16.5k tokens (8.2% of context) as JSON schemas. This is a fixed cost that cannot be reduced by users and becomes increasingly painful as conversations grow toward context limits.

For reference, the full system prompt + tools overhead is ~20.5k tokens (~10%), of which tool schemas are the dominant portion.

Proposal

Use TOON (Token-Oriented Object Notation) for encoding tool definitions in the system prompt.

TOON is a compact, human-readable encoding format designed specifically for LLM input. It provides lossless JSON-compatible representation while significantly reducing token usage.

Benchmarks (from TOON repo)

| Format | Accuracy | Tokens | Efficiency (acc/1K tokens) |
|--------|----------|--------|------------|
| TOON | 73.9% | 2,744 | 26.9/1K |
| JSON compact | 70.7% | 3,081 | 22.9/1K |
| JSON | 69.7% | 4,545 | 15.3/1K |

~39.6% fewer tokens than JSON with comparable or better accuracy across 209 retrieval questions on four LLM models.

Why this fits tool definitions well

Tool definitions are essentially uniform arrays of objects (each tool has name, description, parameters with type/description fields) — exactly the structure where TOON excels.

Estimated impact: ~16.5k → ~10k tokens, freeing ~6.5k tokens (3-4% of context) for actual conversation.

Example

A tool parameter schema in JSON:

{"properties": {"command": {"description": "The command to execute", "type": "string"}, "timeout": {"description": "Optional timeout in milliseconds", "type": "number"}}, "required": ["command"]}

TOON equivalent:

properties{name,description,type}:
  command,The command to execute,string
  timeout,Optional timeout in milliseconds,number
required[1]: command

Alternatives considered

• Lazy-loading tools: Only include schemas for tools likely needed (more complex, may hurt capability)
• Shorter descriptions: Lose useful guidance for the model
• User configuration: Let users disable unused tools (not currently possible)

Additional context

TOON is MIT-licensed, has deterministic lossless JSON round-trips, and is specifically designed for LLM consumption. Repository: https://github.com/toon-format/toon