Session history search is a raw substring match: reordering two words returns zero results

Summary

Session history search appears to be a literal case-insensitive substring match rather than a tokenised full-text search. Reordering two words in a query silently returns zero results, and ranking is dominated by tool output rather than conversation.

This affects the desktop session search and the search_session_transcripts tool that shares its index.

Reproduction

Four queries against the same local corpus (~1,100 sessions):

| Query | Sessions returned |
|---|---|
| diarization | 3 |
| speaker labels not showing | 0 |
| invoice reminder | 3 |
| reminder invoice | 0 |

invoice reminder and reminder invoice differ only in word order: 3 results vs 0.

The tool schema documents the behaviour ("Substring match, case-insensitive"), so this is intended rather than broken — but it makes the feature close to unusable for its actual purpose, since recall requires reproducing the exact contiguous phrasing you used weeks ago.

Four distinct problems

1. No tokenisation. No implicit AND across terms, no stemming (label misses labels / labelling), no prefix or fuzzy matching. Word order is decisive.

2. Tool output is indexed at the same weight as human turns. Measuring my own corpus by volume: tool results are ~209 MB of a ~226 MB total. Text the user actually typed is ~0.6 MB — under 0.3%. So nearly every match is machine exhaust. In the diarization case above, all three results returned the identical snippet: one line of build-script console output that happened to scroll past in three unrelated sessions. The sessions that actually discussed the topic never surfaced.

3. One hit per session, apparently ordered by recency rather than relevance. A session that discussed a term at length ranks below one where it appeared once inside a stack trace.

4. Snippets are returned unescaped — results come back containing literal \\u2014 escape sequences instead of the character.

Suggested fix

SQLite already ships FTS5 with a porter stemmer and BM25, so this needs no new dependency. As a proof of concept I built an external indexer over the same two stores (claude-code-sessions/**/local_*.json for metadata, ~/.claude/projects/**/*.jsonl for messages, joined on cliSessionId):

• Full index of the corpus: ~20 s. Incremental refresh: under 1 s.
• reminder invoice → 23 sessions. speaker labels not showing → 10 sessions. Both previously zero.
• Weighting BM25 by content-block type (user text > assistant text > thinking > tool_use > tool_result) puts real conversation on top. In a controlled fixture where tool output contained a term five times more often than the human turn, the human turn still ranked first.

One wrinkle worth handling in any official implementation: a large share of user-role messages were not typed by the user. System-reminders, hook output, scheduled-task payloads and skill bodies all arrive as user-role turns, several with no wrapper tag at all (skill bodies begin with a bare Base directory for this skill: preamble). Treating those as user speech is a major part of why boilerplate wins the ranking — filtering them reclassified ~25% of my "user" chunks, and about 70% by volume.

Separate but related: retention

Transcripts are pruned on a cleanup cycle while desktop session metadata persists. I currently have ~1,100 sessions with metadata but only ~206 surviving transcripts, so the majority of my history is not searchable at all regardless of the algorithm. If that is intended, it would help for the UI to say so rather than silently returning nothing for older work.

Environment

Claude Code desktop, macOS (Darwin 25.5.0).