[FEATURE] Integrate semantic search capability to reduce token consumption

Preflight Checklist

• [x] I have searched existing requests and this feature hasn't been requested yet
• [x] This is a single feature request (not multiple features)

Problem Statement

When working with large codebases, Claude Code currently relies on basic tools like grep and glob for file discovery. This often results in:

• Listing dozens of files to find relevant code
• Reading files one by one to check relevance
• Launching multiple expensive subagents to dig through directories
• Exponential token consumption that quickly becomes costly

This brute-force approach to codebase exploration is a major bottleneck for productivity and API cost efficiency.

Proposed Solution

Proposed: Hybrid Context Discovery System

Implement a flexible system that allows users to choose their preferred codebase navigation strategy based on project complexity and needs.

Option 1: Semantic Search (for large codebases)

• Integrate local embedding-based semantic search (similar to GrepAI's approach)
• Use Ollama or configurable embedding models for meaning-based code discovery
• Include daemon mode for automatic re-indexing as files change
• Proven benefits: 97% token reduction, 27.5% cost savings (Excalidraw benchmark)

Option 2: Guided Context (for medium projects)

• Support scriptReferences.md and similar documentation patterns
• Allow users to manually curate what Claude should examine first
• Simpler implementation, no embedding dependencies
• This was the top-voted alternative in community discussion

Option 3: Traditional Full-Context (for small/critical tasks)

• Keep existing grep/glob-based full file reading
• Best for scenarios where complete context is essential
• Prevents potential "lazy" code generation from incomplete context

Critical Implementation Requirement:
Quality-first benchmarking - The community's main concern is that cost-focused benchmarks don't measure if reduced context leads to worse code. Before release, benchmark:

• Code correctness/bugs introduced
• Test coverage of generated code
• Need for follow-up corrections/iterations
• User satisfaction with output quality

Recommended Default Behavior:

• Small projects (<10k LOC): Traditional full-context
• Medium projects (10k-100k LOC): Guided context with scriptReferences.md
• Large projects (>100k LOC): Semantic search with quality guardrails

Related Tools to Study:

• Aider's repomap (proven in production)
• LSP-based code intelligence tools (Serena)
• GrepAI implementation details

This hybrid approach addresses both the token efficiency needs AND the community's legitimate concerns about maintaining code quality.

Alternative Solutions

_No response_

Priority

Critical - Blocking my work

Feature Category

CLI commands and flags

Use Case Example

_No response_

Additional Context

_No response_