[FEATURE] Improve the data management side of Claude Code session data - use a SQLite3 db or similar

Preflight Checklist

• [x] I have searched existing requests and this feature hasn't been requested yet
• [x] This is a single feature request (not multiple features)

Problem Statement

Claude Code writes massive JSONL files from long running sessions. Retrieval is not natively well designed for.

Claude typically writes code not to YAGNI or good practice principles (eg the public anguish over slop code) and balloons complex codebases out of control without harnessing capability that is NOT native to Claude Code. Endemic of the problem is these massive JSONL files, IMHO you should be using SQLite3 or similar in structure to register EACH user reply in it's own row related to a conversation, and structure the data for precise retrieval. This is basic data management and Claude Code's design completely fails at doing intelligent data management for its own session data.

Using a proper database (or similar structuring) per project needs to happen like last year. This is why I have had to make my own CLI to further harness how Claude Code operates within my projects. Even with my own CLI based additional harnessing, there is so many edge cases to address and I'm only one human. You have a team of developers and models that can easily refactor the data management side of Claude Code to perform substantially better for both the customers and the models. Please stop dragging your feet and improve Claude Code for the customer base. Clean targeted data management retrieval and verification regarding the user decision lifecycle in projects, without exploding context window usage, is REALLY important to reduce waste and keep projects on point. Massive JSONL file histories, is NOT the answer.

Proposed Solution

SQLite3 with locked schema columns with expansive rows, reasonable table count. Or similar structuring of session data per project.

Alternative Solutions

I work around some of the issues by additional harnessing through having Claude Code use my CLI to develop my codebases. My CLI directs Anthropic models through Claude Code to perform tasks in guided steps and careful context window control when possible. I do a heavily scoped step to develop sprint runs and their related tasks, I then execute and adapt depending on what the model(s) do, verification tests and data management during and there after to better manage the code that these models attempt to make. Then a final retrospective pass before continuing the next cycle of scoped planning. The big easy fix I see for Claude Code is confirming what user has already scoped without guessing and what has been done previously without guessing. Without digging those massive session JSONL histories ballooning the model's context window and without relying on spur of the moment unproven or unreliable tooling it makes to parse that history. CLEAN and ORGANIZED data management should have been a design philosophy from the start. IMHO, it's not.

Priority

High - Significant impact on productivity

Feature Category

Performance and speed

Use Case Example

An example is, a few projects I have been developing and evolving over the course of a few months with many many many millions of tokens via Anthropic Claude Code and OpenAI Codex models end up with 10s of GBs of JSONL session data that is saved in an entirely unmanageable state. Massive JSONL files. While that could be changed by doing shorter sessions, it does not fix the poor data management side of things with Claude Code.
One shot code development is a gimmick and marketing. Quality AI assisted code development takes time and a ton of harnessing, at this time. That can improve very cheaply with just tackling the data management side of the Claude Code harness. Opus 5 and Fable 5 will spend a lot of time trying to track down user answers from JSONL histories and many times will avoid doing so, which verification of design decisions (and actions) history is needed to avoid drift and unwanted code. I know the data management side being structured well does in fact help because I made a CLI (with a SQLite3 database layer) to additionally harness how Claude Code behaves, Claude Code directly controls the CLI but the CLI design requires an order of behaviors that when not ignored by the model, will cause more efficient coding and performance and much less drift. I think the Anthropic team is fully capable of improving the data management and retrieval workflow and design of Claude Code, which imo would make massive worth while improvement in how Claude Code performs for customers.

Additional Context

_No response_