[Feature Request] Implement block attention to enable more nuanced memory and context engineering per request

Bug Description
Currently, Anthropic's LLMs store context linearly, new content is added at the end, and earlier parts can't be modified or removed from context even when they no longer serve a purpose.

This research paper solves the issue of LLM context being linearly additive, even though the needs of practical memory management require that it be possible to add / subtract / expand / contract pieces based on the needs of the current request.

https://arxiv.org/pdf/2409.15355

I think Claude's agentic capabilities and coding capabilities would get a big boost from using this idea.

Note: In the ideal case, users / API callers should have the ability to construct context and have it dynamically match against cached pieces.

Environment Info

• Platform: darwin
• Terminal: ghostty
• Version: 2.1.107
• Feedback ID: eafd1140-fd8d-4fbc-822d-f699d54a7f69