Investigate Memory.cacheLimit for inference optimization

Hypothesis

During LoRA training debugging, we discovered that Memory.cacheLimit (MLX Swift API) can be used to limit the GPU buffer cache size, which prevents memory accumulation during repeated forward passes.

Current implementation (training):

Memory.cacheLimit = 512 * 1024 * 1024  // 512 MB cache limit

Question

Could this same technique improve memory management during inference, especially for:

• Long generation sessions
• Multiple sequential generations
• Image-to-image workflows with repeated encoding

Context

• MLX documentation mentions this for iOS memory-constrained environments
• We observed memory growing unbounded during text encoding in training
• Setting a cache limit forces memory release instead of caching

Testing Needed

1. Benchmark inference memory usage with and without cache limit
2. Test different cache limit values (256 MB, 512 MB, 1 GB)
3. Measure impact on inference speed (cache reuse vs memory pressure)

References

• MLX Swift docs: Source/MLX/Documentation.docc/Articles/running-on-ios.md
• Pattern: MLX.Memory.cacheLimit = 20 * 1024 * 1024 (from MLX examples)