Kalsa
Engines / RaymondHuang210129 / llama.cpp adaptive KV streaming

llama.cpp adaptive KV streaming

NVIDIAForkMIT

Streams the KV cache between host RAM and VRAM in a llama.cpp fork for NVIDIA GPUs. It is a fork of ggml-org/llama.cpp.

Benchmarks

  • On RTX 5070 Ti (Linux) with Qwen3.8-27B (IQ4_XS), it decodes at 26.1 tok/s, against 2.5 tok/s for stock llama.cpp with UVM (10.4×). Source Note: 128K context, target-only; 59.7 tok/s with MTP=3

Listing

Listed: Passed all seven checks.

Security review

Not reviewed yet.

Install

To install it, follow the project's own instructions on GitHub. The engine runs with your user's permissions. macOS doesn't sandbox it, so install only engines you trust.

Something wrong or out of date? Write to help@kalsa.io, or ask us to remove this page.