
spiritbuun/buun-llama-cpp
Adds a trellis-coded 3-bit KV cache codec to llama.cpp for NVIDIA and AMD GPUs. It is a fork of ggml-org/llama.cpp.
What it does better
The trellis-coded 3-bit KV cache codec (turbo3_tcq, 3.25 bits per value) cuts median KL divergence from 0.00279 to 0.00163 against scalar turbo3 (3.5 bits) on Qwen3.6-27B Q6_K at 16K; perplexity moves only from 5.730 to 5.668, 1.1%, so KL divergence is the measure used here
Trade-off: prefill falls from 1015 to 818 tok/s and decode from 28.1 to 26.6 tok/s on a 16K prompt; on weaker-compute hardware such as the Strix Halo iGPU the author says the trellis types can be up to about 40% slower than the scalar ones.
Benchmarks
The author hasn't published benchmarks yet.
Listing
Listed: Passed all seven checks.
Security review
Not reviewed yet.
Install
To install it, follow the project's own instructions on GitHub. The engine runs with your user's permissions. macOS doesn't sandbox it, so install only engines you trust.
Something wrong or out of date? Write to help@kalsa.io, or ask us to remove this page.