A database of local inference engines and forks
What each one is, who made it, its licence, the benchmarks its author published, and how we read them.
What are you running?
Pick a chip and meet the engines that run on it, A to Z.
17 engines run on Apple M1, in alphabetical order. Listed engines only. Solid numbers were measured on this chip family. Dashed ones were measured on another chip, so they say less about yours.

Zig runtime for Apple Silicon built around model-specific speculative decoding and SSD-streamed MoE models.

Inference engine in C built around a few large models (DeepSeek V4 Flash and PRO, GLM 5.x, Qwen3.8), with Metal, CUDA and ROCm backends.

Pure-Rust GGUF loader with quantized Metal kernels and lossless speculative decoding.

Reference upstream across Metal, CUDA, ROCm/HIP, Vulkan, SYCL and CPU.

Rust engine with ISQ/UQFF quantization.

Reference upstream for Apple Silicon (MLX).

MLX engine for text and VLM.

Mac app and command line that runs Qwen models on Apple Silicon using the model's own multi-token prediction heads.

Neural Engine prefill, M1 flash-attention and 8-bit paged KV.
Fork of youssofal/MTPLX
Reference upstream: UX wrapper with a model library.

LLM inference server for Apple Silicon with continuous batching and SSD caching, managed from the macOS menu bar.

Unofficial port of Splash to M1 and M2 Macs, with Metal kernels written for those GPUs.
Fork of incoai/splash
Core ML and XDNA NPU offload.
Fork of ggml-org/llama.cpp
Pure MLX kernels with continuous batching and prefix cache.
Built on ml-explore/mlx-lm
Desktop app and engine for running Qwen3.8 models locally on NVIDIA, AMD and Apple Silicon, with an OpenAI-compatible local API.

Native Swift server for MLX models on Apple Silicon, with SSD streaming of MoE experts and a compressed KV cache.

MLX model server for Apple Silicon with adaptive mixed-precision quantization and partial loading of MoE experts from SSD.
16 engines run on Apple M2, in alphabetical order. Listed engines only. Solid numbers were measured on this chip family. Dashed ones were measured on another chip, so they say less about yours.

Zig runtime for Apple Silicon built around model-specific speculative decoding and SSD-streamed MoE models.

Inference engine in C built around a few large models (DeepSeek V4 Flash and PRO, GLM 5.x, Qwen3.8), with Metal, CUDA and ROCm backends.

Pure-Rust GGUF loader with quantized Metal kernels and lossless speculative decoding.

Reference upstream across Metal, CUDA, ROCm/HIP, Vulkan, SYCL and CPU.

Continuous batching across requests and KV reuse across restarts.

Rust engine with ISQ/UQFF quantization.

Reference upstream for Apple Silicon (MLX).

Mac app and command line that runs Qwen models on Apple Silicon using the model's own multi-token prediction heads.

Reference upstream: UX wrapper with a model library.

LLM inference server for Apple Silicon with continuous batching and SSD caching, managed from the macOS menu bar.

Unofficial port of Splash to M1 and M2 Macs, with Metal kernels written for those GPUs.
Fork of incoai/splash
Core ML and XDNA NPU offload.
Fork of ggml-org/llama.cpp
Pure MLX kernels with continuous batching and prefix cache.
Built on ml-explore/mlx-lm
Desktop app and engine for running Qwen3.8 models locally on NVIDIA, AMD and Apple Silicon, with an OpenAI-compatible local API.

Native Swift server for MLX models on Apple Silicon, with SSD streaming of MoE experts and a compressed KV cache.

MLX model server for Apple Silicon with adaptive mixed-precision quantization and partial loading of MoE experts from SSD.
16 engines run on Apple M3, in alphabetical order. Listed engines only. Solid numbers were measured on this chip family. Dashed ones were measured on another chip, so they say less about yours.

Zig runtime for Apple Silicon built around model-specific speculative decoding and SSD-streamed MoE models.

Inference engine in C built around a few large models (DeepSeek V4 Flash and PRO, GLM 5.x, Qwen3.8), with Metal, CUDA and ROCm backends.

Pure-Rust GGUF loader with quantized Metal kernels and lossless speculative decoding.

GLM-5.3-Flash path, DFlash2 and CUDA batch fixes.
Fork of antirez/ds4
Reference upstream across Metal, CUDA, ROCm/HIP, Vulkan, SYCL and CPU.

Rust engine with ISQ/UQFF quantization.

Reference upstream for Apple Silicon (MLX).

Mac app and command line that runs Qwen models on Apple Silicon using the model's own multi-token prediction heads.

Reference upstream: UX wrapper with a model library.

LLM inference server for Apple Silicon with continuous batching and SSD caching, managed from the macOS menu bar.

Core ML and XDNA NPU offload.
Fork of ggml-org/llama.cpp
Pure MLX kernels with continuous batching and prefix cache.
Built on ml-explore/mlx-lm
Neural Engine prefill and SSD-tiered KV on Apple M3+.

Desktop app and engine for running Qwen3.8 models locally on NVIDIA, AMD and Apple Silicon, with an OpenAI-compatible local API.

Native Swift server for MLX models on Apple Silicon, with SSD streaming of MoE experts and a compressed KV cache.

MLX model server for Apple Silicon with adaptive mixed-precision quantization and partial loading of MoE experts from SSD.
15 engines run on Apple M4, in alphabetical order. Listed engines only. Solid numbers were measured on this chip family. Dashed ones were measured on another chip, so they say less about yours.

Zig runtime for Apple Silicon built around model-specific speculative decoding and SSD-streamed MoE models.

Inference engine in C built around a few large models (DeepSeek V4 Flash and PRO, GLM 5.x, Qwen3.8), with Metal, CUDA and ROCm backends.

Pure-Rust GGUF loader with quantized Metal kernels and lossless speculative decoding.

Reference upstream across Metal, CUDA, ROCm/HIP, Vulkan, SYCL and CPU.

Rust engine with ISQ/UQFF quantization.

Reference upstream for Apple Silicon (MLX).

Mac app and command line that runs Qwen models on Apple Silicon using the model's own multi-token prediction heads.

Reference upstream: UX wrapper with a model library.

LLM inference server for Apple Silicon with continuous batching and SSD caching, managed from the macOS menu bar.

Core ML and XDNA NPU offload.
Fork of ggml-org/llama.cpp
Pure MLX kernels with continuous batching and prefix cache.
Built on ml-explore/mlx-lm
Neural Engine prefill and SSD-tiered KV on Apple M3+.

Desktop app and engine for running Qwen3.8 models locally on NVIDIA, AMD and Apple Silicon, with an OpenAI-compatible local API.

Native Swift server for MLX models on Apple Silicon, with SSD streaming of MoE experts and a compressed KV cache.

MLX model server for Apple Silicon with adaptive mixed-precision quantization and partial loading of MoE experts from SSD.
19 engines run on Apple M5, in alphabetical order. Listed engines only. Solid numbers were measured on this chip family. Dashed ones were measured on another chip, so they say less about yours.

AXQ MXFP4 MTP sidecars on MLX.

DFlash speculative decoding on MLX.

Zig runtime for Apple Silicon built around model-specific speculative decoding and SSD-streamed MoE models.

Inference engine in C built around a few large models (DeepSeek V4 Flash and PRO, GLM 5.x, Qwen3.8), with Metal, CUDA and ROCm backends.

Pure-Rust GGUF loader with quantized Metal kernels and lossless speculative decoding.

KFlash decoding and Nace quants.

Reference upstream across Metal, CUDA, ROCm/HIP, Vulkan, SYCL and CPU.

Rust engine with ISQ/UQFF quantization.

Reference upstream for Apple Silicon (MLX).

MLX engine for text and VLM.

Mac app and command line that runs Qwen models on Apple Silicon using the model's own multi-token prediction heads.

Reference upstream: UX wrapper with a model library.

LLM inference server for Apple Silicon with continuous batching and SSD caching, managed from the macOS menu bar.

Core ML and XDNA NPU offload.
Fork of ggml-org/llama.cpp
Pure MLX kernels with continuous batching and prefix cache.
Built on ml-explore/mlx-lm
Neural Engine prefill and SSD-tiered KV on Apple M3+.

Desktop app and engine for running Qwen3.8 models locally on NVIDIA, AMD and Apple Silicon, with an OpenAI-compatible local API.

Native Swift server for MLX models on Apple Silicon, with SSD streaming of MoE experts and a compressed KV cache.

MLX model server for Apple Silicon with adaptive mixed-precision quantization and partial loading of MoE experts from SSD.
8 engines run on Apple M6, in alphabetical order. Listed engines only. Solid numbers were measured on this chip family. Dashed ones were measured on another chip, so they say less about yours.

Zig runtime for Apple Silicon built around model-specific speculative decoding and SSD-streamed MoE models.

Inference engine in C built around a few large models (DeepSeek V4 Flash and PRO, GLM 5.x, Qwen3.8), with Metal, CUDA and ROCm backends.

Reference upstream for Apple Silicon (MLX).

Mac app and command line that runs Qwen models on Apple Silicon using the model's own multi-token prediction heads.

Core ML and XDNA NPU offload.
Fork of ggml-org/llama.cpp
Pure MLX kernels with continuous batching and prefix cache.
Built on ml-explore/mlx-lm
Desktop app and engine for running Qwen3.8 models locally on NVIDIA, AMD and Apple Silicon, with an OpenAI-compatible local API.

Native Swift server for MLX models on Apple Silicon, with SSD streaming of MoE experts and a compressed KV cache.
16 engines run on NVIDIA, in alphabetical order. Listed engines only. Each number names the card it was measured on.

Two-GPU split with tuned prompt kernels.
Fork of Niko1221/Strata
Inference engine in C built around a few large models (DeepSeek V4 Flash and PRO, GLM 5.x, Qwen3.8), with Metal, CUDA and ROCm backends.

llama.cpp fork that keeps about half of each MoE model's experts as a GPU cache.
Fork of ggml-org/llama.cpp
Expert cache with GPU and RAM tiers.
Fork of ggml-org/llama.cpp
Reference upstream across Metal, CUDA, ROCm/HIP, Vulkan, SYCL and CPU.

KV cache streamed between host RAM and VRAM.
Fork of ggml-org/llama.cpp
VRAM-resident expert cache for MoE.
Fork of ggml-org/llama.cpp
Rust engine with ISQ/UQFF quantization.

Experts streamed from SSD/RAM into GPU passes; prompt passes.
Fork of ggml-org/llama.cpp
Batch prefill for GLM and Hy3.
Fork of antirez/ds4
Reference upstream: UX wrapper with a model library.

Research fork of llama.cpp with new KV cache codecs, including a trellis-coded 3-bit codec.
Fork of ggml-org/llama.cpp
Desktop app and engine for running Qwen3.8 models locally on NVIDIA, AMD and Apple Silicon, with an OpenAI-compatible local API.

MoE expert store with RAM tiers.

llama.cpp fork with TQ3 weight formats and CUDA kernels, including a Blackwell FP4 prompt path.
Fork of ggml-org/llama.cpp
Reference upstream for NVIDIA, AMD, Intel GPU and CPU.
13 engines run on AMD, in alphabetical order. Listed engines only. Each number names the card it was measured on.

Adaptive KV streaming, RDNA 4 kernels and MTP+n-gram.
Fork of ggml-org/llama.cpp
llama.cpp tuned for Qwen 3.8 Flash-Next on AMD Strix Halo, with ROCm kernel patches and a working MTP draft head.
Fork of ggml-org/llama.cpp
Inference engine in C built around a few large models (DeepSeek V4 Flash and PRO, GLM 5.x, Qwen3.8), with Metal, CUDA and ROCm backends.

Reference upstream across Metal, CUDA, ROCm/HIP, Vulkan, SYCL and CPU.

Vulkan LDS stride fix and adaptive draft on Strix Halo.
Fork of ggml-org/llama.cpp
Reference upstream: UX wrapper with a model library.

Core ML and XDNA NPU offload.
Fork of ggml-org/llama.cpp
Pure-HIP engine with decode-graph replay.

llama.cpp fork for AMD RDNA3 and RDNA3.5 GPUs on Vulkan, with the ROCmFPx weight formats and measured RDNA3 patches.
Fork of ggml-org/llama.cpp
Research fork of llama.cpp with new KV cache codecs, including a trellis-coded 3-bit codec.
Fork of ggml-org/llama.cpp
RDNA patches, tensor split and a speculative-decoding clamp.
Fork of ggml-org/llama.cpp
Desktop app and engine for running Qwen3.8 models locally on NVIDIA, AMD and Apple Silicon, with an OpenAI-compatible local API.

Reference upstream for NVIDIA, AMD, Intel GPU and CPU.
2 engines run on Intel, in alphabetical order. Listed engines only. Each number names the card it was measured on.
3 engines run on CPU, in alphabetical order. Listed engines only. Each number names the card it was measured on.

Reference upstream across Metal, CUDA, ROCm/HIP, Vulkan, SYCL and CPU.

Reference upstream: UX wrapper with a model library.

Reference upstream for NVIDIA, AMD, Intel GPU and CPU.
Published comparisons
Against llama.cpp or mlx-lm, as each author ran them. Listed A to Z, not ranked. Each names the chip it was measured on.








Recently added
The last entries in the catalog file. Dates will come from the repository history.

llama.cpp fork for AMD RDNA3 and RDNA3.5 GPUs on Vulkan, with the ROCmFPx weight formats and measured RDNA3 patches.
Fork of ggml-org/llama.cpp
LLM inference server for Apple Silicon with continuous batching and SSD caching, managed from the macOS menu bar.

llama.cpp tuned for Qwen 3.8 Flash-Next on AMD Strix Halo, with ROCm kernel patches and a working MTP draft head.
Fork of ggml-org/llama.cpp
llama.cpp fork with TQ3 weight formats and CUDA kernels, including a Blackwell FP4 prompt path.
Fork of ggml-org/llama.cppThe people behind them
40 developers and organisations so far.
achyudh
antirez
antonellof
architectds
Aristo94
bstnxbt
defai-digital
EricLBuehler
ewhacc
Generated-Computers
GenerelSchwerz
ggml-org
giannisanni
incoai
IngeniousIdiocy
jjang-ai
jundot
lablup
LaurentZuijdwijk
Lidenburg
ml-explore
nace-ai
neurall
Niko1221
oldnordic
ollama
paperniuk
raullenchai
RaymondHuang210129
SelfRef
SharpAI
shunya1810
spiritbuun
stew675
tetherto
thecodacus
turbo-tan
vllm-project
youssofal
zouyeeHow we choose
What gets listed, what a review looks at, and how a published number is read. Read how we choose.