
vMLX
Cuts memory use on Apple Silicon by loading only some of each MoE model's experts.
What it does better
Loading half of the experts in each layer (smelt 50) cuts active RAM for Nemotron-Cascade-2-30B-A3B-JANG_4M from 17,408 MB to 9,529 MB, a 45% saving, on an M3 Ultra with 128 GB
Trade-off: decode falls from 89.9 to 66.5 tok/s because expert swaps hit the SSD on the hot path, and the author published no quality measurement, only that the output stayed coherent. It needs an MoE model in JANG format and turns vision off.
Benchmarks
The author hasn't published benchmarks yet.
Listing
Listed: Passed all seven checks.
Security review
Not reviewed yet.
Install
To install it, follow the project's own instructions on GitHub. The engine runs with your user's permissions. macOS doesn't sandbox it, so install only engines you trust.
Something wrong or out of date? Write to help@kalsa.io, or ask us to remove this page.