Engines / giannisanni / neutronstar

neutronstar
Read this first. Not recommendable: ~0.4 tok/s generation (about 2.5 s per token; the README's own interactive figure is ~2.1 tok/s)
Adds batch prefill for GLM models to ds4 on NVIDIA GPUs. It is a fork of antirez/ds4.
Benchmarks
- On RTX 4060 Ti 16GB with GLM (IQ2_XXS), it reads prompts at 6.5 tok/s, against 0.3 tok/s for token-major prefill (feature off) (21.7×). Source
Listing
Listed, but we don't recommend it: ~0.4 tok/s generation (about 2.5 s per token; the README's own interactive figure is ~2.1 tok/s).
Security review
Not reviewed yet.
Install
To install it, follow the project's own instructions on GitHub. The engine runs with your user's permissions. macOS doesn't sandbox it, so install only engines you trust.
Something wrong or out of date? Write to help@kalsa.io, or ask us to remove this page.