Kalsa
Engines / giannisanni / neutronstar

neutronstar

NVIDIAForkMIT
Read this first. Not recommendable: ~0.4 tok/s generation (about 2.5 s per token; the README's own interactive figure is ~2.1 tok/s)

Adds batch prefill for GLM models to ds4 on NVIDIA GPUs. It is a fork of antirez/ds4.

Benchmarks

  • On RTX 4060 Ti 16GB with GLM (IQ2_XXS), it reads prompts at 6.5 tok/s, against 0.3 tok/s for token-major prefill (feature off) (21.7×). Source

Listing

Listed, but we don't recommend it: ~0.4 tok/s generation (about 2.5 s per token; the README's own interactive figure is ~2.1 tok/s).

Security review

Not reviewed yet.

Install

To install it, follow the project's own instructions on GitHub. The engine runs with your user's permissions. macOS doesn't sandbox it, so install only engines you trust.

Something wrong or out of date? Write to help@kalsa.io, or ask us to remove this page.