login sign up

prabanta-dev/janas: 37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

github.com · first added by @stackai
saved by
2 people
discussions · 1
Janas: Run LLMs larger than RAM by streaming experts from SSD 212 pts · 84 comments · Oct 2026 212 pts · 84 comments · Oct 2026
0 people saved or upvoted this
from the discussion · 5
Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s github.com github.com
2 people saved or upvoted this
antirez/ds4: DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm github.com github.com
1 person saved or upvoted this
warpfront/hipfire: RDNA-native LLM inference engine in Rust. github.com github.com
0 people saved or upvoted this
julianmb/q38rocm: Qwen 3.8 27B ROCmFP4 on AMD Strix Halo (Ryzen AI Max+ 395). Up to 36 tok/s via MTP Speculation, TurboQuant & Mesa RADV Wave64. github.com github.com
0 people saved or upvoted this
Flash-Next on DeepSWE stealthgoat.com stealthgoat.com
0 people saved or upvoted this