Check what can you use and at what rate of token per seconds would it be… It has examples of many models and quantization levels. Huge resource!
Check what can you use and at what rate of token per seconds would it be… It has examples of many models and quantization levels. Huge resource!
This doesn’t seem to take into account CPU MoE, which can make a huge amount of difference - Running a bigger MoE model is better than a small model that fits in your GPU if you have the CPU resources. I run Qwen3.6 (the 30b/e4b version) in MoE at around 40 tok/s on my 5070+Ryzen 9 5950X, and it’s way better than that tool’s recommended 9b.