We got ONNX Runtime’s int4/int8 MatMulNBits path running on the Orange Pi RV2 through MLAS CompInt8 + smt.vmadot. Decode (ms/token, lower better):
| Model | Quant | @4 threads |
|---|---|---|
| Qwen2.5-0.5B | int4 | ~80 ms |
| Qwen2.5-0.5B | int8 | ~159 ms |
| SmolLM2-360M | int4 | ~80 ms |
| SmolLM2-360M | int8 | ~140 ms |
| TinyLlama-1.1B | int4 | ~156 ms |
Qwen used to sit at ~16 s/tok on the wrong path (CompFp32). With accuracy_level=4 + pack/panel kernels that’s ~80 ms int4 and ~159 ms int8 @4t.
Story (results first, then how): ONNX Runtime on X60 — real LLM decode via IME MatMulNBits | OpenSolvers
Code: https://github.com/opensolvers/benchmarks/tree/main/onnx