Repository navigation
Feature matrix
| CPU (AVX/AVX2) | CPU (ARM NEON) | Metal | CUDA | ROCm | SYCL | Vulkan | |
|---|---|---|---|---|---|---|---|
| K-quants | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅⁵ |
| I-quants | ✅ 🐢⁴ | ✅ 🐢⁴ | ✅ 🐢⁴ | ✅ | ✅ | Partial¹ | ✅ 🐢⁴ |
| Parallel Multi-GPU | N/A | N/A | N/A | ✅ | ✅ | Sequential only | ✅ |
| K cache quants | ✅ | ✅ | ✅ | ✅ | ✅ | ❓ | ✅ |
| MoE architecture | ✅ | ✅ | ✅ | ✅ | ✅ | ❓ | ✅ |
| Flash Attention | ✅ | ✅ | ✅ | ✅ | ✅ | ❓ | ✅ |
- ✅: feature works
- 🚫: feature does not work
- ❓: unknown, please contribute if you can test it yourself
- 🐢: feature is slow
- ¹: IQ3_S and IQ1_S, see #5886
- ⁴: Slower than K-quants of comparable size
- ⁵: Generally the CUDA or ROCM backends are faster than Vulkan in most cases. See #10879 for benchmarks.
Useful information for users that doesn't fit into Readme.
- Home
- Feature Matrix
- GGML Tips & Tricks
- Chat Templating
- Metadata Override
- HuggingFace Model Card Metadata Interoperability Consideration
These are information useful for Maintainers and Developers which does not fit into code comments
Click on a badge to jump to workflow. This is here as a useful general view of all the actions so that we may notice quicker if main branch automation is broken and where.
