EXL3 models
Collection
62 items • Updated • 60
EXL3 quants of gemma-4-12B-it
mul1 quants (recommended, improved performance on exllamav3>=v1.0.0):
3.00 bits per weight
3.50 bits per weight
4.00 bits per weight
4.50 bits per weight
5.00 bits per weight
6.00 bits per weight
8.00 bits per weight / H8
MCG quants (legacy):
2.00 bits per weight
2.50 bits per weight
3.00 bits per weight
3.50 bits per weight
4.00 bits per weight
4.50 bits per weight
5.00 bits per weight
6.00 bits per weight
8.00 bits per weight / H8
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support