blossom-v5-14b-GGUF / README.md
bartowski's picture
Llamacpp quants
8600181 verified
metadata
license: apache-2.0
datasets:
  - Azure99/blossom-chat-v3
  - Azure99/blossom-math-v4
  - Azure99/blossom-wizard-v3
  - Azure99/blossom-orca-v3
language:
  - zh
  - en
quantized_by: bartowski
pipeline_tag: text-generation

Llamacpp Quantizations of blossom-v5-14b

Using llama.cpp release b2440 for quantization.

Original model: https://huggingface.co/Azure99/blossom-v5-14b

Download a file (not the whole branch) from below:

Filename Quant type File Size Description
blossom-v5-14b-Q8_0.gguf Q8_0 15.06GB Extremely high quality, generally unneeded but max available quant.
blossom-v5-14b-Q6_K.gguf Q6_K 12.31GB Very high quality, near perfect, recommended.
blossom-v5-14b-Q5_K_M.gguf Q5_K_M 10.53GB High quality, very usable.
blossom-v5-14b-Q5_K_S.gguf Q5_K_S 10.02GB High quality, very usable.
blossom-v5-14b-Q5_0.gguf Q5_0 9.85GB High quality, older format, generally not recommended.
blossom-v5-14b-Q4_K_M.gguf Q4_K_M 9.19GB Good quality, similar to 4.25 bpw.
blossom-v5-14b-Q4_K_S.gguf Q4_K_S 8.56GB Slightly lower quality with small space savings.
blossom-v5-14b-IQ4_NL.gguf IQ4_NL 8.24GB Good quality, similar to Q4_K_S, new method of quanting,
blossom-v5-14b-IQ4_XS.gguf IQ4_XS 7.91GB Decent quality, new method with similar performance to Q4.
blossom-v5-14b-Q4_0.gguf Q4_0 8.17GB Decent quality, older format, generally not recommended.
blossom-v5-14b-IQ3_M.gguf IQ3_M 7.09GB Medium-low quality, new method with decent performance.
blossom-v5-14b-IQ3_S.gguf IQ3_S 6.77GB Lower quality, new method with decent performance, recommended over Q3 quants.
blossom-v5-14b-Q3_K_L.gguf Q3_K_L 7.84GB Lower quality but usable, good for low RAM availability.
blossom-v5-14b-Q3_K_M.gguf Q3_K_M 7.41GB Even lower quality.
blossom-v5-14b-Q3_K_S.gguf Q3_K_S 6.77GB Low quality, not recommended.
blossom-v5-14b-Q2_K.gguf Q2_K 5.91GB Extremely low quality, not recommended.

Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski