Hugging Face's logo

bartowski
/
Hercules-4.0-Mistral-v0.2-7B-GGUF

Text Generation
Transformers
GGUF
English
medical
science
biology
chemistry
Not-For-All-Audiences
Inference Endpoints
conversational
Model card Files Community
1
bartowski commited on
Commit
a2e383f
•
1 Parent(s): e832c0a

Llamacpp quants

Browse files
Files changed (18) hide show
  1. .gitattributes +16 -0
  2. Hercules-4.0-Mistral-v0.2-7B-IQ3_M.gguf +3 -0
  3. Hercules-4.0-Mistral-v0.2-7B-IQ3_S.gguf +3 -0
  4. Hercules-4.0-Mistral-v0.2-7B-IQ4_NL.gguf +3 -0
  5. Hercules-4.0-Mistral-v0.2-7B-IQ4_XS.gguf +3 -0
  6. Hercules-4.0-Mistral-v0.2-7B-Q2_K.gguf +3 -0
  7. Hercules-4.0-Mistral-v0.2-7B-Q3_K_L.gguf +3 -0
  8. Hercules-4.0-Mistral-v0.2-7B-Q3_K_M.gguf +3 -0
  9. Hercules-4.0-Mistral-v0.2-7B-Q3_K_S.gguf +3 -0
  10. Hercules-4.0-Mistral-v0.2-7B-Q4_0.gguf +3 -0
  11. Hercules-4.0-Mistral-v0.2-7B-Q4_K_M.gguf +3 -0
  12. Hercules-4.0-Mistral-v0.2-7B-Q4_K_S.gguf +3 -0
  13. Hercules-4.0-Mistral-v0.2-7B-Q5_0.gguf +3 -0
  14. Hercules-4.0-Mistral-v0.2-7B-Q5_K_M.gguf +3 -0
  15. Hercules-4.0-Mistral-v0.2-7B-Q5_K_S.gguf +3 -0
  16. Hercules-4.0-Mistral-v0.2-7B-Q6_K.gguf +3 -0
  17. Hercules-4.0-Mistral-v0.2-7B-Q8_0.gguf +3 -0
  18. README.md +45 -0
.gitattributes CHANGED
@@ -33,3 +33,19 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ Hercules-4.0-Mistral-v0.2-7B-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
37
+ Hercules-4.0-Mistral-v0.2-7B-IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text
38
+ Hercules-4.0-Mistral-v0.2-7B-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
39
+ Hercules-4.0-Mistral-v0.2-7B-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
40
+ Hercules-4.0-Mistral-v0.2-7B-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
41
+ Hercules-4.0-Mistral-v0.2-7B-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
42
+ Hercules-4.0-Mistral-v0.2-7B-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
43
+ Hercules-4.0-Mistral-v0.2-7B-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
44
+ Hercules-4.0-Mistral-v0.2-7B-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
45
+ Hercules-4.0-Mistral-v0.2-7B-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
46
+ Hercules-4.0-Mistral-v0.2-7B-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
47
+ Hercules-4.0-Mistral-v0.2-7B-Q5_0.gguf filter=lfs diff=lfs merge=lfs -text
48
+ Hercules-4.0-Mistral-v0.2-7B-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
49
+ Hercules-4.0-Mistral-v0.2-7B-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
50
+ Hercules-4.0-Mistral-v0.2-7B-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
51
+ Hercules-4.0-Mistral-v0.2-7B-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
Hercules-4.0-Mistral-v0.2-7B-IQ3_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:67f89b122b3b662395030af03812eab2a8cba1211d170193bf01017bafe749e3
3
+ size 3284892064
Hercules-4.0-Mistral-v0.2-7B-IQ3_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9536d9cf36b10d7ba506ee0be133b310e14480069a1eeab0c02019f50efb2a47
3
+ size 3182393760
Hercules-4.0-Mistral-v0.2-7B-IQ4_NL.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:36ee96c52504caa7c8abd0067661968ffa85f13f60a4585fe89ff3d6dddda5d4
3
+ size 4155054496
Hercules-4.0-Mistral-v0.2-7B-IQ4_XS.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:20c34ba2542cece16ec20dc1cba000bb24877eca4f9f4b143e23a194dfa0bc27
3
+ size 3944389024
Hercules-4.0-Mistral-v0.2-7B-Q2_K.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3d49a77493cba031108b644fda79406f569318b561b1d125bbc1a403fcffa3b4
3
+ size 2719242656
Hercules-4.0-Mistral-v0.2-7B-Q3_K_L.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f18ca44645fe6de1b5dd01de31e3c5eb86800a84432c520c36a0be84ba7a52d0
3
+ size 3822025120
Hercules-4.0-Mistral-v0.2-7B-Q3_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c5f3e221bf1fe6a79a5874bc302ccf0a116c4c16a2a596c25afb6d1697d41f03
3
+ size 3518986656
Hercules-4.0-Mistral-v0.2-7B-Q3_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d56d7ba5a18a84fdd582b1b1bfbea7f62f0d1714287006c929f320e51f949898
3
+ size 3164567968
Hercules-4.0-Mistral-v0.2-7B-Q4_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:73c780b8387e977d2308c03e86d977d9d523221bfcd9f353df4bf1237dd6671b
3
+ size 4108917152
Hercules-4.0-Mistral-v0.2-7B-Q4_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fded06c6f2ff7156d7eb54aebe271869dcc9f7661e3aaa68758135ed1d9e6a44
3
+ size 4368439712
Hercules-4.0-Mistral-v0.2-7B-Q4_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f416f79e78b428f64450cad50b1d3176282abb3b508deba9e193e04a01fe0777
3
+ size 4140374432
Hercules-4.0-Mistral-v0.2-7B-Q5_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4e0cae4ca5a5e6a752830b1f9bb2a504459593ffb06aaebeb52f56aa3cfe62e5
3
+ size 4997716384
Hercules-4.0-Mistral-v0.2-7B-Q5_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f7e0602b0c4e3aff862609e8313a8943607a9f20b1ca363c8311164ce95e759b
3
+ size 5131409824
Hercules-4.0-Mistral-v0.2-7B-Q5_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:06f92dd62a7abc092d3249014570330473450748ef4e14ccbfbd8d35e190a739
3
+ size 4997716384
Hercules-4.0-Mistral-v0.2-7B-Q6_K.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9c928bb6a92b395a24cea7ba27f7065c8a5ed04955311f63ff9277d19f06cb0a
3
+ size 5942065568
Hercules-4.0-Mistral-v0.2-7B-Q8_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a3fbf7cebb62d8477050b7382d6204f508a262aae804fc6ba94a011c68128ab6
3
+ size 7695858080
README.md ADDED
@@ -0,0 +1,45 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ tags:
4
+ - medical
5
+ - science
6
+ - biology
7
+ - chemistry
8
+ - not-for-all-audiences
9
+ license: apache-2.0
10
+ datasets:
11
+ - Locutusque/hercules-v4.0
12
+ language:
13
+ - en
14
+ quantized_by: bartowski
15
+ pipeline_tag: text-generation
16
+ ---
17
+
18
+ ## Llamacpp Quantizations of Hercules-4.0-Mistral-v0.2-7B
19
+
20
+ Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b2536">b2536</a> for quantization.
21
+
22
+ Original model: https://huggingface.co/Locutusque/Hercules-4.0-Mistral-v0.2-7B
23
+
24
+ Download a file (not the whole branch) from below:
25
+
26
+ | Filename | Quant type | File Size | Description |
27
+ | -------- | ---------- | --------- | ----------- |
28
+ | [Hercules-4.0-Mistral-v0.2-7B-Q8_0.gguf](https://huggingface.co/bartowski/Hercules-4.0-Mistral-v0.2-7B-GGUF/blob/main/Hercules-4.0-Mistral-v0.2-7B-Q8_0.gguf) | Q8_0 | 7.69GB | Extremely high quality, generally unneeded but max available quant. |
29
+ | [Hercules-4.0-Mistral-v0.2-7B-Q6_K.gguf](https://huggingface.co/bartowski/Hercules-4.0-Mistral-v0.2-7B-GGUF/blob/main/Hercules-4.0-Mistral-v0.2-7B-Q6_K.gguf) | Q6_K | 5.94GB | Very high quality, near perfect, *recommended*. |
30
+ | [Hercules-4.0-Mistral-v0.2-7B-Q5_K_M.gguf](https://huggingface.co/bartowski/Hercules-4.0-Mistral-v0.2-7B-GGUF/blob/main/Hercules-4.0-Mistral-v0.2-7B-Q5_K_M.gguf) | Q5_K_M | 5.13GB | High quality, very usable. |
31
+ | [Hercules-4.0-Mistral-v0.2-7B-Q5_K_S.gguf](https://huggingface.co/bartowski/Hercules-4.0-Mistral-v0.2-7B-GGUF/blob/main/Hercules-4.0-Mistral-v0.2-7B-Q5_K_S.gguf) | Q5_K_S | 4.99GB | High quality, very usable. |
32
+ | [Hercules-4.0-Mistral-v0.2-7B-Q5_0.gguf](https://huggingface.co/bartowski/Hercules-4.0-Mistral-v0.2-7B-GGUF/blob/main/Hercules-4.0-Mistral-v0.2-7B-Q5_0.gguf) | Q5_0 | 4.99GB | High quality, older format, generally not recommended. |
33
+ | [Hercules-4.0-Mistral-v0.2-7B-Q4_K_M.gguf](https://huggingface.co/bartowski/Hercules-4.0-Mistral-v0.2-7B-GGUF/blob/main/Hercules-4.0-Mistral-v0.2-7B-Q4_K_M.gguf) | Q4_K_M | 4.36GB | Good quality, uses about 4.83 bits per weight. |
34
+ | [Hercules-4.0-Mistral-v0.2-7B-Q4_K_S.gguf](https://huggingface.co/bartowski/Hercules-4.0-Mistral-v0.2-7B-GGUF/blob/main/Hercules-4.0-Mistral-v0.2-7B-Q4_K_S.gguf) | Q4_K_S | 4.14GB | Slightly lower quality with small space savings. |
35
+ | [Hercules-4.0-Mistral-v0.2-7B-IQ4_NL.gguf](https://huggingface.co/bartowski/Hercules-4.0-Mistral-v0.2-7B-GGUF/blob/main/Hercules-4.0-Mistral-v0.2-7B-IQ4_NL.gguf) | IQ4_NL | 4.15GB | Decent quality, similar to Q4_K_S, new method of quanting, |
36
+ | [Hercules-4.0-Mistral-v0.2-7B-IQ4_XS.gguf](https://huggingface.co/bartowski/Hercules-4.0-Mistral-v0.2-7B-GGUF/blob/main/Hercules-4.0-Mistral-v0.2-7B-IQ4_XS.gguf) | IQ4_XS | 3.94GB | Decent quality, new method with similar performance to Q4. |
37
+ | [Hercules-4.0-Mistral-v0.2-7B-Q4_0.gguf](https://huggingface.co/bartowski/Hercules-4.0-Mistral-v0.2-7B-GGUF/blob/main/Hercules-4.0-Mistral-v0.2-7B-Q4_0.gguf) | Q4_0 | 4.10GB | Decent quality, older format, generally not recommended. |
38
+ | [Hercules-4.0-Mistral-v0.2-7B-Q3_K_L.gguf](https://huggingface.co/bartowski/Hercules-4.0-Mistral-v0.2-7B-GGUF/blob/main/Hercules-4.0-Mistral-v0.2-7B-Q3_K_L.gguf) | Q3_K_L | 3.82GB | Lower quality but usable, good for low RAM availability. |
39
+ | [Hercules-4.0-Mistral-v0.2-7B-Q3_K_M.gguf](https://huggingface.co/bartowski/Hercules-4.0-Mistral-v0.2-7B-GGUF/blob/main/Hercules-4.0-Mistral-v0.2-7B-Q3_K_M.gguf) | Q3_K_M | 3.51GB | Even lower quality. |
40
+ | [Hercules-4.0-Mistral-v0.2-7B-IQ3_M.gguf](https://huggingface.co/bartowski/Hercules-4.0-Mistral-v0.2-7B-GGUF/blob/main/Hercules-4.0-Mistral-v0.2-7B-IQ3_M.gguf) | IQ3_M | 3.28GB | Medium-low quality, new method with decent performance. |
41
+ | [Hercules-4.0-Mistral-v0.2-7B-IQ3_S.gguf](https://huggingface.co/bartowski/Hercules-4.0-Mistral-v0.2-7B-GGUF/blob/main/Hercules-4.0-Mistral-v0.2-7B-IQ3_S.gguf) | IQ3_S | 3.18GB | Lower quality, new method with decent performance, recommended over Q3 quants. |
42
+ | [Hercules-4.0-Mistral-v0.2-7B-Q3_K_S.gguf](https://huggingface.co/bartowski/Hercules-4.0-Mistral-v0.2-7B-GGUF/blob/main/Hercules-4.0-Mistral-v0.2-7B-Q3_K_S.gguf) | Q3_K_S | 3.16GB | Low quality, not recommended. |
43
+ | [Hercules-4.0-Mistral-v0.2-7B-Q2_K.gguf](https://huggingface.co/bartowski/Hercules-4.0-Mistral-v0.2-7B-GGUF/blob/main/Hercules-4.0-Mistral-v0.2-7B-Q2_K.gguf) | Q2_K | 2.71GB | Extremely low quality, *not* recommended.
44
+
45
+ Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski