DavidAU commited on
Commit
2687c6e
·
verified ·
1 Parent(s): 8892b4a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +3 -1
README.md CHANGED
@@ -39,7 +39,9 @@ pipeline_tag: text-generation
39
  - All quants have also been upgraded with "more bits" for output tensor (all set at Q8_0) and embed for better performance (this is in addition to the "refresh")
40
  - New specialized quants (in addition to the new refresh/upgrades): "max, max-cpu" (will include this in the file name) for quants "Q2K" (max cpu only), "IQ4_XS", "Q6_K" and "Q8_0"
41
  - "MAX": output tensor / embed at float 16. You get better instruction following/output generation than standard/upgraded quants.
42
- - "MAX-CPU": output tensor / embed at bfloat 16, which forces both of these on to the CPU (Nvidia cards / other will vary), this frees up vram at cost of token/second and you get better instruction following/output generation too. Example: q8_0 Max-CPU : 2004 mb will load on to CPU/RAM, 7073 mb will load onto the GPU/vram. Extra Vram can be used for context. NOTE: "Math" on the CPU is slightly more accurate than GPU, so you may get a better generation.
 
 
43
  - Q8_0 (Max,Max-CPU) now clocks in at 9.5 bits per weight (average).
44
 
45
  <h2>L3-Dark-Planet-8B-GGUF</h2>
 
39
  - All quants have also been upgraded with "more bits" for output tensor (all set at Q8_0) and embed for better performance (this is in addition to the "refresh")
40
  - New specialized quants (in addition to the new refresh/upgrades): "max, max-cpu" (will include this in the file name) for quants "Q2K" (max cpu only), "IQ4_XS", "Q6_K" and "Q8_0"
41
  - "MAX": output tensor / embed at float 16. You get better instruction following/output generation than standard/upgraded quants.
42
+ - "MAX-CPU": output tensor / embed at bfloat 16, which forces both of these on to the CPU (Nvidia cards / other will vary), this frees up vram at cost of token/second and you get better instruction following/output generation too.
43
+ - "MAX-CPU": Example 1: q8_0 Max-CPU : 2004 mb will load on to CPU/RAM, 7073 mb will load onto the GPU/vram. Extra Vram can be used for context. NOTE: "Math" on the CPU is slightly more accurate than GPU, so you may get a better generation.
44
+ - "MAX-CPU": Example 2: q2_k Max-CPU : 2004 mb will load on to CPU/RAM, 2449 mb will load onto the GPU/vram. Extra Vram can be used for context. NOTE: "Math" on the CPU is slightly more accurate than GPU, so you may get a better generation. You could run this model/quant on a 4GB vram card.
45
  - Q8_0 (Max,Max-CPU) now clocks in at 9.5 bits per weight (average).
46
 
47
  <h2>L3-Dark-Planet-8B-GGUF</h2>