squeeze-ai-lab commited on
Commit
a6292a8
1 Parent(s): 1c09791

Update README

Browse files
Files changed (1) hide show
  1. README.md +29 -1
README.md CHANGED
@@ -1,3 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: mit
3
- ---
 
1
+ **KVQuant** is a methodology for efficient KV cache quantization that incorporates several innovations to acheive accurate low-precision quantization,
2
+ thereby enabling efficient long context length inference.
3
+
4
+ **TLDR:** KVQuant addresses the memory bottleneck with long context length inference by quantizing the KV cache to low precision.
5
+ KVQuant achieves high accuracy with low-precision KV cache quantization by considering several consistent patterns observed in cached KV values across different LLMs,
6
+ and by developing methods to exploit these patterns, including:
7
+
8
+ - **Per-channel, Pre-RoPE** Key quantization to better match the outlier channels in Keys
9
+ - Non-Uniform Quantization (**NUQ**) to better represent the non-uniform activations
10
+ - **Dense-and-Sparse Quantization** to mitigate the impacts of numerical outliers on quantization difficulty
11
+ - **Q-Norm** to mitigate distribution shift at ultra low precisions (eg. 2-bit)
12
+ - **Attention-Sink Aware Quantization** to avoid quantization error with the first token, which is disproportionately sensitive to quantization error
13
+
14
+ For more details please check out our [paper](https://arxiv.org/abs/2401.18079.pdf).
15
+
16
+ ## Model description
17
+
18
+ Quantizer file for running DBRX with 2-bit KV cache using KVQuant.
19
+
20
+ * **Base Model:** [DBRX-Instruct](https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm)
21
+ * **Bitwidth:** 2-bit
22
+ * **Sparsity Level:** 1%
23
+
24
+ ## Links
25
+
26
+ * **Paper**: [https://arxiv.org/abs/2401.18079.pdf](https://arxiv.org/abs/2401.18079.pdf)
27
+ * **Code**: [https://github.com/SqueezeAILab/KVQuant](https://github.com/SqueezeAILab/KVQuant)
28
+
29
  ---
30
  license: mit
31
+ ---