Update readme on quants

README.md

@@ -50,9 +50,37 @@ For training data, we generate long contexts by augmenting [SlimPajama](https://
 | GPU Type               | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S |
 | Minutes to Train (Wall)| 202       | 555       | 61        | 87        |
 
-**Quants**:
-- [GGUF](https://huggingface.co/crusoeai/Llama-3-8B-Instruct-1048k-GGUF)
+
+**Evaluation:**
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/6585dc9be92bc5f258156bd6/mWxIGZNi3ejlmeIDWafKu.png)
+
+```
+EVAL_MAX_CONTEXT_LENGTH=1040200
+EVAL_MIN_CONTEXT_LENGTH=100
+EVAL_CONTEXT_INTERVAL=86675
+EVAL_DEPTH_INTERVAL=0.2
+EVAL_RND_NUMBER_DIGITS=8
+
+HAYSTACK1:
+    EVAL_GENERATOR_TOKENS=25
+
+HAYSTACK2:
+    EVAL_CONTEXT_INTERVAL=173350
+    EVAL_GENERATOR_TOKENS=150000
+
+HAYSTACK3:
+    EVAL_GENERATOR_TOKENS=925000
+```
+
+All boxes not pictured for Haystack 1 and 3 are 100% accurate. Haystacks 1,2 and 3 are further detailed in this [blog post](https://gradient.ai/blog/the-haystack-matters-for-niah-evals).
+
+**Quants:**
+- [GGUF by Crusoe](https://huggingface.co/crusoeai/Llama-3-8B-Instruct-1048k-GGUF). Note that you need to add 128009 as [special token with llama.cpp](https://huggingface.co/gradientai/Llama-3-8B-Instruct-262k/discussions/13).
 - [MLX-4bit](https://huggingface.co/mlx-community/Llama-3-8B-Instruct-1048k-4bit)
+- [Ollama](https://ollama.com/library/llama3-gradient)
+- vLLM docker image, recommended to load via `--max-model-len 32768`
+- If you are interested in a hosted version, drop us a mail below.
 
 ## The Gradient AI Team