Update README.md
Browse files
README.md
CHANGED
@@ -19,18 +19,27 @@ GGUF quantized models of [mattshumer/ref_70_e3](https://huggingface.co/mattshume
|
|
19 |
|
20 |
**Reflection Llama-3.1 70B is (currently) the world's top open-source LLM, trained with a new technique called Reflection-Tuning that teaches a LLM to detect mistakes in its reasoning and correct course.**
|
21 |
|
22 |
-
| Quantization | Size |
|
23 |
-
| ------------ | ------ |
|
24 |
-
| FP16 | 141GB |
|
25 |
-
| Q8_0_L | 73GB |
|
26 |
-
| Q6_K_L | 56.2GB |
|
27 |
-
|
|
28 |
-
|
|
29 |
-
|
|
30 |
-
|
|
31 |
-
|
|
32 |
-
|
33 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
34 |
|
35 |
## Benchmarks
|
36 |
![image/png](https://cdn-uploads.huggingface.co/production/uploads/60518f3731c5be7f3dd5ebc3/zNs-ZFs0SbnomH7mikiOU.png)
|
@@ -39,7 +48,7 @@ All benchmarks tested have been checked for contamination by running [LMSys's LL
|
|
39 |
|
40 |
Trained from Llama 3.1 70B Instruct, you can sample from Reflection Llama-3.1 70B using the same code, pipelines, etc. as any other Llama model. It even uses the stock Llama 3.1 chat template format (though, we've trained in a few new special tokens to aid in reasoning and reflection).
|
41 |
|
42 |
-
During sampling, the model will start by
|
43 |
|
44 |
This enables the model to separate its internal thoughts and reasoning from its final answer, improving the experience for the user.
|
45 |
|
@@ -57,25 +66,23 @@ We recommend using this exact system prompt to get the best results from Reflect
|
|
57 |
|
58 |
## Chat Format
|
59 |
|
60 |
-
|
61 |
|
62 |
```
|
63 |
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
|
64 |
|
65 |
You are a world-class AI system, capable of complex reasoning and reflection. Reason through the query inside <thinking> tags, and then provide your final response inside <output> tags. If you detect that you made a mistake in your reasoning at any point, correct yourself inside <reflection> tags.<|eot_id|><|start_header_id|>user<|end_header_id|>
|
66 |
|
67 |
-
|
68 |
```
|
69 |
|
70 |
## Tips for Performance
|
71 |
|
72 |
-
- We
|
73 |
- For increased accuracy, append `Think carefully.` at the end of your messages.
|
74 |
|
75 |
## Dataset / Report
|
76 |
|
77 |
Both the dataset and a brief report detailing how we trained this model will be released next week, alongside our Reflection 405B model that we expect will be the top-performing LLM in the world, including closed-source models.
|
78 |
|
79 |
-
---
|
80 |
-
|
81 |
Thanks to Jason Kuperberg and Josh Bickett from the [HyperWrite](https://hyperwriteai.com) team for reviewing drafts of the report we'll be releasing next week.
|
|
|
19 |
|
20 |
**Reflection Llama-3.1 70B is (currently) the world's top open-source LLM, trained with a new technique called Reflection-Tuning that teaches a LLM to detect mistakes in its reasoning and correct course.**
|
21 |
|
22 |
+
| Quantization | Size | Split | iMatrix |
|
23 |
+
| ------------ | ------ | ----- | ------- |
|
24 |
+
| FP16 | 141GB | true | false |
|
25 |
+
| Q8_0_L | 73GB | true | false |
|
26 |
+
| Q6_K_L | 56.2GB | true | false |
|
27 |
+
| Q6_K | ??.?GB | true | false |
|
28 |
+
| Q5_K_L | 52.6GB | true | false |
|
29 |
+
| Q5_K_M | ??.?GB | true | false |
|
30 |
+
| Q5_K_S | ??.?GB | false | false |
|
31 |
+
| Q4_K_L | 42.1GB | false | false |
|
32 |
+
| Q4_K_M | ??.?GB | false | false |
|
33 |
+
| Q4_K_S | ??.?GB | false | false |
|
34 |
+
| Q3_K_XL | ??.?GB | false | false |
|
35 |
+
| Q3_K_L | 40GB | false | false |
|
36 |
+
| Q3_K_M | ??.?GB | false | false |
|
37 |
+
| Q3_K_S | ??.?GB | false | false |
|
38 |
+
| Q2_K_L | 29.4GB | false | false |
|
39 |
+
| Q2_K | ??.?GB | false | false |
|
40 |
+
| Q2_K_S | ??.?GB | false | true |
|
41 |
+
|
42 |
+
The `_L` or `_XL` suffix means that the token embeddings and output weight are at fp16 precision.
|
43 |
|
44 |
## Benchmarks
|
45 |
![image/png](https://cdn-uploads.huggingface.co/production/uploads/60518f3731c5be7f3dd5ebc3/zNs-ZFs0SbnomH7mikiOU.png)
|
|
|
48 |
|
49 |
Trained from Llama 3.1 70B Instruct, you can sample from Reflection Llama-3.1 70B using the same code, pipelines, etc. as any other Llama model. It even uses the stock Llama 3.1 chat template format (though, we've trained in a few new special tokens to aid in reasoning and reflection).
|
50 |
|
51 |
+
During sampling, the model will start by generating reasoning inside `<thinking>` and `</thinking>` tags, and then once it is satisfied with its reasoning, it will output the final answer inside `<output>` and `</output>` tags. Each of these tags are special tokens, trained into the model.
|
52 |
|
53 |
This enables the model to separate its internal thoughts and reasoning from its final answer, improving the experience for the user.
|
54 |
|
|
|
66 |
|
67 |
## Chat Format
|
68 |
|
69 |
+
The model uses the standard Llama 3.1 chat format. Here’s an example:
|
70 |
|
71 |
```
|
72 |
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
|
73 |
|
74 |
You are a world-class AI system, capable of complex reasoning and reflection. Reason through the query inside <thinking> tags, and then provide your final response inside <output> tags. If you detect that you made a mistake in your reasoning at any point, correct yourself inside <reflection> tags.<|eot_id|><|start_header_id|>user<|end_header_id|>
|
75 |
|
76 |
+
What is 2+2?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
|
77 |
```
|
78 |
|
79 |
## Tips for Performance
|
80 |
|
81 |
+
- We recommend a `temperature` of `.7` and a `top_p` of `.95`.
|
82 |
- For increased accuracy, append `Think carefully.` at the end of your messages.
|
83 |
|
84 |
## Dataset / Report
|
85 |
|
86 |
Both the dataset and a brief report detailing how we trained this model will be released next week, alongside our Reflection 405B model that we expect will be the top-performing LLM in the world, including closed-source models.
|
87 |
|
|
|
|
|
88 |
Thanks to Jason Kuperberg and Josh Bickett from the [HyperWrite](https://hyperwriteai.com) team for reviewing drafts of the report we'll be releasing next week.
|