Update README.md
Browse files
README.md
CHANGED
@@ -10,4 +10,8 @@ Pythia-6.9B Deduped 4K is a [Pythia-6.9B Deduped](https://huggingface.co/Eleuthe
|
|
10 |
Training resumed from their 143,000 step checkpoint and continued on The Pile v1 Deduped (threshold=0.87).
|
11 |
This particular model is from a checkpoint captured at step 175,500 for an extra 134,217,728,000 tokens of training.
|
12 |
|
13 |
-
Note: Sequence length warmup was not used to move up from 2048 but, in hindsight, should have been applied.
|
|
|
|
|
|
|
|
|
|
10 |
Training resumed from their 143,000 step checkpoint and continued on The Pile v1 Deduped (threshold=0.87).
|
11 |
This particular model is from a checkpoint captured at step 175,500 for an extra 134,217,728,000 tokens of training.
|
12 |
|
13 |
+
Note: Sequence length warmup was not used to move up from 2048 but, in hindsight, should have been applied.
|
14 |
+
|
15 |
+
## Acknoweldgements
|
16 |
+
|
17 |
+
This work would not have been possible without the support of [Stability AI](https://stability.ai/).
|