Update README.md

README.md

@@ -19,9 +19,4 @@ rotary_dim - 64
 tokenizer - gpt-j
 ```
 
-First, trained on 4,194,304 samples from the [c4](https://hf.co/datasets/c4) dataset, at a length of 128 tokens each, that comes out to 536,870,912 (0.53B) tokens seen during training. A batch size of 16 with 128 gradient accumulation steps was used, making the effective batch size 2048. A cosine learning rate schedule was used starting at 1e-3.
-
-Second, the same settings, with double the accumulation steps, were used on [EleutherAI/the_pile_deduplicated](https://hf.co/EleutherAI/the_pile_deduplicated) with a learning rate of 1e-5 with a linearly decreasing schedule after the training on c4 was done. (for a total of ~1.06B tokens seen)
-
-
-it's like this because i forgot to train on the pile in the first place and i wasn't going to let somewhere around 13 hours of gpu time go to waste I couldn't play minecraft that entire time
+First, trained on 4,194,304 samples from the [c4](https://hf.co/datasets/c4) dataset, at a length of 128 tokens each, that comes out to 536,870,912 (0.53B) tokens seen during training. A batch size of 16 with 128 gradient accumulation steps was used, making the effective batch size 2048. A cosine learning rate schedule was used starting at 1e-3.