andreaskoepf
commited on
Commit
•
39f6fc8
1
Parent(s):
934bda9
Update README.md
Browse files
README.md
CHANGED
@@ -6,6 +6,7 @@ license: apache-2.0
|
|
6 |
|
7 |
This is an intermediate model used as base-model for further pythia 12b SFT-8 experiments.
|
8 |
It was trained on a wider set of instruction-tuning datasets for >12.5k steps with batch-size 128 and a context size of 2048.
|
|
|
9 |
|
10 |
|
11 |
- wandb: https://wandb.ai/open-assistant/supervised-finetuning/runs/sytsyhrp
|
|
|
6 |
|
7 |
This is an intermediate model used as base-model for further pythia 12b SFT-8 experiments.
|
8 |
It was trained on a wider set of instruction-tuning datasets for >12.5k steps with batch-size 128 and a context size of 2048.
|
9 |
+
The gpt4all dataset had "as a language model" *contamination* (>1.8k entries). We added filtering later, but this model (pre-v8) was trained on the raw unfildered gpt4all dataset.
|
10 |
|
11 |
|
12 |
- wandb: https://wandb.ai/open-assistant/supervised-finetuning/runs/sytsyhrp
|