andreaskoepf commited on
Commit
39f6fc8
1 Parent(s): 934bda9

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -0
README.md CHANGED
@@ -6,6 +6,7 @@ license: apache-2.0
6
 
7
  This is an intermediate model used as base-model for further pythia 12b SFT-8 experiments.
8
  It was trained on a wider set of instruction-tuning datasets for >12.5k steps with batch-size 128 and a context size of 2048.
 
9
 
10
 
11
  - wandb: https://wandb.ai/open-assistant/supervised-finetuning/runs/sytsyhrp
 
6
 
7
  This is an intermediate model used as base-model for further pythia 12b SFT-8 experiments.
8
  It was trained on a wider set of instruction-tuning datasets for >12.5k steps with batch-size 128 and a context size of 2048.
9
+ The gpt4all dataset had "as a language model" *contamination* (>1.8k entries). We added filtering later, but this model (pre-v8) was trained on the raw unfildered gpt4all dataset.
10
 
11
 
12
  - wandb: https://wandb.ai/open-assistant/supervised-finetuning/runs/sytsyhrp