Update README.md

README.md

@@ -33,6 +33,66 @@ Trained on wikipedia datasets:
 Trained MLM and NSP pre-training scheme from [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://arxiv.org/abs/1904.00962).
 Trained on 16 Graphcore Mk2 IPUs.
 
+Command lines:
+
+Phase 1:
+```
+python examples/language-modeling/run_pretraining.py \
+  --config_name bert-base-uncased \
+  --tokenizer_name bert-base-uncased \
+  --do_train \
+  --logging_steps 5 \
+  --max_seq_length 128 \
+  --ipu_config_name Graphcore/bert-base-ipu \
+  --dataset_name Graphcore/wikipedia-bert-128 \
+  --max_steps 10500 \
+  --is_already_preprocessed \
+  --dataloader_num_workers 64 \
+  --dataloader_mode async_rebatched \
+  --lamb \
+  --lamb_no_bias_correction \
+  --per_device_train_batch_size 32 \
+  --gradient_accumulation_steps 512 \
+  --learning_rate 0.006 \
+  --lr_scheduler_type linear \
+  --loss_scaling 16384 \
+  --weight_decay 0.01 \
+  --warmup_ratio 0.28 \
+  --save_steps 100 \
+  --config_overrides "layer_norm_eps=0.001" \
+  --ipu_config_overrides "device_iterations=1" \
+  --output_dir output-pretrain-bert-base-phase1
+```
+
+Phase 2:
+```
+python examples/language-modeling/run_pretraining.py \
+  --config_name bert-base-uncased \
+  --tokenizer_name bert-base-uncased \
+  --model_name_or_path ./output-pretrain-bert-base-phase1 \
+  --do_train \
+  --logging_steps 5 \
+  --max_seq_length 512 \
+  --ipu_config_name Graphcore/bert-base-ipu \
+  --dataset_name Graphcore/wikipedia-bert-512 \
+  --max_steps 2038 \
+  --is_already_preprocessed \
+  --dataloader_num_workers 128 \
+  --dataloader_mode async_rebatched \
+  --lamb \
+  --lamb_no_bias_correction \
+  --per_device_train_batch_size 8 \
+  --gradient_accumulation_steps 512 \
+  --learning_rate 0.002828 \
+  --lr_scheduler_type linear \
+  --loss_scaling 128.0 \
+  --weight_decay 0.01 \
+  --warmup_ratio 0.128 \
+  --config_overrides "layer_norm_eps=0.001" \
+  --ipu_config_overrides "device_iterations=1,embedding_serialization_factor=2,matmul_proportion=0.22" \
+  --output_dir output-pretrain-bert-base-phase2
+```
+
 ### Training hyperparameters
 
 The following hyperparameters were used during phase 1 training: