Pile_training_log.txt

NeoXArgs.configure_distributed_args() using world size: 64 and model-parallel size: 1 
> building HFTokenizer tokenizer ...
 > padded vocab (size: 50277) with 27 dummy tokens (new size: 50304)
> setting tensorboard ...
> initializing torch distributed ...
> initializing model parallel with size 1
MPU DP: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
MPU PP: [0]
MPU PP: [1]
MPU PP: [2]
MPU PP: [3]
MPU PP: [4]
MPU PP: [5]
MPU PP: [6]
MPU PP: [7]
MPU PP: [8]
MPU PP: [9]
MPU PP: [10]
MPU PP: [11]
MPU PP: [12]
MPU PP: [13]
MPU PP: [14]
MPU PP: [15]
MPU PP: [16]
MPU PP: [17]
MPU PP: [18]
MPU PP: [19]
MPU PP: [20]
MPU PP: [21]
MPU PP: [22]
MPU PP: [23]
MPU PP: [24]
MPU PP: [25]
MPU PP: [26]
MPU PP: [27]
MPU PP: [28]
MPU PP: [29]
MPU PP: [30]
MPU PP: [31]
MPU PP: [32]
MPU PP: [33]
MPU PP: [34]
MPU PP: [35]
MPU PP: [36]
MPU PP: [37]
MPU PP: [38]
MPU PP: [39]
MPU PP: [40]
MPU PP: [41]
MPU PP: [42]
MPU PP: [43]
MPU PP: [44]
MPU PP: [45]
MPU PP: [46]
MPU PP: [47]
MPU PP: [48]
MPU PP: [49]
MPU PP: [50]
MPU PP: [51]
MPU PP: [52]
MPU PP: [53]
MPU PP: [54]
MPU PP: [55]
MPU PP: [56]
MPU PP: [57]
MPU PP: [58]
MPU PP: [59]
MPU PP: [60]
MPU PP: [61]
MPU PP: [62]
MPU PP: [63]
MPU MP: [0]
MPU MP: [1]
MPU MP: [2]
MPU MP: [3]
MPU MP: [4]
MPU MP: [5]
MPU MP: [6]
MPU MP: [7]
MPU MP: [8]
MPU MP: [9]
MPU MP: [10]
MPU MP: [11]
MPU MP: [12]
MPU MP: [13]
MPU MP: [14]
MPU MP: [15]
MPU MP: [16]
MPU MP: [17]
MPU MP: [18]
MPU MP: [19]
MPU MP: [20]
MPU MP: [21]
MPU MP: [22]
MPU MP: [23]
MPU MP: [24]
MPU MP: [25]
MPU MP: [26]
MPU MP: [27]
MPU MP: [28]
MPU MP: [29]
MPU MP: [30]
MPU MP: [31]
MPU MP: [32]
MPU MP: [33]
MPU MP: [34]
MPU MP: [35]
MPU MP: [36]
MPU MP: [37]
MPU MP: [38]
MPU MP: [39]
MPU MP: [40]
MPU MP: [41]
MPU MP: [42]
MPU MP: [43]
MPU MP: [44]
MPU MP: [45]
MPU MP: [46]
MPU MP: [47]
MPU MP: [48]
MPU MP: [49]
MPU MP: [50]
MPU MP: [51]
MPU MP: [52]
MPU MP: [53]
MPU MP: [54]
MPU MP: [55]
MPU MP: [56]
MPU MP: [57]
MPU MP: [58]
MPU MP: [59]
MPU MP: [60]
MPU MP: [61]
MPU MP: [62]
MPU MP: [63]
> setting random seeds to 1234 ...
building GPT2 model ...
SEED_LAYERS=False BASE_SEED=1234 SEED_FN=None
Using topology: {ProcessCoord(pipe=0, data=0, model=0): 0, ProcessCoord(pipe=0, data=1, model=0): 1, ProcessCoord(pipe=0, data=2, model=0): 2, ProcessCoord(pipe=0, data=3, model=0): 3, ProcessCoord(pipe=0, data=4, model=0): 4, ProcessCoord(pipe=0, data=5, model=0): 5, ProcessCoord(pipe=0, data=6, model=0): 6, ProcessCoord(pipe=0, data=7, model=0): 7, ProcessCoord(pipe=0, data=8, model=0): 8, ProcessCoord(pipe=0, data=9, model=0): 9, ProcessCoord(pipe=0, data=10, model=0): 10, ProcessCoord(pipe=0, data=11, model=0): 11, ProcessCoord(pipe=0, data=12, model=0): 12, ProcessCoord(pipe=0, data=13, model=0): 13, ProcessCoord(pipe=0, data=14, model=0): 14, ProcessCoord(pipe=0, data=15, model=0): 15, ProcessCoord(pipe=0, data=16, model=0): 16, ProcessCoord(pipe=0, data=17, model=0): 17, ProcessCoord(pipe=0, data=18, model=0): 18, ProcessCoord(pipe=0, data=19, model=0): 19, ProcessCoord(pipe=0, data=20, model=0): 20, ProcessCoord(pipe=0, data=21, model=0): 21, ProcessCoord(pipe=0, data=22, model=0): 22, ProcessCoord(pipe=0, data=23, model=0): 23, ProcessCoord(pipe=0, data=24, model=0): 24, ProcessCoord(pipe=0, data=25, model=0): 25, ProcessCoord(pipe=0, data=26, model=0): 26, ProcessCoord(pipe=0, data=27, model=0): 27, ProcessCoord(pipe=0, data=28, model=0): 28, ProcessCoord(pipe=0, data=29, model=0): 29, ProcessCoord(pipe=0, data=30, model=0): 30, ProcessCoord(pipe=0, data=31, model=0): 31, ProcessCoord(pipe=0, data=32, model=0): 32, ProcessCoord(pipe=0, data=33, model=0): 33, ProcessCoord(pipe=0, data=34, model=0): 34, ProcessCoord(pipe=0, data=35, model=0): 35, ProcessCoord(pipe=0, data=36, model=0): 36, ProcessCoord(pipe=0, data=37, model=0): 37, ProcessCoord(pipe=0, data=38, model=0): 38, ProcessCoord(pipe=0, data=39, model=0): 39, ProcessCoord(pipe=0, data=40, model=0): 40, ProcessCoord(pipe=0, data=41, model=0): 41, ProcessCoord(pipe=0, data=42, model=0): 42, ProcessCoord(pipe=0, data=43, model=0): 43, ProcessCoord(pipe=0, data=44, model=0): 44, ProcessCoord(pipe=0, data=45, model=0): 45, ProcessCoord(pipe=0, data=46, model=0): 46, ProcessCoord(pipe=0, data=47, model=0): 47, ProcessCoord(pipe=0, data=48, model=0): 48, ProcessCoord(pipe=0, data=49, model=0): 49, ProcessCoord(pipe=0, data=50, model=0): 50, ProcessCoord(pipe=0, data=51, model=0): 51, ProcessCoord(pipe=0, data=52, model=0): 52, ProcessCoord(pipe=0, data=53, model=0): 53, ProcessCoord(pipe=0, data=54, model=0): 54, ProcessCoord(pipe=0, data=55, model=0): 55, ProcessCoord(pipe=0, data=56, model=0): 56, ProcessCoord(pipe=0, data=57, model=0): 57, ProcessCoord(pipe=0, data=58, model=0): 58, ProcessCoord(pipe=0, data=59, model=0): 59, ProcessCoord(pipe=0, data=60, model=0): 60, ProcessCoord(pipe=0, data=61, model=0): 61, ProcessCoord(pipe=0, data=62, model=0): 62, ProcessCoord(pipe=0, data=63, model=0): 63}
stage=0 layers=45
     0: EmbeddingPipe
     1: _pre_transformer_block
     2: ParallelFlownetLayerPipe
     3: ParallelFlownetLayerPipe
     4: ParallelFlownetLayerPipe
     5: ParallelFlownetLayerPipe
     6: ParallelFlownetLayerPipe
     7: ParallelFlownetLayerPipe
     8: ParallelFlownetLayerPipe
     9: ParallelFlownetLayerPipe
    10: ParallelFlownetLayerPipe
    11: ParallelFlownetLayerPipe
    12: ParallelFlownetLayerPipe
    13: ParallelFlownetLayerPipe
    14: ParallelFlownetLayerPipe
    15: ParallelFlownetLayerPipe
    16: ParallelFlownetLayerPipe
    17: ParallelFlownetLayerPipe
    18: ParallelFlownetLayerPipe
    19: ParallelFlownetLayerPipe
    20: ParallelFlownetLayerPipe
    21: ParallelFlownetLayerPipe
    22: ParallelFlownetLayerPipe
    23: ParallelFlownetLayerPipe
    24: ParallelFlownetLayerPipe
    25: ParallelFlownetLayerPipe
    26: ParallelFlownetLayerPipe
    27: ParallelFlownetLayerPipe
    28: ParallelFlownetLayerPipe
    29: ParallelFlownetLayerPipe
    30: ParallelFlownetLayerPipe
    31: ParallelFlownetLayerPipe
    32: ParallelFlownetLayerPipe
    33: ParallelFlownetLayerPipe
    34: ParallelFlownetLayerPipe
    35: ParallelFlownetLayerPipe
    36: ParallelFlownetLayerPipe
    37: ParallelFlownetLayerPipe
    38: ParallelFlownetLayerPipe
    39: ParallelFlownetLayerPipe
    40: ParallelFlownetLayerPipe
    41: ParallelFlownetLayerPipe
    42: _post_transformer_block
    43: NormPipe
    44: EmbeddingPipe
  loss: partial
Configuring Optimizer type: Adam with params: {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}
> learning rate decay style: cosine
DeepSpeed is enabled.
 > number of parameters on model parallel rank 0: 1587219456
 > total params: 1,587,219,456
Unable to load checkpoint.
Loading checkpoint and starting from iteration 0
> building train, validation, and test datasets ...
    reading sizes...
    reading pointers...
    reading document index...
    creating numpy buffer of mmap...
    creating memory view of numpy buffer...
 > dataset split:
    train:
     document indices in [0, 130154259) total of 130154259 documents
    validation:
     document indices in [130154259, 134183803) total of 4029544 documents
    test:
     document indices in [134183803, 134318121) total of 134318 documents
 > loading doc-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_train_indexmap_146432000ns_2048sl_1234s_doc_idx.npy
 > loading sample-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_train_indexmap_146432000ns_2048sl_1234s_sample_idx.npy
 > loading shuffle-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_train_indexmap_146432000ns_2048sl_1234s_shuffle_idx.npy
    loaded indexed file in 0.251 seconds
    total number of samples: 195915016
    total number of epochs: 2
 > loading doc-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_valid_indexmap_2969600ns_2048sl_1234s_doc_idx.npy
 > loading sample-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_valid_indexmap_2969600ns_2048sl_1234s_sample_idx.npy
 > loading shuffle-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_valid_indexmap_2969600ns_2048sl_1234s_shuffle_idx.npy
    loaded indexed file in 0.218 seconds
    total number of samples: 3097460
    total number of epochs: 1
 > loading doc-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_test_indexmap_102400ns_2048sl_1234s_doc_idx.npy
 > loading sample-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_test_indexmap_102400ns_2048sl_1234s_sample_idx.npy
 > loading shuffle-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_test_indexmap_102400ns_2048sl_1234s_shuffle_idx.npy
    loaded indexed file in 0.144 seconds
    total number of samples: 102462
    total number of epochs: 1
setting training data start iteration to 0
setting validation data start iteration to 0
done with setups ...
time (ms) | model and optimizer: 4507.04 | train/valid/test data iterators: 6699.92
training ...
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step0
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step1
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step2
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step4
 samples/sec: 221.838 | iteration      500/  143000 | elapsed time per iteration (ms): 4616.0 | learning rate: 2.098E-04 | approx flops per GPU: 81.9TFLOPS | lm_loss: 6.314383E+00 | loss scale: 4096.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
after 500 iterations memory (MB) | allocated: 6356.6455078125 | max allocated: 16125.09765625 | reserved: 17282.0 | max reserved: 17282.0
time (ms) | forward: 940.28 | backward: 3469.40 | backward-backward: 3469.33 | backward-allreduce: 0.00 | optimizer: 138.22 | batch generator: 3.18
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step8
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step16
 samples/sec: 225.774 | iteration     1000/  143000 | elapsed time per iteration (ms): 4535.5 | learning rate: 4.196E-04 | approx flops per GPU: 83.3TFLOPS | lm_loss: 3.695211E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.00 | backward: 3446.64 | backward-backward: 3446.57 | backward-allreduce: 0.00 | optimizer: 138.24 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step32
 samples/sec: 226.139 | iteration     1500/  143000 | elapsed time per iteration (ms): 4528.2 | learning rate: 6.000E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 3.099024E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.80 | backward: 3449.78 | backward-backward: 3449.70 | backward-allreduce: 0.00 | optimizer: 138.24 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step64
 samples/sec: 226.333 | iteration     2000/  143000 | elapsed time per iteration (ms): 4524.3 | learning rate: 6.000E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.830897E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.27 | backward: 3444.95 | backward-backward: 3444.88 | backward-allreduce: 0.00 | optimizer: 138.16 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step128
 samples/sec: 226.182 | iteration     2500/  143000 | elapsed time per iteration (ms): 4527.3 | learning rate: 5.999E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.682815E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.73 | backward: 3448.22 | backward-backward: 3448.14 | backward-allreduce: 0.00 | optimizer: 138.18 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step256
 samples/sec: 226.165 | iteration     3000/  143000 | elapsed time per iteration (ms): 4527.7 | learning rate: 5.998E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.592844E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.02 | backward: 3449.06 | backward-backward: 3448.99 | backward-allreduce: 0.00 | optimizer: 137.99 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step500
 samples/sec: 225.553 | iteration     3500/  143000 | elapsed time per iteration (ms): 4539.9 | learning rate: 5.997E-04 | approx flops per GPU: 83.3TFLOPS | lm_loss: 2.534201E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.25 | backward: 3459.57 | backward-backward: 3459.49 | backward-allreduce: 0.00 | optimizer: 139.31 | batch generator: 2.86
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step512
 samples/sec: 226.121 | iteration     4000/  143000 | elapsed time per iteration (ms): 4528.6 | learning rate: 5.996E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.490065E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.60 | backward: 3445.28 | backward-backward: 3445.21 | backward-allreduce: 0.00 | optimizer: 137.65 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step1000
 samples/sec: 226.309 | iteration     4500/  143000 | elapsed time per iteration (ms): 4524.8 | learning rate: 5.994E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.456325E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.59 | backward: 3446.43 | backward-backward: 3446.35 | backward-allreduce: 0.00 | optimizer: 137.77 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step1500
 samples/sec: 226.310 | iteration     5000/  143000 | elapsed time per iteration (ms): 4524.8 | learning rate: 5.992E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.428536E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.51 | backward: 3446.61 | backward-backward: 3446.54 | backward-allreduce: 0.00 | optimizer: 137.67 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step2000
---------------------------------------------------------------------------------------------------------
 validation results at iteration 5000 | lm_loss value: 2.367668E+00 | lm_loss_ppl value: 1.067248E+01 | 
---------------------------------------------------------------------------------------------------------
 samples/sec: 216.965 | iteration     5500/  143000 | elapsed time per iteration (ms): 4719.7 | learning rate: 5.989E-04 | approx flops per GPU: 80.1TFLOPS | lm_loss: 2.405654E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.09 | backward: 3446.48 | backward-backward: 3446.41 | backward-allreduce: 0.00 | optimizer: 137.96 | batch generator: 2.67
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step2500
 samples/sec: 226.293 | iteration     6000/  143000 | elapsed time per iteration (ms): 4525.1 | learning rate: 5.986E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.386074E+00 | loss scale: 131072.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.28 | backward: 3447.20 | backward-backward: 3447.13 | backward-allreduce: 0.00 | optimizer: 137.87 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step3000
 samples/sec: 226.315 | iteration     6500/  143000 | elapsed time per iteration (ms): 4524.7 | learning rate: 5.983E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.367453E+00 | loss scale: 65536.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 931.23 | backward: 3446.88 | backward-backward: 3446.81 | backward-allreduce: 0.00 | optimizer: 137.21 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step3500
 samples/sec: 226.530 | iteration     7000/  143000 | elapsed time per iteration (ms): 4520.4 | learning rate: 5.979E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.354038E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.58 | backward: 3442.72 | backward-backward: 3442.65 | backward-allreduce: 0.00 | optimizer: 137.52 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step4000
 samples/sec: 226.107 | iteration     7500/  143000 | elapsed time per iteration (ms): 4528.8 | learning rate: 5.976E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.342027E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 933.75 | backward: 3448.55 | backward-backward: 3448.47 | backward-allreduce: 0.00 | optimizer: 137.72 | batch generator: 4.74
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step4500
 samples/sec: 226.502 | iteration     8000/  143000 | elapsed time per iteration (ms): 4520.9 | learning rate: 5.971E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.329950E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.50 | backward: 3443.02 | backward-backward: 3442.94 | backward-allreduce: 0.00 | optimizer: 137.78 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step5000
 samples/sec: 226.361 | iteration     8500/  143000 | elapsed time per iteration (ms): 4523.7 | learning rate: 5.967E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.319938E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.40 | backward: 3446.50 | backward-backward: 3446.43 | backward-allreduce: 0.00 | optimizer: 137.63 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step5500
 samples/sec: 226.503 | iteration     9000/  143000 | elapsed time per iteration (ms): 4520.9 | learning rate: 5.962E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.308959E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.68 | backward: 3442.73 | backward-backward: 3442.66 | backward-allreduce: 0.00 | optimizer: 137.98 | batch generator: 2.36
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step6000
 samples/sec: 226.511 | iteration     9500/  143000 | elapsed time per iteration (ms): 4520.8 | learning rate: 5.957E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.300303E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.52 | backward: 3442.84 | backward-backward: 3442.77 | backward-allreduce: 0.00 | optimizer: 137.91 | batch generator: 2.37
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step6500
 samples/sec: 226.280 | iteration    10000/  143000 | elapsed time per iteration (ms): 4525.4 | learning rate: 5.951E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.292035E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.50 | backward: 3447.59 | backward-backward: 3447.51 | backward-allreduce: 0.00 | optimizer: 138.00 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step7000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 10000 | lm_loss value: 2.242038E+00 | lm_loss_ppl value: 9.412495E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 216.743 | iteration    10500/  143000 | elapsed time per iteration (ms): 4724.5 | learning rate: 5.946E-04 | approx flops per GPU: 80.0TFLOPS | lm_loss: 2.285537E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.29 | backward: 3450.05 | backward-backward: 3449.97 | backward-allreduce: 0.00 | optimizer: 137.96 | batch generator: 2.72
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step7500
 samples/sec: 226.311 | iteration    11000/  143000 | elapsed time per iteration (ms): 4524.8 | learning rate: 5.939E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.278703E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.52 | backward: 3446.94 | backward-backward: 3446.87 | backward-allreduce: 0.00 | optimizer: 137.93 | batch generator: 2.38
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step8000
 samples/sec: 226.524 | iteration    11500/  143000 | elapsed time per iteration (ms): 4520.5 | learning rate: 5.933E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.271160E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.23 | backward: 3442.70 | backward-backward: 3442.62 | backward-allreduce: 0.00 | optimizer: 137.78 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step8500
 samples/sec: 226.499 | iteration    12000/  143000 | elapsed time per iteration (ms): 4521.0 | learning rate: 5.926E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.267786E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.45 | backward: 3442.90 | backward-backward: 3442.82 | backward-allreduce: 0.00 | optimizer: 138.05 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step9000
 samples/sec: 226.322 | iteration    12500/  143000 | elapsed time per iteration (ms): 4524.5 | learning rate: 5.919E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.260543E+00 | loss scale: 131072.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.32 | backward: 3446.52 | backward-backward: 3446.44 | backward-allreduce: 0.00 | optimizer: 138.03 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step9500
 samples/sec: 226.500 | iteration    13000/  143000 | elapsed time per iteration (ms): 4521.0 | learning rate: 5.912E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.254842E+00 | loss scale: 65536.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 931.57 | backward: 3442.73 | backward-backward: 3442.66 | backward-allreduce: 0.00 | optimizer: 137.55 | batch generator: 2.35
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step10000
 samples/sec: 226.529 | iteration    13500/  143000 | elapsed time per iteration (ms): 4520.4 | learning rate: 5.904E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.249845E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.33 | backward: 3442.60 | backward-backward: 3442.53 | backward-allreduce: 0.00 | optimizer: 137.84 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step10500
 samples/sec: 226.095 | iteration    14000/  143000 | elapsed time per iteration (ms): 4529.1 | learning rate: 5.896E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.246879E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.50 | backward: 3451.09 | backward-backward: 3451.02 | backward-allreduce: 0.00 | optimizer: 138.01 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step11000
 samples/sec: 226.482 | iteration    14500/  143000 | elapsed time per iteration (ms): 4521.3 | learning rate: 5.887E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.241458E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.36 | backward: 3443.34 | backward-backward: 3443.27 | backward-allreduce: 0.00 | optimizer: 138.32 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step11500
 samples/sec: 226.480 | iteration    15000/  143000 | elapsed time per iteration (ms): 4521.4 | learning rate: 5.879E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.238571E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.53 | backward: 3443.34 | backward-backward: 3443.27 | backward-allreduce: 0.00 | optimizer: 138.29 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step12000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 15000 | lm_loss value: 2.186796E+00 | lm_loss_ppl value: 8.906630E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 217.174 | iteration    15500/  143000 | elapsed time per iteration (ms): 4715.1 | learning rate: 5.870E-04 | approx flops per GPU: 80.2TFLOPS | lm_loss: 2.232844E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.40 | backward: 3442.74 | backward-backward: 3442.66 | backward-allreduce: 0.00 | optimizer: 138.22 | batch generator: 2.66
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step12500
 samples/sec: 226.318 | iteration    16000/  143000 | elapsed time per iteration (ms): 4524.6 | learning rate: 5.860E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.229282E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.56 | backward: 3446.90 | backward-backward: 3446.82 | backward-allreduce: 0.00 | optimizer: 137.88 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step13000
 samples/sec: 226.542 | iteration    16500/  143000 | elapsed time per iteration (ms): 4520.1 | learning rate: 5.851E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.226802E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.36 | backward: 3442.35 | backward-backward: 3442.27 | backward-allreduce: 0.00 | optimizer: 138.10 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step13500
 samples/sec: 226.520 | iteration    17000/  143000 | elapsed time per iteration (ms): 4520.6 | learning rate: 5.841E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.221625E+00 | loss scale: 131072.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.55 | backward: 3442.55 | backward-backward: 3442.47 | backward-allreduce: 0.00 | optimizer: 138.02 | batch generator: 2.26
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step14000
 samples/sec: 226.189 | iteration    17500/  143000 | elapsed time per iteration (ms): 4527.2 | learning rate: 5.830E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.219367E+00 | loss scale: 131072.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.38 | backward: 3449.64 | backward-backward: 3449.56 | backward-allreduce: 0.00 | optimizer: 137.94 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step14500
 samples/sec: 226.330 | iteration    18000/  143000 | elapsed time per iteration (ms): 4524.4 | learning rate: 5.820E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.214526E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.54 | backward: 3446.65 | backward-backward: 3446.57 | backward-allreduce: 0.00 | optimizer: 138.01 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step15000
 samples/sec: 226.518 | iteration    18500/  143000 | elapsed time per iteration (ms): 4520.6 | learning rate: 5.809E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.213750E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.77 | backward: 3442.23 | backward-backward: 3442.15 | backward-allreduce: 0.00 | optimizer: 137.94 | batch generator: 2.33
 samples/sec: 225.724 | iteration    19000/  143000 | elapsed time per iteration (ms): 4536.5 | learning rate: 5.798E-04 | approx flops per GPU: 83.3TFLOPS | lm_loss: 2.210246E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
after 19000 iterations memory (MB) | allocated: 6356.6455078125 | max allocated: 16125.47265625 | reserved: 17262.0 | max reserved: 17262.0
time (ms) | forward: 951.22 | backward: 3445.88 | backward-backward: 3445.81 | backward-allreduce: 0.00 | optimizer: 137.97 | batch generator: 6.18
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step16000
 samples/sec: 225.617 | iteration    19500/  143000 | elapsed time per iteration (ms): 4538.7 | learning rate: 5.786E-04 | approx flops per GPU: 83.3TFLOPS | lm_loss: 2.207177E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 933.88 | backward: 3443.56 | backward-backward: 3443.49 | backward-allreduce: 0.00 | optimizer: 137.98 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step16500
 samples/sec: 225.475 | iteration    20000/  143000 | elapsed time per iteration (ms): 4541.5 | learning rate: 5.774E-04 | approx flops per GPU: 83.2TFLOPS | lm_loss: 2.203768E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 933.87 | backward: 3461.44 | backward-backward: 3461.37 | backward-allreduce: 0.00 | optimizer: 137.74 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step17000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 20000 | lm_loss value: 2.160887E+00 | lm_loss_ppl value: 8.678828E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 216.413 | iteration    20500/  143000 | elapsed time per iteration (ms): 4731.7 | learning rate: 5.762E-04 | approx flops per GPU: 79.9TFLOPS | lm_loss: 2.202244E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 940.35 | backward: 3449.80 | backward-backward: 3449.72 | backward-allreduce: 0.00 | optimizer: 138.01 | batch generator: 9.14
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step17500
 samples/sec: 226.318 | iteration    21000/  143000 | elapsed time per iteration (ms): 4524.6 | learning rate: 5.750E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.197923E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 933.77 | backward: 3444.45 | backward-backward: 3444.37 | backward-allreduce: 0.00 | optimizer: 137.89 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step18000
 samples/sec: 226.340 | iteration    21500/  143000 | elapsed time per iteration (ms): 4524.2 | learning rate: 5.737E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.197335E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 934.07 | backward: 3443.71 | backward-backward: 3443.64 | backward-allreduce: 0.00 | optimizer: 137.94 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step18500
 samples/sec: 226.407 | iteration    22000/  143000 | elapsed time per iteration (ms): 4522.8 | learning rate: 5.724E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.194903E+00 | loss scale: 65536.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 933.73 | backward: 3443.05 | backward-backward: 3442.98 | backward-allreduce: 0.00 | optimizer: 137.60 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step19000
 samples/sec: 226.400 | iteration    22500/  143000 | elapsed time per iteration (ms): 4523.0 | learning rate: 5.711E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.192983E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 933.65 | backward: 3442.85 | backward-backward: 3442.78 | backward-allreduce: 0.00 | optimizer: 137.89 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step19500
 samples/sec: 226.458 | iteration    23000/  143000 | elapsed time per iteration (ms): 4521.8 | learning rate: 5.697E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.190313E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 933.23 | backward: 3442.11 | backward-backward: 3442.04 | backward-allreduce: 0.00 | optimizer: 138.12 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step20000
 samples/sec: 226.448 | iteration    23500/  143000 | elapsed time per iteration (ms): 4522.0 | learning rate: 5.683E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.188262E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 933.45 | backward: 3442.17 | backward-backward: 3442.10 | backward-allreduce: 0.00 | optimizer: 138.00 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step20500
 samples/sec: 225.516 | iteration    24000/  143000 | elapsed time per iteration (ms): 4540.7 | learning rate: 5.669E-04 | approx flops per GPU: 83.3TFLOPS | lm_loss: 2.187867E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 933.22 | backward: 3461.56 | backward-backward: 3461.48 | backward-allreduce: 0.00 | optimizer: 137.67 | batch generator: 2.25
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step21000
 samples/sec: 226.430 | iteration    24500/  143000 | elapsed time per iteration (ms): 4522.4 | learning rate: 5.655E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.183607E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 933.46 | backward: 3442.57 | backward-backward: 3442.49 | backward-allreduce: 0.00 | optimizer: 137.80 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step21500
 samples/sec: 226.430 | iteration    25000/  143000 | elapsed time per iteration (ms): 4522.4 | learning rate: 5.640E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.182082E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 933.36 | backward: 3442.07 | backward-backward: 3442.00 | backward-allreduce: 0.00 | optimizer: 138.04 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step22000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 25000 | lm_loss value: 2.139077E+00 | lm_loss_ppl value: 8.491594E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 217.078 | iteration    25500/  143000 | elapsed time per iteration (ms): 4717.2 | learning rate: 5.625E-04 | approx flops per GPU: 80.1TFLOPS | lm_loss: 2.180659E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 933.52 | backward: 3442.36 | backward-backward: 3442.29 | backward-allreduce: 0.00 | optimizer: 138.09 | batch generator: 2.72
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step22500
 samples/sec: 225.913 | iteration    26000/  143000 | elapsed time per iteration (ms): 4532.7 | learning rate: 5.609E-04 | approx flops per GPU: 83.4TFLOPS | lm_loss: 2.179621E+00 | loss scale: 32768.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 933.34 | backward: 3453.62 | backward-backward: 3453.55 | backward-allreduce: 0.00 | optimizer: 137.43 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step23000
 samples/sec: 226.421 | iteration    26500/  143000 | elapsed time per iteration (ms): 4522.5 | learning rate: 5.594E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.176574E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 933.55 | backward: 3442.57 | backward-backward: 3442.49 | backward-allreduce: 0.00 | optimizer: 138.20 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step23500
 samples/sec: 224.621 | iteration    27000/  143000 | elapsed time per iteration (ms): 4558.8 | learning rate: 5.578E-04 | approx flops per GPU: 82.9TFLOPS | lm_loss: 2.174954E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 933.43 | backward: 3478.89 | backward-backward: 3478.82 | backward-allreduce: 0.00 | optimizer: 137.91 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step24000
 samples/sec: 225.213 | iteration    27500/  143000 | elapsed time per iteration (ms): 4546.8 | learning rate: 5.561E-04 | approx flops per GPU: 83.1TFLOPS | lm_loss: 2.173187E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 933.43 | backward: 3466.81 | backward-backward: 3466.74 | backward-allreduce: 0.00 | optimizer: 137.76 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step24500
 samples/sec: 226.355 | iteration    28000/  143000 | elapsed time per iteration (ms): 4523.9 | learning rate: 5.545E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.171455E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 933.10 | backward: 3442.67 | backward-backward: 3442.60 | backward-allreduce: 0.00 | optimizer: 137.85 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step25000
 samples/sec: 226.433 | iteration    28500/  143000 | elapsed time per iteration (ms): 4522.3 | learning rate: 5.528E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.170265E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 933.57 | backward: 3442.43 | backward-backward: 3442.36 | backward-allreduce: 0.00 | optimizer: 138.05 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step25500
 samples/sec: 226.176 | iteration    29000/  143000 | elapsed time per iteration (ms): 4527.4 | learning rate: 5.511E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.167834E+00 | loss scale: 32768.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 933.40 | backward: 3447.70 | backward-backward: 3447.62 | backward-allreduce: 0.00 | optimizer: 137.73 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step26000
 samples/sec: 226.140 | iteration    29500/  143000 | elapsed time per iteration (ms): 4528.2 | learning rate: 5.494E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.167481E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 933.66 | backward: 3447.72 | backward-backward: 3447.65 | backward-allreduce: 0.00 | optimizer: 138.03 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step26500
 samples/sec: 225.350 | iteration    30000/  143000 | elapsed time per iteration (ms): 4544.0 | learning rate: 5.476E-04 | approx flops per GPU: 83.2TFLOPS | lm_loss: 2.164627E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 937.78 | backward: 3458.98 | backward-backward: 3458.90 | backward-allreduce: 0.00 | optimizer: 138.88 | batch generator: 2.93
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step27000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 30000 | lm_loss value: 2.117252E+00 | lm_loss_ppl value: 8.308276E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 216.502 | iteration    30500/  143000 | elapsed time per iteration (ms): 4729.8 | learning rate: 5.458E-04 | approx flops per GPU: 79.9TFLOPS | lm_loss: 2.163256E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 933.08 | backward: 3453.37 | backward-backward: 3453.30 | backward-allreduce: 0.00 | optimizer: 137.85 | batch generator: 2.67
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step27500
 samples/sec: 225.404 | iteration    31000/  143000 | elapsed time per iteration (ms): 4543.0 | learning rate: 5.440E-04 | approx flops per GPU: 83.2TFLOPS | lm_loss: 2.162730E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 940.38 | backward: 3456.07 | backward-backward: 3456.00 | backward-allreduce: 0.00 | optimizer: 138.11 | batch generator: 9.40
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step28000
 samples/sec: 226.458 | iteration    31500/  143000 | elapsed time per iteration (ms): 4521.8 | learning rate: 5.422E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.158811E+00 | loss scale: 32768.0 | number of skipped iterations:   3 | number of nan iterations:   0 |
time (ms) | forward: 933.20 | backward: 3443.11 | backward-backward: 3443.04 | backward-allreduce: 0.00 | optimizer: 137.23 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step28500
 samples/sec: 225.334 | iteration    32000/  143000 | elapsed time per iteration (ms): 4544.4 | learning rate: 5.403E-04 | approx flops per GPU: 83.2TFLOPS | lm_loss: 2.158880E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.52 | backward: 3464.64 | backward-backward: 3464.57 | backward-allreduce: 0.00 | optimizer: 138.13 | batch generator: 2.26
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step29000
 samples/sec: 226.434 | iteration    32500/  143000 | elapsed time per iteration (ms): 4522.3 | learning rate: 5.385E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.157467E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 933.45 | backward: 3442.48 | backward-backward: 3442.41 | backward-allreduce: 0.00 | optimizer: 137.94 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step29500
 samples/sec: 226.459 | iteration    33000/  143000 | elapsed time per iteration (ms): 4521.8 | learning rate: 5.365E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.157562E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.24 | backward: 3443.55 | backward-backward: 3443.48 | backward-allreduce: 0.00 | optimizer: 137.72 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step30000
 samples/sec: 226.422 | iteration    33500/  143000 | elapsed time per iteration (ms): 4522.5 | learning rate: 5.346E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.155031E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 933.57 | backward: 3442.48 | backward-backward: 3442.41 | backward-allreduce: 0.00 | optimizer: 138.09 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step30500
 samples/sec: 226.450 | iteration    34000/  143000 | elapsed time per iteration (ms): 4522.0 | learning rate: 5.326E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.155254E+00 | loss scale: 131072.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 933.38 | backward: 3442.51 | backward-backward: 3442.44 | backward-allreduce: 0.00 | optimizer: 137.73 | batch generator: 2.37
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step31000
 samples/sec: 226.450 | iteration    34500/  143000 | elapsed time per iteration (ms): 4522.0 | learning rate: 5.306E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.152287E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 933.62 | backward: 3442.32 | backward-backward: 3442.25 | backward-allreduce: 0.00 | optimizer: 137.72 | batch generator: 2.36
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step31500
 samples/sec: 226.424 | iteration    35000/  143000 | elapsed time per iteration (ms): 4522.5 | learning rate: 5.286E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.150846E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 933.11 | backward: 3443.06 | backward-backward: 3442.98 | backward-allreduce: 0.00 | optimizer: 138.14 | batch generator: 2.35
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step32000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 35000 | lm_loss value: 2.106865E+00 | lm_loss_ppl value: 8.222427E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 217.015 | iteration    35500/  143000 | elapsed time per iteration (ms): 4718.6 | learning rate: 5.266E-04 | approx flops per GPU: 80.1TFLOPS | lm_loss: 2.149761E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.62 | backward: 3443.63 | backward-backward: 3443.56 | backward-allreduce: 0.00 | optimizer: 137.91 | batch generator: 2.70
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step32500
 samples/sec: 223.571 | iteration    36000/  143000 | elapsed time per iteration (ms): 4580.2 | learning rate: 5.245E-04 | approx flops per GPU: 82.5TFLOPS | lm_loss: 2.148680E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 944.12 | backward: 3489.30 | backward-backward: 3489.23 | backward-allreduce: 0.00 | optimizer: 137.83 | batch generator: 14.06
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step33000
 samples/sec: 226.405 | iteration    36500/  143000 | elapsed time per iteration (ms): 4522.9 | learning rate: 5.224E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.147072E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.93 | backward: 3443.61 | backward-backward: 3443.54 | backward-allreduce: 0.00 | optimizer: 138.10 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step33500
 samples/sec: 226.403 | iteration    37000/  143000 | elapsed time per iteration (ms): 4522.9 | learning rate: 5.203E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.146458E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.72 | backward: 3443.64 | backward-backward: 3443.57 | backward-allreduce: 0.00 | optimizer: 138.07 | batch generator: 2.34
 samples/sec: 225.655 | iteration    37500/  143000 | elapsed time per iteration (ms): 4537.9 | learning rate: 5.182E-04 | approx flops per GPU: 83.3TFLOPS | lm_loss: 2.145587E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
after 37500 iterations memory (MB) | allocated: 6356.6455078125 | max allocated: 16125.47265625 | reserved: 17262.0 | max reserved: 17262.0
time (ms) | forward: 936.44 | backward: 3461.87 | backward-backward: 3461.80 | backward-allreduce: 0.00 | optimizer: 138.14 | batch generator: 4.85
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step34500
 samples/sec: 226.290 | iteration    38000/  143000 | elapsed time per iteration (ms): 4525.2 | learning rate: 5.160E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.145178E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.92 | backward: 3444.95 | backward-backward: 3444.87 | backward-allreduce: 0.00 | optimizer: 138.01 | batch generator: 2.25
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step35000
 samples/sec: 225.845 | iteration    38500/  143000 | elapsed time per iteration (ms): 4534.1 | learning rate: 5.139E-04 | approx flops per GPU: 83.4TFLOPS | lm_loss: 2.143805E+00 | loss scale: 32768.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 931.67 | backward: 3456.75 | backward-backward: 3456.68 | backward-allreduce: 0.00 | optimizer: 137.49 | batch generator: 2.26
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step35500
 samples/sec: 226.413 | iteration    39000/  143000 | elapsed time per iteration (ms): 4522.7 | learning rate: 5.117E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.144680E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.45 | backward: 3444.96 | backward-backward: 3444.89 | backward-allreduce: 0.00 | optimizer: 137.88 | batch generator: 2.22
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step36000
 samples/sec: 226.429 | iteration    39500/  143000 | elapsed time per iteration (ms): 4522.4 | learning rate: 5.094E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.141239E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.29 | backward: 3444.59 | backward-backward: 3444.52 | backward-allreduce: 0.00 | optimizer: 138.08 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step36500
 samples/sec: 226.392 | iteration    40000/  143000 | elapsed time per iteration (ms): 4523.1 | learning rate: 5.072E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.139953E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.60 | backward: 3445.22 | backward-backward: 3445.15 | backward-allreduce: 0.00 | optimizer: 137.93 | batch generator: 2.23
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step37000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 40000 | lm_loss value: 2.094210E+00 | lm_loss_ppl value: 8.119024E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 216.121 | iteration    40500/  143000 | elapsed time per iteration (ms): 4738.1 | learning rate: 5.049E-04 | approx flops per GPU: 79.8TFLOPS | lm_loss: 2.138332E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.34 | backward: 3465.60 | backward-backward: 3465.53 | backward-allreduce: 0.00 | optimizer: 137.94 | batch generator: 2.60
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step37500
 samples/sec: 226.328 | iteration    41000/  143000 | elapsed time per iteration (ms): 4524.4 | learning rate: 5.026E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.137843E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.66 | backward: 3446.21 | backward-backward: 3446.14 | backward-allreduce: 0.00 | optimizer: 137.61 | batch generator: 2.25
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step38000
 samples/sec: 226.428 | iteration    41500/  143000 | elapsed time per iteration (ms): 4522.4 | learning rate: 5.003E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.136726E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.46 | backward: 3444.97 | backward-backward: 3444.90 | backward-allreduce: 0.00 | optimizer: 137.60 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step38500
 samples/sec: 226.393 | iteration    42000/  143000 | elapsed time per iteration (ms): 4523.1 | learning rate: 4.980E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.134204E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.68 | backward: 3445.07 | backward-backward: 3445.00 | backward-allreduce: 0.00 | optimizer: 137.91 | batch generator: 2.26
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step39000
 samples/sec: 225.566 | iteration    42500/  143000 | elapsed time per iteration (ms): 4539.7 | learning rate: 4.956E-04 | approx flops per GPU: 83.3TFLOPS | lm_loss: 2.135012E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.54 | backward: 3461.99 | backward-backward: 3461.92 | backward-allreduce: 0.00 | optimizer: 137.81 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step39500
 samples/sec: 226.354 | iteration    43000/  143000 | elapsed time per iteration (ms): 4523.9 | learning rate: 4.933E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.132556E+00 | loss scale: 32768.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 931.80 | backward: 3446.47 | backward-backward: 3446.39 | backward-allreduce: 0.00 | optimizer: 137.33 | batch generator: 2.35
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step40000
 samples/sec: 226.408 | iteration    43500/  143000 | elapsed time per iteration (ms): 4522.8 | learning rate: 4.909E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.132889E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.59 | backward: 3444.90 | backward-backward: 3444.83 | backward-allreduce: 0.00 | optimizer: 137.83 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step40500
 samples/sec: 226.407 | iteration    44000/  143000 | elapsed time per iteration (ms): 4522.8 | learning rate: 4.885E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.140665E+00 | loss scale: 16384.0 | number of skipped iterations:   3 | number of nan iterations:   0 |
time (ms) | forward: 931.81 | backward: 3445.44 | backward-backward: 3445.37 | backward-allreduce: 0.00 | optimizer: 137.08 | batch generator: 2.35
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step41000
 samples/sec: 226.171 | iteration    44500/  143000 | elapsed time per iteration (ms): 4527.5 | learning rate: 4.860E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.130905E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.57 | backward: 3449.85 | backward-backward: 3449.78 | backward-allreduce: 0.00 | optimizer: 137.81 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step41500
 samples/sec: 226.393 | iteration    45000/  143000 | elapsed time per iteration (ms): 4523.1 | learning rate: 4.836E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.130383E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.79 | backward: 3445.17 | backward-backward: 3445.09 | backward-allreduce: 0.00 | optimizer: 137.80 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step42000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 45000 | lm_loss value: 2.082156E+00 | lm_loss_ppl value: 8.021745E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 217.058 | iteration    45500/  143000 | elapsed time per iteration (ms): 4717.6 | learning rate: 4.811E-04 | approx flops per GPU: 80.1TFLOPS | lm_loss: 2.128915E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.13 | backward: 3444.63 | backward-backward: 3444.55 | backward-allreduce: 0.00 | optimizer: 137.65 | batch generator: 2.75
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step42500
 samples/sec: 226.374 | iteration    46000/  143000 | elapsed time per iteration (ms): 4523.5 | learning rate: 4.786E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.131272E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 933.19 | backward: 3443.99 | backward-backward: 3443.91 | backward-allreduce: 0.00 | optimizer: 137.89 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step43000
 samples/sec: 225.788 | iteration    46500/  143000 | elapsed time per iteration (ms): 4535.2 | learning rate: 4.761E-04 | approx flops per GPU: 83.4TFLOPS | lm_loss: 2.126276E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.25 | backward: 3456.69 | backward-backward: 3456.62 | backward-allreduce: 0.00 | optimizer: 137.88 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step43500
 samples/sec: 226.419 | iteration    47000/  143000 | elapsed time per iteration (ms): 4522.6 | learning rate: 4.736E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.124412E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.96 | backward: 3443.73 | backward-backward: 3443.65 | backward-allreduce: 0.00 | optimizer: 137.64 | batch generator: 2.37
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step44000
 samples/sec: 226.138 | iteration    47500/  143000 | elapsed time per iteration (ms): 4528.2 | learning rate: 4.710E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.125073E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.71 | backward: 3447.40 | backward-backward: 3447.32 | backward-allreduce: 0.00 | optimizer: 139.79 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step44500
 samples/sec: 226.368 | iteration    48000/  143000 | elapsed time per iteration (ms): 4523.6 | learning rate: 4.685E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.123639E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.95 | backward: 3443.89 | backward-backward: 3443.82 | backward-allreduce: 0.00 | optimizer: 137.89 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step45000
 samples/sec: 225.690 | iteration    48500/  143000 | elapsed time per iteration (ms): 4537.2 | learning rate: 4.659E-04 | approx flops per GPU: 83.3TFLOPS | lm_loss: 2.122041E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.76 | backward: 3457.96 | backward-backward: 3457.89 | backward-allreduce: 0.00 | optimizer: 137.66 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step45500
 samples/sec: 226.421 | iteration    49000/  143000 | elapsed time per iteration (ms): 4522.5 | learning rate: 4.633E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.121722E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.90 | backward: 3443.48 | backward-backward: 3443.40 | backward-allreduce: 0.00 | optimizer: 137.64 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step46000
 samples/sec: 226.442 | iteration    49500/  143000 | elapsed time per iteration (ms): 4522.1 | learning rate: 4.607E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.121354E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.67 | backward: 3443.22 | backward-backward: 3443.15 | backward-allreduce: 0.00 | optimizer: 137.69 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step46500
 samples/sec: 226.385 | iteration    50000/  143000 | elapsed time per iteration (ms): 4523.3 | learning rate: 4.581E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.118096E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.88 | backward: 3443.73 | backward-backward: 3443.66 | backward-allreduce: 0.00 | optimizer: 138.04 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step47000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 50000 | lm_loss value: 2.074510E+00 | lm_loss_ppl value: 7.960646E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 217.097 | iteration    50500/  143000 | elapsed time per iteration (ms): 4716.8 | learning rate: 4.554E-04 | approx flops per GPU: 80.1TFLOPS | lm_loss: 2.118645E+00 | loss scale: 16384.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 932.74 | backward: 3443.50 | backward-backward: 3443.42 | backward-allreduce: 0.00 | optimizer: 137.44 | batch generator: 2.70
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step47500
 samples/sec: 226.404 | iteration    51000/  143000 | elapsed time per iteration (ms): 4522.9 | learning rate: 4.528E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.117642E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.94 | backward: 3443.56 | backward-backward: 3443.49 | backward-allreduce: 0.00 | optimizer: 138.02 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step48000
 samples/sec: 226.128 | iteration    51500/  143000 | elapsed time per iteration (ms): 4528.4 | learning rate: 4.501E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.115632E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 933.88 | backward: 3447.99 | backward-backward: 3447.92 | backward-allreduce: 0.00 | optimizer: 138.00 | batch generator: 3.43
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step48500
 samples/sec: 226.437 | iteration    52000/  143000 | elapsed time per iteration (ms): 4522.2 | learning rate: 4.474E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.114921E+00 | loss scale: 16384.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 933.03 | backward: 3443.46 | backward-backward: 3443.39 | backward-allreduce: 0.00 | optimizer: 137.38 | batch generator: 2.40
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step49000
 samples/sec: 226.426 | iteration    52500/  143000 | elapsed time per iteration (ms): 4522.4 | learning rate: 4.447E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.116263E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.80 | backward: 3443.39 | backward-backward: 3443.32 | backward-allreduce: 0.00 | optimizer: 137.95 | batch generator: 2.39
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step49500
 samples/sec: 226.399 | iteration    53000/  143000 | elapsed time per iteration (ms): 4523.0 | learning rate: 4.420E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.113496E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.93 | backward: 3443.73 | backward-backward: 3443.65 | backward-allreduce: 0.00 | optimizer: 137.96 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step50000
 samples/sec: 225.906 | iteration    53500/  143000 | elapsed time per iteration (ms): 4532.9 | learning rate: 4.393E-04 | approx flops per GPU: 83.4TFLOPS | lm_loss: 2.112594E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.77 | backward: 3453.39 | backward-backward: 3453.32 | backward-allreduce: 0.00 | optimizer: 137.91 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step50500
 samples/sec: 226.419 | iteration    54000/  143000 | elapsed time per iteration (ms): 4522.6 | learning rate: 4.365E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.111097E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.94 | backward: 3443.74 | backward-backward: 3443.67 | backward-allreduce: 0.00 | optimizer: 137.66 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step51000
 samples/sec: 226.425 | iteration    54500/  143000 | elapsed time per iteration (ms): 4522.5 | learning rate: 4.338E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.110641E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.82 | backward: 3443.40 | backward-backward: 3443.33 | backward-allreduce: 0.00 | optimizer: 138.02 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step51500
 samples/sec: 226.292 | iteration    55000/  143000 | elapsed time per iteration (ms): 4525.1 | learning rate: 4.310E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.108976E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 933.01 | backward: 3443.68 | backward-backward: 3443.61 | backward-allreduce: 0.00 | optimizer: 137.68 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step52000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 55000 | lm_loss value: 2.066541E+00 | lm_loss_ppl value: 7.897462E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 217.052 | iteration    55500/  143000 | elapsed time per iteration (ms): 4717.8 | learning rate: 4.282E-04 | approx flops per GPU: 80.1TFLOPS | lm_loss: 2.109255E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.80 | backward: 3443.79 | backward-backward: 3443.72 | backward-allreduce: 0.00 | optimizer: 138.04 | batch generator: 2.71
 samples/sec: 226.094 | iteration    56000/  143000 | elapsed time per iteration (ms): 4529.1 | learning rate: 4.254E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.107156E+00 | loss scale: 16384.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
after 56000 iterations memory (MB) | allocated: 6504.0234375 | max allocated: 16272.8505859375 | reserved: 17262.0 | max reserved: 17262.0
time (ms) | forward: 944.02 | backward: 3446.39 | backward-backward: 3446.32 | backward-allreduce: 0.00 | optimizer: 137.41 | batch generator: 6.60
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step53000
 samples/sec: 226.246 | iteration    56500/  143000 | elapsed time per iteration (ms): 4526.0 | learning rate: 4.226E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.106588E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 933.10 | backward: 3444.77 | backward-backward: 3444.70 | backward-allreduce: 0.00 | optimizer: 138.30 | batch generator: 2.38
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step53500
 samples/sec: 226.382 | iteration    57000/  143000 | elapsed time per iteration (ms): 4523.3 | learning rate: 4.198E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.105888E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.53 | backward: 3444.35 | backward-backward: 3444.28 | backward-allreduce: 0.00 | optimizer: 138.01 | batch generator: 2.38
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step54000
 samples/sec: 226.417 | iteration    57500/  143000 | elapsed time per iteration (ms): 4522.6 | learning rate: 4.170E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.105284E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.95 | backward: 3443.36 | backward-backward: 3443.29 | backward-allreduce: 0.00 | optimizer: 138.05 | batch generator: 2.38
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step54500
 samples/sec: 226.449 | iteration    58000/  143000 | elapsed time per iteration (ms): 4522.0 | learning rate: 4.141E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.105197E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.73 | backward: 3443.99 | backward-backward: 3443.92 | backward-allreduce: 0.00 | optimizer: 137.86 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step55000
 samples/sec: 226.451 | iteration    58500/  143000 | elapsed time per iteration (ms): 4522.0 | learning rate: 4.113E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.100466E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.42 | backward: 3444.72 | backward-backward: 3444.65 | backward-allreduce: 0.00 | optimizer: 137.65 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step55500
 samples/sec: 226.421 | iteration    59000/  143000 | elapsed time per iteration (ms): 4522.6 | learning rate: 4.084E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.103008E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.10 | backward: 3444.86 | backward-backward: 3444.79 | backward-allreduce: 0.00 | optimizer: 137.94 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step56000
 samples/sec: 226.280 | iteration    59500/  143000 | elapsed time per iteration (ms): 4525.4 | learning rate: 4.055E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.101373E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.00 | backward: 3447.52 | backward-backward: 3447.46 | backward-allreduce: 0.00 | optimizer: 137.75 | batch generator: 2.36
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step56500
 samples/sec: 226.525 | iteration    60000/  143000 | elapsed time per iteration (ms): 4520.5 | learning rate: 4.027E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.101090E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.53 | backward: 3442.61 | backward-backward: 3442.54 | backward-allreduce: 0.00 | optimizer: 138.36 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step57000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 60000 | lm_loss value: 2.057104E+00 | lm_loss_ppl value: 7.823278E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 217.150 | iteration    60500/  143000 | elapsed time per iteration (ms): 4715.6 | learning rate: 3.998E-04 | approx flops per GPU: 80.2TFLOPS | lm_loss: 2.098890E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.24 | backward: 3443.51 | backward-backward: 3443.45 | backward-allreduce: 0.00 | optimizer: 138.01 | batch generator: 2.67
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step57500
 samples/sec: 226.500 | iteration    61000/  143000 | elapsed time per iteration (ms): 4521.0 | learning rate: 3.969E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.097962E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.10 | backward: 3443.24 | backward-backward: 3443.18 | backward-allreduce: 0.00 | optimizer: 138.27 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step58000
 samples/sec: 226.527 | iteration    61500/  143000 | elapsed time per iteration (ms): 4520.4 | learning rate: 3.940E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.099420E+00 | loss scale: 16384.0 | number of skipped iterations:   3 | number of nan iterations:   0 |
time (ms) | forward: 931.74 | backward: 3443.04 | backward-backward: 3442.97 | backward-allreduce: 0.00 | optimizer: 137.61 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step58500
 samples/sec: 226.503 | iteration    62000/  143000 | elapsed time per iteration (ms): 4520.9 | learning rate: 3.911E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.094402E+00 | loss scale: 8192.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.48 | backward: 3442.99 | backward-backward: 3442.92 | backward-allreduce: 0.00 | optimizer: 138.20 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step59000
 samples/sec: 226.381 | iteration    62500/  143000 | elapsed time per iteration (ms): 4523.4 | learning rate: 3.882E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.096752E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.25 | backward: 3445.58 | backward-backward: 3445.51 | backward-allreduce: 0.00 | optimizer: 138.48 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step59500
 samples/sec: 226.477 | iteration    63000/  143000 | elapsed time per iteration (ms): 4521.4 | learning rate: 3.852E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.092951E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 930.82 | backward: 3444.18 | backward-backward: 3444.11 | backward-allreduce: 0.00 | optimizer: 138.38 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step60000
 samples/sec: 226.484 | iteration    63500/  143000 | elapsed time per iteration (ms): 4521.3 | learning rate: 3.823E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.093923E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 930.87 | backward: 3444.25 | backward-backward: 3444.18 | backward-allreduce: 0.00 | optimizer: 138.36 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step60500
 samples/sec: 226.490 | iteration    64000/  143000 | elapsed time per iteration (ms): 4521.2 | learning rate: 3.794E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.095593E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 930.65 | backward: 3443.79 | backward-backward: 3443.72 | backward-allreduce: 0.00 | optimizer: 138.48 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step61000
 samples/sec: 226.484 | iteration    64500/  143000 | elapsed time per iteration (ms): 4521.3 | learning rate: 3.764E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.091592E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 930.95 | backward: 3443.91 | backward-backward: 3443.84 | backward-allreduce: 0.00 | optimizer: 138.46 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step61500
 samples/sec: 226.517 | iteration    65000/  143000 | elapsed time per iteration (ms): 4520.6 | learning rate: 3.735E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.093253E+00 | loss scale: 16384.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 931.57 | backward: 3442.90 | backward-backward: 3442.83 | backward-allreduce: 0.00 | optimizer: 137.96 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step62000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 65000 | lm_loss value: 2.048048E+00 | lm_loss_ppl value: 7.752753E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 215.731 | iteration    65500/  143000 | elapsed time per iteration (ms): 4746.6 | learning rate: 3.705E-04 | approx flops per GPU: 79.6TFLOPS | lm_loss: 2.091273E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.14 | backward: 3473.09 | backward-backward: 3473.02 | backward-allreduce: 0.00 | optimizer: 138.58 | batch generator: 2.73
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step62500
 samples/sec: 226.485 | iteration    66000/  143000 | elapsed time per iteration (ms): 4521.3 | learning rate: 3.675E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.089873E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.91 | backward: 3442.71 | backward-backward: 3442.64 | backward-allreduce: 0.00 | optimizer: 138.54 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step63000
 samples/sec: 226.476 | iteration    66500/  143000 | elapsed time per iteration (ms): 4521.5 | learning rate: 3.646E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.091106E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.90 | backward: 3443.01 | backward-backward: 3442.94 | backward-allreduce: 0.00 | optimizer: 138.46 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step63500
 samples/sec: 226.095 | iteration    67000/  143000 | elapsed time per iteration (ms): 4529.1 | learning rate: 3.616E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.088570E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.66 | backward: 3450.80 | backward-backward: 3450.73 | backward-allreduce: 0.00 | optimizer: 138.48 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step64000
 samples/sec: 226.400 | iteration    67500/  143000 | elapsed time per iteration (ms): 4523.0 | learning rate: 3.586E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.088293E+00 | loss scale: 16384.0 | number of skipped iterations:   3 | number of nan iterations:   0 |
time (ms) | forward: 931.72 | backward: 3445.55 | backward-backward: 3445.48 | backward-allreduce: 0.00 | optimizer: 137.63 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step64500
 samples/sec: 226.472 | iteration    68000/  143000 | elapsed time per iteration (ms): 4521.5 | learning rate: 3.557E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.084413E+00 | loss scale: 8192.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.59 | backward: 3443.56 | backward-backward: 3443.49 | backward-allreduce: 0.00 | optimizer: 138.19 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step65000
 samples/sec: 226.419 | iteration    68500/  143000 | elapsed time per iteration (ms): 4522.6 | learning rate: 3.527E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.085778E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.79 | backward: 3444.39 | backward-backward: 3444.32 | backward-allreduce: 0.00 | optimizer: 138.44 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step65500
 samples/sec: 226.438 | iteration    69000/  143000 | elapsed time per iteration (ms): 4522.2 | learning rate: 3.497E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.084865E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.89 | backward: 3443.59 | backward-backward: 3443.52 | backward-allreduce: 0.00 | optimizer: 138.56 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step66000
 samples/sec: 226.442 | iteration    69500/  143000 | elapsed time per iteration (ms): 4522.1 | learning rate: 3.467E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.085569E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.08 | backward: 3443.30 | backward-backward: 3443.23 | backward-allreduce: 0.00 | optimizer: 138.56 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step66500
 samples/sec: 226.469 | iteration    70000/  143000 | elapsed time per iteration (ms): 4521.6 | learning rate: 3.437E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.085879E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.64 | backward: 3443.57 | backward-backward: 3443.50 | backward-allreduce: 0.00 | optimizer: 138.22 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step67000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 70000 | lm_loss value: 2.040694E+00 | lm_loss_ppl value: 7.695945E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 217.092 | iteration    70500/  143000 | elapsed time per iteration (ms): 4716.9 | learning rate: 3.407E-04 | approx flops per GPU: 80.1TFLOPS | lm_loss: 2.084102E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.25 | backward: 3443.36 | backward-backward: 3443.29 | backward-allreduce: 0.00 | optimizer: 138.45 | batch generator: 2.73
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step67500
 samples/sec: 226.474 | iteration    71000/  143000 | elapsed time per iteration (ms): 4521.5 | learning rate: 3.377E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.082498E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.59 | backward: 3443.56 | backward-backward: 3443.49 | backward-allreduce: 0.00 | optimizer: 138.14 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step68000
 samples/sec: 225.495 | iteration    71500/  143000 | elapsed time per iteration (ms): 4541.1 | learning rate: 3.347E-04 | approx flops per GPU: 83.2TFLOPS | lm_loss: 2.080401E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 935.16 | backward: 3459.36 | backward-backward: 3459.30 | backward-allreduce: 0.00 | optimizer: 138.51 | batch generator: 5.47
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step68500
 samples/sec: 226.383 | iteration    72000/  143000 | elapsed time per iteration (ms): 4523.3 | learning rate: 3.318E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.078601E+00 | loss scale: 32768.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 931.87 | backward: 3445.31 | backward-backward: 3445.24 | backward-allreduce: 0.00 | optimizer: 137.92 | batch generator: 2.39
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step69000
 samples/sec: 226.443 | iteration    72500/  143000 | elapsed time per iteration (ms): 4522.1 | learning rate: 3.288E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.076446E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.05 | backward: 3444.50 | backward-backward: 3444.44 | backward-allreduce: 0.00 | optimizer: 138.50 | batch generator: 2.35
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step69500
 samples/sec: 226.475 | iteration    73000/  143000 | elapsed time per iteration (ms): 4521.5 | learning rate: 3.258E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.077394E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 930.83 | backward: 3444.39 | backward-backward: 3444.33 | backward-allreduce: 0.00 | optimizer: 138.30 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step70000
 samples/sec: 225.822 | iteration    73500/  143000 | elapsed time per iteration (ms): 4534.5 | learning rate: 3.228E-04 | approx flops per GPU: 83.4TFLOPS | lm_loss: 2.077776E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.02 | backward: 3451.49 | backward-backward: 3451.42 | backward-allreduce: 0.00 | optimizer: 138.54 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step70500
 samples/sec: 226.469 | iteration    74000/  143000 | elapsed time per iteration (ms): 4521.6 | learning rate: 3.198E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.076031E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 930.85 | backward: 3444.28 | backward-backward: 3444.22 | backward-allreduce: 0.00 | optimizer: 138.54 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step71000
 samples/sec: 226.461 | iteration    74500/  143000 | elapsed time per iteration (ms): 4521.8 | learning rate: 3.168E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.074688E+00 | loss scale: 16384.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 931.04 | backward: 3444.62 | backward-backward: 3444.56 | backward-allreduce: 0.00 | optimizer: 137.98 | batch generator: 2.29
 samples/sec: 225.912 | iteration    75000/  143000 | elapsed time per iteration (ms): 4532.7 | learning rate: 3.138E-04 | approx flops per GPU: 83.4TFLOPS | lm_loss: 2.073882E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
after 75000 iterations memory (MB) | allocated: 6356.6455078125 | max allocated: 16125.47265625 | reserved: 17262.0 | max reserved: 17262.0
time (ms) | forward: 942.32 | backward: 3450.45 | backward-backward: 3450.37 | backward-allreduce: 0.00 | optimizer: 138.50 | batch generator: 4.07
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step72000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 75000 | lm_loss value: 2.030104E+00 | lm_loss_ppl value: 7.614878E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 216.913 | iteration    75500/  143000 | elapsed time per iteration (ms): 4720.8 | learning rate: 3.108E-04 | approx flops per GPU: 80.1TFLOPS | lm_loss: 2.071406E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 933.06 | backward: 3444.85 | backward-backward: 3444.78 | backward-allreduce: 0.00 | optimizer: 138.47 | batch generator: 2.61
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step72500
 samples/sec: 226.171 | iteration    76000/  143000 | elapsed time per iteration (ms): 4527.5 | learning rate: 3.078E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.071908E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.83 | backward: 3448.08 | backward-backward: 3448.01 | backward-allreduce: 0.00 | optimizer: 138.17 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step73000
 samples/sec: 225.725 | iteration    76500/  143000 | elapsed time per iteration (ms): 4536.5 | learning rate: 3.049E-04 | approx flops per GPU: 83.3TFLOPS | lm_loss: 2.071142E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 933.06 | backward: 3452.02 | backward-backward: 3451.95 | backward-allreduce: 0.00 | optimizer: 138.44 | batch generator: 2.23
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step73500
 samples/sec: 226.390 | iteration    77000/  143000 | elapsed time per iteration (ms): 4523.2 | learning rate: 3.019E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.070703E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 933.13 | backward: 3443.48 | backward-backward: 3443.41 | backward-allreduce: 0.00 | optimizer: 138.17 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step74000
 samples/sec: 226.357 | iteration    77500/  143000 | elapsed time per iteration (ms): 4523.8 | learning rate: 2.989E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.068652E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 933.12 | backward: 3443.79 | backward-backward: 3443.72 | backward-allreduce: 0.00 | optimizer: 138.45 | batch generator: 2.25
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step74500
 samples/sec: 225.792 | iteration    78000/  143000 | elapsed time per iteration (ms): 4535.1 | learning rate: 2.960E-04 | approx flops per GPU: 83.4TFLOPS | lm_loss: 2.080577E+00 | loss scale: 8192.0 | number of skipped iterations:   3 | number of nan iterations:   0 |
time (ms) | forward: 932.50 | backward: 3454.17 | backward-backward: 3454.09 | backward-allreduce: 0.00 | optimizer: 140.16 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step75000
 samples/sec: 224.596 | iteration    78500/  143000 | elapsed time per iteration (ms): 4559.3 | learning rate: 2.930E-04 | approx flops per GPU: 82.9TFLOPS | lm_loss: 2.070003E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 944.61 | backward: 3463.94 | backward-backward: 3463.87 | backward-allreduce: 0.00 | optimizer: 142.46 | batch generator: 14.38
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step75500
 samples/sec: 226.331 | iteration    79000/  143000 | elapsed time per iteration (ms): 4524.4 | learning rate: 2.900E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.067264E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.28 | backward: 3445.37 | backward-backward: 3445.30 | backward-allreduce: 0.00 | optimizer: 138.43 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step76000
 samples/sec: 226.158 | iteration    79500/  143000 | elapsed time per iteration (ms): 4527.8 | learning rate: 2.871E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.072462E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.58 | backward: 3448.83 | backward-backward: 3448.76 | backward-allreduce: 0.00 | optimizer: 138.18 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step76500
 samples/sec: 226.174 | iteration    80000/  143000 | elapsed time per iteration (ms): 4527.5 | learning rate: 2.841E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.066358E+00 | loss scale: 8192.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.38 | backward: 3448.79 | backward-backward: 3448.72 | backward-allreduce: 0.00 | optimizer: 138.17 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step77000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 80000 | lm_loss value: 2.020702E+00 | lm_loss_ppl value: 7.543620E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 216.981 | iteration    80500/  143000 | elapsed time per iteration (ms): 4719.3 | learning rate: 2.812E-04 | approx flops per GPU: 80.1TFLOPS | lm_loss: 2.066406E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.51 | backward: 3445.28 | backward-backward: 3445.21 | backward-allreduce: 0.00 | optimizer: 138.45 | batch generator: 2.65
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step77500
 samples/sec: 226.341 | iteration    81000/  143000 | elapsed time per iteration (ms): 4524.1 | learning rate: 2.782E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.064181E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.25 | backward: 3445.26 | backward-backward: 3445.19 | backward-allreduce: 0.00 | optimizer: 138.42 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step78000
 samples/sec: 226.123 | iteration    81500/  143000 | elapsed time per iteration (ms): 4528.5 | learning rate: 2.753E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.064041E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.46 | backward: 3449.21 | backward-backward: 3449.14 | backward-allreduce: 0.00 | optimizer: 138.43 | batch generator: 2.22
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step78500
 samples/sec: 225.974 | iteration    82000/  143000 | elapsed time per iteration (ms): 4531.5 | learning rate: 2.724E-04 | approx flops per GPU: 83.4TFLOPS | lm_loss: 2.062567E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.36 | backward: 3452.75 | backward-backward: 3452.68 | backward-allreduce: 0.00 | optimizer: 138.16 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step79000
 samples/sec: 226.113 | iteration    82500/  143000 | elapsed time per iteration (ms): 4528.7 | learning rate: 2.694E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.060072E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.54 | backward: 3449.46 | backward-backward: 3449.38 | backward-allreduce: 0.00 | optimizer: 138.46 | batch generator: 2.25
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step79500
 samples/sec: 226.160 | iteration    83000/  143000 | elapsed time per iteration (ms): 4527.8 | learning rate: 2.665E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.059083E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 933.17 | backward: 3447.65 | backward-backward: 3447.58 | backward-allreduce: 0.00 | optimizer: 138.55 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step80000
 samples/sec: 226.348 | iteration    83500/  143000 | elapsed time per iteration (ms): 4524.0 | learning rate: 2.636E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.058454E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.67 | backward: 3444.30 | backward-backward: 3444.22 | backward-allreduce: 0.00 | optimizer: 138.57 | batch generator: 2.25
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step80500
 samples/sec: 226.213 | iteration    84000/  143000 | elapsed time per iteration (ms): 4526.7 | learning rate: 2.607E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.060353E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.50 | backward: 3447.62 | backward-backward: 3447.55 | backward-allreduce: 0.00 | optimizer: 138.19 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step81000
 samples/sec: 226.382 | iteration    84500/  143000 | elapsed time per iteration (ms): 4523.3 | learning rate: 2.578E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.055604E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.65 | backward: 3443.86 | backward-backward: 3443.79 | backward-allreduce: 0.00 | optimizer: 138.47 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step81500
 samples/sec: 226.410 | iteration    85000/  143000 | elapsed time per iteration (ms): 4522.8 | learning rate: 2.550E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.054995E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.38 | backward: 3443.93 | backward-backward: 3443.86 | backward-allreduce: 0.00 | optimizer: 138.15 | batch generator: 2.22
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step82000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 85000 | lm_loss value: 2.010824E+00 | lm_loss_ppl value: 7.469468E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 216.915 | iteration    85500/  143000 | elapsed time per iteration (ms): 4720.7 | learning rate: 2.521E-04 | approx flops per GPU: 80.1TFLOPS | lm_loss: 2.052863E+00 | loss scale: 16384.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 932.61 | backward: 3444.80 | backward-backward: 3444.73 | backward-allreduce: 0.00 | optimizer: 137.92 | batch generator: 3.74
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step82500
 samples/sec: 226.366 | iteration    86000/  143000 | elapsed time per iteration (ms): 4523.7 | learning rate: 2.492E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.052566E+00 | loss scale: 8192.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.35 | backward: 3444.94 | backward-backward: 3444.87 | backward-allreduce: 0.00 | optimizer: 138.18 | batch generator: 2.18
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step83000
 samples/sec: 226.303 | iteration    86500/  143000 | elapsed time per iteration (ms): 4524.9 | learning rate: 2.464E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.054447E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.59 | backward: 3445.39 | backward-backward: 3445.32 | backward-allreduce: 0.00 | optimizer: 138.47 | batch generator: 2.22
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step83500
 samples/sec: 226.329 | iteration    87000/  143000 | elapsed time per iteration (ms): 4524.4 | learning rate: 2.435E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.052134E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.47 | backward: 3445.10 | backward-backward: 3445.03 | backward-allreduce: 0.00 | optimizer: 138.50 | batch generator: 2.25
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step84000
 samples/sec: 226.334 | iteration    87500/  143000 | elapsed time per iteration (ms): 4524.3 | learning rate: 2.407E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.049812E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.68 | backward: 3444.91 | backward-backward: 3444.84 | backward-allreduce: 0.00 | optimizer: 138.49 | batch generator: 2.26
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step84500
 samples/sec: 226.352 | iteration    88000/  143000 | elapsed time per iteration (ms): 4523.9 | learning rate: 2.379E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.050551E+00 | loss scale: 8192.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 932.46 | backward: 3445.23 | backward-backward: 3445.16 | backward-allreduce: 0.00 | optimizer: 137.95 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step85000
 samples/sec: 226.292 | iteration    88500/  143000 | elapsed time per iteration (ms): 4525.1 | learning rate: 2.351E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.049685E+00 | loss scale: 4096.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.61 | backward: 3446.11 | backward-backward: 3446.04 | backward-allreduce: 0.00 | optimizer: 138.19 | batch generator: 2.23
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step85500
 samples/sec: 226.244 | iteration    89000/  143000 | elapsed time per iteration (ms): 4526.1 | learning rate: 2.323E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.051894E+00 | loss scale: 4096.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.66 | backward: 3445.85 | backward-backward: 3445.78 | backward-allreduce: 0.00 | optimizer: 138.43 | batch generator: 2.25
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step86000
 samples/sec: 226.285 | iteration    89500/  143000 | elapsed time per iteration (ms): 4525.3 | learning rate: 2.295E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.048874E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.85 | backward: 3445.68 | backward-backward: 3445.61 | backward-allreduce: 0.00 | optimizer: 138.44 | batch generator: 2.22
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step86500
 samples/sec: 226.324 | iteration    90000/  143000 | elapsed time per iteration (ms): 4524.5 | learning rate: 2.267E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.047533E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.72 | backward: 3445.02 | backward-backward: 3444.95 | backward-allreduce: 0.00 | optimizer: 138.46 | batch generator: 2.21
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step87000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 90000 | lm_loss value: 2.006134E+00 | lm_loss_ppl value: 7.434518E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 216.988 | iteration    90500/  143000 | elapsed time per iteration (ms): 4719.2 | learning rate: 2.240E-04 | approx flops per GPU: 80.1TFLOPS | lm_loss: 2.046744E+00 | loss scale: 8192.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 932.69 | backward: 3445.57 | backward-backward: 3445.50 | backward-allreduce: 0.00 | optimizer: 137.92 | batch generator: 2.57
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step87500
 samples/sec: 226.325 | iteration    91000/  143000 | elapsed time per iteration (ms): 4524.5 | learning rate: 2.212E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.044981E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.49 | backward: 3445.31 | backward-backward: 3445.24 | backward-allreduce: 0.00 | optimizer: 138.45 | batch generator: 2.21
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step88000
 samples/sec: 226.122 | iteration    91500/  143000 | elapsed time per iteration (ms): 4528.5 | learning rate: 2.185E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.045153E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.71 | backward: 3449.31 | backward-backward: 3449.24 | backward-allreduce: 0.00 | optimizer: 138.14 | batch generator: 2.22
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step88500
 samples/sec: 225.974 | iteration    92000/  143000 | elapsed time per iteration (ms): 4531.5 | learning rate: 2.158E-04 | approx flops per GPU: 83.4TFLOPS | lm_loss: 2.043000E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.49 | backward: 3452.34 | backward-backward: 3452.27 | backward-allreduce: 0.00 | optimizer: 138.44 | batch generator: 2.21
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step89000
 samples/sec: 226.325 | iteration    92500/  143000 | elapsed time per iteration (ms): 4524.5 | learning rate: 2.131E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.041921E+00 | loss scale: 8192.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.69 | backward: 3444.82 | backward-backward: 3444.75 | backward-allreduce: 0.00 | optimizer: 138.25 | batch generator: 2.22
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step89500
 samples/sec: 226.314 | iteration    93000/  143000 | elapsed time per iteration (ms): 4524.7 | learning rate: 2.104E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.043737E+00 | loss scale: 4096.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.49 | backward: 3445.49 | backward-backward: 3445.42 | backward-allreduce: 0.00 | optimizer: 138.21 | batch generator: 2.22
 samples/sec: 226.103 | iteration    93500/  143000 | elapsed time per iteration (ms): 4528.9 | learning rate: 2.077E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.040963E+00 | loss scale: 4096.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
after 93500 iterations memory (MB) | allocated: 6356.6455078125 | max allocated: 16125.47265625 | reserved: 17262.0 | max reserved: 17262.0
time (ms) | forward: 942.02 | backward: 3447.72 | backward-backward: 3447.65 | backward-allreduce: 0.00 | optimizer: 137.69 | batch generator: 5.50
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step90500
 samples/sec: 225.987 | iteration    94000/  143000 | elapsed time per iteration (ms): 4531.2 | learning rate: 2.050E-04 | approx flops per GPU: 83.4TFLOPS | lm_loss: 2.040425E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.05 | backward: 3452.38 | backward-backward: 3452.30 | backward-allreduce: 0.00 | optimizer: 137.47 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step91000
 samples/sec: 220.427 | iteration    94500/  143000 | elapsed time per iteration (ms): 4645.5 | learning rate: 2.024E-04 | approx flops per GPU: 81.4TFLOPS | lm_loss: 2.040609E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 947.40 | backward: 3552.13 | backward-backward: 3552.05 | backward-allreduce: 0.00 | optimizer: 137.51 | batch generator: 17.77
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step91500
 samples/sec: 226.294 | iteration    95000/  143000 | elapsed time per iteration (ms): 4525.1 | learning rate: 1.998E-04 | approx flops per GPU: 83.5TFLOPS | lm_loss: 2.037358E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.00 | backward: 3446.65 | backward-backward: 3446.58 | backward-allreduce: 0.00 | optimizer: 137.41 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step92000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 95000 | lm_loss value: 1.994387E+00 | lm_loss_ppl value: 7.347695E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 216.811 | iteration    95500/  143000 | elapsed time per iteration (ms): 4723.0 | learning rate: 1.971E-04 | approx flops per GPU: 80.0TFLOPS | lm_loss: 2.037165E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.85 | backward: 3446.85 | backward-backward: 3446.78 | backward-allreduce: 0.00 | optimizer: 137.46 | batch generator: 2.64
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step92500
 samples/sec: 226.371 | iteration    96000/  143000 | elapsed time per iteration (ms): 4523.6 | learning rate: 1.946E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.037140E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.08 | backward: 3446.07 | backward-backward: 3446.00 | backward-allreduce: 0.00 | optimizer: 137.09 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step93000
 samples/sec: 226.439 | iteration    96500/  143000 | elapsed time per iteration (ms): 4522.2 | learning rate: 1.920E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.034181E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.93 | backward: 3444.47 | backward-backward: 3444.40 | backward-allreduce: 0.00 | optimizer: 137.50 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step93500
 samples/sec: 226.442 | iteration    97000/  143000 | elapsed time per iteration (ms): 4522.1 | learning rate: 1.894E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.032874E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.04 | backward: 3444.58 | backward-backward: 3444.51 | backward-allreduce: 0.00 | optimizer: 137.21 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step94000
 samples/sec: 226.458 | iteration    97500/  143000 | elapsed time per iteration (ms): 4521.8 | learning rate: 1.869E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.032629E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.57 | backward: 3444.59 | backward-backward: 3444.52 | backward-allreduce: 0.00 | optimizer: 137.47 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step94500
 samples/sec: 226.455 | iteration    98000/  143000 | elapsed time per iteration (ms): 4521.9 | learning rate: 1.843E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.030533E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.20 | backward: 3443.94 | backward-backward: 3443.87 | backward-allreduce: 0.00 | optimizer: 137.44 | batch generator: 2.26
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step95000
 samples/sec: 226.502 | iteration    98500/  143000 | elapsed time per iteration (ms): 4520.9 | learning rate: 1.818E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.029609E+00 | loss scale: 16384.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 931.45 | backward: 3444.42 | backward-backward: 3444.34 | backward-allreduce: 0.00 | optimizer: 136.86 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step95500
 samples/sec: 226.448 | iteration    99000/  143000 | elapsed time per iteration (ms): 4522.0 | learning rate: 1.793E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.029894E+00 | loss scale: 8192.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.27 | backward: 3444.39 | backward-backward: 3444.32 | backward-allreduce: 0.00 | optimizer: 137.16 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step96000
 samples/sec: 226.397 | iteration    99500/  143000 | elapsed time per iteration (ms): 4523.0 | learning rate: 1.769E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.031221E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.82 | backward: 3444.43 | backward-backward: 3444.36 | backward-allreduce: 0.00 | optimizer: 137.40 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step96500
 samples/sec: 221.829 | iteration   100000/  143000 | elapsed time per iteration (ms): 4616.2 | learning rate: 1.744E-04 | approx flops per GPU: 81.9TFLOPS | lm_loss: 2.028838E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.62 | backward: 3538.91 | backward-backward: 3538.83 | backward-allreduce: 0.00 | optimizer: 137.43 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step97000
-----------------------------------------------------------------------------------------------------------
 validation results at iteration 100000 | lm_loss value: 1.986333E+00 | lm_loss_ppl value: 7.288755E+00 | 
-----------------------------------------------------------------------------------------------------------
 samples/sec: 204.854 | iteration   100500/  143000 | elapsed time per iteration (ms): 4998.7 | learning rate: 1.720E-04 | approx flops per GPU: 75.6TFLOPS | lm_loss: 2.026396E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.61 | backward: 3451.18 | backward-backward: 3451.11 | backward-allreduce: 0.00 | optimizer: 137.16 | batch generator: 109.10
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step97500
 samples/sec: 226.371 | iteration   101000/  143000 | elapsed time per iteration (ms): 4523.5 | learning rate: 1.695E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.025244E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.75 | backward: 3445.78 | backward-backward: 3445.71 | backward-allreduce: 0.00 | optimizer: 137.41 | batch generator: 2.22
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step98000
 samples/sec: 226.380 | iteration   101500/  143000 | elapsed time per iteration (ms): 4523.4 | learning rate: 1.671E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.025261E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.56 | backward: 3446.23 | backward-backward: 3446.16 | backward-allreduce: 0.00 | optimizer: 137.38 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step98500
 samples/sec: 226.365 | iteration   102000/  143000 | elapsed time per iteration (ms): 4523.7 | learning rate: 1.648E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.023159E+00 | loss scale: 16384.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 933.14 | backward: 3445.27 | backward-backward: 3445.20 | backward-allreduce: 0.00 | optimizer: 136.88 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step99000
 samples/sec: 226.375 | iteration   102500/  143000 | elapsed time per iteration (ms): 4523.5 | learning rate: 1.624E-04 | approx flops per GPU: 83.6TFLOPS | lm_loss: 2.022871E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.94 | backward: 3444.78 | backward-backward: 3444.71 | backward-allreduce: 0.00 | optimizer: 137.41 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step99500
 samples/sec: 220.809 | iteration   103000/  143000 | elapsed time per iteration (ms): 4637.5 | learning rate: 1.601E-04 | approx flops per GPU: 81.5TFLOPS | lm_loss: 2.021728E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 933.10 | backward: 3556.67 | backward-backward: 3556.60 | backward-allreduce: 0.00 | optimizer: 139.21 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step100000
 samples/sec: 218.880 | iteration   103500/  143000 | elapsed time per iteration (ms): 4678.4 | learning rate: 1.578E-04 | approx flops per GPU: 80.8TFLOPS | lm_loss: 2.020923E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.81 | backward: 3596.01 | backward-backward: 3595.94 | backward-allreduce: 0.00 | optimizer: 141.17 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step100500
 samples/sec: 219.680 | iteration   104000/  143000 | elapsed time per iteration (ms): 4661.3 | learning rate: 1.555E-04 | approx flops per GPU: 81.1TFLOPS | lm_loss: 2.019912E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 933.10 | backward: 3579.17 | backward-backward: 3579.10 | backward-allreduce: 0.00 | optimizer: 140.82 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step101000
 samples/sec: 216.468 | iteration   104500/  143000 | elapsed time per iteration (ms): 4730.5 | learning rate: 1.532E-04 | approx flops per GPU: 79.9TFLOPS | lm_loss: 2.018320E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.56 | backward: 3646.51 | backward-backward: 3646.43 | backward-allreduce: 0.00 | optimizer: 144.16 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step101500
 samples/sec: 218.269 | iteration   105000/  143000 | elapsed time per iteration (ms): 4691.5 | learning rate: 1.510E-04 | approx flops per GPU: 80.6TFLOPS | lm_loss: 2.019081E+00 | loss scale: 16384.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 931.74 | backward: 3610.08 | backward-backward: 3610.01 | backward-allreduce: 0.00 | optimizer: 141.36 | batch generator: 2.25
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step102000
-----------------------------------------------------------------------------------------------------------
 validation results at iteration 105000 | lm_loss value: 1.975867E+00 | lm_loss_ppl value: 7.212868E+00 | 
-----------------------------------------------------------------------------------------------------------
 samples/sec: 208.821 | iteration   105500/  143000 | elapsed time per iteration (ms): 4903.7 | learning rate: 1.487E-04 | approx flops per GPU: 77.1TFLOPS | lm_loss: 2.017798E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.47 | backward: 3615.37 | backward-backward: 3615.29 | backward-allreduce: 0.00 | optimizer: 141.38 | batch generator: 2.72
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step102500
 samples/sec: 218.759 | iteration   106000/  143000 | elapsed time per iteration (ms): 4680.9 | learning rate: 1.465E-04 | approx flops per GPU: 80.8TFLOPS | lm_loss: 2.015763E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.91 | backward: 3600.95 | backward-backward: 3600.88 | backward-allreduce: 0.00 | optimizer: 139.83 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step103000
 samples/sec: 218.762 | iteration   106500/  143000 | elapsed time per iteration (ms): 4680.9 | learning rate: 1.443E-04 | approx flops per GPU: 80.8TFLOPS | lm_loss: 2.015442E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 933.09 | backward: 3599.10 | backward-backward: 3599.03 | backward-allreduce: 0.00 | optimizer: 140.59 | batch generator: 2.40
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step103500
 samples/sec: 219.684 | iteration   107000/  143000 | elapsed time per iteration (ms): 4661.3 | learning rate: 1.422E-04 | approx flops per GPU: 81.1TFLOPS | lm_loss: 2.015300E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.82 | backward: 3581.73 | backward-backward: 3581.65 | backward-allreduce: 0.00 | optimizer: 138.25 | batch generator: 2.38
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step104000
 samples/sec: 216.268 | iteration   107500/  143000 | elapsed time per iteration (ms): 4734.9 | learning rate: 1.400E-04 | approx flops per GPU: 79.8TFLOPS | lm_loss: 2.013739E+00 | loss scale: 16384.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 932.84 | backward: 3652.74 | backward-backward: 3652.66 | backward-allreduce: 0.00 | optimizer: 140.93 | batch generator: 2.37
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step104500
 samples/sec: 216.421 | iteration   108000/  143000 | elapsed time per iteration (ms): 4731.5 | learning rate: 1.379E-04 | approx flops per GPU: 79.9TFLOPS | lm_loss: 2.011742E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.93 | backward: 3649.62 | backward-backward: 3649.54 | backward-allreduce: 0.00 | optimizer: 140.49 | batch generator: 2.36
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step105000, /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step105500
 samples/sec: 217.444 | iteration   108500/  143000 | elapsed time per iteration (ms): 4709.3 | learning rate: 1.358E-04 | approx flops per GPU: 80.3TFLOPS | lm_loss: 2.011313E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.67 | backward: 3627.85 | backward-backward: 3627.78 | backward-allreduce: 0.00 | optimizer: 140.32 | batch generator: 2.36
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step106000
 samples/sec: 216.706 | iteration   109000/  143000 | elapsed time per iteration (ms): 4725.3 | learning rate: 1.338E-04 | approx flops per GPU: 80.0TFLOPS | lm_loss: 2.010411E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.89 | backward: 3642.45 | backward-backward: 3642.37 | backward-allreduce: 0.00 | optimizer: 141.69 | batch generator: 2.36
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step106500
 samples/sec: 182.057 | iteration   109500/  143000 | elapsed time per iteration (ms): 5624.6 | learning rate: 1.317E-04 | approx flops per GPU: 67.2TFLOPS | lm_loss: 2.009421E+00 | loss scale: 16384.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 932.83 | backward: 4540.06 | backward-backward: 4539.97 | backward-allreduce: 0.00 | optimizer: 143.41 | batch generator: 2.39
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step107000
 samples/sec: 160.675 | iteration   110000/  143000 | elapsed time per iteration (ms): 6373.1 | learning rate: 1.297E-04 | approx flops per GPU: 59.3TFLOPS | lm_loss: 2.011936E+00 | loss scale: 8192.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 933.36 | backward: 5287.17 | backward-backward: 5287.07 | backward-allreduce: 0.00 | optimizer: 141.64 | batch generator: 2.39
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step107500
-----------------------------------------------------------------------------------------------------------
 validation results at iteration 110000 | lm_loss value: 1.967269E+00 | lm_loss_ppl value: 7.151122E+00 | 
-----------------------------------------------------------------------------------------------------------
 samples/sec: 156.196 | iteration   110500/  143000 | elapsed time per iteration (ms): 6555.9 | learning rate: 1.277E-04 | approx flops per GPU: 57.7TFLOPS | lm_loss: 2.009842E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 933.13 | backward: 5247.84 | backward-backward: 5247.75 | backward-allreduce: 0.00 | optimizer: 140.58 | batch generator: 2.77
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step108000
 samples/sec: 202.852 | iteration   111000/  143000 | elapsed time per iteration (ms): 5048.0 | learning rate: 1.257E-04 | approx flops per GPU: 74.9TFLOPS | lm_loss: 2.008719E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.81 | backward: 3969.42 | backward-backward: 3969.34 | backward-allreduce: 0.00 | optimizer: 137.44 | batch generator: 2.32
 samples/sec: 225.329 | iteration   111500/  143000 | elapsed time per iteration (ms): 4544.5 | learning rate: 1.238E-04 | approx flops per GPU: 83.2TFLOPS | lm_loss: 2.007216E+00 | loss scale: 8192.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
after 111500 iterations memory (MB) | allocated: 6504.0234375 | max allocated: 16272.8505859375 | reserved: 17262.0 | max reserved: 17262.0
time (ms) | forward: 939.82 | backward: 3460.14 | backward-backward: 3460.06 | backward-allreduce: 0.00 | optimizer: 143.04 | batch generator: 3.21
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step109000
 samples/sec: 225.211 | iteration   112000/  143000 | elapsed time per iteration (ms): 4546.8 | learning rate: 1.219E-04 | approx flops per GPU: 83.1TFLOPS | lm_loss: 2.005311E+00 | loss scale: 4096.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.05 | backward: 3461.05 | backward-backward: 3460.98 | backward-allreduce: 0.00 | optimizer: 144.40 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step109500
 samples/sec: 225.447 | iteration   112500/  143000 | elapsed time per iteration (ms): 4542.1 | learning rate: 1.200E-04 | approx flops per GPU: 83.2TFLOPS | lm_loss: 2.006976E+00 | loss scale: 4096.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.97 | backward: 3458.53 | backward-backward: 3458.46 | backward-allreduce: 0.00 | optimizer: 143.09 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step110000
 samples/sec: 225.540 | iteration   113000/  143000 | elapsed time per iteration (ms): 4540.2 | learning rate: 1.181E-04 | approx flops per GPU: 83.3TFLOPS | lm_loss: 2.003492E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.16 | backward: 3456.82 | backward-backward: 3456.74 | backward-allreduce: 0.00 | optimizer: 142.91 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step110500
 samples/sec: 225.284 | iteration   113500/  143000 | elapsed time per iteration (ms): 4545.4 | learning rate: 1.163E-04 | approx flops per GPU: 83.2TFLOPS | lm_loss: 2.005670E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.95 | backward: 3461.47 | backward-backward: 3461.40 | backward-allreduce: 0.00 | optimizer: 143.60 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step111000
 samples/sec: 225.669 | iteration   114000/  143000 | elapsed time per iteration (ms): 4537.6 | learning rate: 1.145E-04 | approx flops per GPU: 83.3TFLOPS | lm_loss: 2.003099E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.15 | backward: 3454.95 | backward-backward: 3454.87 | backward-allreduce: 0.00 | optimizer: 142.32 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step111500
 samples/sec: 225.517 | iteration   114500/  143000 | elapsed time per iteration (ms): 4540.7 | learning rate: 1.127E-04 | approx flops per GPU: 83.3TFLOPS | lm_loss: 2.001999E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.29 | backward: 3457.36 | backward-backward: 3457.29 | backward-allreduce: 0.00 | optimizer: 143.74 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step112000
 samples/sec: 225.680 | iteration   115000/  143000 | elapsed time per iteration (ms): 4537.4 | learning rate: 1.109E-04 | approx flops per GPU: 83.3TFLOPS | lm_loss: 2.000839E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.59 | backward: 3454.93 | backward-backward: 3454.86 | backward-allreduce: 0.00 | optimizer: 142.49 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step112500
-----------------------------------------------------------------------------------------------------------
 validation results at iteration 115000 | lm_loss value: 1.955436E+00 | lm_loss_ppl value: 7.066997E+00 | 
-----------------------------------------------------------------------------------------------------------
 samples/sec: 216.285 | iteration   115500/  143000 | elapsed time per iteration (ms): 4734.5 | learning rate: 1.092E-04 | approx flops per GPU: 79.8TFLOPS | lm_loss: 1.998555E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.49 | backward: 3456.30 | backward-backward: 3456.23 | backward-allreduce: 0.00 | optimizer: 143.39 | batch generator: 2.69
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step113000
 samples/sec: 225.487 | iteration   116000/  143000 | elapsed time per iteration (ms): 4541.3 | learning rate: 1.075E-04 | approx flops per GPU: 83.2TFLOPS | lm_loss: 1.999399E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.68 | backward: 3457.55 | backward-backward: 3457.48 | backward-allreduce: 0.00 | optimizer: 143.64 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step113500
 samples/sec: 225.586 | iteration   116500/  143000 | elapsed time per iteration (ms): 4539.3 | learning rate: 1.058E-04 | approx flops per GPU: 83.3TFLOPS | lm_loss: 1.999006E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.44 | backward: 3456.64 | backward-backward: 3456.57 | backward-allreduce: 0.00 | optimizer: 143.06 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step114000
 samples/sec: 225.423 | iteration   117000/  143000 | elapsed time per iteration (ms): 4542.6 | learning rate: 1.041E-04 | approx flops per GPU: 83.2TFLOPS | lm_loss: 1.996142E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.58 | backward: 3459.00 | backward-backward: 3458.93 | backward-allreduce: 0.00 | optimizer: 143.70 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step114500
 samples/sec: 225.438 | iteration   117500/  143000 | elapsed time per iteration (ms): 4542.3 | learning rate: 1.025E-04 | approx flops per GPU: 83.2TFLOPS | lm_loss: 1.996410E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.36 | backward: 3459.75 | backward-backward: 3459.67 | backward-allreduce: 0.00 | optimizer: 142.92 | batch generator: 2.26
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step115000
 samples/sec: 225.567 | iteration   118000/  143000 | elapsed time per iteration (ms): 4539.7 | learning rate: 1.009E-04 | approx flops per GPU: 83.3TFLOPS | lm_loss: 1.995015E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.61 | backward: 3456.66 | backward-backward: 3456.59 | backward-allreduce: 0.00 | optimizer: 143.18 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step115500
 samples/sec: 225.598 | iteration   118500/  143000 | elapsed time per iteration (ms): 4539.1 | learning rate: 9.934E-05 | approx flops per GPU: 83.3TFLOPS | lm_loss: 1.994024E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.42 | backward: 3456.18 | backward-backward: 3456.11 | backward-allreduce: 0.00 | optimizer: 143.02 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step116000
 samples/sec: 225.606 | iteration   119000/  143000 | elapsed time per iteration (ms): 4538.9 | learning rate: 9.780E-05 | approx flops per GPU: 83.3TFLOPS | lm_loss: 1.992871E+00 | loss scale: 8192.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 931.63 | backward: 3456.50 | backward-backward: 3456.43 | backward-allreduce: 0.00 | optimizer: 142.56 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step116500
 samples/sec: 225.580 | iteration   119500/  143000 | elapsed time per iteration (ms): 4539.4 | learning rate: 9.628E-05 | approx flops per GPU: 83.3TFLOPS | lm_loss: 1.993649E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.37 | backward: 3456.79 | backward-backward: 3456.72 | backward-allreduce: 0.00 | optimizer: 143.02 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step117000
 samples/sec: 225.358 | iteration   120000/  143000 | elapsed time per iteration (ms): 4543.9 | learning rate: 9.480E-05 | approx flops per GPU: 83.2TFLOPS | lm_loss: 1.992405E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.52 | backward: 3459.29 | backward-backward: 3459.22 | backward-allreduce: 0.00 | optimizer: 144.44 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step117500
-----------------------------------------------------------------------------------------------------------
 validation results at iteration 120000 | lm_loss value: 1.947511E+00 | lm_loss_ppl value: 7.011218E+00 | 
-----------------------------------------------------------------------------------------------------------
 samples/sec: 215.281 | iteration   120500/  143000 | elapsed time per iteration (ms): 4756.6 | learning rate: 9.334E-05 | approx flops per GPU: 79.5TFLOPS | lm_loss: 1.990981E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.20 | backward: 3477.25 | backward-backward: 3477.18 | backward-allreduce: 0.00 | optimizer: 144.07 | batch generator: 3.48
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step118000
 samples/sec: 225.449 | iteration   121000/  143000 | elapsed time per iteration (ms): 4542.0 | learning rate: 9.192E-05 | approx flops per GPU: 83.2TFLOPS | lm_loss: 1.991275E+00 | loss scale: 16384.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 931.54 | backward: 3458.47 | backward-backward: 3458.40 | backward-allreduce: 0.00 | optimizer: 143.71 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step118500
 samples/sec: 225.481 | iteration   121500/  143000 | elapsed time per iteration (ms): 4541.4 | learning rate: 9.052E-05 | approx flops per GPU: 83.2TFLOPS | lm_loss: 1.990468E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.35 | backward: 3457.82 | backward-backward: 3457.74 | backward-allreduce: 0.00 | optimizer: 143.99 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step119000
 samples/sec: 225.296 | iteration   122000/  143000 | elapsed time per iteration (ms): 4545.1 | learning rate: 8.916E-05 | approx flops per GPU: 83.2TFLOPS | lm_loss: 1.989551E+00 | loss scale: 8192.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.08 | backward: 3460.02 | backward-backward: 3459.94 | backward-allreduce: 0.00 | optimizer: 144.81 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step119500
 samples/sec: 225.517 | iteration   122500/  143000 | elapsed time per iteration (ms): 4540.7 | learning rate: 8.782E-05 | approx flops per GPU: 83.3TFLOPS | lm_loss: 1.990237E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.14 | backward: 3456.68 | backward-backward: 3456.60 | backward-allreduce: 0.00 | optimizer: 143.58 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step120000
 samples/sec: 225.527 | iteration   123000/  143000 | elapsed time per iteration (ms): 4540.5 | learning rate: 8.651E-05 | approx flops per GPU: 83.3TFLOPS | lm_loss: 1.990290E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.40 | backward: 3456.23 | backward-backward: 3456.16 | backward-allreduce: 0.00 | optimizer: 143.28 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step120500
 samples/sec: 225.440 | iteration   123500/  143000 | elapsed time per iteration (ms): 4542.2 | learning rate: 8.523E-05 | approx flops per GPU: 83.2TFLOPS | lm_loss: 1.988435E+00 | loss scale: 8192.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 932.29 | backward: 3458.84 | backward-backward: 3458.76 | backward-allreduce: 0.00 | optimizer: 142.21 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step121000
 samples/sec: 225.527 | iteration   124000/  143000 | elapsed time per iteration (ms): 4540.5 | learning rate: 8.398E-05 | approx flops per GPU: 83.3TFLOPS | lm_loss: 1.987477E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.51 | backward: 3456.31 | backward-backward: 3456.23 | backward-allreduce: 0.00 | optimizer: 143.24 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step121500
 samples/sec: 225.616 | iteration   124500/  143000 | elapsed time per iteration (ms): 4538.7 | learning rate: 8.276E-05 | approx flops per GPU: 83.3TFLOPS | lm_loss: 1.985991E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.29 | backward: 3455.22 | backward-backward: 3455.15 | backward-allreduce: 0.00 | optimizer: 142.93 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step122000
 samples/sec: 225.397 | iteration   125000/  143000 | elapsed time per iteration (ms): 4543.1 | learning rate: 8.158E-05 | approx flops per GPU: 83.2TFLOPS | lm_loss: 1.987353E+00 | loss scale: 8192.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 932.41 | backward: 3459.92 | backward-backward: 3459.84 | backward-allreduce: 0.00 | optimizer: 142.36 | batch generator: 2.26
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step122500
-----------------------------------------------------------------------------------------------------------
 validation results at iteration 125000 | lm_loss value: 1.944666E+00 | lm_loss_ppl value: 6.991295E+00 | 
-----------------------------------------------------------------------------------------------------------
 samples/sec: 216.255 | iteration   125500/  143000 | elapsed time per iteration (ms): 4735.1 | learning rate: 8.042E-05 | approx flops per GPU: 79.8TFLOPS | lm_loss: 1.986155E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.27 | backward: 3456.56 | backward-backward: 3456.49 | backward-allreduce: 0.00 | optimizer: 143.35 | batch generator: 2.67
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step123000
 samples/sec: 225.567 | iteration   126000/  143000 | elapsed time per iteration (ms): 4539.7 | learning rate: 7.929E-05 | approx flops per GPU: 83.3TFLOPS | lm_loss: 1.984752E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.46 | backward: 3455.84 | backward-backward: 3455.77 | backward-allreduce: 0.00 | optimizer: 142.79 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step123500
 samples/sec: 225.798 | iteration   126500/  143000 | elapsed time per iteration (ms): 4535.0 | learning rate: 7.820E-05 | approx flops per GPU: 83.4TFLOPS | lm_loss: 1.984365E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.25 | backward: 3452.51 | backward-backward: 3452.44 | backward-allreduce: 0.00 | optimizer: 141.40 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step124000
 samples/sec: 226.376 | iteration   127000/  143000 | elapsed time per iteration (ms): 4523.5 | learning rate: 7.713E-05 | approx flops per GPU: 83.6TFLOPS | lm_loss: 1.984910E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.50 | backward: 3444.29 | backward-backward: 3444.21 | backward-allreduce: 0.00 | optimizer: 138.41 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step124500
 samples/sec: 226.400 | iteration   127500/  143000 | elapsed time per iteration (ms): 4523.0 | learning rate: 7.610E-05 | approx flops per GPU: 83.6TFLOPS | lm_loss: 1.983438E+00 | loss scale: 4096.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 932.25 | backward: 3444.51 | backward-backward: 3444.43 | backward-allreduce: 0.00 | optimizer: 137.90 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step125000
 samples/sec: 225.756 | iteration   128000/  143000 | elapsed time per iteration (ms): 4535.9 | learning rate: 7.510E-05 | approx flops per GPU: 83.3TFLOPS | lm_loss: 1.984966E+00 | loss scale: 4096.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.79 | backward: 3456.62 | backward-backward: 3456.55 | backward-allreduce: 0.00 | optimizer: 138.55 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step125500
 samples/sec: 226.353 | iteration   128500/  143000 | elapsed time per iteration (ms): 4523.9 | learning rate: 7.413E-05 | approx flops per GPU: 83.6TFLOPS | lm_loss: 1.982671E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.66 | backward: 3444.98 | backward-backward: 3444.91 | backward-allreduce: 0.00 | optimizer: 138.52 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step126000
 samples/sec: 225.946 | iteration   129000/  143000 | elapsed time per iteration (ms): 4532.1 | learning rate: 7.319E-05 | approx flops per GPU: 83.4TFLOPS | lm_loss: 1.982815E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.96 | backward: 3451.04 | backward-backward: 3450.97 | backward-allreduce: 0.00 | optimizer: 140.82 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step126500
 samples/sec: 226.182 | iteration   129500/  143000 | elapsed time per iteration (ms): 4527.3 | learning rate: 7.228E-05 | approx flops per GPU: 83.5TFLOPS | lm_loss: 1.979980E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 931.76 | backward: 3447.87 | backward-backward: 3447.80 | backward-allreduce: 0.00 | optimizer: 139.33 | batch generator: 2.35
 samples/sec: 226.215 | iteration   130000/  143000 | elapsed time per iteration (ms): 4526.7 | learning rate: 7.140E-05 | approx flops per GPU: 83.5TFLOPS | lm_loss: 1.980654E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
after 130000 iterations memory (MB) | allocated: 6356.6455078125 | max allocated: 16125.47265625 | reserved: 17262.0 | max reserved: 17262.0
time (ms) | forward: 938.54 | backward: 3448.36 | backward-backward: 3448.28 | backward-allreduce: 0.00 | optimizer: 138.30 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step127500
-----------------------------------------------------------------------------------------------------------
 validation results at iteration 130000 | lm_loss value: 1.940109E+00 | lm_loss_ppl value: 6.959507E+00 | 
-----------------------------------------------------------------------------------------------------------
 samples/sec: 216.827 | iteration   130500/  143000 | elapsed time per iteration (ms): 4722.7 | learning rate: 7.056E-05 | approx flops per GPU: 80.0TFLOPS | lm_loss: 1.980375E+00 | loss scale: 16384.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 932.71 | backward: 3447.81 | backward-backward: 3447.73 | backward-allreduce: 0.00 | optimizer: 137.99 | batch generator: 2.69
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step128000
 samples/sec: 226.254 | iteration   131000/  143000 | elapsed time per iteration (ms): 4525.9 | learning rate: 6.975E-05 | approx flops per GPU: 83.5TFLOPS | lm_loss: 1.980156E+00 | loss scale: 8192.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.47 | backward: 3447.13 | backward-backward: 3447.05 | backward-allreduce: 0.00 | optimizer: 138.01 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step128500
 samples/sec: 226.163 | iteration   131500/  143000 | elapsed time per iteration (ms): 4527.7 | learning rate: 6.897E-05 | approx flops per GPU: 83.5TFLOPS | lm_loss: 1.977921E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.61 | backward: 3448.57 | backward-backward: 3448.50 | backward-allreduce: 0.00 | optimizer: 138.31 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step129000
 samples/sec: 226.278 | iteration   132000/  143000 | elapsed time per iteration (ms): 4525.4 | learning rate: 6.822E-05 | approx flops per GPU: 83.5TFLOPS | lm_loss: 1.978234E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.16 | backward: 3446.87 | backward-backward: 3446.80 | backward-allreduce: 0.00 | optimizer: 138.20 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step129500
 samples/sec: 226.315 | iteration   132500/  143000 | elapsed time per iteration (ms): 4524.7 | learning rate: 6.750E-05 | approx flops per GPU: 83.5TFLOPS | lm_loss: 1.978484E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.45 | backward: 3445.82 | backward-backward: 3445.75 | backward-allreduce: 0.00 | optimizer: 138.20 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step130000
 samples/sec: 226.367 | iteration   133000/  143000 | elapsed time per iteration (ms): 4523.6 | learning rate: 6.682E-05 | approx flops per GPU: 83.6TFLOPS | lm_loss: 1.976798E+00 | loss scale: 16384.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 932.16 | backward: 3445.26 | backward-backward: 3445.19 | backward-allreduce: 0.00 | optimizer: 137.67 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step130500
 samples/sec: 226.347 | iteration   133500/  143000 | elapsed time per iteration (ms): 4524.0 | learning rate: 6.616E-05 | approx flops per GPU: 83.6TFLOPS | lm_loss: 1.976104E+00 | loss scale: 8192.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.37 | backward: 3445.61 | backward-backward: 3445.54 | backward-allreduce: 0.00 | optimizer: 137.96 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step131000
 samples/sec: 226.304 | iteration   134000/  143000 | elapsed time per iteration (ms): 4524.9 | learning rate: 6.555E-05 | approx flops per GPU: 83.5TFLOPS | lm_loss: 1.975728E+00 | loss scale: 4096.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.21 | backward: 3446.62 | backward-backward: 3446.55 | backward-allreduce: 0.00 | optimizer: 137.98 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step131500
 samples/sec: 226.213 | iteration   134500/  143000 | elapsed time per iteration (ms): 4526.7 | learning rate: 6.496E-05 | approx flops per GPU: 83.5TFLOPS | lm_loss: 1.976804E+00 | loss scale: 4096.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.42 | backward: 3447.29 | backward-backward: 3447.21 | backward-allreduce: 0.00 | optimizer: 138.36 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step132000
 samples/sec: 225.753 | iteration   135000/  143000 | elapsed time per iteration (ms): 4535.9 | learning rate: 6.440E-05 | approx flops per GPU: 83.3TFLOPS | lm_loss: 1.979001E+00 | loss scale: 2048.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.16 | backward: 3448.04 | backward-backward: 3447.97 | backward-allreduce: 0.00 | optimizer: 137.94 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step132500
-----------------------------------------------------------------------------------------------------------
 validation results at iteration 135000 | lm_loss value: 1.938240E+00 | lm_loss_ppl value: 6.946512E+00 | 
-----------------------------------------------------------------------------------------------------------
 samples/sec: 216.839 | iteration   135500/  143000 | elapsed time per iteration (ms): 4722.4 | learning rate: 6.388E-05 | approx flops per GPU: 80.1TFLOPS | lm_loss: 1.978531E+00 | loss scale: 2048.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.46 | backward: 3448.34 | backward-backward: 3448.27 | backward-allreduce: 0.00 | optimizer: 138.42 | batch generator: 2.78
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step133000
 samples/sec: 226.258 | iteration   136000/  143000 | elapsed time per iteration (ms): 4525.8 | learning rate: 6.339E-05 | approx flops per GPU: 83.5TFLOPS | lm_loss: 1.979827E+00 | loss scale: 4096.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.18 | backward: 3447.08 | backward-backward: 3447.01 | backward-allreduce: 0.00 | optimizer: 138.30 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step133500
 samples/sec: 226.261 | iteration   136500/  143000 | elapsed time per iteration (ms): 4525.7 | learning rate: 6.293E-05 | approx flops per GPU: 83.5TFLOPS | lm_loss: 1.976965E+00 | loss scale: 4096.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.47 | backward: 3446.74 | backward-backward: 3446.67 | backward-allreduce: 0.00 | optimizer: 138.28 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step134000
 samples/sec: 226.281 | iteration   137000/  143000 | elapsed time per iteration (ms): 4525.4 | learning rate: 6.251E-05 | approx flops per GPU: 83.5TFLOPS | lm_loss: 1.978071E+00 | loss scale: 8192.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.32 | backward: 3446.61 | backward-backward: 3446.54 | backward-allreduce: 0.00 | optimizer: 138.16 | batch generator: 2.35
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step134500, /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step135000, /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step135500
 samples/sec: 226.271 | iteration   137500/  143000 | elapsed time per iteration (ms): 4525.5 | learning rate: 6.212E-05 | approx flops per GPU: 83.5TFLOPS | lm_loss: 1.975112E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.47 | backward: 3446.24 | backward-backward: 3446.17 | backward-allreduce: 0.00 | optimizer: 138.49 | batch generator: 2.29
 samples/sec: 226.344 | iteration   138000/  143000 | elapsed time per iteration (ms): 4524.1 | learning rate: 6.176E-05 | approx flops per GPU: 83.6TFLOPS | lm_loss: 1.973767E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.21 | backward: 3445.70 | backward-backward: 3445.62 | backward-allreduce: 0.00 | optimizer: 138.04 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step136000
 samples/sec: 226.317 | iteration   138500/  143000 | elapsed time per iteration (ms): 4524.6 | learning rate: 6.144E-05 | approx flops per GPU: 83.5TFLOPS | lm_loss: 1.975881E+00 | loss scale: 8192.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.38 | backward: 3445.91 | backward-backward: 3445.84 | backward-allreduce: 0.00 | optimizer: 138.03 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step136500
 samples/sec: 226.304 | iteration   139000/  143000 | elapsed time per iteration (ms): 4524.9 | learning rate: 6.115E-05 | approx flops per GPU: 83.5TFLOPS | lm_loss: 1.974844E+00 | loss scale: 4096.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.15 | backward: 3446.45 | backward-backward: 3446.37 | backward-allreduce: 0.00 | optimizer: 138.04 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step136500 copy
 samples/sec: 226.245 | iteration   139500/  143000 | elapsed time per iteration (ms): 4526.1 | learning rate: 6.089E-05 | approx flops per GPU: 83.5TFLOPS | lm_loss: 1.974050E+00 | loss scale: 4096.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.40 | backward: 3446.90 | backward-backward: 3446.83 | backward-allreduce: 0.00 | optimizer: 138.50 | batch generator: 2.26
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step137000
 samples/sec: 226.308 | iteration   140000/  143000 | elapsed time per iteration (ms): 4524.8 | learning rate: 6.066E-05 | approx flops per GPU: 83.5TFLOPS | lm_loss: 1.973476E+00 | loss scale: 8192.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.18 | backward: 3446.36 | backward-backward: 3446.29 | backward-allreduce: 0.00 | optimizer: 138.04 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step137500
-----------------------------------------------------------------------------------------------------------
 validation results at iteration 140000 | lm_loss value: 1.933604E+00 | lm_loss_ppl value: 6.914384E+00 | 
-----------------------------------------------------------------------------------------------------------
 samples/sec: 216.940 | iteration   140500/  143000 | elapsed time per iteration (ms): 4720.2 | learning rate: 6.047E-05 | approx flops per GPU: 80.1TFLOPS | lm_loss: 1.973758E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.43 | backward: 3446.15 | backward-backward: 3446.07 | backward-allreduce: 0.00 | optimizer: 138.39 | batch generator: 2.64
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step138000
 samples/sec: 226.321 | iteration   141000/  143000 | elapsed time per iteration (ms): 4524.6 | learning rate: 6.031E-05 | approx flops per GPU: 83.6TFLOPS | lm_loss: 1.973787E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.29 | backward: 3445.49 | backward-backward: 3445.42 | backward-allreduce: 0.00 | optimizer: 138.58 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step138500
 samples/sec: 226.332 | iteration   141500/  143000 | elapsed time per iteration (ms): 4524.3 | learning rate: 6.018E-05 | approx flops per GPU: 83.6TFLOPS | lm_loss: 1.972484E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.51 | backward: 3445.27 | backward-backward: 3445.20 | backward-allreduce: 0.00 | optimizer: 138.29 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step139000
 samples/sec: 226.337 | iteration   142000/  143000 | elapsed time per iteration (ms): 4524.2 | learning rate: 6.009E-05 | approx flops per GPU: 83.6TFLOPS | lm_loss: 1.973384E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.25 | backward: 3445.18 | backward-backward: 3445.10 | backward-allreduce: 0.00 | optimizer: 138.55 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step139500
 samples/sec: 226.311 | iteration   142500/  143000 | elapsed time per iteration (ms): 4524.7 | learning rate: 6.003E-05 | approx flops per GPU: 83.5TFLOPS | lm_loss: 1.971734E+00 | loss scale: 8192.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 932.45 | backward: 3445.57 | backward-backward: 3445.50 | backward-allreduce: 0.00 | optimizer: 138.36 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step140000
 samples/sec: 226.315 | iteration   143000/  143000 | elapsed time per iteration (ms): 4524.7 | learning rate: 6.000E-05 | approx flops per GPU: 83.5TFLOPS | lm_loss: 1.971988E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 932.27 | backward: 3445.57 | backward-backward: 3445.50 | backward-allreduce: 0.00 | optimizer: 138.65 | batch generator: 2.26
---------------------------------------------------------------------------------------------------------------------------
 validation results at the end of training for val data | lm_loss value: 1.933715E+00 | lm_loss_ppl value: 6.915151E+00 | 
---------------------------------------------------------------------------------------------------------------------------
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/1-5B_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step140500
----------------------------------------------------------------------------------------------------------------------
 test results at the end of training for test data | lm_loss value: 1.927591E+00 | lm_loss_ppl value: 6.872936E+00 | 
----------------------------------------------------------------------------------------------------------------------