llama-1B/64_GPUS/dp-1_tp-16_pp-4_mbz-64/log.out

========================
START TIME: Wed Jul  3 02:11:24 UTC 2024
python3 version = Python 3.10.14
========================
The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well.
Token is valid (permission: write).
Your token has been saved to /admin/home/ferdinand_mom/.cache/huggingface/token
Login successful
Already on 'bench_cluster'
M	examples/config_tiny_llama.py
M	examples/config_tiny_llama.yaml
M	examples/train_tiny_llama.sh
M	src/nanotron/models/llama.py
M	src/nanotron/trainer.py
Your branch is up to date with 'origin/bench_cluster'.
Job status: RUNNING
W0703 02:11:27.417000 140025732028224 torch/distributed/run.py:757] 
W0703 02:11:27.417000 140025732028224 torch/distributed/run.py:757] *****************************************
W0703 02:11:27.417000 140025732028224 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0703 02:11:27.417000 140025732028224 torch/distributed/run.py:757] *****************************************
W0703 02:11:27.417000 140181892396864 torch/distributed/run.py:757] 
W0703 02:11:27.417000 140181892396864 torch/distributed/run.py:757] *****************************************
W0703 02:11:27.417000 140181892396864 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0703 02:11:27.417000 140181892396864 torch/distributed/run.py:757] *****************************************
W0703 02:11:27.419000 140088508618560 torch/distributed/run.py:757] 
W0703 02:11:27.419000 140088508618560 torch/distributed/run.py:757] *****************************************
W0703 02:11:27.419000 140088508618560 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0703 02:11:27.419000 140088508618560 torch/distributed/run.py:757] *****************************************
W0703 02:11:27.426000 140471672833856 torch/distributed/run.py:757] 
W0703 02:11:27.426000 140471672833856 torch/distributed/run.py:757] *****************************************
W0703 02:11:27.426000 140471672833856 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0703 02:11:27.426000 140471672833856 torch/distributed/run.py:757] *****************************************
W0703 02:11:27.428000 140515281438528 torch/distributed/run.py:757] 
W0703 02:11:27.428000 140515281438528 torch/distributed/run.py:757] *****************************************
W0703 02:11:27.428000 140515281438528 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0703 02:11:27.428000 140515281438528 torch/distributed/run.py:757] *****************************************
W0703 02:11:27.430000 140513055242048 torch/distributed/run.py:757] 
W0703 02:11:27.430000 140513055242048 torch/distributed/run.py:757] *****************************************
W0703 02:11:27.430000 140513055242048 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0703 02:11:27.430000 140513055242048 torch/distributed/run.py:757] *****************************************
W0703 02:11:27.432000 140355073300288 torch/distributed/run.py:757] 
W0703 02:11:27.432000 140355073300288 torch/distributed/run.py:757] *****************************************
W0703 02:11:27.432000 140355073300288 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0703 02:11:27.432000 140355073300288 torch/distributed/run.py:757] *****************************************
W0703 02:11:27.498000 139778136758080 torch/distributed/run.py:757] 
W0703 02:11:27.498000 139778136758080 torch/distributed/run.py:757] *****************************************
W0703 02:11:27.498000 139778136758080 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0703 02:11:27.498000 139778136758080 torch/distributed/run.py:757] *****************************************
[default0]:07/03/2024 02:11:48 [WARNING|DP=0|PP=0|TP=0|ip-26-0-160-192]: [Vocab Size Padding] Padded vocab (size: 50257) with 15 dummy tokens (new size: 50272)
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Config:
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Config(general=GeneralArgs(project='bench_cluster',
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                            run='%date_%jobid',
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                            seed=42,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                            step=None,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                            consumed_train_samples=None,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                            benchmark_csv_path=None,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                            ignore_sanity_checks=True),
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        parallelism=ParallelismArgs(dp=1,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    pp=4,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    tp=16,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    pp_engine=<nanotron.parallel.pipeline_parallel.engine.OneForwardOneBackwardPipelineEngine object at 0x7f98a08d0670>,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    tp_mode=<TensorParallelLinearMode.REDUCE_SCATTER: 2>,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    tp_linear_async_communication=False,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    expert_parallel_size=1),
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        model=ModelArgs(model_config=LlamaConfig(bos_token_id=1,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 eos_token_id=2,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 hidden_act='silu',
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 hidden_size=2048,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 initializer_range=0.02,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 intermediate_size=4096,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 is_llama_config=True,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 max_position_embeddings=4096,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 num_attention_heads=32,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 num_hidden_layers=24,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 num_key_value_heads=32,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 pad_token_id=None,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 pretraining_tp=1,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 rms_norm_eps=1e-05,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 rope_scaling=None,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 rope_theta=10000.0,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 tie_word_embeddings=True,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 use_cache=True,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 vocab_size=50272),
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                        init_method=RandomInit(std=0.025),
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                        dtype=torch.bfloat16,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                        make_vocab_size_divisible_by=1,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                        ddp_bucket_cap_mb=25),
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        tokenizer=TokenizerArgs(tokenizer_name_or_path='openai-community/gpt2',
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                tokenizer_revision=None,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                tokenizer_max_length=None),
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        checkpoints=CheckpointsArgs(checkpoints_path=Path('/dev/null'),
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    checkpoint_interval=100000,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    save_initial_state=False,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    resume_checkpoint_path=None,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    checkpoints_path_is_shared_file_system=False),
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        logging=LoggingArgs(log_level='info',
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                            log_level_replica='info',
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                            iteration_step_info_interval=1),
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        tokens=TokensArgs(sequence_length=4096,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                          train_steps=20,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                          micro_batch_size=64,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                          batch_accumulation_per_replica=16,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                          val_check_interval=-1,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                          limit_val_batches=0,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                          limit_test_batches=0),
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        optimizer=OptimizerArgs(optimizer_factory=AdamWOptimizerArgs(adam_eps=1e-08,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                     adam_beta1=0.9,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                     adam_beta2=0.95,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                     torch_adam_is_fused=True,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                     name='adamW'),
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                zero_stage=1,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                weight_decay=0.01,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                clip_grad=1.0,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                accumulate_grad_in_fp32=True,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                learning_rate_scheduler=LRSchedulerArgs(learning_rate=0.0001,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                        lr_warmup_steps=1,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                        lr_warmup_style='linear',
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                        lr_decay_style='linear',
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                        lr_decay_steps=19,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                        lr_decay_starting_step=None,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                        min_decay_lr=1e-05)),
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        data_stages=[DatasetStageArgs(name='Training Stage',
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                      start_training_step=1,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                      data=DataArgs(dataset=PretrainDatasetsArgs(hf_dataset_or_datasets='roneneldan/TinyStories',
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                                 hf_dataset_splits='train',
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                                 hf_dataset_config_name=None,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                                 dataset_processing_num_proc_per_process=64,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                                 dataset_overwrite_cache=False,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                                 text_column_name='text'),
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                    seed=42,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                    num_loading_workers=0))],
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        profiler=ProfilerArgs(profiler_export_path=Path('/fsx/ferdinandmom/ferdinand-hf/bench_cluster/results/llama-1B/64_GPUS/dp-1_tp-16_pp-4_mbz-64')),
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        lighteval=None)
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Model Config:
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: LlamaConfig(bos_token_id=1,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             eos_token_id=2,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             hidden_act='silu',
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             hidden_size=2048,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             initializer_range=0.02,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             intermediate_size=4096,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             is_llama_config=True,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             max_position_embeddings=4096,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             num_attention_heads=32,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             num_hidden_layers=24,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             num_key_value_heads=32,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             pad_token_id=None,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             pretraining_tp=1,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             rms_norm_eps=1e-05,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             rope_scaling=None,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             rope_theta=10000.0,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             tie_word_embeddings=True,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             use_cache=True,
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             vocab_size=50272)
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Building model..
[default0]:07/03/2024 02:11:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Setting PP block ranks...
[default4]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=12|ip-26-0-163-226]: Local number of parameters: 18.4M (35.05MiB)
[default4]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=12|ip-26-0-163-226]: [After model building] Memory usage: 43.07MiB. Peak allocated: 45.10MiB Peak reserved: 60.00MiB
[default4]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=12|ip-26-0-163-226]: No checkpoint path provided.
[default3]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=11|ip-26-0-163-226]: Local number of parameters: 18.4M (35.05MiB)
[default3]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=11|ip-26-0-163-226]: [After model building] Memory usage: 43.07MiB. Peak allocated: 45.10MiB Peak reserved: 60.00MiB
[default3]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=11|ip-26-0-163-226]: No checkpoint path provided.
[default3]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=11|ip-26-0-161-178]: Local number of parameters: 24.8M (47.33MiB)
[default3]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=11|ip-26-0-161-178]: [After model building] Memory usage: 55.07MiB. Peak allocated: 57.10MiB Peak reserved: 74.00MiB
[default1]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=9|ip-26-0-161-178]: Local number of parameters: 24.8M (47.33MiB)
[default1]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=9|ip-26-0-161-178]: [After model building] Memory usage: 55.07MiB. Peak allocated: 57.10MiB Peak reserved: 74.00MiB
[default1]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=9|ip-26-0-161-178]: No checkpoint path provided.
[default3]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=11|ip-26-0-161-178]: No checkpoint path provided.
[default5]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=13|ip-26-0-163-226]: Local number of parameters: 18.4M (35.05MiB)
[default5]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=13|ip-26-0-163-226]: [After model building] Memory usage: 43.07MiB. Peak allocated: 45.10MiB Peak reserved: 60.00MiB
[default5]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=13|ip-26-0-163-226]: No checkpoint path provided.
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=8|ip-26-0-161-178]: Local number of parameters: 24.8M (47.33MiB)
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=8|ip-26-0-161-178]: [After model building] Memory usage: 55.07MiB. Peak allocated: 57.10MiB Peak reserved: 74.00MiB
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=8|ip-26-0-161-178]: No checkpoint path provided.
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=8|ip-26-0-163-226]: Local number of parameters: 18.4M (35.05MiB)
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=8|ip-26-0-163-226]: [After model building] Memory usage: 43.07MiB. Peak allocated: 45.10MiB Peak reserved: 60.00MiB
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=8|ip-26-0-163-226]: No checkpoint path provided.
[default7]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=7|ip-26-0-172-57]: Local number of parameters: 16.9M (32.31MiB)
[default6]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=6|ip-26-0-172-57]: Local number of parameters: 16.9M (32.31MiB)
[default6]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=6|ip-26-0-172-57]: [After model building] Memory usage: 36.32MiB. Peak allocated: 38.35MiB Peak reserved: 48.00MiB
[default7]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=7|ip-26-0-172-57]: [After model building] Memory usage: 36.32MiB. Peak allocated: 38.35MiB Peak reserved: 48.00MiB
[default7]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=7|ip-26-0-172-57]: No checkpoint path provided.
[default6]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=6|ip-26-0-172-57]: No checkpoint path provided.
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=8|ip-26-0-172-73]: Local number of parameters: 16.9M (32.31MiB)
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=8|ip-26-0-172-73]: [After model building] Memory usage: 36.32MiB. Peak allocated: 38.35MiB Peak reserved: 48.00MiB
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=8|ip-26-0-172-73]: No checkpoint path provided.
[default4]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=12|ip-26-0-172-73]: Local number of parameters: 16.9M (32.31MiB)
[default4]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=12|ip-26-0-172-73]: [After model building] Memory usage: 36.32MiB. Peak allocated: 38.35MiB Peak reserved: 48.00MiB
[default4]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=12|ip-26-0-172-73]: No checkpoint path provided.
[default5]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=13|ip-26-0-172-73]: Local number of parameters: 16.9M (32.31MiB)
[default5]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=13|ip-26-0-172-73]: [After model building] Memory usage: 36.32MiB. Peak allocated: 38.35MiB Peak reserved: 48.00MiB
[default5]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=13|ip-26-0-172-73]: No checkpoint path provided.
[default4]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=12|ip-26-0-161-178]: Local number of parameters: 24.8M (47.33MiB)
[default4]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=12|ip-26-0-161-178]: [After model building] Memory usage: 55.07MiB. Peak allocated: 57.10MiB Peak reserved: 74.00MiB
[default2]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=2|ip-26-0-163-220]: Local number of parameters: 18.4M (35.05MiB)
[default2]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=2|ip-26-0-163-220]: [After model building] Memory usage: 43.07MiB. Peak allocated: 45.10MiB Peak reserved: 60.00MiB
[default2]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=2|ip-26-0-163-220]: No checkpoint path provided.
[default4]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=12|ip-26-0-161-178]: No checkpoint path provided.
[default4]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=4|ip-26-0-163-220]: Local number of parameters: 18.4M (35.05MiB)
[default4]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=4|ip-26-0-163-220]: [After model building] Memory usage: 43.07MiB. Peak allocated: 45.10MiB Peak reserved: 60.00MiB
[default4]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=4|ip-26-0-163-220]: No checkpoint path provided.
[default5]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=13|ip-26-0-161-178]: Local number of parameters: 24.8M (47.33MiB)
[default5]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=13|ip-26-0-161-178]: [After model building] Memory usage: 55.07MiB. Peak allocated: 57.10MiB Peak reserved: 74.00MiB
[default5]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=13|ip-26-0-161-178]: No checkpoint path provided.
[default6]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=6|ip-26-0-163-220]: Local number of parameters: 18.4M (35.05MiB)
[default6]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=6|ip-26-0-163-220]: [After model building] Memory usage: 43.07MiB. Peak allocated: 45.10MiB Peak reserved: 60.00MiB
[default1]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=9|ip-26-0-163-226]: Local number of parameters: 18.4M (35.05MiB)
[default1]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=9|ip-26-0-163-226]: [After model building] Memory usage: 43.07MiB. Peak allocated: 45.10MiB Peak reserved: 60.00MiB
[default2]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=2|ip-26-0-168-238]: Local number of parameters: 15.8M (30.05MiB)
[default2]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=2|ip-26-0-168-238]: [After model building] Memory usage: 37.06MiB. Peak allocated: 39.09MiB Peak reserved: 58.00MiB
[default2]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=2|ip-26-0-168-238]: No checkpoint path provided.
[default6]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=6|ip-26-0-163-220]: No checkpoint path provided.
[default1]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=9|ip-26-0-163-226]: No checkpoint path provided.
[default7]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=7|ip-26-0-168-238]: Local number of parameters: 15.8M (30.05MiB)
[default7]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=7|ip-26-0-168-238]: [After model building] Memory usage: 37.06MiB. Peak allocated: 39.09MiB Peak reserved: 58.00MiB
[default7]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=7|ip-26-0-168-238]: No checkpoint path provided.
[default5]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=5|ip-26-0-163-220]: Local number of parameters: 18.4M (35.05MiB)
[default5]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=5|ip-26-0-163-220]: [After model building] Memory usage: 43.07MiB. Peak allocated: 45.10MiB Peak reserved: 60.00MiB
[default5]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=5|ip-26-0-163-220]: No checkpoint path provided.
[default1]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=9|ip-26-0-169-86]: Local number of parameters: 15.8M (30.05MiB)
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=8|ip-26-0-169-86]: Local number of parameters: 15.8M (30.05MiB)
[default1]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=9|ip-26-0-169-86]: [After model building] Memory usage: 37.06MiB. Peak allocated: 39.09MiB Peak reserved: 58.00MiB
[default1]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=9|ip-26-0-169-86]: No checkpoint path provided.
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=8|ip-26-0-169-86]: [After model building] Memory usage: 37.06MiB. Peak allocated: 39.09MiB Peak reserved: 58.00MiB
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=8|ip-26-0-169-86]: No checkpoint path provided.
[default4]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=12|ip-26-0-169-86]: Local number of parameters: 15.8M (30.05MiB)
[default4]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=12|ip-26-0-169-86]: [After model building] Memory usage: 37.06MiB. Peak allocated: 39.09MiB Peak reserved: 58.00MiB
[default4]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=12|ip-26-0-169-86]: No checkpoint path provided.
[default6]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=6|ip-26-0-168-238]: Local number of parameters: 15.8M (30.05MiB)
[default6]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=6|ip-26-0-168-238]: [After model building] Memory usage: 37.06MiB. Peak allocated: 39.09MiB Peak reserved: 58.00MiB
[default6]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=6|ip-26-0-168-238]: No checkpoint path provided.
[default7]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=7|ip-26-0-163-220]: Local number of parameters: 18.4M (35.05MiB)
[default7]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=7|ip-26-0-163-220]: [After model building] Memory usage: 43.07MiB. Peak allocated: 45.10MiB Peak reserved: 60.00MiB
[default7]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=7|ip-26-0-163-220]: No checkpoint path provided.
[default3]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=3|ip-26-0-172-57]: Local number of parameters: 16.9M (32.31MiB)
[default3]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=3|ip-26-0-172-57]: [After model building] Memory usage: 36.32MiB. Peak allocated: 38.35MiB Peak reserved: 48.00MiB
[default3]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=3|ip-26-0-172-57]: No checkpoint path provided.
[default1]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=1|ip-26-0-160-192]: Local number of parameters: 24.8M (47.33MiB)
[default1]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=1|ip-26-0-160-192]: [After model building] Memory usage: 55.07MiB. Peak allocated: 57.10MiB Peak reserved: 74.00MiB
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Total number of parameters: 1.21G (2315.81MiB)
[default1]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=9|ip-26-0-172-73]: Local number of parameters: 16.9M (32.31MiB)
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Local number of parameters: 24.8M (47.33MiB)
[default1]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=9|ip-26-0-172-73]: [After model building] Memory usage: 36.32MiB. Peak allocated: 38.35MiB Peak reserved: 48.00MiB
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [After model building] Memory usage: 55.07MiB. Peak allocated: 57.10MiB Peak reserved: 74.00MiB
[default5]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=5|ip-26-0-160-192]: Local number of parameters: 24.8M (47.33MiB)
[default5]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=5|ip-26-0-160-192]: [After model building] Memory usage: 55.07MiB. Peak allocated: 57.10MiB Peak reserved: 74.00MiB
[default5]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=5|ip-26-0-160-192]: No checkpoint path provided.
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: No checkpoint path provided.
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Parametrizing model parameters using StandardParametrizator
[default1]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=9|ip-26-0-172-73]: No checkpoint path provided.
[default3]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=11|ip-26-0-172-73]: Local number of parameters: 16.9M (32.31MiB)
[default3]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=11|ip-26-0-172-73]: [After model building] Memory usage: 36.32MiB. Peak allocated: 38.35MiB Peak reserved: 48.00MiB
[default3]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=11|ip-26-0-172-73]: No checkpoint path provided.
[default7]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=7|ip-26-0-160-192]: Local number of parameters: 24.8M (47.33MiB)
[default7]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=7|ip-26-0-160-192]: [After model building] Memory usage: 55.07MiB. Peak allocated: 57.10MiB Peak reserved: 74.00MiB
[default7]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=7|ip-26-0-160-192]: No checkpoint path provided.
[default2]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=2|ip-26-0-160-192]: Local number of parameters: 24.8M (47.33MiB)
[default2]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=2|ip-26-0-160-192]: [After model building] Memory usage: 55.07MiB. Peak allocated: 57.10MiB Peak reserved: 74.00MiB
[default2]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=2|ip-26-0-160-192]: No checkpoint path provided.
[default3]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=3|ip-26-0-160-192]: Local number of parameters: 24.8M (47.33MiB)
[default3]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=3|ip-26-0-160-192]: [After model building] Memory usage: 55.07MiB. Peak allocated: 57.10MiB Peak reserved: 74.00MiB
[default3]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=3|ip-26-0-160-192]: No checkpoint path provided.
[default4]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=4|ip-26-0-160-192]: Local number of parameters: 24.8M (47.33MiB)
[default4]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=4|ip-26-0-160-192]: [After model building] Memory usage: 55.07MiB. Peak allocated: 57.10MiB Peak reserved: 74.00MiB
[default4]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=4|ip-26-0-160-192]: No checkpoint path provided.
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=0|ip-26-0-168-238]: Local number of parameters: 15.8M (30.05MiB)
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=0|ip-26-0-168-238]: [After model building] Memory usage: 37.06MiB. Peak allocated: 39.09MiB Peak reserved: 58.00MiB
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=0|ip-26-0-168-238]: No checkpoint path provided.
[default5]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=5|ip-26-0-172-57]: Local number of parameters: 16.9M (32.31MiB)
[default1]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=1|ip-26-0-172-57]: Local number of parameters: 16.9M (32.31MiB)
[default5]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=13|ip-26-0-169-86]: Local number of parameters: 15.8M (30.05MiB)
[default5]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=13|ip-26-0-169-86]: [After model building] Memory usage: 37.06MiB. Peak allocated: 39.09MiB Peak reserved: 58.00MiB
[default5]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=13|ip-26-0-169-86]: No checkpoint path provided.
[default5]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=5|ip-26-0-172-57]: [After model building] Memory usage: 36.32MiB. Peak allocated: 38.35MiB Peak reserved: 48.00MiB
[default5]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=5|ip-26-0-172-57]: No checkpoint path provided.
[default1]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=1|ip-26-0-172-57]: [After model building] Memory usage: 36.32MiB. Peak allocated: 38.35MiB Peak reserved: 48.00MiB
[default2]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=2|ip-26-0-172-57]: Local number of parameters: 16.9M (32.31MiB)
[default2]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=2|ip-26-0-172-57]: [After model building] Memory usage: 36.32MiB. Peak allocated: 38.35MiB Peak reserved: 48.00MiB
[default2]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=2|ip-26-0-172-57]: No checkpoint path provided.
[default6]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=6|ip-26-0-160-192]: Local number of parameters: 24.8M (47.33MiB)
[default6]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=6|ip-26-0-160-192]: [After model building] Memory usage: 55.07MiB. Peak allocated: 57.10MiB Peak reserved: 74.00MiB
[default6]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=6|ip-26-0-160-192]: No checkpoint path provided.
[default4]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=4|ip-26-0-172-57]: Local number of parameters: 16.9M (32.31MiB)
[default4]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=4|ip-26-0-172-57]: [After model building] Memory usage: 36.32MiB. Peak allocated: 38.35MiB Peak reserved: 48.00MiB
[default4]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=4|ip-26-0-172-57]: No checkpoint path provided.
[default1]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=1|ip-26-0-168-238]: Local number of parameters: 15.8M (30.05MiB)
[default1]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=1|ip-26-0-168-238]: [After model building] Memory usage: 37.06MiB. Peak allocated: 39.09MiB Peak reserved: 58.00MiB
[default1]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=1|ip-26-0-168-238]: No checkpoint path provided.
[default3]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=3|ip-26-0-168-238]: Local number of parameters: 15.8M (30.05MiB)
[default3]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=3|ip-26-0-168-238]: [After model building] Memory usage: 37.06MiB. Peak allocated: 39.09MiB Peak reserved: 58.00MiB
[default3]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=3|ip-26-0-168-238]: No checkpoint path provided.
[default4]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=4|ip-26-0-168-238]: Local number of parameters: 15.8M (30.05MiB)
[default4]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=4|ip-26-0-168-238]: [After model building] Memory usage: 37.06MiB. Peak allocated: 39.09MiB Peak reserved: 58.00MiB
[default4]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=4|ip-26-0-168-238]: No checkpoint path provided.
[default5]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=5|ip-26-0-168-238]: Local number of parameters: 15.8M (30.05MiB)
[default5]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=5|ip-26-0-168-238]: [After model building] Memory usage: 37.06MiB. Peak allocated: 39.09MiB Peak reserved: 58.00MiB
[default5]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=5|ip-26-0-168-238]: No checkpoint path provided.
[default3]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=11|ip-26-0-169-86]: Local number of parameters: 15.8M (30.05MiB)
[default3]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=11|ip-26-0-169-86]: [After model building] Memory usage: 37.06MiB. Peak allocated: 39.09MiB Peak reserved: 58.00MiB
[default3]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=11|ip-26-0-169-86]: No checkpoint path provided.
[default6]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=14|ip-26-0-169-86]: Local number of parameters: 15.8M (30.05MiB)
[default6]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=14|ip-26-0-169-86]: [After model building] Memory usage: 37.06MiB. Peak allocated: 39.09MiB Peak reserved: 58.00MiB
[default6]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=14|ip-26-0-169-86]: No checkpoint path provided.
[default2]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=10|ip-26-0-161-178]: Local number of parameters: 24.8M (47.33MiB)
[default2]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=10|ip-26-0-161-178]: [After model building] Memory usage: 55.07MiB. Peak allocated: 57.10MiB Peak reserved: 74.00MiB
[default2]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=10|ip-26-0-161-178]: No checkpoint path provided.
[default6]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=14|ip-26-0-163-226]: Local number of parameters: 18.4M (35.05MiB)
[default6]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=14|ip-26-0-163-226]: [After model building] Memory usage: 43.07MiB. Peak allocated: 45.10MiB Peak reserved: 60.00MiB
[default6]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=14|ip-26-0-163-226]: No checkpoint path provided.
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=0|ip-26-0-172-57]: Local number of parameters: 16.9M (32.31MiB)
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=0|ip-26-0-172-57]: [After model building] Memory usage: 36.32MiB. Peak allocated: 38.35MiB Peak reserved: 48.00MiB
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=0|ip-26-0-172-57]: No checkpoint path provided.
[default3]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=3|ip-26-0-163-220]: Local number of parameters: 18.4M (35.05MiB)
[default3]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=3|ip-26-0-163-220]: [After model building] Memory usage: 43.07MiB. Peak allocated: 45.10MiB Peak reserved: 60.00MiB
[default3]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=3|ip-26-0-163-220]: No checkpoint path provided.
[default1]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=1|ip-26-0-163-220]: Local number of parameters: 18.4M (35.05MiB)
[default1]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=1|ip-26-0-163-220]: [After model building] Memory usage: 43.07MiB. Peak allocated: 45.10MiB Peak reserved: 60.00MiB
[default1]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=1|ip-26-0-163-220]: No checkpoint path provided.
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=0|ip-26-0-163-220]: Local number of parameters: 18.4M (35.05MiB)
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=0|ip-26-0-163-220]: [After model building] Memory usage: 43.07MiB. Peak allocated: 45.10MiB Peak reserved: 60.00MiB
[default0]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=0|ip-26-0-163-220]: No checkpoint path provided.
[default7]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=15|ip-26-0-161-178]: Local number of parameters: 24.8M (47.33MiB)
[default7]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=15|ip-26-0-161-178]: [After model building] Memory usage: 55.07MiB. Peak allocated: 57.10MiB Peak reserved: 74.00MiB
[default7]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=15|ip-26-0-161-178]: No checkpoint path provided.
[default6]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=14|ip-26-0-161-178]: Local number of parameters: 24.8M (47.33MiB)
[default6]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=14|ip-26-0-161-178]: [After model building] Memory usage: 55.07MiB. Peak allocated: 57.10MiB Peak reserved: 74.00MiB
[default7]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=15|ip-26-0-163-226]: Local number of parameters: 18.4M (35.05MiB)
[default7]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=15|ip-26-0-163-226]: [After model building] Memory usage: 43.07MiB. Peak allocated: 45.10MiB Peak reserved: 60.00MiB
[default7]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=15|ip-26-0-163-226]: No checkpoint path provided.
[default6]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=14|ip-26-0-161-178]: No checkpoint path provided.
[default2]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=10|ip-26-0-163-226]: Local number of parameters: 18.4M (35.05MiB)
[default2]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=10|ip-26-0-163-226]: [After model building] Memory usage: 43.07MiB. Peak allocated: 45.10MiB Peak reserved: 60.00MiB
[default2]:07/03/2024 02:12:05 [INFO|DP=0|PP=1|TP=10|ip-26-0-163-226]: No checkpoint path provided.
[default2]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=10|ip-26-0-169-86]: Local number of parameters: 15.8M (30.05MiB)
[default2]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=10|ip-26-0-169-86]: [After model building] Memory usage: 37.06MiB. Peak allocated: 39.09MiB Peak reserved: 58.00MiB
[default2]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=10|ip-26-0-169-86]: No checkpoint path provided.
[default2]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=10|ip-26-0-172-73]: Local number of parameters: 16.9M (32.31MiB)
[default2]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=10|ip-26-0-172-73]: [After model building] Memory usage: 36.32MiB. Peak allocated: 38.35MiB Peak reserved: 48.00MiB
[default2]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=10|ip-26-0-172-73]: No checkpoint path provided.
[default7]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=15|ip-26-0-172-73]: Local number of parameters: 16.9M (32.31MiB)
[default7]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=15|ip-26-0-172-73]: [After model building] Memory usage: 36.32MiB. Peak allocated: 38.35MiB Peak reserved: 48.00MiB
[default7]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=15|ip-26-0-172-73]: No checkpoint path provided.
[default1]:07/03/2024 02:12:05 [INFO|DP=0|PP=0|TP=1|ip-26-0-160-192]: No checkpoint path provided.
[default6]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=14|ip-26-0-172-73]: Local number of parameters: 16.9M (32.31MiB)
[default6]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=14|ip-26-0-172-73]: [After model building] Memory usage: 36.32MiB. Peak allocated: 38.35MiB Peak reserved: 48.00MiB
[default6]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=14|ip-26-0-172-73]: No checkpoint path provided.
[default7]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=15|ip-26-0-169-86]: Local number of parameters: 15.8M (30.05MiB)
[default7]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=15|ip-26-0-169-86]: [After model building] Memory usage: 37.06MiB. Peak allocated: 39.09MiB Peak reserved: 58.00MiB
[default7]:07/03/2024 02:12:05 [INFO|DP=0|PP=2|TP=15|ip-26-0-169-86]: No checkpoint path provided.
[default1]:07/03/2024 02:12:05 [INFO|DP=0|PP=3|TP=1|ip-26-0-172-57]: No checkpoint path provided.
[default0]:07/03/2024 02:12:07 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [Optimizer Building] Using LearningRateForSP as learning rate
[default0]:07/03/2024 02:12:07 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [ZeRO sharding] Size of optimizer params per rank:
[default0]:07/03/2024 02:12:07 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [ZeRO sharding] DP Rank 0 has 24.8M out of 24.8M (100.00%) params' optimizer states
[default0]:07/03/2024 02:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [Training Plan] Stage Training Stage has 19 remaining training steps and has consumed 0 samples
[default0]:07/03/2024 02:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Using `datasets` library
[default0]:07/03/2024 02:12:09 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Loading tokenizer from openai-community/gpt2 and transformers/hf_hub versions ('4.41.2', '0.23.4')
[default0]:07/03/2024 02:12:09 [WARNING|DP=0|PP=0|TP=0|ip-26-0-160-192]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/03/2024 02:12:11 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [Training Plan] There are 1 training stages 
[default0]:07/03/2024 02:12:11 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [Stage Training Stage] start from step 1 
[default0]:07/03/2024 02:12:11 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: 
[default0]:07/03/2024 02:12:11 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [Start training] datetime: 2024-07-03 02:12:11.369396 | mbs: 64 | grad_accum: 16 | global_batch_size: 1024 | sequence_length: 4096 | train_steps: 20 | start_iteration_step: 0 | consumed_train_samples: 0
[default0]:07/03/2024 02:12:11 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Resuming training from stage Training Stage, it has trained for 0 samples and has 19 remaining train steps
[default0]:07/03/2024 02:12:11 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:  Memory usage: 244.38MiB. Peak allocated 244.38MiB. Peak reserved: 266.00MiB
[default3]:07/03/2024 02:12:11 [WARNING|DP=0|PP=1|TP=3|ip-26-0-163-220]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/03/2024 02:12:11 [WARNING|DP=0|PP=1|TP=8|ip-26-0-163-226]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/03/2024 02:12:11 [WARNING|DP=0|PP=1|TP=0|ip-26-0-163-220]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/03/2024 02:12:11 [WARNING|DP=0|PP=3|TP=13|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/03/2024 02:12:11 [WARNING|DP=0|PP=1|TP=10|ip-26-0-163-226]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/03/2024 02:12:11 [WARNING|DP=0|PP=2|TP=12|ip-26-0-169-86]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/03/2024 02:12:11 [WARNING|DP=0|PP=0|TP=2|ip-26-0-160-192]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/03/2024 02:12:11 [WARNING|DP=0|PP=2|TP=13|ip-26-0-169-86]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/03/2024 02:12:11 [WARNING|DP=0|PP=2|TP=15|ip-26-0-169-86]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/03/2024 02:12:11 [WARNING|DP=0|PP=3|TP=4|ip-26-0-172-57]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/03/2024 02:12:11 [WARNING|DP=0|PP=2|TP=11|ip-26-0-169-86]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/03/2024 02:12:11 [WARNING|DP=0|PP=0|TP=9|ip-26-0-161-178]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/03/2024 02:12:11 [WARNING|DP=0|PP=1|TP=11|ip-26-0-163-226]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/03/2024 02:12:11 [WARNING|DP=0|PP=0|TP=8|ip-26-0-161-178]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/03/2024 02:12:11 [WARNING|DP=0|PP=3|TP=0|ip-26-0-172-57]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/03/2024 02:12:11 [WARNING|DP=0|PP=1|TP=1|ip-26-0-163-220]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/03/2024 02:12:11 [WARNING|DP=0|PP=3|TP=6|ip-26-0-172-57]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/03/2024 02:12:11 [WARNING|DP=0|PP=3|TP=7|ip-26-0-172-57]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/03/2024 02:12:11 [WARNING|DP=0|PP=0|TP=12|ip-26-0-161-178]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/03/2024 02:12:11 [WARNING|DP=0|PP=3|TP=8|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/03/2024 02:12:11 [WARNING|DP=0|PP=1|TP=4|ip-26-0-163-220]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/03/2024 02:12:11 [WARNING|DP=0|PP=1|TP=15|ip-26-0-163-226]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/03/2024 02:12:11 [WARNING|DP=0|PP=1|TP=9|ip-26-0-163-226]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/03/2024 02:12:11 [WARNING|DP=0|PP=0|TP=13|ip-26-0-161-178]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/03/2024 02:12:11 [WARNING|DP=0|PP=0|TP=14|ip-26-0-161-178]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/03/2024 02:12:11 [WARNING|DP=0|PP=2|TP=6|ip-26-0-168-238]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/03/2024 02:12:11 [WARNING|DP=0|PP=2|TP=8|ip-26-0-169-86]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/03/2024 02:12:11 [WARNING|DP=0|PP=2|TP=10|ip-26-0-169-86]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/03/2024 02:12:11 [WARNING|DP=0|PP=3|TP=15|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/03/2024 02:12:11 [WARNING|DP=0|PP=0|TP=1|ip-26-0-160-192]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/03/2024 02:12:11 [WARNING|DP=0|PP=3|TP=3|ip-26-0-172-57]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/03/2024 02:12:11 [WARNING|DP=0|PP=0|TP=5|ip-26-0-160-192]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/03/2024 02:12:11 [WARNING|DP=0|PP=3|TP=9|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/03/2024 02:12:11 [WARNING|DP=0|PP=0|TP=4|ip-26-0-160-192]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/03/2024 02:12:11 [WARNING|DP=0|PP=0|TP=6|ip-26-0-160-192]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/03/2024 02:12:11 [WARNING|DP=0|PP=2|TP=1|ip-26-0-168-238]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/03/2024 02:12:11 [WARNING|DP=0|PP=2|TP=3|ip-26-0-168-238]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/03/2024 02:12:11 [WARNING|DP=0|PP=2|TP=5|ip-26-0-168-238]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/03/2024 02:12:11 [WARNING|DP=0|PP=2|TP=14|ip-26-0-169-86]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/03/2024 02:12:11 [WARNING|DP=0|PP=0|TP=10|ip-26-0-161-178]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/03/2024 02:12:11 [WARNING|DP=0|PP=0|TP=11|ip-26-0-161-178]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/03/2024 02:12:11 [WARNING|DP=0|PP=1|TP=13|ip-26-0-163-226]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/03/2024 02:12:11 [WARNING|DP=0|PP=0|TP=15|ip-26-0-161-178]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/03/2024 02:12:11 [WARNING|DP=0|PP=3|TP=12|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/03/2024 02:12:11 [WARNING|DP=0|PP=1|TP=2|ip-26-0-163-220]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/03/2024 02:12:11 [WARNING|DP=0|PP=2|TP=2|ip-26-0-168-238]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/03/2024 02:12:11 [WARNING|DP=0|PP=1|TP=6|ip-26-0-163-220]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/03/2024 02:12:11 [WARNING|DP=0|PP=1|TP=7|ip-26-0-163-220]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/03/2024 02:12:11 [WARNING|DP=0|PP=3|TP=14|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/03/2024 02:12:11 [WARNING|DP=0|PP=3|TP=2|ip-26-0-172-57]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/03/2024 02:12:11 [WARNING|DP=0|PP=3|TP=5|ip-26-0-172-57]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/03/2024 02:12:11 [WARNING|DP=0|PP=1|TP=12|ip-26-0-163-226]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/03/2024 02:12:11 [WARNING|DP=0|PP=2|TP=7|ip-26-0-168-238]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/03/2024 02:12:11 [WARNING|DP=0|PP=1|TP=5|ip-26-0-163-220]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/03/2024 02:12:11 [WARNING|DP=0|PP=3|TP=11|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/03/2024 02:12:11 [WARNING|DP=0|PP=2|TP=0|ip-26-0-168-238]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/03/2024 02:12:11 [WARNING|DP=0|PP=3|TP=1|ip-26-0-172-57]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/03/2024 02:12:12 [WARNING|DP=0|PP=1|TP=14|ip-26-0-163-226]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/03/2024 02:12:12 [WARNING|DP=0|PP=0|TP=3|ip-26-0-160-192]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/03/2024 02:12:12 [WARNING|DP=0|PP=2|TP=9|ip-26-0-169-86]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/03/2024 02:12:12 [WARNING|DP=0|PP=3|TP=10|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/03/2024 02:12:12 [WARNING|DP=0|PP=2|TP=4|ip-26-0-168-238]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/03/2024 02:12:12 [WARNING|DP=0|PP=0|TP=7|ip-26-0-160-192]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:[rank9]: Traceback (most recent call last):
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default1]:[rank9]:     trainer.train(dataloader)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default1]:[rank9]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default1]:[rank9]:     outputs = self.pipeline_engine.train_batch_iter(
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default1]:[rank9]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default1]:[rank9]:     output = model(**micro_batch)
[default5]:[rank13]: Traceback (most recent call last):
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default6]:[rank14]: Traceback (most recent call last):
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default5]:[rank13]:     trainer.train(dataloader)
[default3]:[rank11]: Traceback (most recent call last):
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank9]:     return self._call_impl(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank14]:     trainer.train(dataloader)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default1]:[rank9]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default6]:[rank14]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default5]:[rank13]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default3]:[rank11]:     trainer.train(dataloader)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default5]:[rank13]:     outputs = self.pipeline_engine.train_batch_iter(
[default1]:[rank9]:     sharded_logits = self.model(
[default2]:[rank10]: Traceback (most recent call last):
[default3]:[rank11]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank14]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank13]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default2]:[rank10]:     trainer.train(dataloader)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default1]:[rank9]:     return self._call_impl(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank13]:     output = model(**micro_batch)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default3]:[rank11]:     outputs = self.pipeline_engine.train_batch_iter(
[default1]:[rank9]:     return forward_call(*args, **kwargs)
[default2]:[rank10]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default6]:[rank14]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default5]:[rank13]:     return self._call_impl(*args, **kwargs)
[default3]:[rank11]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default2]:[rank10]:     outputs = self.pipeline_engine.train_batch_iter(
[default6]:[rank14]:     output = model(**micro_batch)
[default1]:[rank9]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank11]:     output = model(**micro_batch)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default6]:[rank14]:     return self._call_impl(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank13]:     return forward_call(*args, **kwargs)
[default2]:[rank10]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default1]:[rank9]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank11]:     return self._call_impl(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default2]:[rank10]:     output = model(**micro_batch)
[default6]:[rank14]:     return forward_call(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank10]:     return self._call_impl(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank13]:     sharded_logits = self.model(
[default1]:[rank9]:     return self._call_impl(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank9]:     return forward_call(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default6]:[rank14]:     sharded_logits = self.model(
[default3]:[rank11]:     return forward_call(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default5]:[rank13]:     return self._call_impl(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank11]:     sharded_logits = self.model(
[default2]:[rank10]:     return forward_call(*args, **kwargs)
[default1]:[rank9]:     output = self.pp_block(**new_kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default6]:[rank14]:     return self._call_impl(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank13]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:     return self._call_impl(*args, **kwargs)
[default6]:[rank14]:     return forward_call(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default6]:[rank14]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default2]:[rank10]:     sharded_logits = self.model(
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank15]: Traceback (most recent call last):
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default5]:[rank13]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default3]:[rank11]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default4]:[rank12]: Traceback (most recent call last):
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default6]:[rank14]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank15]:     trainer.train(dataloader)
[default4]:[rank12]:     trainer.train(dataloader)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank11]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default1]:[rank9]:     return self._call_impl(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default6]:[rank14]:     return self._call_impl(*args, **kwargs)
[default2]:[rank10]:     return self._call_impl(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank9]:     return forward_call(*args, **kwargs)
[default7]:[rank15]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default5]:[rank13]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default6]:[rank14]:     return forward_call(*args, **kwargs)
[default4]:[rank12]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default3]:[rank11]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default4]:[rank12]:     outputs = self.pipeline_engine.train_batch_iter(
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default7]:[rank15]:     outputs = self.pipeline_engine.train_batch_iter(
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank10]:     return forward_call(*args, **kwargs)
[default6]:[rank14]:     output = self.pp_block(**new_kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default3]:[rank11]:     return self._call_impl(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default2]:[rank10]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default7]:[rank15]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank14]:     return self._call_impl(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank12]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default1]:[rank9]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default3]:[rank11]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default5]:[rank13]:     return self._call_impl(*args, **kwargs)
[default3]:[rank11]:     output = self.pp_block(**new_kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default2]:[rank10]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default7]:[rank15]:     output = model(**micro_batch)
[default6]:[rank14]:     return forward_call(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank10]:     return self._call_impl(*args, **kwargs)
[default5]:[rank13]:     return forward_call(*args, **kwargs)
[default6]:[rank14]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank10]:     return forward_call(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default1]:[rank9]:     return self._call_impl(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default5]:[rank13]:     output = self.pp_block(**new_kwargs)
[default2]:[rank10]:     output = self.pp_block(**new_kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank9]:     return forward_call(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default5]:[rank13]:     return self._call_impl(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank13]:     return forward_call(*args, **kwargs)
[default4]:[rank12]:     output = model(**micro_batch)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default1]:[rank9]:     output = self.o_proj(attention_output)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank12]:     return self._call_impl(*args, **kwargs)
[default5]:[rank13]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank14]:     return self._call_impl(*args, **kwargs)
[default7]:[rank15]:     return self._call_impl(*args, **kwargs)
[default2]:[rank10]:     return self._call_impl(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank11]:     return self._call_impl(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank9]:     return self._call_impl(*args, **kwargs)
[default5]:[rank13]:     return self._call_impl(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank15]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank14]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default2]:[rank10]:     return forward_call(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default4]:[rank12]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank10]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank9]:     return forward_call(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default4]:[rank12]:     sharded_logits = self.model(
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 565, in forward
[default7]:[rank15]:     sharded_logits = self.model(
[default2]:[rank10]:     return self._call_impl(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank14]:     key_value_states = key_value_states.permute(1, 2, 0, 3, 4).contiguous()
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank13]:     return forward_call(*args, **kwargs)
[default4]:[rank12]:     return self._call_impl(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank12]:     return forward_call(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank10]:     return forward_call(*args, **kwargs)
[default6]:[rank14]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 128.00 MiB. GPU  has a total capacity of 79.33 GiB of which 79.94 MiB is free. Including non-PyTorch memory, this process has 79.24 GiB memory in use. Of the allocated memory 69.33 GiB is allocated by PyTorch, and 61.28 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default7]:[rank15]:     return self._call_impl(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 565, in forward
[default2]:[rank10]:     key_value_states = key_value_states.permute(1, 2, 0, 3, 4).contiguous()
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default3]:[rank11]:     return self._call_impl(*args, **kwargs)
[default7]:[rank15]:     return forward_call(*args, **kwargs)
[default1]:[rank9]:     return row_linear(
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 565, in forward
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default7]:[rank15]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default4]:[rank12]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default2]:[rank10]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 128.00 MiB. GPU  has a total capacity of 79.33 GiB of which 79.94 MiB is free. Including non-PyTorch memory, this process has 79.24 GiB memory in use. Of the allocated memory 69.33 GiB is allocated by PyTorch, and 61.28 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default3]:[rank11]:     return forward_call(*args, **kwargs)
[default1]:[rank9]:     out = F.linear(input, weight, bias)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default5]:[rank13]:     key_value_states = key_value_states.permute(1, 2, 0, 3, 4).contiguous()
[default1]:[rank9]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1024.00 MiB. GPU 
 has a total capacity of 79.33 GiB of which 39.94 MiB is free. Including non-PyTorch memory, this process has 79.28 GiB memory in use. Of the allocated memory 69.46 GiB is allocated by PyTorch, and 59.27 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default3]:[rank11]:     output = self.o_proj(attention_output)
[default4]:[rank12]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default5]:[rank13]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 128.00 MiB. GPU  has a total capacity of 79.33 GiB of which 103.94 MiB is free. Including non-PyTorch memory, this process has 79.22 GiB memory in use. Of the allocated memory 69.33 GiB is allocated by PyTorch, and 125.28 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default7]:[rank15]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank11]:     return self._call_impl(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank12]:     return self._call_impl(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank15]:     return self._call_impl(*args, **kwargs)
[default3]:[rank11]:     return forward_call(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank15]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:     return row_linear(
[default4]:[rank12]:     return forward_call(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default7]:[rank15]:     output = self.pp_block(**new_kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default4]:[rank12]:     output = self.pp_block(**new_kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default3]:[rank11]:     out = F.linear(input, weight, bias)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank12]:     return self._call_impl(*args, **kwargs)
[default3]:[rank11]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1024.00 MiB. GPU  has a total capacity of 79.33 GiB of which 39.94 MiB is free. Including non-PyTorch memory, this process has 79.28 GiB memory in use. Of the allocated memory 69.46 GiB is allocated by PyTorch, and 59.27 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default7]:[rank15]:     return self._call_impl(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank12]:     return forward_call(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default7]:[rank15]:     return forward_call(*args, **kwargs)
[default4]:[rank12]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank12]:     return self._call_impl(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default7]:[rank15]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default4]:[rank12]:     return forward_call(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 565, in forward
[default4]:[rank12]:     key_value_states = key_value_states.permute(1, 2, 0, 3, 4).contiguous()
[default7]:[rank15]:     return self._call_impl(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank12]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 128.00 MiB. GPU  has a total capacity of 79.33 GiB of which 79.94 MiB is free. Including non-PyTorch memory, this process has 79.24 GiB memory in use. Of the allocated memory 69.33 GiB is allocated by PyTorch, and 61.28 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default7]:[rank15]:     return forward_call(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default7]:[rank15]:     output = self.o_proj(attention_output)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank15]:     return self._call_impl(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank15]:     return forward_call(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default7]:[rank15]:     return row_linear(
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default7]:[rank15]:     out = F.linear(input, weight, bias)
[default7]:[rank15]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1024.00 MiB. GPU  has a total capacity of 79.33 GiB of which 39.94 MiB is free. Including non-PyTorch memory, this process has 79.28 GiB memory in use. Of the allocated memory 69.46 GiB is allocated by PyTorch, and 59.27 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default0]:[rank8]: Traceback (most recent call last):
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default0]:[rank8]:     trainer.train(dataloader)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default0]:[rank8]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default0]:[rank8]:     outputs = self.pipeline_engine.train_batch_iter(
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default0]:[rank8]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default0]:[rank8]:     output = model(**micro_batch)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank8]:     return self._call_impl(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank8]:     return forward_call(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default0]:[rank8]:     sharded_logits = self.model(
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank8]:     return self._call_impl(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank8]:     return forward_call(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default0]:[rank8]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default0]:[rank8]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank8]:     return self._call_impl(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank8]:     return forward_call(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default0]:[rank8]:     output = self.pp_block(**new_kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank8]:     return self._call_impl(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank8]:     return forward_call(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default0]:[rank8]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank8]:     return self._call_impl(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank8]:     return forward_call(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 565, in forward
[default0]:[rank8]:     key_value_states = key_value_states.permute(1, 2, 0, 3, 4).contiguous()
[default0]:[rank8]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 128.00 MiB. GPU 
[default6]:[rank6]: Traceback (most recent call last):
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default6]:[rank6]:     trainer.train(dataloader)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default6]:[rank6]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default6]:[rank6]:     outputs = self.pipeline_engine.train_batch_iter(
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default6]:[rank6]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default6]:[rank6]:     output = model(**micro_batch)
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank6]:     return self._call_impl(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank6]:     return forward_call(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default6]:[rank6]:     sharded_logits = self.model(
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank6]:     return self._call_impl(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank6]:     return forward_call(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default6]:[rank6]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default6]:[rank6]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank6]:     return self._call_impl(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank6]:     return forward_call(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default6]:[rank6]:     output = self.pp_block(**new_kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank6]:     return self._call_impl(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank6]:     return forward_call(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default6]:[rank6]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank6]:     return self._call_impl(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank6]:     return forward_call(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default6]:[rank6]:     output = self.o_proj(attention_output)
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank6]:     return self._call_impl(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank6]:     return forward_call(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default6]:[rank6]:     return row_linear(
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default6]:[rank6]:     out = F.linear(input, weight, bias)
[default6]:[rank6]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1024.00 MiB. GPU  has a total capacity of 79.33 GiB of which 575.94 MiB is free. Including non-PyTorch memory, this process has 78.76 GiB memory in use. Of the allocated memory 69.46 GiB is allocated by PyTorch, and 123.27 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default7]:[rank7]: Traceback (most recent call last):
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default7]:[rank7]:     trainer.train(dataloader)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default7]:[rank7]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default7]:[rank7]:     outputs = self.pipeline_engine.train_batch_iter(
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default7]:[rank7]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default7]:[rank7]:     output = model(**micro_batch)
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank7]:     return self._call_impl(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank7]:     return forward_call(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default7]:[rank7]:     sharded_logits = self.model(
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank7]:     return self._call_impl(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank7]:     return forward_call(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default7]:[rank7]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default7]:[rank7]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank7]:     return self._call_impl(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank7]:     return forward_call(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default7]:[rank7]:     output = self.pp_block(**new_kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank7]:     return self._call_impl(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank7]:     return forward_call(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default7]:[rank7]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank7]:     return self._call_impl(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank7]:     return forward_call(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default7]:[rank7]:     output = self.o_proj(attention_output)
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank7]:     return self._call_impl(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank7]:     return forward_call(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default7]:[rank7]:     return row_linear(
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default7]:[rank7]:     out = F.linear(input, weight, bias)
[default7]:[rank7]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1024.00 MiB. GPU  has a total capacity of 79.33 GiB of which 487.94 MiB is free. Including non-PyTorch memory, this process has 78.84 GiB memory in use. Of the allocated memory 69.46 GiB is allocated by PyTorch, and 123.27 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default3]:[rank3]: Traceback (most recent call last):
[default2]:[rank2]: Traceback (most recent call last):
[default0]:[rank0]: Traceback (most recent call last):
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default3]:[rank3]:     trainer.train(dataloader)
[default1]:[rank1]: Traceback (most recent call last):
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default2]:[rank2]:     trainer.train(dataloader)
[default1]:[rank1]:     trainer.train(dataloader)
[default0]:[rank0]:     trainer.train(dataloader)
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default2]:[rank2]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default1]:[rank1]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default3]:[rank3]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default2]:[rank2]:     outputs = self.pipeline_engine.train_batch_iter(
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default3]:[rank3]:     outputs = self.pipeline_engine.train_batch_iter(
[default0]:[rank0]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default2]:[rank2]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default1]:[rank1]:     outputs = self.pipeline_engine.train_batch_iter(
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default0]:[rank0]:     outputs = self.pipeline_engine.train_batch_iter(
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default0]:[rank0]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default1]:[rank1]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default2]:[rank2]:     output = model(**micro_batch)
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank1]:     output = model(**micro_batch)
[default0]:[rank0]:     output = model(**micro_batch)
[default2]:[rank2]:     return self._call_impl(*args, **kwargs)
[default1]:[rank1]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank3]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank1]:     return self._call_impl(*args, **kwargs)
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default0]:[rank0]:     return self._call_impl(*args, **kwargs)
[default1]:[rank1]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank2]:     return forward_call(*args, **kwargs)
[default3]:[rank3]:     output = model(**micro_batch)
[default1]:[rank1]:     return forward_call(*args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank3]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default0]:[rank0]:     return forward_call(*args, **kwargs)
[default2]:[rank2]:     sharded_logits = self.model(
[default1]:[rank1]:     sharded_logits = self.model(
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default3]:[rank3]:     return self._call_impl(*args, **kwargs)
[default1]:[rank1]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank0]:     sharded_logits = self.model(
[default2]:[rank2]:     return self._call_impl(*args, **kwargs)
[default3]:[rank3]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank1]:     return self._call_impl(*args, **kwargs)
[default3]:[rank3]:     return forward_call(*args, **kwargs)
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default3]:[rank3]:     sharded_logits = self.model(
[default1]:[rank1]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank1]:     return forward_call(*args, **kwargs)
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default3]:[rank3]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank3]:     return self._call_impl(*args, **kwargs)
[default0]:[rank0]:     return self._call_impl(*args, **kwargs)
[default1]:[rank1]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default2]:[rank2]:     return forward_call(*args, **kwargs)
[default3]:[rank3]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank1]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default3]:[rank3]:     return forward_call(*args, **kwargs)
[default1]:[rank1]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank0]:     return forward_call(*args, **kwargs)
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default1]:[rank1]:     return self._call_impl(*args, **kwargs)
[default2]:[rank2]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default3]:[rank3]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default1]:[rank1]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default1]:[rank1]:     return forward_call(*args, **kwargs)
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default3]:[rank3]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default2]:[rank2]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default0]:[rank0]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default1]:[rank1]:     output = self.pp_block(**new_kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank2]:     return self._call_impl(*args, **kwargs)
[default1]:[rank1]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank1]:     return self._call_impl(*args, **kwargs)
[default3]:[rank3]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank2]:     return forward_call(*args, **kwargs)
[default1]:[rank1]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank0]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default1]:[rank1]:     return forward_call(*args, **kwargs)
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank2]:     output = self.pp_block(**new_kwargs)
[default3]:[rank3]:     return self._call_impl(*args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank0]:     return self._call_impl(*args, **kwargs)
[default3]:[rank3]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank2]:     return self._call_impl(*args, **kwargs)
[default3]:[rank3]:     return forward_call(*args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default2]:[rank2]:     return forward_call(*args, **kwargs)
[default1]:[rank1]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default1]:[rank1]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank0]:     return forward_call(*args, **kwargs)
[default3]:[rank3]:     output = self.pp_block(**new_kwargs)
[default1]:[rank1]:     return self._call_impl(*args, **kwargs)
[default3]:[rank3]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank1]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default3]:[rank3]:     return self._call_impl(*args, **kwargs)
[default1]:[rank1]:     return forward_call(*args, **kwargs)
[default2]:[rank2]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default1]:[rank1]:     output = self.o_proj(attention_output)
[default0]:[rank0]:     output = self.pp_block(**new_kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank1]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank2]:     return self._call_impl(*args, **kwargs)
[default1]:[rank1]:     return self._call_impl(*args, **kwargs)
[default0]:[rank0]:     return self._call_impl(*args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank2]:     return forward_call(*args, **kwargs)
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank3]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default0]:[rank0]:     return forward_call(*args, **kwargs)
[default3]:[rank3]:     return forward_call(*args, **kwargs)
[default1]:[rank1]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default2]:[rank2]:     output = self.o_proj(attention_output)
[default1]:[rank1]:     return forward_call(*args, **kwargs)
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default0]:[rank0]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default3]:[rank3]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank0]:     return self._call_impl(*args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank3]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank1]:     return row_linear(
[default3]:[rank3]:     return self._call_impl(*args, **kwargs)
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank3]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank1]:     out = F.linear(input, weight, bias)
[default0]:[rank0]:     return forward_call(*args, **kwargs)
[default3]:[rank3]:     return forward_call(*args, **kwargs)
[default1]:[rank1]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1024.00 MiB. GPU 
 has a total capacity of 79.33 GiB of which 487.94 MiB is free. Including non-PyTorch memory, this process has 78.84 GiB memory in use. Of the allocated memory 69.46 GiB is allocated by PyTorch, and 123.27 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default2]:[rank2]:     return self._call_impl(*args, **kwargs)
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank3]:     output = self.o_proj(attention_output)
[default0]:[rank0]:     output = self.o_proj(attention_output)
[default2]:[rank2]:     return forward_call(*args, **kwargs)
[default3]:[rank3]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank3]:     return self._call_impl(*args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default3]:[rank3]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank2]:     return row_linear(
[default3]:[rank3]:     return forward_call(*args, **kwargs)
[default0]:[rank0]:     return self._call_impl(*args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default2]:[rank2]:     out = F.linear(input, weight, bias)
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank3]:     return row_linear(
[default0]:[rank0]:     return forward_call(*args, **kwargs)
[default2]:[rank2]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1024.00 MiB. GPU  has a total capacity of 79.33 GiB of which 575.94 MiB is free. Including non-PyTorch memory, this process has 78.76 GiB memory in use. Of the allocated memory 69.46 GiB is allocated by PyTorch, and 123.27 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default0]:[rank0]:     return row_linear(
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default0]:[rank0]:     out = F.linear(input, weight, bias)
[default0]:[rank0]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1024.00 MiB. GPU 
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default3]:[rank3]:     out = F.linear(input, weight, bias)
[default3]:[rank3]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1024.00 MiB. GPU  has a total capacity of 79.33 GiB of which 487.94 MiB is free. Including non-PyTorch memory, this process has 78.84 GiB memory in use. Of the allocated memory 69.46 GiB is allocated by PyTorch, and 123.27 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default5]:[rank5]: Traceback (most recent call last):
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default5]:[rank5]:     trainer.train(dataloader)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default5]:[rank5]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default5]:[rank5]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default5]:[rank5]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default5]:[rank5]:     output = model(**micro_batch)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank5]:     return self._call_impl(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank5]:     return forward_call(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default5]:[rank5]:     sharded_logits = self.model(
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank5]:     return self._call_impl(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank5]:     return forward_call(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default5]:[rank5]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default5]:[rank5]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank5]:     return self._call_impl(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank5]:     return forward_call(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default5]:[rank5]:     output = self.pp_block(**new_kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank5]:     return self._call_impl(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank5]:     return forward_call(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default5]:[rank5]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank5]:     return self._call_impl(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank5]:     return forward_call(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default5]:[rank5]:     output = self.o_proj(attention_output)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank5]:     return self._call_impl(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank5]:     return forward_call(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default5]:[rank5]:     return row_linear(
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default5]:[rank5]:     out = F.linear(input, weight, bias)
[default5]:[rank5]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1024.00 MiB. GPU  has a total capacity of 79.33 GiB of which 487.94 MiB is free. Including non-PyTorch memory, this process has 78.84 GiB memory in use. Of the allocated memory 69.46 GiB is allocated by PyTorch, and 123.27 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default4]:[rank4]: Traceback (most recent call last):
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default4]:[rank4]:     trainer.train(dataloader)
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default4]:[rank4]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default4]:[rank4]:     outputs = self.pipeline_engine.train_batch_iter(
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default4]:[rank4]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default4]:[rank4]:     output = model(**micro_batch)
[default4]:[rank4]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank4]:     return self._call_impl(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank4]:     return forward_call(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default4]:[rank4]:     sharded_logits = self.model(
[default4]:[rank4]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank4]:     return self._call_impl(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank4]:     return forward_call(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default4]:[rank4]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default4]:[rank4]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default4]:[rank4]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank4]:     return self._call_impl(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank4]:     return forward_call(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default4]:[rank4]:     output = self.pp_block(**new_kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank4]:     return self._call_impl(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank4]:     return forward_call(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default4]:[rank4]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default4]:[rank4]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank4]:     return self._call_impl(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank4]:     return forward_call(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default4]:[rank4]:     output = self.o_proj(attention_output)
[default4]:[rank4]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank4]:     return self._call_impl(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank4]:     return forward_call(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default4]:[rank4]:     return row_linear(
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default4]:[rank4]:     out = F.linear(input, weight, bias)
[default4]:[rank4]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1024.00 MiB. GPU  has a total capacity of 79.33 GiB of which 575.94 MiB is free. Including non-PyTorch memory, this process has 78.76 GiB memory in use. Of the allocated memory 69.46 GiB is allocated by PyTorch, and 123.27 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default5]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
[default5]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[default4]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
[default4]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[default3]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
[default3]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[default7]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
[default7]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[default0]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
[default1]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
[default0]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[default1]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[default4]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
[default4]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[default7]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
[default7]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[default2]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
[default2]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[default0]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
[default0]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[default1]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
[default1]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[default3]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
[default3]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[default6]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
[default6]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[default5]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
[default5]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[default6]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
[default6]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[default2]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
[default2]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
E0703 02:12:44.911000 140513055242048 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 511973) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10
E0703 02:12:44.913000 140515281438528 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 1125019) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-07-03_02:12:44
  host      : ip-26-0-160-192.ec2.internal
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 1125020)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2024-07-03_02:12:44
  host      : ip-26-0-160-192.ec2.internal
  rank      : 2 (local_rank: 2)
  exitcode  : 1 (pid: 1125021)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
  time      : 2024-07-03_02:12:44
  host      : ip-26-0-160-192.ec2.internal
  rank      : 3 (local_rank: 3)
  exitcode  : 1 (pid: 1125022)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[4]:
  time      : 2024-07-03_02:12:44
  host      : ip-26-0-160-192.ec2.internal
  rank      : 4 (local_rank: 4)
  exitcode  : 1 (pid: 1125023)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[5]:
  time      : 2024-07-03_02:12:44
  host      : ip-26-0-160-192.ec2.internal
  rank      : 5 (local_rank: 5)
  exitcode  : 1 (pid: 1125024)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[6]:
  time      : 2024-07-03_02:12:44
  host      : ip-26-0-160-192.ec2.internal
  rank      : 6 (local_rank: 6)
  exitcode  : 1 (pid: 1125025)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[7]:
  time      : 2024-07-03_02:12:44
  host      : ip-26-0-160-192.ec2.internal
  rank      : 7 (local_rank: 7)
  exitcode  : 1 (pid: 1125026)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-03_02:12:44
  host      : ip-26-0-160-192.ec2.internal
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 1125019)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-07-03_02:12:44
  host      : ip-26-0-161-178.ec2.internal
  rank      : 9 (local_rank: 1)
  exitcode  : 1 (pid: 511974)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2024-07-03_02:12:44
  host      : ip-26-0-161-178.ec2.internal
  rank      : 10 (local_rank: 2)
  exitcode  : 1 (pid: 511975)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
  time      : 2024-07-03_02:12:44
  host      : ip-26-0-161-178.ec2.internal
  rank      : 11 (local_rank: 3)
  exitcode  : 1 (pid: 511976)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[4]:
  time      : 2024-07-03_02:12:44
  host      : ip-26-0-161-178.ec2.internal
  rank      : 12 (local_rank: 4)
  exitcode  : 1 (pid: 511977)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[5]:
  time      : 2024-07-03_02:12:44
  host      : ip-26-0-161-178.ec2.internal
  rank      : 13 (local_rank: 5)
  exitcode  : 1 (pid: 511978)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[6]:
  time      : 2024-07-03_02:12:44
  host      : ip-26-0-161-178.ec2.internal
  rank      : 14 (local_rank: 6)
  exitcode  : 1 (pid: 511979)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[7]:
  time      : 2024-07-03_02:12:44
  host      : ip-26-0-161-178.ec2.internal
  rank      : 15 (local_rank: 7)
  exitcode  : 1 (pid: 511980)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-03_02:12:44
  host      : ip-26-0-161-178.ec2.internal
  rank      : 8 (local_rank: 0)
  exitcode  : 1 (pid: 511973)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
srun: error: ip-26-0-161-178: task 1: Exited with exit code 1
srun: error: ip-26-0-160-192: task 0: Exited with exit code 1
W0703 02:12:48.783000 140020071294720 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-163-220.ec2.internal_761797_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 02:12:49.549000 140082847885056 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-172-57.ec2.internal_1048111_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 02:12:49.699000 140349412566784 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-169-86.ec2.internal_1822398_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 02:12:49.702000 140176231663360 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-172-73.ec2.internal_890715_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 02:12:49.728000 139772476024576 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-168-238.ec2.internal_1848651_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 02:12:49.747000 140466012100352 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-163-226.ec2.internal_3207894_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 02:12:49.797000 140471672833856 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3207965 closing signal SIGTERM
W0703 02:12:49.795000 140025732028224 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 761868 closing signal SIGTERM
W0703 02:12:49.797000 140471672833856 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3207966 closing signal SIGTERM
W0703 02:12:49.795000 140025732028224 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 761869 closing signal SIGTERM
W0703 02:12:49.797000 140471672833856 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3207967 closing signal SIGTERM
W0703 02:12:49.795000 140025732028224 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 761870 closing signal SIGTERM
W0703 02:12:49.797000 140471672833856 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3207968 closing signal SIGTERM
W0703 02:12:49.795000 140025732028224 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 761871 closing signal SIGTERM
W0703 02:12:49.797000 140025732028224 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 761872 closing signal SIGTERM
W0703 02:12:49.798000 140471672833856 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3207969 closing signal SIGTERM
W0703 02:12:49.799000 140088508618560 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1048182 closing signal SIGTERM
W0703 02:12:49.797000 140025732028224 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 761873 closing signal SIGTERM
W0703 02:12:49.799000 140088508618560 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1048183 closing signal SIGTERM
W0703 02:12:49.797000 140025732028224 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 761874 closing signal SIGTERM
W0703 02:12:49.799000 140088508618560 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1048184 closing signal SIGTERM
W0703 02:12:49.799000 140088508618560 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1048185 closing signal SIGTERM
W0703 02:12:49.798000 140025732028224 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 761875 closing signal SIGTERM
W0703 02:12:49.799000 140181892396864 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 890786 closing signal SIGTERM
W0703 02:12:49.801000 140088508618560 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1048186 closing signal SIGTERM
W0703 02:12:49.799000 140181892396864 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 890787 closing signal SIGTERM
W0703 02:12:49.800000 139778136758080 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1848721 closing signal SIGTERM
W0703 02:12:49.801000 139778136758080 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1848722 closing signal SIGTERM
W0703 02:12:49.801000 139778136758080 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1848723 closing signal SIGTERM
W0703 02:12:49.802000 140471672833856 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3207970 closing signal SIGTERM
W0703 02:12:49.799000 140181892396864 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 890788 closing signal SIGTERM
W0703 02:12:49.800000 140181892396864 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 890789 closing signal SIGTERM
W0703 02:12:49.801000 140181892396864 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 890790 closing signal SIGTERM
W0703 02:12:49.801000 139778136758080 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1848724 closing signal SIGTERM
W0703 02:12:49.801000 140181892396864 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 890791 closing signal SIGTERM
W0703 02:12:49.804000 140471672833856 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3207971 closing signal SIGTERM
W0703 02:12:49.804000 140471672833856 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3207972 closing signal SIGTERM
W0703 02:12:49.802000 140181892396864 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 890792 closing signal SIGTERM
W0703 02:12:49.804000 140088508618560 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1048187 closing signal SIGTERM
W0703 02:12:49.804000 140088508618560 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1048188 closing signal SIGTERM
W0703 02:12:49.803000 140181892396864 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 890793 closing signal SIGTERM
W0703 02:12:49.803000 140355073300288 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1822468 closing signal SIGTERM
W0703 02:12:49.803000 140355073300288 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1822469 closing signal SIGTERM
W0703 02:12:49.805000 140088508618560 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1048189 closing signal SIGTERM
W0703 02:12:49.803000 140355073300288 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1822470 closing signal SIGTERM
W0703 02:12:49.805000 139778136758080 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1848725 closing signal SIGTERM
W0703 02:12:49.804000 140355073300288 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1822471 closing signal SIGTERM
W0703 02:12:49.806000 139778136758080 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1848726 closing signal SIGTERM
W0703 02:12:49.806000 139778136758080 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1848727 closing signal SIGTERM
W0703 02:12:49.806000 139778136758080 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1848728 closing signal SIGTERM
W0703 02:12:49.806000 140355073300288 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1822472 closing signal SIGTERM
W0703 02:12:49.807000 140355073300288 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1822473 closing signal SIGTERM
W0703 02:12:49.807000 140355073300288 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1822474 closing signal SIGTERM
W0703 02:12:49.808000 140355073300288 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1822475 closing signal SIGTERM
W0703 02:12:53.787000 140020071294720 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-163-220.ec2.internal_761797_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 02:12:54.554000 140082847885056 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-172-57.ec2.internal_1048111_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 02:12:54.704000 140349412566784 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-169-86.ec2.internal_1822398_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 02:12:54.707000 140176231663360 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-172-73.ec2.internal_890715_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 02:12:54.732000 139772476024576 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-168-238.ec2.internal_1848651_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 02:12:54.752000 140466012100352 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-163-226.ec2.internal_3207894_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 02:12:58.791000 140020071294720 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-163-220.ec2.internal_761797_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 02:12:58.935000 140181892396864 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-172-73.ec2.internal_890715_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 02:12:58.946000 140181892396864 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-172-73.ec2.internal_890715_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 113, in _call_store
    return getattr(self._store, store_op)(*args, **kwargs)
torch.distributed.DistNetworkError: Broken pipe

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 254, in launch_agent
    result = agent.run()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 123, in wrapper
    result = f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 733, in run
    result = self._invoke_run(role)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 908, in _invoke_run
    num_nodes_waiting = rdzv_handler.num_nodes_waiting()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 1174, in num_nodes_waiting
    self._state_holder.sync()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 419, in sync
    get_response = self._backend.get_state()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 73, in get_state
    base64_state: bytes = self._call_store("get", self._key)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 115, in _call_store
    raise RendezvousConnectionError(
torch.distributed.elastic.rendezvous.api.RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details.
srun: error: ip-26-0-172-73: task 7: Exited with exit code 1
W0703 02:12:59.558000 140082847885056 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-172-57.ec2.internal_1048111_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 02:12:59.708000 140349412566784 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-169-86.ec2.internal_1822398_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 02:12:59.738000 139772476024576 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-168-238.ec2.internal_1848651_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 02:12:59.756000 140466012100352 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-163-226.ec2.internal_3207894_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 02:13:02.744000 140088508618560 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-172-57.ec2.internal_1048111_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 02:13:02.754000 140088508618560 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-172-57.ec2.internal_1048111_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 113, in _call_store
    return getattr(self._store, store_op)(*args, **kwargs)
torch.distributed.DistNetworkError: Broken pipe

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 254, in launch_agent
    result = agent.run()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 123, in wrapper
    result = f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 733, in run
    result = self._invoke_run(role)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 908, in _invoke_run
    num_nodes_waiting = rdzv_handler.num_nodes_waiting()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 1174, in num_nodes_waiting
    self._state_holder.sync()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 419, in sync
    get_response = self._backend.get_state()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 73, in get_state
    base64_state: bytes = self._call_store("get", self._key)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 115, in _call_store
    raise RendezvousConnectionError(
torch.distributed.elastic.rendezvous.api.RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details.
srun: error: ip-26-0-172-57: task 6: Exited with exit code 1
W0703 02:13:03.239000 140025732028224 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-163-220.ec2.internal_761797_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 02:13:03.252000 140025732028224 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-163-220.ec2.internal_761797_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 113, in _call_store
    return getattr(self._store, store_op)(*args, **kwargs)
torch.distributed.DistNetworkError: Broken pipe

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 254, in launch_agent
    result = agent.run()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 123, in wrapper
    result = f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 733, in run
    result = self._invoke_run(role)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 908, in _invoke_run
    num_nodes_waiting = rdzv_handler.num_nodes_waiting()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 1174, in num_nodes_waiting
    self._state_holder.sync()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 419, in sync
    get_response = self._backend.get_state()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 73, in get_state
    base64_state: bytes = self._call_store("get", self._key)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 115, in _call_store
    raise RendezvousConnectionError(
torch.distributed.elastic.rendezvous.api.RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details.
srun: error: ip-26-0-163-220: task 2: Exited with exit code 1
W0703 02:13:03.645000 140355073300288 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-169-86.ec2.internal_1822398_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 02:13:03.656000 140355073300288 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-169-86.ec2.internal_1822398_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 113, in _call_store
    return getattr(self._store, store_op)(*args, **kwargs)
torch.distributed.DistNetworkError: Broken pipe

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 254, in launch_agent
    result = agent.run()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 123, in wrapper
    result = f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 733, in run
    result = self._invoke_run(role)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 908, in _invoke_run
    num_nodes_waiting = rdzv_handler.num_nodes_waiting()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 1174, in num_nodes_waiting
    self._state_holder.sync()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 419, in sync
    get_response = self._backend.get_state()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 73, in get_state
    base64_state: bytes = self._call_store("get", self._key)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 115, in _call_store
    raise RendezvousConnectionError(
torch.distributed.elastic.rendezvous.api.RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details.
W0703 02:13:03.944000 139778136758080 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-168-238.ec2.internal_1848651_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 02:13:03.957000 139778136758080 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-168-238.ec2.internal_1848651_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 113, in _call_store
    return getattr(self._store, store_op)(*args, **kwargs)
torch.distributed.DistNetworkError: Broken pipe

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 254, in launch_agent
    result = agent.run()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 123, in wrapper
    result = f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 733, in run
    result = self._invoke_run(role)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 908, in _invoke_run
    num_nodes_waiting = rdzv_handler.num_nodes_waiting()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 1174, in num_nodes_waiting
    self._state_holder.sync()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 419, in sync
    get_response = self._backend.get_state()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 73, in get_state
    base64_state: bytes = self._call_store("get", self._key)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 115, in _call_store
    raise RendezvousConnectionError(
torch.distributed.elastic.rendezvous.api.RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details.
srun: error: ip-26-0-169-86: task 5: Exited with exit code 1
W0703 02:13:04.139000 140471672833856 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-163-226.ec2.internal_3207894_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 02:13:04.150000 140471672833856 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-163-226.ec2.internal_3207894_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 113, in _call_store
    return getattr(self._store, store_op)(*args, **kwargs)
torch.distributed.DistNetworkError: Broken pipe

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 254, in launch_agent
    result = agent.run()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 123, in wrapper
    result = f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 733, in run
    result = self._invoke_run(role)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 908, in _invoke_run
    num_nodes_waiting = rdzv_handler.num_nodes_waiting()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 1174, in num_nodes_waiting
    self._state_holder.sync()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 419, in sync
    get_response = self._backend.get_state()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 73, in get_state
    base64_state: bytes = self._call_store("get", self._key)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 115, in _call_store
    raise RendezvousConnectionError(
torch.distributed.elastic.rendezvous.api.RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details.
srun: error: ip-26-0-168-238: task 4: Exited with exit code 1
srun: error: ip-26-0-163-226: task 3: Exited with exit code 1
Consider using `hf_transfer` for faster uploads. This solution comes with some limitations. See https://huggingface.co/docs/huggingface_hub/hf_transfer for more details.