========================
START TIME: Wed Jul  3 00:12:51 UTC 2024
python3 version = Python 3.10.14
========================
The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well.
Token is valid (permission: write).
Your token has been saved to /admin/home/ferdinand_mom/.cache/huggingface/token
Login successful
Already on 'bench_cluster'
M	examples/config_tiny_llama.py
M	examples/config_tiny_llama.yaml
M	examples/train_tiny_llama.sh
M	src/nanotron/models/llama.py
M	src/nanotron/trainer.py
Your branch is up to date with 'origin/bench_cluster'.
Job status: RUNNING
W0703 00:12:56.920000 140445265241920 torch/distributed/run.py:757] 
W0703 00:12:56.920000 140445265241920 torch/distributed/run.py:757] *****************************************
W0703 00:12:56.920000 140445265241920 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0703 00:12:56.920000 140445265241920 torch/distributed/run.py:757] *****************************************
W0703 00:12:56.921000 139985663354688 torch/distributed/run.py:757] 
W0703 00:12:56.921000 139985663354688 torch/distributed/run.py:757] *****************************************
W0703 00:12:56.921000 139985663354688 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0703 00:12:56.921000 139985663354688 torch/distributed/run.py:757] *****************************************
W0703 00:12:56.921000 139905265825600 torch/distributed/run.py:757] 
W0703 00:12:56.921000 139905265825600 torch/distributed/run.py:757] *****************************************
W0703 00:12:56.921000 139905265825600 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0703 00:12:56.921000 139905265825600 torch/distributed/run.py:757] *****************************************
W0703 00:12:56.922000 140507061364544 torch/distributed/run.py:757] 
W0703 00:12:56.922000 140507061364544 torch/distributed/run.py:757] *****************************************
W0703 00:12:56.922000 140507061364544 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0703 00:12:56.922000 140507061364544 torch/distributed/run.py:757] *****************************************
W0703 00:12:57.013000 140629515450176 torch/distributed/run.py:757] 
W0703 00:12:57.013000 140629515450176 torch/distributed/run.py:757] *****************************************
W0703 00:12:57.013000 140629515450176 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0703 00:12:57.013000 140629515450176 torch/distributed/run.py:757] *****************************************
W0703 00:12:57.051000 139849737688896 torch/distributed/run.py:757] 
W0703 00:12:57.051000 139849737688896 torch/distributed/run.py:757] *****************************************
W0703 00:12:57.051000 139849737688896 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0703 00:12:57.051000 139849737688896 torch/distributed/run.py:757] *****************************************
W0703 00:12:57.259000 139716723873600 torch/distributed/run.py:757] 
W0703 00:12:57.259000 139716723873600 torch/distributed/run.py:757] *****************************************
W0703 00:12:57.259000 139716723873600 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0703 00:12:57.259000 139716723873600 torch/distributed/run.py:757] *****************************************
W0703 00:12:57.264000 140671516706624 torch/distributed/run.py:757] 
W0703 00:12:57.264000 140671516706624 torch/distributed/run.py:757] *****************************************
W0703 00:12:57.264000 140671516706624 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0703 00:12:57.264000 140671516706624 torch/distributed/run.py:757] *****************************************
[default0]:07/03/2024 00:13:23 [WARNING|DP=0|PP=0|TP=0|ip-26-0-160-192]: [Vocab Size Padding] Padded vocab (size: 50257) with 15 dummy tokens (new size: 50272)
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Config:
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Config(general=GeneralArgs(project='bench_cluster',
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                            run='%date_%jobid',
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                            seed=42,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                            step=None,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                            consumed_train_samples=None,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                            benchmark_csv_path=None,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                            ignore_sanity_checks=True),
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        parallelism=ParallelismArgs(dp=2,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    pp=2,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    tp=16,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    pp_engine=<nanotron.parallel.pipeline_parallel.engine.OneForwardOneBackwardPipelineEngine object at 0x7f8d20d84940>,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    tp_mode=<TensorParallelLinearMode.REDUCE_SCATTER: 2>,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    tp_linear_async_communication=False,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    expert_parallel_size=1),
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        model=ModelArgs(model_config=LlamaConfig(bos_token_id=1,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 eos_token_id=2,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 hidden_act='silu',
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 hidden_size=2048,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 initializer_range=0.02,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 intermediate_size=4096,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 is_llama_config=True,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 max_position_embeddings=4096,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 num_attention_heads=32,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 num_hidden_layers=24,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 num_key_value_heads=32,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 pad_token_id=None,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 pretraining_tp=1,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 rms_norm_eps=1e-05,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 rope_scaling=None,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 rope_theta=10000.0,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 tie_word_embeddings=True,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 use_cache=True,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                 vocab_size=50272),
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                        init_method=RandomInit(std=0.025),
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                        dtype=torch.bfloat16,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                        make_vocab_size_divisible_by=1,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                        ddp_bucket_cap_mb=25),
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        tokenizer=TokenizerArgs(tokenizer_name_or_path='openai-community/gpt2',
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                tokenizer_revision=None,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                tokenizer_max_length=None),
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        checkpoints=CheckpointsArgs(checkpoints_path=Path('/dev/null'),
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    checkpoint_interval=100000,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    save_initial_state=False,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    resume_checkpoint_path=None,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                    checkpoints_path_is_shared_file_system=False),
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        logging=LoggingArgs(log_level='info',
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                            log_level_replica='info',
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                            iteration_step_info_interval=1),
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        tokens=TokensArgs(sequence_length=4096,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                          train_steps=20,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                          micro_batch_size=128,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                          batch_accumulation_per_replica=4,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                          val_check_interval=-1,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                          limit_val_batches=0,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                          limit_test_batches=0),
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        optimizer=OptimizerArgs(optimizer_factory=AdamWOptimizerArgs(adam_eps=1e-08,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                     adam_beta1=0.9,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                     adam_beta2=0.95,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                     torch_adam_is_fused=True,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                     name='adamW'),
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                zero_stage=1,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                weight_decay=0.01,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                clip_grad=1.0,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                accumulate_grad_in_fp32=True,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                learning_rate_scheduler=LRSchedulerArgs(learning_rate=0.0001,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                        lr_warmup_steps=1,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                        lr_warmup_style='linear',
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                        lr_decay_style='linear',
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                        lr_decay_steps=19,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                        lr_decay_starting_step=None,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                        min_decay_lr=1e-05)),
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        data_stages=[DatasetStageArgs(name='Training Stage',
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                      start_training_step=1,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                      data=DataArgs(dataset=PretrainDatasetsArgs(hf_dataset_or_datasets='roneneldan/TinyStories',
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                                 hf_dataset_splits='train',
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                                 hf_dataset_config_name=None,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                                 dataset_processing_num_proc_per_process=64,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                                 dataset_overwrite_cache=False,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                                                 text_column_name='text'),
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                    seed=42,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:                                                    num_loading_workers=0))],
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        profiler=ProfilerArgs(profiler_export_path=Path('/fsx/ferdinandmom/ferdinand-hf/bench_cluster/results/llama-1B/64_GPUS/dp-2_tp-16_pp-2_mbz-128')),
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:        lighteval=None)
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Model Config:
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: LlamaConfig(bos_token_id=1,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             eos_token_id=2,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             hidden_act='silu',
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             hidden_size=2048,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             initializer_range=0.02,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             intermediate_size=4096,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             is_llama_config=True,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             max_position_embeddings=4096,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             num_attention_heads=32,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             num_hidden_layers=24,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             num_key_value_heads=32,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             pad_token_id=None,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             pretraining_tp=1,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             rms_norm_eps=1e-05,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             rope_scaling=None,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             rope_theta=10000.0,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             tie_word_embeddings=True,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             use_cache=True,
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:             vocab_size=50272)
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Building model..
[default0]:07/03/2024 00:13:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Setting PP block ranks...
[default3]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=3|ip-26-0-168-238]: Local number of parameters: 32.7M (62.36MiB)
[default3]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=3|ip-26-0-168-238]: [After model building] Memory usage: 73.37MiB. Peak allocated: 75.40MiB Peak reserved: 82.00MiB
[default3]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=3|ip-26-0-168-238]: No checkpoint path provided.
[default1]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=1|ip-26-0-168-238]: Local number of parameters: 32.7M (62.36MiB)
[default1]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=1|ip-26-0-168-238]: [After model building] Memory usage: 73.37MiB. Peak allocated: 75.40MiB Peak reserved: 82.00MiB
[default1]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=1|ip-26-0-168-238]: No checkpoint path provided.
[default6]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=6|ip-26-0-168-238]: Local number of parameters: 32.7M (62.36MiB)
[default6]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=6|ip-26-0-168-238]: [After model building] Memory usage: 73.37MiB. Peak allocated: 75.40MiB Peak reserved: 82.00MiB
[default6]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=6|ip-26-0-168-238]: No checkpoint path provided.
[default2]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=2|ip-26-0-168-238]: Local number of parameters: 32.7M (62.36MiB)
[default2]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=2|ip-26-0-168-238]: [After model building] Memory usage: 73.37MiB. Peak allocated: 75.40MiB Peak reserved: 82.00MiB
[default2]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=2|ip-26-0-168-238]: No checkpoint path provided.
[default4]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=4|ip-26-0-168-238]: Local number of parameters: 32.7M (62.36MiB)
[default4]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=4|ip-26-0-168-238]: [After model building] Memory usage: 73.37MiB. Peak allocated: 75.40MiB Peak reserved: 82.00MiB
[default4]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=4|ip-26-0-168-238]: No checkpoint path provided.
[default5]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=5|ip-26-0-168-238]: Local number of parameters: 32.7M (62.36MiB)
[default5]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=5|ip-26-0-168-238]: [After model building] Memory usage: 73.37MiB. Peak allocated: 75.40MiB Peak reserved: 82.00MiB
[default5]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=5|ip-26-0-168-238]: No checkpoint path provided.
[default3]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=11|ip-26-0-161-178]: Local number of parameters: 43.2M (82.38MiB)
[default3]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=11|ip-26-0-161-178]: [After model building] Memory usage: 98.13MiB. Peak allocated: 100.16MiB Peak reserved: 112.00MiB
[default3]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=11|ip-26-0-161-178]: No checkpoint path provided.
[default4]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=12|ip-26-0-161-178]: Local number of parameters: 43.2M (82.38MiB)
[default4]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=12|ip-26-0-161-178]: [After model building] Memory usage: 98.13MiB. Peak allocated: 100.16MiB Peak reserved: 112.00MiB
[default4]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=12|ip-26-0-161-178]: No checkpoint path provided.
[default2]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=10|ip-26-0-161-178]: Local number of parameters: 43.2M (82.38MiB)
[default2]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=10|ip-26-0-161-178]: [After model building] Memory usage: 98.13MiB. Peak allocated: 100.16MiB Peak reserved: 112.00MiB
[default2]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=10|ip-26-0-161-178]: No checkpoint path provided.
[default7]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=15|ip-26-0-161-178]: Local number of parameters: 43.2M (82.38MiB)
[default7]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=15|ip-26-0-161-178]: [After model building] Memory usage: 98.13MiB. Peak allocated: 100.16MiB Peak reserved: 112.00MiB
[default7]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=15|ip-26-0-161-178]: No checkpoint path provided.
[default0]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=0|ip-26-0-168-238]: Local number of parameters: 32.7M (62.36MiB)
[default0]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=0|ip-26-0-168-238]: [After model building] Memory usage: 73.37MiB. Peak allocated: 75.40MiB Peak reserved: 82.00MiB
[default0]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=0|ip-26-0-168-238]: No checkpoint path provided.
[default1]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=9|ip-26-0-161-178]: Local number of parameters: 43.2M (82.38MiB)
[default1]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=9|ip-26-0-161-178]: [After model building] Memory usage: 98.13MiB. Peak allocated: 100.16MiB Peak reserved: 112.00MiB
[default1]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=9|ip-26-0-161-178]: No checkpoint path provided.
[default0]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=8|ip-26-0-161-178]: Local number of parameters: 43.2M (82.38MiB)
[default0]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=8|ip-26-0-161-178]: [After model building] Memory usage: 98.13MiB. Peak allocated: 100.16MiB Peak reserved: 112.00MiB
[default0]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=8|ip-26-0-161-178]: No checkpoint path provided.
[default6]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=14|ip-26-0-161-178]: Local number of parameters: 43.2M (82.38MiB)
[default6]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=14|ip-26-0-161-178]: [After model building] Memory usage: 98.13MiB. Peak allocated: 100.16MiB Peak reserved: 112.00MiB
[default6]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=14|ip-26-0-161-178]: No checkpoint path provided.
[default5]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=13|ip-26-0-161-178]: Local number of parameters: 43.2M (82.38MiB)
[default5]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=13|ip-26-0-161-178]: [After model building] Memory usage: 98.13MiB. Peak allocated: 100.16MiB Peak reserved: 112.00MiB
[default5]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=13|ip-26-0-161-178]: No checkpoint path provided.
[default1]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=1|ip-26-0-160-192]: Local number of parameters: 43.2M (82.38MiB)
[default1]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=1|ip-26-0-160-192]: [After model building] Memory usage: 98.13MiB. Peak allocated: 100.16MiB Peak reserved: 112.00MiB
[default1]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=1|ip-26-0-160-192]: No checkpoint path provided.
[default7]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=7|ip-26-0-160-192]: Local number of parameters: 43.2M (82.38MiB)
[default7]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=7|ip-26-0-160-192]: [After model building] Memory usage: 98.13MiB. Peak allocated: 100.16MiB Peak reserved: 112.00MiB
[default7]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=7|ip-26-0-160-192]: No checkpoint path provided.
[default0]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Total number of parameters: 1.21G (2315.81MiB)
[default0]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Local number of parameters: 43.2M (82.38MiB)
[default0]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [After model building] Memory usage: 98.13MiB. Peak allocated: 100.16MiB Peak reserved: 112.00MiB
[default0]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: No checkpoint path provided.
[default0]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Parametrizing model parameters using StandardParametrizator
[default4]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=4|ip-26-0-160-192]: Local number of parameters: 43.2M (82.38MiB)
[default4]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=4|ip-26-0-160-192]: [After model building] Memory usage: 98.13MiB. Peak allocated: 100.16MiB Peak reserved: 112.00MiB
[default4]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=4|ip-26-0-160-192]: No checkpoint path provided.
[default2]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=2|ip-26-0-160-192]: Local number of parameters: 43.2M (82.38MiB)
[default2]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=2|ip-26-0-160-192]: [After model building] Memory usage: 98.13MiB. Peak allocated: 100.16MiB Peak reserved: 112.00MiB
[default2]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=2|ip-26-0-160-192]: No checkpoint path provided.
[default3]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=3|ip-26-0-160-192]: Local number of parameters: 43.2M (82.38MiB)
[default3]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=3|ip-26-0-160-192]: [After model building] Memory usage: 98.13MiB. Peak allocated: 100.16MiB Peak reserved: 112.00MiB
[default3]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=3|ip-26-0-160-192]: No checkpoint path provided.
[default6]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=6|ip-26-0-160-192]: Local number of parameters: 43.2M (82.38MiB)
[default6]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=6|ip-26-0-160-192]: [After model building] Memory usage: 98.13MiB. Peak allocated: 100.16MiB Peak reserved: 112.00MiB
[default6]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=6|ip-26-0-160-192]: No checkpoint path provided.
[default7]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=7|ip-26-0-168-238]: Local number of parameters: 32.7M (62.36MiB)
[default7]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=7|ip-26-0-168-238]: [After model building] Memory usage: 73.37MiB. Peak allocated: 75.40MiB Peak reserved: 82.00MiB
[default7]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=7|ip-26-0-168-238]: No checkpoint path provided.
[default0]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=8|ip-26-0-169-86]: Local number of parameters: 32.7M (62.36MiB)
[default0]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=8|ip-26-0-169-86]: [After model building] Memory usage: 73.37MiB. Peak allocated: 75.40MiB Peak reserved: 82.00MiB
[default0]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=8|ip-26-0-169-86]: No checkpoint path provided.
[default5]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=5|ip-26-0-160-192]: Local number of parameters: 43.2M (82.38MiB)
[default5]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=5|ip-26-0-160-192]: [After model building] Memory usage: 98.13MiB. Peak allocated: 100.16MiB Peak reserved: 112.00MiB
[default5]:07/03/2024 00:13:41 [INFO|DP=0|PP=0|TP=5|ip-26-0-160-192]: No checkpoint path provided.
[default7]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=15|ip-26-0-169-86]: Local number of parameters: 32.7M (62.36MiB)
[default5]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=13|ip-26-0-169-86]: Local number of parameters: 32.7M (62.36MiB)
[default5]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=13|ip-26-0-169-86]: [After model building] Memory usage: 73.37MiB. Peak allocated: 75.40MiB Peak reserved: 82.00MiB
[default7]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=15|ip-26-0-169-86]: [After model building] Memory usage: 73.37MiB. Peak allocated: 75.40MiB Peak reserved: 82.00MiB
[default7]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=15|ip-26-0-169-86]: No checkpoint path provided.
[default5]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=13|ip-26-0-169-86]: No checkpoint path provided.
[default4]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=12|ip-26-0-169-86]: Local number of parameters: 32.7M (62.36MiB)
[default4]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=12|ip-26-0-169-86]: [After model building] Memory usage: 73.37MiB. Peak allocated: 75.40MiB Peak reserved: 82.00MiB
[default4]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=12|ip-26-0-169-86]: No checkpoint path provided.
[default1]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=9|ip-26-0-169-86]: Local number of parameters: 32.7M (62.36MiB)
[default1]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=9|ip-26-0-169-86]: [After model building] Memory usage: 73.37MiB. Peak allocated: 75.40MiB Peak reserved: 82.00MiB
[default6]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=14|ip-26-0-169-86]: Local number of parameters: 32.7M (62.36MiB)
[default6]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=14|ip-26-0-169-86]: [After model building] Memory usage: 73.37MiB. Peak allocated: 75.40MiB Peak reserved: 82.00MiB
[default1]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=9|ip-26-0-169-86]: No checkpoint path provided.
[default6]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=14|ip-26-0-169-86]: No checkpoint path provided.
[default3]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=11|ip-26-0-169-86]: Local number of parameters: 32.7M (62.36MiB)
[default3]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=11|ip-26-0-169-86]: [After model building] Memory usage: 73.37MiB. Peak allocated: 75.40MiB Peak reserved: 82.00MiB
[default3]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=11|ip-26-0-169-86]: No checkpoint path provided.
[default2]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=10|ip-26-0-169-86]: Local number of parameters: 32.7M (62.36MiB)
[default2]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=10|ip-26-0-169-86]: [After model building] Memory usage: 73.37MiB. Peak allocated: 75.40MiB Peak reserved: 82.00MiB
[default2]:07/03/2024 00:13:41 [INFO|DP=0|PP=1|TP=10|ip-26-0-169-86]: No checkpoint path provided.
[default1]:07/03/2024 00:13:41 [INFO|DP=1|PP=1|TP=9|ip-26-0-172-73]: No checkpoint path provided.
[default5]:07/03/2024 00:13:41 [INFO|DP=1|PP=1|TP=13|ip-26-0-172-73]: No checkpoint path provided.
[default4]:07/03/2024 00:13:41 [INFO|DP=1|PP=1|TP=12|ip-26-0-172-73]: No checkpoint path provided.
[default3]:07/03/2024 00:13:41 [INFO|DP=1|PP=1|TP=11|ip-26-0-172-73]: No checkpoint path provided.
[default7]:07/03/2024 00:13:41 [INFO|DP=1|PP=1|TP=15|ip-26-0-172-73]: No checkpoint path provided.
[default6]:07/03/2024 00:13:41 [INFO|DP=1|PP=1|TP=14|ip-26-0-172-73]: No checkpoint path provided.
[default2]:07/03/2024 00:13:41 [INFO|DP=1|PP=1|TP=10|ip-26-0-172-73]: No checkpoint path provided.
[default6]:07/03/2024 00:13:41 [INFO|DP=1|PP=1|TP=6|ip-26-0-172-57]: No checkpoint path provided.
[default0]:07/03/2024 00:13:41 [INFO|DP=1|PP=1|TP=8|ip-26-0-172-73]: No checkpoint path provided.
[default7]:07/03/2024 00:13:41 [INFO|DP=1|PP=0|TP=15|ip-26-0-165-24]: No checkpoint path provided.
[default4]:07/03/2024 00:13:41 [INFO|DP=1|PP=0|TP=12|ip-26-0-165-24]: No checkpoint path provided.
[default0]:07/03/2024 00:13:41 [INFO|DP=1|PP=0|TP=8|ip-26-0-165-24]: No checkpoint path provided.
[default5]:07/03/2024 00:13:41 [INFO|DP=1|PP=0|TP=13|ip-26-0-165-24]: No checkpoint path provided.
[default3]:07/03/2024 00:13:41 [INFO|DP=1|PP=0|TP=11|ip-26-0-165-24]: No checkpoint path provided.
[default6]:07/03/2024 00:13:41 [INFO|DP=1|PP=0|TP=14|ip-26-0-165-24]: No checkpoint path provided.
[default3]:07/03/2024 00:13:41 [INFO|DP=1|PP=1|TP=3|ip-26-0-172-57]: No checkpoint path provided.
[default1]:07/03/2024 00:13:41 [INFO|DP=1|PP=0|TP=9|ip-26-0-165-24]: No checkpoint path provided.
[default0]:07/03/2024 00:13:41 [INFO|DP=1|PP=0|TP=0|ip-26-0-163-226]: No checkpoint path provided.
[default7]:07/03/2024 00:13:41 [INFO|DP=1|PP=1|TP=7|ip-26-0-172-57]: No checkpoint path provided.
[default0]:07/03/2024 00:13:41 [INFO|DP=1|PP=1|TP=0|ip-26-0-172-57]: No checkpoint path provided.
[default2]:07/03/2024 00:13:41 [INFO|DP=1|PP=0|TP=10|ip-26-0-165-24]: No checkpoint path provided.
[default4]:07/03/2024 00:13:41 [INFO|DP=1|PP=1|TP=4|ip-26-0-172-57]: No checkpoint path provided.
[default2]:07/03/2024 00:13:41 [INFO|DP=1|PP=1|TP=2|ip-26-0-172-57]: No checkpoint path provided.
[default5]:07/03/2024 00:13:41 [INFO|DP=1|PP=1|TP=5|ip-26-0-172-57]: No checkpoint path provided.
[default1]:07/03/2024 00:13:41 [INFO|DP=1|PP=1|TP=1|ip-26-0-172-57]: No checkpoint path provided.
[default1]:07/03/2024 00:13:41 [INFO|DP=1|PP=0|TP=1|ip-26-0-163-226]: No checkpoint path provided.
[default2]:07/03/2024 00:13:41 [INFO|DP=1|PP=0|TP=2|ip-26-0-163-226]: No checkpoint path provided.
[default4]:07/03/2024 00:13:41 [INFO|DP=1|PP=0|TP=4|ip-26-0-163-226]: No checkpoint path provided.
[default5]:07/03/2024 00:13:41 [INFO|DP=1|PP=0|TP=5|ip-26-0-163-226]: No checkpoint path provided.
[default3]:07/03/2024 00:13:41 [INFO|DP=1|PP=0|TP=3|ip-26-0-163-226]: No checkpoint path provided.
[default7]:07/03/2024 00:13:41 [INFO|DP=1|PP=0|TP=7|ip-26-0-163-226]: No checkpoint path provided.
[default6]:07/03/2024 00:13:41 [INFO|DP=1|PP=0|TP=6|ip-26-0-163-226]: No checkpoint path provided.
[default0]:07/03/2024 00:13:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [Optimizer Building] Using LearningRateForSP as learning rate
[default0]:07/03/2024 00:13:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [ZeRO sharding] Size of optimizer params per rank:
[default0]:07/03/2024 00:13:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [ZeRO sharding] DP Rank 0 has 21.6M out of 43.2M (50.00%) params' optimizer states
[default0]:07/03/2024 00:13:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [ZeRO sharding] DP Rank 1 has 21.6M out of 43.2M (50.00%) params' optimizer states
[default0]:07/03/2024 00:13:44 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [Training Plan] Stage Training Stage has 19 remaining training steps and has consumed 0 samples
[default0]:07/03/2024 00:13:44 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Using `datasets` library
[default0]:07/03/2024 00:13:44 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Loading tokenizer from openai-community/gpt2 and transformers/hf_hub versions ('4.41.2', '0.23.4')
[default0]:07/03/2024 00:13:45 [WARNING|DP=0|PP=0|TP=0|ip-26-0-160-192]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/03/2024 00:13:51 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [Training Plan] There are 1 training stages 
[default0]:07/03/2024 00:13:51 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [Stage Training Stage] start from step 1 
[default0]:07/03/2024 00:13:51 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: 
[default0]:07/03/2024 00:13:51 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: [Start training] datetime: 2024-07-03 00:13:51.450590 | mbs: 128 | grad_accum: 4 | global_batch_size: 1024 | sequence_length: 4096 | train_steps: 20 | start_iteration_step: 0 | consumed_train_samples: 0
[default0]:07/03/2024 00:13:51 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]: Resuming training from stage Training Stage, it has trained for 0 samples and has 19 remaining train steps
[default0]:07/03/2024 00:13:51 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-192]:  Memory usage: 345.27MiB. Peak allocated 345.27MiB. Peak reserved: 362.00MiB
[default1]:07/03/2024 00:13:52 [WARNING|DP=0|PP=1|TP=1|ip-26-0-168-238]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/03/2024 00:13:52 [WARNING|DP=0|PP=1|TP=7|ip-26-0-168-238]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/03/2024 00:13:52 [WARNING|DP=0|PP=1|TP=11|ip-26-0-169-86]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/03/2024 00:13:52 [WARNING|DP=1|PP=0|TP=8|ip-26-0-165-24]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/03/2024 00:13:52 [WARNING|DP=1|PP=1|TP=15|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/03/2024 00:13:52 [WARNING|DP=0|PP=0|TP=10|ip-26-0-161-178]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/03/2024 00:13:52 [WARNING|DP=1|PP=1|TP=3|ip-26-0-172-57]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/03/2024 00:13:52 [WARNING|DP=0|PP=0|TP=12|ip-26-0-161-178]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/03/2024 00:13:52 [WARNING|DP=0|PP=0|TP=8|ip-26-0-161-178]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/03/2024 00:13:52 [WARNING|DP=1|PP=0|TP=0|ip-26-0-163-226]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/03/2024 00:13:52 [WARNING|DP=1|PP=1|TP=2|ip-26-0-172-57]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/03/2024 00:13:52 [WARNING|DP=0|PP=0|TP=15|ip-26-0-161-178]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/03/2024 00:13:52 [WARNING|DP=0|PP=0|TP=7|ip-26-0-160-192]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/03/2024 00:13:52 [WARNING|DP=0|PP=0|TP=4|ip-26-0-160-192]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/03/2024 00:13:52 [WARNING|DP=0|PP=0|TP=6|ip-26-0-160-192]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/03/2024 00:13:52 [WARNING|DP=0|PP=0|TP=14|ip-26-0-161-178]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/03/2024 00:13:52 [WARNING|DP=1|PP=1|TP=4|ip-26-0-172-57]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/03/2024 00:13:52 [WARNING|DP=0|PP=0|TP=2|ip-26-0-160-192]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/03/2024 00:13:52 [WARNING|DP=0|PP=1|TP=4|ip-26-0-168-238]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/03/2024 00:13:52 [WARNING|DP=0|PP=0|TP=11|ip-26-0-161-178]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/03/2024 00:13:52 [WARNING|DP=0|PP=0|TP=13|ip-26-0-161-178]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/03/2024 00:13:52 [WARNING|DP=1|PP=0|TP=15|ip-26-0-165-24]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/03/2024 00:13:52 [WARNING|DP=1|PP=1|TP=7|ip-26-0-172-57]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/03/2024 00:13:52 [WARNING|DP=1|PP=0|TP=3|ip-26-0-163-226]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/03/2024 00:13:52 [WARNING|DP=1|PP=0|TP=4|ip-26-0-163-226]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/03/2024 00:13:52 [WARNING|DP=0|PP=1|TP=0|ip-26-0-168-238]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/03/2024 00:13:52 [WARNING|DP=1|PP=0|TP=11|ip-26-0-165-24]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/03/2024 00:13:52 [WARNING|DP=1|PP=0|TP=14|ip-26-0-165-24]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/03/2024 00:13:52 [WARNING|DP=1|PP=0|TP=9|ip-26-0-165-24]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/03/2024 00:13:52 [WARNING|DP=1|PP=0|TP=5|ip-26-0-163-226]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/03/2024 00:13:52 [WARNING|DP=0|PP=1|TP=10|ip-26-0-169-86]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/03/2024 00:13:52 [WARNING|DP=1|PP=1|TP=13|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/03/2024 00:13:52 [WARNING|DP=1|PP=1|TP=12|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/03/2024 00:13:52 [WARNING|DP=1|PP=1|TP=14|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/03/2024 00:13:52 [WARNING|DP=1|PP=1|TP=10|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/03/2024 00:13:52 [WARNING|DP=1|PP=1|TP=6|ip-26-0-172-57]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/03/2024 00:13:52 [WARNING|DP=1|PP=0|TP=12|ip-26-0-165-24]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/03/2024 00:13:52 [WARNING|DP=1|PP=0|TP=2|ip-26-0-163-226]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/03/2024 00:13:52 [WARNING|DP=1|PP=0|TP=7|ip-26-0-163-226]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/03/2024 00:13:52 [WARNING|DP=1|PP=0|TP=10|ip-26-0-165-24]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/03/2024 00:13:52 [WARNING|DP=0|PP=0|TP=3|ip-26-0-160-192]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/03/2024 00:13:52 [WARNING|DP=0|PP=1|TP=12|ip-26-0-169-86]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/03/2024 00:13:53 [WARNING|DP=0|PP=1|TP=3|ip-26-0-168-238]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/03/2024 00:13:52 [WARNING|DP=0|PP=1|TP=6|ip-26-0-168-238]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/03/2024 00:13:53 [WARNING|DP=0|PP=1|TP=2|ip-26-0-168-238]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/03/2024 00:13:53 [WARNING|DP=0|PP=0|TP=9|ip-26-0-161-178]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/03/2024 00:13:52 [WARNING|DP=0|PP=0|TP=1|ip-26-0-160-192]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/03/2024 00:13:53 [WARNING|DP=1|PP=1|TP=0|ip-26-0-172-57]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/03/2024 00:13:52 [WARNING|DP=1|PP=0|TP=1|ip-26-0-163-226]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/03/2024 00:13:52 [WARNING|DP=0|PP=1|TP=13|ip-26-0-169-86]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/03/2024 00:13:52 [WARNING|DP=0|PP=1|TP=15|ip-26-0-169-86]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/03/2024 00:13:52 [WARNING|DP=0|PP=1|TP=9|ip-26-0-169-86]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/03/2024 00:13:53 [WARNING|DP=0|PP=1|TP=5|ip-26-0-168-238]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/03/2024 00:13:53 [WARNING|DP=1|PP=1|TP=9|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/03/2024 00:13:53 [WARNING|DP=1|PP=1|TP=11|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/03/2024 00:13:53 [WARNING|DP=1|PP=1|TP=8|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/03/2024 00:13:53 [WARNING|DP=0|PP=0|TP=5|ip-26-0-160-192]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/03/2024 00:13:53 [WARNING|DP=1|PP=1|TP=1|ip-26-0-172-57]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/03/2024 00:13:53 [WARNING|DP=0|PP=1|TP=14|ip-26-0-169-86]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/03/2024 00:13:53 [WARNING|DP=0|PP=1|TP=8|ip-26-0-169-86]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/03/2024 00:13:53 [WARNING|DP=1|PP=0|TP=13|ip-26-0-165-24]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/03/2024 00:13:53 [WARNING|DP=1|PP=1|TP=5|ip-26-0-172-57]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/03/2024 00:13:53 [WARNING|DP=1|PP=0|TP=6|ip-26-0-163-226]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:[rank31]: Traceback (most recent call last):
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default7]:[rank31]:     trainer.train(dataloader)
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default7]:[rank31]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default7]:[rank31]:     outputs = self.pipeline_engine.train_batch_iter(
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default7]:[rank31]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default7]:[rank31]:     output = model(**micro_batch)
[default7]:[rank31]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank31]:     return self._call_impl(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank31]:     return forward_call(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default7]:[rank31]:     sharded_logits = self.model(
[default7]:[rank31]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank31]:     return self._call_impl(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank31]:     return forward_call(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default7]:[rank31]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default7]:[rank31]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default7]:[rank31]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank31]:     return self._call_impl(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank31]:     return forward_call(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default7]:[rank31]:     output = self.pp_block(**new_kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank31]:     return self._call_impl(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank31]:     return forward_call(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward
[default7]:[rank31]:     hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"]
[default7]:[rank31]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank31]:     return self._call_impl(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank31]:     return forward_call(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward
[default7]:[rank31]:     hidden_states = self.down_proj(self.split_silu_mul(merged_states))
[default7]:[rank31]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank31]:     return self._call_impl(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank31]:     return forward_call(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default7]:[rank31]:     return row_linear(
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default7]:[rank31]:     out = F.linear(input, weight, bias)
[default7]:[rank31]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU  has a total capacity of 79.33 GiB of which 1.82 GiB is free. Including non-PyTorch memory, this process has 77.50 GiB memory in use. Of the allocated memory 68.05 GiB is allocated by PyTorch, and 79.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default7]:[rank23]: Traceback (most recent call last):
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default7]:[rank23]:     trainer.train(dataloader)
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default7]:[rank23]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default7]:[rank23]:     outputs = self.pipeline_engine.train_batch_iter(
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default7]:[rank23]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default7]:[rank23]:     output = model(**micro_batch)
[default7]:[rank23]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank23]:     return self._call_impl(*args, **kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank23]:     return forward_call(*args, **kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default7]:[rank23]:     sharded_logits = self.model(
[default7]:[rank23]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank23]:     return self._call_impl(*args, **kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank23]:     return forward_call(*args, **kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default7]:[rank23]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default7]:[rank23]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default7]:[rank23]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank23]:     return self._call_impl(*args, **kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank23]:     return forward_call(*args, **kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default7]:[rank23]:     output = self.pp_block(**new_kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank23]:     return self._call_impl(*args, **kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank23]:     return forward_call(*args, **kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward
[default7]:[rank23]:     hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"]
[default7]:[rank23]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank23]:     return self._call_impl(*args, **kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank23]:     return forward_call(*args, **kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward
[default7]:[rank23]:     hidden_states = self.down_proj(self.split_silu_mul(merged_states))
[default7]:[rank23]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank23]:     return self._call_impl(*args, **kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank23]:     return forward_call(*args, **kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default7]:[rank23]:     return row_linear(
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default7]:[rank23]:     out = F.linear(input, weight, bias)
[default7]:[rank23]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU  has a total capacity of 79.33 GiB of which 1.77 GiB is free. Including non-PyTorch memory, this process has 77.54 GiB memory in use. Of the allocated memory 68.05 GiB is allocated by PyTorch, and 79.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default4]:[rank28]: Traceback (most recent call last):
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default4]:[rank28]:     trainer.train(dataloader)
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default0]:[rank24]: Traceback (most recent call last):
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default4]:[rank28]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default3]:[rank27]: Traceback (most recent call last):
[default0]:[rank16]: Traceback (most recent call last):
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default0]:[rank16]:     trainer.train(dataloader)
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default0]:[rank16]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default0]:[rank16]:     outputs = self.pipeline_engine.train_batch_iter(
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default0]:[rank16]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanot[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default4]:[rank28]:     outputs = self.pipeline_engine.train_batch_iter(
[default0]:[rank24]:     trainer.train(dataloader)
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default3]:[rank27]:     trainer.train(dataloader)
[default4]:[rank28]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
ron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default0]:[rank24]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default0]:[rank16]:     output = model(**micro_batch)
[default0]:[rank16]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank16]:     return self._call_impl(*args, **kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank16]:     return forward_call(*args, **kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default0]:[rank16]:     sharded_logits = self.model(
[default0]:[rank16]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank16]:     return self._call_impl(*args, **kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank16]:     return forward_call(*args, **kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default0]:[rank16]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default0]:[rank16]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default0]:[rank16]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank16]:     return self._call_impl(*args, **kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in [default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
_call_impl
[default0]:[rank16]:     return forward_call(*args, **kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default0]:[rank16]:     output = self.pp_block(**new_kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank16]:     return self._call_impl(*args, **kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank16]:     return forward_call(*args, **kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default0]:[rank16]:     hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"]
[default0]:[rank16]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank16]:     return self._call_impl(*args, **kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank16]:     return forward_call(*args, **kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward
[default0]:[rank16]:     hidden_states = self.down_proj(self.split_silu_mul(merged_states))
[default0]:[rank16]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank16]:     return self._call_impl(*args, **kwargs)
[[default3]:[rank27]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
default0]:[rank16]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank16]:     return forward_call(*args, **kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default0]:[rank16]:     return row_linear(
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default0]:[rank16]:     out = F.linear(input, weight, bias)
[default0]:[rank16]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU 
[default0]:[rank24]:     outputs = self.pipeline_engine.train_batch_iter(
[default4]:[rank28]:     output = model(**micro_batch)
[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default3]:[rank27]:     outputs = self.pipeline_engine.train_batch_iter(
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default4]:[rank28]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank27]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default0]:[rank24]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default4]:[rank28]:     return self._call_impl(*args, **kwargs)
[default3]:[rank27]:     output = model(**micro_batch)
[default0]:[rank24]:     output = model(**micro_batch)
[default3]:[rank27]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank24]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank28]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank24]:     return self._call_impl(*args, **kwargs)
[default3]:[rank27]:     return self._call_impl(*args, **kwargs)
[default4]:[rank28]:     return forward_call(*args, **kwargs)
[default0]:[rank24]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank27]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank24]:     return forward_call(*args, **kwargs)
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default4]:[rank28]:     sharded_logits = self.model(
[default0]:[rank24]:     sharded_logits = self.model(
[default4]:[rank28]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank27]:     return forward_call(*args, **kwargs)
[default4]:[rank28]:     return self._call_impl(*args, **kwargs)
[default0]:[rank24]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default4]:[rank28]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank24]:     return self._call_impl(*args, **kwargs)
[default4]:[rank28]:     return forward_call(*args, **kwargs)
[default3]:[rank27]:     sharded_logits = self.model(
[default0]:[rank24]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank27]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default0]:[rank24]:     return forward_call(*args, **kwargs)
[default3]:[rank27]:     return self._call_impl(*args, **kwargs)
[default4]:[rank28]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default3]:[rank27]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank24]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default3]:[rank27]:     return forward_call(*args, **kwargs)
[default0]:[rank24]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default3]:[rank27]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default4]:[rank28]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default0]:[rank24]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank28]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default0]:[rank24]:     return self._call_impl(*args, **kwargs)
[default4]:[rank28]:     return self._call_impl(*args, **kwargs)
[default3]:[rank27]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default4]:[rank28]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank24]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank27]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank28]:     return forward_call(*args, **kwargs)
[default0]:[rank24]:     return forward_call(*args, **kwargs)
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default3]:[rank27]:     return self._call_impl(*args, **kwargs)
[default4]:[rank28]:     output = self.pp_block(**new_kwargs)
[default0]:[rank24]:     output = self.pp_block(**new_kwargs)
[default0]:[rank24]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank28]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank27]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank28]:     return self._call_impl(*args, **kwargs)
[default0]:[rank24]:     return self._call_impl(*args, **kwargs)
[default4]:[rank28]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank27]:     return forward_call(*args, **kwargs)
[default4]:[rank28]:     return forward_call(*args, **kwargs)
[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward
[default0]:[rank24]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank27]:     output = self.pp_block(**new_kwargs)
[default4]:[rank28]:     hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"]
[default0]:[rank24]:     return forward_call(*args, **kwargs)
[default3]:[rank27]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank28]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank27]:     return self._call_impl(*args, **kwargs)
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward
[default4]:[rank28]:     return self._call_impl(*args, **kwargs)
[default0]:[rank24]:     hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"]
[default4]:[rank28]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank28]:     return forward_call(*args, **kwargs)
[default3]:[rank27]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank24]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward
[default3]:[rank27]:     return forward_call(*args, **kwargs)
[default0]:[rank24]:     return self._call_impl(*args, **kwargs)
[default0]:[rank24]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward
[default4]:[rank28]:     hidden_states = self.down_proj(self.split_silu_mul(merged_states))
[default0]:[rank24]:     return forward_call(*args, **kwargs)
[default3]:[rank27]:     hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"]
[default4]:[rank28]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward
[default4]:[rank28]:     return self._call_impl(*args, **kwargs)
[default3]:[rank27]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank24]:     hidden_states = self.down_proj(self.split_silu_mul(merged_states))
[default3]:[rank27]:     return self._call_impl(*args, **kwargs)
[default4]:[rank28]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank27]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank28]:     return forward_call(*args, **kwargs)
[default0]:[rank24]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default3]:[rank27]:     return forward_call(*args, **kwargs)
[default0]:[rank24]:     return self._call_impl(*args, **kwargs)
[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward
[default0]:[rank24]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank28]:     return row_linear(
[default3]:[rank27]:     hidden_states = self.down_proj(self.split_silu_mul(merged_states))
[default0]:[rank24]:     return forward_call(*args, **kwargs)
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default3]:[rank27]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default4]:[rank28]:     out = F.linear(input, weight, bias)
[default3]:[rank27]:     return self._call_impl(*args, **kwargs)
[default0]:[rank24]:     return row_linear(
[default4]:[rank28]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU  has a total capacity of 79.33 GiB of which 1.66 GiB is free. Including non-PyTorch memory, this process has 77.66 GiB memory in use. Of the allocated memory 68.05 GiB is allocated by PyTorch, and 79.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default3]:[rank27]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default0]:[rank24]:     out = F.linear(input, weight, bias)
[default3]:[rank27]:     return forward_call(*args, **kwargs)
[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default3]:[rank27]:     return row_linear(
[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default3]:[rank27]:     out = F.linear(input, weight, bias)
[default0]:[rank24]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU 
[default3]:[rank27]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU  has a total capacity of 79.33 GiB of which 1.82 GiB is free. Including non-PyTorch memory, this process has 77.50 GiB memory in use. Of the allocated memory 68.05 GiB is allocated by PyTorch, and 79.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default6]:[rank30]: Traceback (most recent call last):
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default6]:[rank30]:     trainer.train(dataloader)
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default1]:[rank25]: Traceback (most recent call last):
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default1]:[rank25]:     trainer.train(dataloader)
[default6]:[rank30]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default6]:[rank30]:     outputs = self.pipeline_engine.train_batch_iter(
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default6]:[rank30]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default6]:[rank30]:     output = model(**micro_batch)
[default1]:[rank25]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default6]:[rank30]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank30]:     return self._call_impl(*args, **kwargs)
[default6]:[rank30]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank25]:     outputs = self.pipeline_engine.train_batch_iter(
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default1]:[rank25]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default1]:[rank25]:     output = model(**micro_batch)
[default1]:[rank25]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank30]:     return forward_call(*args, **kwargs)
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default6]:[rank30]:     sharded_logits = self.model(
[default6]:[rank30]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank30]:     return self._call_impl(*args, **kwargs)
[default6]:[rank30]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank25]:     return self._call_impl(*args, **kwargs)
[default6]:[rank30]:     return forward_call(*args, **kwargs)
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default6]:[rank30]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default1]:[rank25]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default6]:[rank30]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default6]:[rank30]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank30]:     return self._call_impl(*args, **kwargs)
[default1]:[rank25]:     return forward_call(*args, **kwargs)
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default1]:[rank25]:     sharded_logits = self.model(
[default1]:[rank25]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank30]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank25]:     return self._call_impl(*args, **kwargs)
[default1]:[rank25]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank30]:     return forward_call(*args, **kwargs)
[default1]:[rank25]:     return forward_call(*args, **kwargs)
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default1]:[rank25]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default2]:[rank26]: Traceback (most recent call last):
[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default1]:[rank25]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default1]:[rank25]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank26]:     trainer.train(dataloader)
[default6]:[rank30]:     output = self.pp_block(**new_kwargs)
[default1]:[rank25]:     return self._call_impl(*args, **kwargs)
[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default1]:[rank25]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank25]:     return forward_call(*args, **kwargs)
[default2]:[rank26]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default6]:[rank30]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default1]:[rank25]:     output = self.pp_block(**new_kwargs)
[default2]:[rank26]:     outputs = self.pipeline_engine.train_batch_iter(
[default6]:[rank30]:     return self._call_impl(*args, **kwargs)
[default6]:[rank30]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank25]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank25]:     return self._call_impl(*args, **kwargs)
[default1]:[rank25]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank22]: Traceback (most recent call last):
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default6]:[rank22]:     trainer.train(dataloader)
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default6]:[rank22]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default6]:[rank22]:     outputs = self.pipeline_engine.train_batch_iter(
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default6]:[rank22]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default6]:[rank22]:     output = model(**micro_batch)
[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default2]:[rank26]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default6]:[rank22]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank17]: Traceback (most recent call last):
[default6]:[rank30]:     return forward_call(*args, **kwargs)
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default6]:[rank22]:     return self._call_impl(*args, **kwargs)
[default1]:[rank25]:     return forward_call(*args, **kwargs)
[default1]:[rank17]:     trainer.train(dataloader)
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default1]:[rank17]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward
[default6]:[rank22]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default6]:[rank22]:     return forward_call(*args, **kwargs)
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default2]:[rank26]:     output = model(**micro_batch)
[default1]:[rank25]:     hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"]
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default1]:[rank25]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank22]:     sharded_logits = self.model(
[default6]:[rank22]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank26]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank30]:     hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"]
[default6]:[rank22]:     return self._call_impl(*args, **kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank22]:     return forward_call(*args, **kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default2]:[rank26]:     return self._call_impl(*args, **kwargs)
[default1]:[rank25]:     return self._call_impl(*args, **kwargs)
[default2]:[rank26]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank22]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default1]:[rank25]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank25]:     return forward_call(*args, **kwargs)
[default6]:[rank22]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward
[default2]:[rank26]:     return forward_call(*args, **kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank22]:     return self._call_impl(*args, **kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default1]:[rank17]:     outputs = self.pipeline_engine.train_batch_iter(
[default6]:[rank30]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default6]:[rank22]:     return forward_call(*args, **kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default1]:[rank17]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default6]:[rank22]:     output = self.pp_block(**new_kwargs)
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default6]:[rank22]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank22]:     return self._call_impl(*args, **kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank22]:     return forward_call(*args, **kwargs)
[default1]:[rank17]:     output = model(**micro_batch)
[default2]:[rank26]:     sharded_logits = self.model(
[default1]:[rank17]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank26]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank30]:     return self._call_impl(*args, **kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward
[default2]:[rank26]:     return self._call_impl(*args, **kwargs)
[default2]:[rank26]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank17]:     return self._call_impl(*args, **kwargs)
[default1]:[rank25]:     hidden_states = self.down_proj(self.split_silu_mul(merged_states))
[default6]:[rank30]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank22]:     hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"]
[default1]:[rank25]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank17]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank22]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank26]:     return forward_call(*args, **kwargs)
[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default1]:[rank17]:     return forward_call(*args, **kwargs)
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default6]:[rank30]:     return forward_call(*args, **kwargs)
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward
[default1]:[rank17]:     sharded_logits = self.model(
[default1]:[rank17]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank26]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default1]:[rank17]:     return self._call_impl(*args, **kwargs)
[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default1]:[rank17]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank25]:     return self._call_impl(*args, **kwargs)
[default1]:[rank17]:     return forward_call(*args, **kwargs)
[default1]:[rank25]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank25]:     return forward_call(*args, **kwargs)
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default6]:[rank22]:     return self._call_impl(*args, **kwargs)
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default1]:[rank17]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank30]:     hidden_states = self.down_proj(self.split_silu_mul(merged_states))
[default1]:[rank25]:     return row_linear(
[default6]:[rank22]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank30]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default1]:[rank17]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default2]:[rank26]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default1]:[rank17]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank26]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank30]:     return self._call_impl(*args, **kwargs)
[default6]:[rank30]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank17]:     return self._call_impl(*args, **kwargs)
[default1]:[rank17]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default1]:[rank17]:     return forward_call(*args, **kwargs)
[default6]:[rank22]:     return forward_call(*args, **kwargs)
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default1]:[rank17]:     output = self.pp_block(**new_kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward
[default1]:[rank17]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank22]:     hidden_states = self.down_proj(self.split_silu_mul(merged_states))
[default6]:[rank22]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank22]:     return self._call_impl(*args, **kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank22]:     return forward_call(*args, **kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default1]:[rank17]:     return self._call_impl(*args, **kwargs)
[default2]:[rank26]:     return self._call_impl(*args, **kwargs)
[default6]:[rank30]:     return forward_call(*args, **kwargs)
[default1]:[rank17]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default6]:[rank22]:     return row_linear(
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default2]:[rank26]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank25]:     out = F.linear(input, weight, bias)
[default6]:[rank30]:     return row_linear(
[default2]:[rank26]:     return forward_call(*args, **kwargs)
[default6]:[rank22]:     out = F.linear(input, weight, bias)
[default1]:[rank17]:     return forward_call(*args, **kwargs)
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward
[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default1]:[rank17]:     hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"]
[default1]:[rank25]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU 
 has a total capacity of 79.33 GiB of which 1.82 GiB is free. Including non-PyTorch memory, this process has 77.50 GiB memory in use. Of the allocated memory 68.05 GiB is allocated by PyTorch, and 79.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default6]:[rank22]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU  has a total capacity of 79.33 GiB of which 1.86 GiB is free. Including non-PyTorch memory, this process has 77.46 GiB memory in use. Of the allocated memory 68.05 GiB is allocated by PyTorch, and 79.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default1]:[rank17]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank17]:     return self._call_impl(*args, **kwargs)
[default2]:[rank26]:     output = self.pp_block(**new_kwargs)
[default2]:[rank26]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank17]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank17]:     return forward_call(*args, **kwargs)
[default2]:[rank26]:     return self._call_impl(*args, **kwargs)
[default2]:[rank26]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward
[default1]:[rank17]:     hidden_states = self.down_proj(self.split_silu_mul(merged_states))
[default2]:[rank26]:     return forward_call(*args, **kwargs)
[default1]:[rank17]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank17]:     return self._call_impl(*args, **kwargs)
[default1]:[rank17]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank17]:     return forward_call(*args, **kwargs)
[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default1]:[rank17]:     return row_linear(
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default6]:[rank30]:     out = F.linear(input, weight, bias)
[default1]:[rank17]:     out = F.linear(input, weight, bias)
[default2]:[rank26]:     hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"]
[default2]:[rank26]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank17]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU 
 has a total capacity of 79.33 GiB of which 1.77 GiB is free. Including non-PyTorch memory, this process has 77.54 GiB memory in use. Of the allocated memory 68.05 GiB is allocated by PyTorch, and 79.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default4]:[rank20]: Traceback (most recent call last):
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default4]:[rank20]:     trainer.train(dataloader)
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default4]:[rank20]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default4]:[rank20]:     outputs = self.pipeline_engine.train_batch_iter(
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default4]:[rank20]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default4]:[rank20]:     output = model(**micro_batch)
[default4]:[rank20]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank29]: Traceback (most recent call last):
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default2]:[rank26]:     return self._call_impl(*args, **kwargs)
[default2]:[rank26]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank20]:     return self._call_impl(*args, **kwargs)
[default4]:[rank20]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank20]:     return forward_call(*args, **kwargs)
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default4]:[rank20]:     sharded_logits = self.model(
[default4]:[rank20]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank20]:     return self._call_impl(*args, **kwargs)
[default4]:[rank20]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank20]:     return forward_call(*args, **kwargs)
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotro[default6]:[rank30]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU  has a total capacity of 79.33 GiB of which 1.66 GiB is free. Including non-PyTorch memory, this process has 77.66 GiB memory in use. Of the allocated memory 68.05 GiB is allocated by PyTorch, and 79.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
n/src/nanotron/models/llama.py", line 764, in forward
[default4]:[rank20]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default4]:[rank20]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default4]:[rank20]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank20]:     return self._call_impl(*args, **kwargs)
[default4]:[rank20]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank29]:     trainer.train(dataloader)
[default2]:[rank26]:     return forward_call(*args, **kwargs)
[default4]:[rank20]:     return forward_call(*args, **kwargs)
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default4]:[rank20]:     output = self.pp_block(**new_kwargs)
[default4]:[rank20]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank20]:     return self._call_impl(*args, **kwargs)
[default4]:[rank20]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank20]:     return forward_call(*args, **kwargs)
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward
[default4]:[rank20]:     hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"]
[default4]:[rank20]:   File "/fsx/ferdinandmom/mini[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward
forge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank20]:     return self._call_impl(*args, **kwargs)
[default4]:[rank20]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank20]:     return forward_call(*args, **kwargs)
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward
[default4]:[rank20]:     hidden_states = self.down_proj(self.split_silu_mul(merged_states))
[default4]:[rank20]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank20]:     return self._call_impl(*args, **kwargs)
[default4]:[rank20]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541[default2]:[rank26]:     hidden_states = self.down_proj(self.split_silu_mul(merged_states))
, in _call_impl
[default4]:[rank20]:     return forward_call(*args, **kwargs)
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default2]:[rank26]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank26]:     return self._call_impl(*args, **kwargs)
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default4]:[rank20]:     return row_linear(
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default4]:[rank20]:     out = F.linear(input, weight, bias)
[default5]:[rank29]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default4]:[rank20]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU  has a total capacity of 79.33 GiB of which 1.86 GiB is free. Including non-PyTorch memory, this process has 77.46 GiB memory in use. Of the allocated memory 68.05 GiB is allocated by PyTorch, and 79.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default5]:[rank29]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default5]:[rank29]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default2]:[rank26]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default5]:[rank29]:     output = model(**micro_batch)
[default5]:[rank29]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank29]:     return self._call_impl(*args, **kwargs)
[default2]:[rank26]:     return forward_call(*args, **kwargs)
[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default2]:[rank26]:     return row_linear(
[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default2]:[rank26]:     out = F.linear(input, weight, bias)
[default5]:[rank29]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank29]:     return forward_call(*args, **kwargs)
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default5]:[rank29]:     sharded_logits = self.model(
[default5]:[rank29]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank26]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU  has a total capacity of 79.33 GiB of which 1.66 GiB is free. Including non-PyTorch memory, this process has 77.66 GiB memory in use. Of the allocated memory 68.05 GiB is allocated by PyTorch, and 79.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default5]:[rank29]:     return self._call_impl(*args, **kwargs)
[default5]:[rank29]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank29]:     return forward_call(*args, **kwargs)
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default5]:[rank29]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default5]:[rank29]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default5]:[rank29]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank29]:     return self._call_impl(*args, **kwargs)
[default5]:[rank29]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank29]:     return forward_call(*args, **kwargs)
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default5]:[rank29]:     output = self.pp_block(**new_kwargs)
[default5]:[rank29]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank29]:     return self._call_impl(*args, **kwargs)
[default5]:[rank29]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank29]:     return forward_call(*args, **kwargs)
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward
[default5]:[rank29]:     hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"]
[default5]:[rank29]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank29]:     return self._call_impl(*args, **kwargs)
[default5]:[rank29]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank29]:     return forward_call(*args, **kwargs)
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward
[default5]:[rank29]:     hidden_states = self.down_proj(self.split_silu_mul(merged_states))
[default5]:[rank29]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank29]:     return self._call_impl(*args, **kwargs)
[default5]:[rank29]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank29]:     return forward_call(*args, **kwargs)
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default5]:[rank29]:     return row_linear(
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default5]:[rank29]:     out = F.linear(input, weight, bias)
[default5]:[rank29]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU  has a total capacity of 79.33 GiB of which 1.82 GiB is free. Including non-PyTorch memory, this process has 77.50 GiB memory in use. Of the allocated memory 68.05 GiB is allocated by PyTorch, and 79.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default2]:[rank18]: Traceback (most recent call last):
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default3]:[rank19]: Traceback (most recent call last):
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default3]:[rank19]:     trainer.train(dataloader)
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default3]:[rank19]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default5]:[rank21]: Traceback (most recent call last):
[default2]:[rank18]:     trainer.train(dataloader)
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default5]:[rank21]:     trainer.train(dataloader)
[default3]:[rank19]:     outputs = self.pipeline_engine.train_batch_iter(
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default5]:[rank21]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default3]:[rank19]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default2]:[rank18]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default5]:[rank21]:     outputs = self.pipeline_engine.train_batch_iter(
[default3]:[rank19]:     output = model(**micro_batch)
[default3]:[rank19]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank18]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default3]:[rank19]:     return self._call_impl(*args, **kwargs)
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default5]:[rank21]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default2]:[rank18]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default3]:[rank19]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default2]:[rank18]:     output = model(**micro_batch)
[default3]:[rank19]:     return forward_call(*args, **kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default5]:[rank21]:     output = model(**micro_batch)
[default2]:[rank18]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank21]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank18]:     return self._call_impl(*args, **kwargs)
[default2]:[rank18]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank19]:     sharded_logits = self.model(
[default5]:[rank21]:     return self._call_impl(*args, **kwargs)
[default2]:[rank18]:     return forward_call(*args, **kwargs)
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default5]:[rank21]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank19]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank21]:     return forward_call(*args, **kwargs)
[default2]:[rank18]:     sharded_logits = self.model(
[default3]:[rank19]:     return self._call_impl(*args, **kwargs)
[default3]:[rank19]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default2]:[rank18]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank21]:     sharded_logits = self.model(
[default2]:[rank18]:     return self._call_impl(*args, **kwargs)
[default3]:[rank19]:     return forward_call(*args, **kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank18]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank21]:     return self._call_impl(*args, **kwargs)
[default2]:[rank18]:     return forward_call(*args, **kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default5]:[rank21]:     return forward_call(*args, **kwargs)
[default2]:[rank18]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default3]:[rank19]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default5]:[rank21]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default5]:[rank21]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default5]:[rank21]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank19]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default2]:[rank18]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default5]:[rank21]:     return self._call_impl(*args, **kwargs)
[default3]:[rank19]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank21]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank18]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank18]:     return self._call_impl(*args, **kwargs)
[default5]:[rank21]:     return forward_call(*args, **kwargs)
[default3]:[rank19]:     return self._call_impl(*args, **kwargs)
[default2]:[rank18]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank19]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank18]:     return forward_call(*args, **kwargs)
[default3]:[rank19]:     return forward_call(*args, **kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default5]:[rank21]:     output = self.pp_block(**new_kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank19]:     output = self.pp_block(**new_kwargs)
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default3]:[rank19]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank21]:     return self._call_impl(*args, **kwargs)
[default2]:[rank18]:     output = self.pp_block(**new_kwargs)
[default3]:[rank19]:     return self._call_impl(*args, **kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank19]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank18]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank21]:     return forward_call(*args, **kwargs)
[default3]:[rank19]:     return forward_call(*args, **kwargs)
[default2]:[rank18]:     return self._call_impl(*args, **kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward
[default5]:[rank21]:     hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"]
[default2]:[rank18]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank18]:     return forward_call(*args, **kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank19]:     hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"]
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward
[default5]:[rank21]:     return self._call_impl(*args, **kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank18]:     hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"]
[default3]:[rank19]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank18]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank19]:     return self._call_impl(*args, **kwargs)
[default2]:[rank18]:     return self._call_impl(*args, **kwargs)
[default3]:[rank19]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank18]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank21]:     return forward_call(*args, **kwargs)
[default3]:[rank19]:     return forward_call(*args, **kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward
[default5]:[rank21]:     hidden_states = self.down_proj(self.split_silu_mul(merged_states))
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward
[default2]:[rank18]:     return forward_call(*args, **kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward
[default5]:[rank21]:     return self._call_impl(*args, **kwargs)
[default2]:[rank18]:     hidden_states = self.down_proj(self.split_silu_mul(merged_states))
[default5]:[rank21]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank19]:     hidden_states = self.down_proj(self.split_silu_mul(merged_states))
[default2]:[rank18]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank21]:     return forward_call(*args, **kwargs)
[default3]:[rank19]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank18]:     return self._call_impl(*args, **kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default3]:[rank19]:     return self._call_impl(*args, **kwargs)
[default3]:[rank19]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank18]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank21]:     return row_linear(
[default3]:[rank19]:     return forward_call(*args, **kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default5]:[rank21]:     out = F.linear(input, weight, bias)
[default2]:[rank18]:     return forward_call(*args, **kwargs)
[default3]:[rank19]:     return row_linear(
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default5]:[rank21]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU  has a total capacity of 79.33 GiB of which 1.77 GiB is free. Including non-PyTorch memory, this process has 77.54 GiB memory in use. Of the allocated memory 68.05 GiB is allocated by PyTorch, and 79.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default3]:[rank19]:     out = F.linear(input, weight, bias)
[default2]:[rank18]:     return row_linear(
[default3]:[rank19]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU  has a total capacity of 79.33 GiB of which 1.77 GiB is free. Including non-PyTorch memory, this process has 77.54 GiB memory in use. Of the allocated memory 68.05 GiB is allocated by PyTorch, and 79.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default2]:[rank18]:     out = F.linear(input, weight, bias)
[default2]:[rank18]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU  has a total capacity of 79.33 GiB of which 1.86 GiB is free. Including non-PyTorch memory, this process has 77.46 GiB memory in use. Of the allocated memory 68.05 GiB is allocated by PyTorch, and 79.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default1]:[rank9]: Traceback (most recent call last):
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default1]:[rank9]:     trainer.train(dataloader)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default1]:[rank9]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default1]:[rank9]:     outputs = self.pipeline_engine.train_batch_iter(
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default1]:[rank9]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default1]:[rank9]:     output = model(**micro_batch)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank9]:     return self._call_impl(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank9]:     return forward_call(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default1]:[rank9]:     sharded_logits = self.model(
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank9]:     return self._call_impl(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank9]:     return forward_call(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default1]:[rank9]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default1]:[rank9]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank9]:     return self._call_impl(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank9]:     return forward_call(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default1]:[rank9]:     output = self.pp_block(**new_kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank9]:     return self._call_impl(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank9]:     return forward_call(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward
[default1]:[rank9]:     hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"]
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank9]:     return self._call_impl(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank9]:     return forward_call(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward
[default1]:[rank9]:     hidden_states = self.down_proj(self.split_silu_mul(merged_states))
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank9]:     return self._call_impl(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank9]:     return forward_call(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default1]:[rank9]:     return row_linear(
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default1]:[rank9]:     out = F.linear(input, weight, bias)
[default1]:[rank9]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU 
 has a total capacity of 79.33 GiB of which 1.86 GiB is free. Including non-PyTorch memory, this process has 77.46 GiB memory in use. Of the allocated memory 68.05 GiB is allocated by PyTorch, and 79.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default7]:[rank15]: Traceback (most recent call last):
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default7]:[rank15]:     trainer.train(dataloader)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default7]:[rank15]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default7]:[rank15]:     outputs = self.pipeline_engine.train_batch_iter(
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default7]:[rank15]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default7]:[rank15]:     output = model(**micro_batch)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank15]:     return self._call_impl(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank15]:     return forward_call(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default7]:[rank15]:     sharded_logits = self.model(
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank15]:     return self._call_impl(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank15]:     return forward_call(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default7]:[rank15]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default7]:[rank15]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank15]:     return self._call_impl(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank15]:     return forward_call(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default7]:[rank15]:     output = self.pp_block(**new_kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank15]:     return self._call_impl(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank15]:     return forward_call(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward
[default7]:[rank15]:     hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"]
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank15]:     return self._call_impl(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank15]:     return forward_call(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward
[default7]:[rank15]:     hidden_states = self.down_proj(self.split_silu_mul(merged_states))
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank15]:     return self._call_impl(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank15]:     return forward_call(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default7]:[rank15]:     return row_linear(
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default7]:[rank15]:     out = F.linear(input, weight, bias)
[default7]:[rank15]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU  has a total capacity of 79.33 GiB of which 1.86 GiB is free. Including non-PyTorch memory, this process has 77.46 GiB memory in use. Of the allocated memory 68.05 GiB is allocated by PyTorch, and 79.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default0]:[rank8]: Traceback (most recent call last):
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default0]:[rank8]:     trainer.train(dataloader)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default0]:[rank8]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default0]:[rank8]:     outputs = self.pipeline_engine.train_batch_iter(
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default0]:[rank8]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default0]:[rank8]:     output = model(**micro_batch)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank8]:     return self._call_impl(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank8]:     return forward_call(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default0]:[rank8]:     sharded_logits = self.model(
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank8]:     return self._call_impl(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank8]:     return forward_call(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default0]:[rank8]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default0]:[rank8]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank8]:     return self._call_impl(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank8]:     return forward_call(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default0]:[rank8]:     output = self.pp_block(**new_kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank8]:     return self._call_impl(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank8]:     return forward_call(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward
[default0]:[rank8]:     hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"]
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank8]:     return self._call_impl(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank8]:     return forward_call(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward
[default0]:[rank8]:     hidden_states = self.down_proj(self.split_silu_mul(merged_states))
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank8]:     return self._call_impl(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank8]:     return forward_call(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default0]:[rank8]:     return row_linear(
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default0]:[rank8]:     out = F.linear(input, weight, bias)
[default0]:[rank8]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU 
[default5]:[rank13]: Traceback (most recent call last):
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default5]:[rank13]:     trainer.train(dataloader)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default5]:[rank13]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default5]:[rank13]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default5]:[rank13]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default5]:[rank13]:     output = model(**micro_batch)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank13]:     return self._call_impl(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank13]:     return forward_call(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default5]:[rank13]:     sharded_logits = self.model(
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank13]:     return self._call_impl(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank13]:     return forward_call(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default5]:[rank13]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default5]:[rank13]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank13]:     return self._call_impl(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank13]:     return forward_call(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default5]:[rank13]:     output = self.pp_block(**new_kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank13]:     return self._call_impl(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank13]:     return forward_call(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward
[default5]:[rank13]:     hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"]
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank13]:     return self._call_impl(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank13]:     return forward_call(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward
[default5]:[rank13]:     hidden_states = self.down_proj(self.split_silu_mul(merged_states))
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank13]:     return self._call_impl(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank13]:     return forward_call(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default5]:[rank13]:     return row_linear(
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default5]:[rank13]:     out = F.linear(input, weight, bias)
[default5]:[rank13]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU  has a total capacity of 79.33 GiB of which 1.86 GiB is free. Including non-PyTorch memory, this process has 77.46 GiB memory in use. Of the allocated memory 68.05 GiB is allocated by PyTorch, and 79.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default4]:[rank12]: Traceback (most recent call last):
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default4]:[rank12]:     trainer.train(dataloader)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default4]:[rank12]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default4]:[rank12]:     outputs = self.pipeline_engine.train_batch_iter(
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default4]:[rank12]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default4]:[rank12]:     output = model(**micro_batch)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank12]:     return self._call_impl(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank14]: Traceback (most recent call last):
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default6]:[rank14]:     trainer.train(dataloader)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default6]:[rank14]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default6]:[rank14]:     outputs = self.pipeline_engine.train_batch_iter(
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default6]:[rank14]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default6]:[rank14]:     output = model(**micro_batch)
[default4]:[rank12]:     return forward_call(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank12]:     sharded_logits = self.model(
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank14]:     return self._call_impl(*args, **kwargs)
[default4]:[rank12]:     return self._call_impl(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank12]:     return forward_call(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank14]:     return forward_call(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default4]:[rank12]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default4]:[rank12]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank12]:     return self._call_impl(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default6]:[rank14]:     sharded_logits = self.model(
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank14]:     return self._call_impl(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank12]:     return forward_call(*args, **kwargs)
[default6]:[rank14]:     return forward_call(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default6]:[rank14]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default4]:[rank12]:     output = self.pp_block(**new_kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank14]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank12]:     return self._call_impl(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank12]:     return forward_call(*args, **kwargs)
[default6]:[rank14]:     return self._call_impl(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank14]:     return forward_call(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward
[default4]:[rank12]:     hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"]
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank12]:     return self._call_impl(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank12]:     return forward_call(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward
[default4]:[rank12]:     hidden_states = self.down_proj(self.split_silu_mul(merged_states))
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default6]:[rank14]:     output = self.pp_block(**new_kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank14]:     return self._call_impl(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank14]:     return forward_call(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank12]:     return self._call_impl(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward
[default6]:[rank14]:     hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"]
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank14]:     return self._call_impl(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank12]:     return forward_call(*args, **kwargs)
[default6]:[rank14]:     return forward_call(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default4]:[rank12]:     return row_linear(
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward
[default6]:[rank14]:     hidden_states = self.down_proj(self.split_silu_mul(merged_states))
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default4]:[rank12]:     out = F.linear(input, weight, bias)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank14]:     return self._call_impl(*args, **kwargs)
[default4]:[rank12]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU  has a total capacity of 79.33 GiB of which 1.77 GiB is free. Including non-PyTorch memory, this process has 77.54 GiB memory in use. Of the allocated memory 68.05 GiB is allocated by PyTorch, and 79.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank14]:     return forward_call(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default6]:[rank14]:     return row_linear(
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default6]:[rank14]:     out = F.linear(input, weight, bias)
[default6]:[rank14]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU  has a total capacity of 79.33 GiB of which 1.77 GiB is free. Including non-PyTorch memory, this process has 77.54 GiB memory in use. Of the allocated memory 68.05 GiB is allocated by PyTorch, and 79.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default2]:[rank10]: Traceback (most recent call last):
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default3]:[rank11]: Traceback (most recent call last):
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default3]:[rank11]:     trainer.train(dataloader)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default3]:[rank11]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default3]:[rank11]:     outputs = self.pipeline_engine.train_batch_iter(
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default2]:[rank10]:     trainer.train(dataloader)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default2]:[rank10]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default3]:[rank11]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default3]:[rank11]:     output = model(**micro_batch)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank10]:     outputs = self.pipeline_engine.train_batch_iter(
[default3]:[rank11]:     return self._call_impl(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank11]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default3]:[rank11]:     sharded_logits = self.model(
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank10]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default3]:[rank11]:     return self._call_impl(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank11]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default3]:[rank11]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default2]:[rank10]:     output = model(**micro_batch)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank10]:     return self._call_impl(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank10]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default3]:[rank11]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank11]:     return self._call_impl(*args, **kwargs)
[default2]:[rank10]:     sharded_logits = self.model(
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank11]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default2]:[rank10]:     return self._call_impl(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank10]:     return forward_call(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default3]:[rank11]:     output = self.pp_block(**new_kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank10]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default3]:[rank11]:     return self._call_impl(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank11]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward
[default2]:[rank10]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default3]:[rank11]:     hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"]
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank11]:     return self._call_impl(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank10]:     return self._call_impl(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank10]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:     return forward_call(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward
[default3]:[rank11]:     hidden_states = self.down_proj(self.split_silu_mul(merged_states))
[default2]:[rank10]:     output = self.pp_block(**new_kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank10]:     return self._call_impl(*args, **kwargs)
[default3]:[rank11]:     return self._call_impl(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank10]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward
[default3]:[rank11]:     return forward_call(*args, **kwargs)
[default2]:[rank10]:     hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"]
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default3]:[rank11]:     return row_linear(
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default2]:[rank10]:     return self._call_impl(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank11]:     out = F.linear(input, weight, bias)
[default2]:[rank10]:     return forward_call(*args, **kwargs)
[default3]:[rank11]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU  has a total capacity of 79.33 GiB of which 1.86 GiB is free. Including non-PyTorch memory, this process has 77.46 GiB memory in use. Of the allocated memory 68.05 GiB is allocated by PyTorch, and 79.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 172, in forward
[default2]:[rank10]:     hidden_states = self.down_proj(self.split_silu_mul(merged_states))
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank10]:     return self._call_impl(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank10]:     return forward_call(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default2]:[rank10]:     return row_linear(
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default2]:[rank10]:     out = F.linear(input, weight, bias)
[default2]:[rank10]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU  has a total capacity of 79.33 GiB of which 1.77 GiB is free. Including non-PyTorch memory, this process has 77.54 GiB memory in use. Of the allocated memory 68.05 GiB is allocated by PyTorch, and 79.45 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
W0703 00:14:19.268000 139905265825600 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3192000 closing signal SIGTERM
W0703 00:14:19.268000 139905265825600 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3192003 closing signal SIGTERM
W0703 00:14:19.268000 139905265825600 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3192004 closing signal SIGTERM
W0703 00:14:19.268000 139905265825600 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3192005 closing signal SIGTERM
W0703 00:14:19.275000 140507061364544 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 495736 closing signal SIGTERM
W0703 00:14:19.275000 140507061364544 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 495737 closing signal SIGTERM
W0703 00:14:19.276000 140507061364544 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 495739 closing signal SIGTERM
W0703 00:14:19.276000 140507061364544 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 495740 closing signal SIGTERM
W0703 00:14:19.279000 140671516706624 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 868707 closing signal SIGTERM
W0703 00:14:19.279000 140671516706624 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 868708 closing signal SIGTERM
W0703 00:14:19.280000 140671516706624 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 868709 closing signal SIGTERM
W0703 00:14:19.280000 140671516706624 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 868711 closing signal SIGTERM
W0703 00:14:19.280000 140671516706624 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 868713 closing signal SIGTERM
E0703 00:14:20.398000 140507061364544 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 495735) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10
E0703 00:14:20.404000 139905265825600 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 3191999) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-07-03_00:14:19
  host      : ip-26-0-161-178.ec2.internal
  rank      : 11 (local_rank: 3)
  exitcode  : 1 (pid: 495738)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2024-07-03_00:14:19
  host      : ip-26-0-161-178.ec2.internal
  rank      : 14 (local_rank: 6)
  exitcode  : 1 (pid: 495741)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
  time      : 2024-07-03_00:14:19
  host      : ip-26-0-161-178.ec2.internal
  rank      : 15 (local_rank: 7)
  exitcode  : 1 (pid: 495742)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-03_00:14:19
  host      : ip-26-0-161-178.ec2.internal
  rank      : 8 (local_rank: 0)
  exitcode  : 1 (pid: 495735)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-07-03_00:14:19
  host      : ip-26-0-163-226.ec2.internal
  rank      : 18 (local_rank: 2)
  exitcode  : 1 (pid: 3192001)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2024-07-03_00:14:19
  host      : ip-26-0-163-226.ec2.internal
  rank      : 19 (local_rank: 3)
  exitcode  : 1 (pid: 3192002)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
  time      : 2024-07-03_00:14:19
  host      : ip-26-0-163-226.ec2.internal
  rank      : 23 (local_rank: 7)
  exitcode  : 1 (pid: 3192006)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-03_00:14:19
  host      : ip-26-0-163-226.ec2.internal
  rank      : 16 (local_rank: 0)
  exitcode  : 1 (pid: 3191999)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
srun: error: ip-26-0-163-226: task 2: Exited with exit code 1
srun: error: ip-26-0-161-178: task 1: Exited with exit code 1
E0703 00:14:20.806000 140671516706624 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 3 (pid: 868710) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-07-03_00:14:19
  host      : ip-26-0-165-24.ec2.internal
  rank      : 29 (local_rank: 5)
  exitcode  : 1 (pid: 868712)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2024-07-03_00:14:19
  host      : ip-26-0-165-24.ec2.internal
  rank      : 31 (local_rank: 7)
  exitcode  : 1 (pid: 868714)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-03_00:14:19
  host      : ip-26-0-165-24.ec2.internal
  rank      : 27 (local_rank: 3)
  exitcode  : 1 (pid: 868710)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
srun: error: ip-26-0-165-24: task 3: Exited with exit code 1
[default4]:[rank4]:[E ProcessGroupNCCL.cpp:1316] [PG 2 Rank 4] Heartbeat monitor timed out! Process will be terminated after dumping debug info. workMetaList_.size()=5
[default4]:[rank4]:[E ProcessGroupNCCL.cpp:1153] [PG 2 Rank 4] ProcessGroupNCCL preparing to dump debug info.
[default4]:[rank4]:[F ProcessGroupNCCL.cpp:1169] [PG 2 Rank 4] [PG 2 Rank 4] ProcessGroupNCCL's watchdog got stuck for 600 seconds without making progress in monitoring enqueued collectives. This typically indicates a NCCL/CUDA API hang blocking the watchdog, and could be triggered by another thread holding the GIL inside a CUDA api, or other deadlock-prone behaviors.If you suspect the watchdog is not actually stuck and a longer timeout would help, you can either increase the timeout (TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC) to a larger value or disable the heartbeat monitor (TORCH_NCCL_ENABLE_MONITORING=0).If either of aforementioned helps, feel free to file an issue to PyTorch about the short timeout or false positive abort; otherwise, please attempt to debug the hang. workMetaList_.size() = 5
[default1]:[rank1]:[E ProcessGroupNCCL.cpp:1316] [PG 2 Rank 1] Heartbeat monitor timed out! Process will be terminated after dumping debug info. workMetaList_.size()=13
[default1]:[rank1]:[E ProcessGroupNCCL.cpp:1153] [PG 2 Rank 1] ProcessGroupNCCL preparing to dump debug info.
[default3]:[rank3]:[E ProcessGroupNCCL.cpp:1316] [PG 2 Rank 3] Heartbeat monitor timed out! Process will be terminated after dumping debug info. workMetaList_.size()=13
[default3]:[rank3]:[E ProcessGroupNCCL.cpp:1153] [PG 2 Rank 3] ProcessGroupNCCL preparing to dump debug info.
[default1]:[rank1]:[F ProcessGroupNCCL.cpp:1169] [PG 2 Rank 1] [PG 2 Rank 1] ProcessGroupNCCL's watchdog got stuck for 600 seconds without making progress in monitoring enqueued collectives. This typically indicates a NCCL/CUDA API hang blocking the watchdog, and could be triggered by another thread holding the GIL inside a CUDA api, or other deadlock-prone behaviors.If you suspect the watchdog is not actually stuck and a longer timeout would help, you can either increase the timeout (TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC) to a larger value or disable the heartbeat monitor (TORCH_NCCL_ENABLE_MONITORING=0).If either of aforementioned helps, feel free to file an issue to PyTorch about the short timeout or false positive abort; otherwise, please attempt to debug the hang. workMetaList_.size() = 13
[default3]:[rank3]:[F ProcessGroupNCCL.cpp:1169] [PG 2 Rank 3] [PG 2 Rank 3] ProcessGroupNCCL's watchdog got stuck for 600 seconds without making progress in monitoring enqueued collectives. This typically indicates a NCCL/CUDA API hang blocking the watchdog, and could be triggered by another thread holding the GIL inside a CUDA api, or other deadlock-prone behaviors.If you suspect the watchdog is not actually stuck and a longer timeout would help, you can either increase the timeout (TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC) to a larger value or disable the heartbeat monitor (TORCH_NCCL_ENABLE_MONITORING=0).If either of aforementioned helps, feel free to file an issue to PyTorch about the short timeout or false positive abort; otherwise, please attempt to debug the hang. workMetaList_.size() = 13
[default7]:[rank7]:[E ProcessGroupNCCL.cpp:1316] [PG 2 Rank 7] Heartbeat monitor timed out! Process will be terminated after dumping debug info. workMetaList_.size()=12
[default7]:[rank7]:[E ProcessGroupNCCL.cpp:1153] [PG 2 Rank 7] ProcessGroupNCCL preparing to dump debug info.
[default7]:[rank7]:[F ProcessGroupNCCL.cpp:1169] [PG 2 Rank 7] [PG 2 Rank 7] ProcessGroupNCCL's watchdog got stuck for 600 seconds without making progress in monitoring enqueued collectives. This typically indicates a NCCL/CUDA API hang blocking the watchdog, and could be triggered by another thread holding the GIL inside a CUDA api, or other deadlock-prone behaviors.If you suspect the watchdog is not actually stuck and a longer timeout would help, you can either increase the timeout (TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC) to a larger value or disable the heartbeat monitor (TORCH_NCCL_ENABLE_MONITORING=0).If either of aforementioned helps, feel free to file an issue to PyTorch about the short timeout or false positive abort; otherwise, please attempt to debug the hang. workMetaList_.size() = 12
[default6]:[rank6]:[E ProcessGroupNCCL.cpp:1316] [PG 2 Rank 6] Heartbeat monitor timed out! Process will be terminated after dumping debug info. workMetaList_.size()=13
[default6]:[rank6]:[E ProcessGroupNCCL.cpp:1153] [PG 2 Rank 6] ProcessGroupNCCL preparing to dump debug info.
[default6]:[rank6]:[F ProcessGroupNCCL.cpp:1169] [PG 2 Rank 6] [PG 2 Rank 6] ProcessGroupNCCL's watchdog got stuck for 600 seconds without making progress in monitoring enqueued collectives. This typically indicates a NCCL/CUDA API hang blocking the watchdog, and could be triggered by another thread holding the GIL inside a CUDA api, or other deadlock-prone behaviors.If you suspect the watchdog is not actually stuck and a longer timeout would help, you can either increase the timeout (TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC) to a larger value or disable the heartbeat monitor (TORCH_NCCL_ENABLE_MONITORING=0).If either of aforementioned helps, feel free to file an issue to PyTorch about the short timeout or false positive abort; otherwise, please attempt to debug the hang. workMetaList_.size() = 13
[default2]:[rank2]:[E ProcessGroupNCCL.cpp:1316] [PG 2 Rank 2] Heartbeat monitor timed out! Process will be terminated after dumping debug info. workMetaList_.size()=12
[default2]:[rank2]:[E ProcessGroupNCCL.cpp:1153] [PG 2 Rank 2] ProcessGroupNCCL preparing to dump debug info.
[default2]:[rank2]:[F ProcessGroupNCCL.cpp:1169] [PG 2 Rank 2] [PG 2 Rank 2] ProcessGroupNCCL's watchdog got stuck for 600 seconds without making progress in monitoring enqueued collectives. This typically indicates a NCCL/CUDA API hang blocking the watchdog, and could be triggered by another thread holding the GIL inside a CUDA api, or other deadlock-prone behaviors.If you suspect the watchdog is not actually stuck and a longer timeout would help, you can either increase the timeout (TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC) to a larger value or disable the heartbeat monitor (TORCH_NCCL_ENABLE_MONITORING=0).If either of aforementioned helps, feel free to file an issue to PyTorch about the short timeout or false positive abort; otherwise, please attempt to debug the hang. workMetaList_.size() = 12
[default0]:[rank0]:[E ProcessGroupNCCL.cpp:1316] [PG 2 Rank 0] Heartbeat monitor timed out! Process will be terminated after dumping debug info. workMetaList_.size()=5
[default0]:[rank0]:[E ProcessGroupNCCL.cpp:1153] [PG 2 Rank 0] ProcessGroupNCCL preparing to dump debug info.
[default0]:[rank0]:[F ProcessGroupNCCL.cpp:1169] [PG 2 Rank 0] [PG 2 Rank 0] ProcessGroupNCCL's watchdog got stuck for 600 seconds without making progress in monitoring enqueued collectives. This typically indicates a NCCL/CUDA API hang blocking the watchdog, and could be triggered by another thread holding the GIL inside a CUDA api, or other deadlock-prone behaviors.If you suspect the watchdog is not actually stuck and a longer timeout would help, you can either increase the timeout (TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC) to a larger value or disable the heartbeat monitor (TORCH_NCCL_ENABLE_MONITORING=0).If either of aforementioned helps, feel free to file an issue to PyTorch about the short timeout or false positive abort; otherwise, please attempt to debug the hang. workMetaList_.size() = 5
[default5]:[rank5]:[E ProcessGroupNCCL.cpp:1316] [PG 2 Rank 5] Heartbeat monitor timed out! Process will be terminated after dumping debug info. workMetaList_.size()=5
[default5]:[rank5]:[E ProcessGroupNCCL.cpp:1153] [PG 2 Rank 5] ProcessGroupNCCL preparing to dump debug info.
[default5]:[rank5]:[F ProcessGroupNCCL.cpp:1169] [PG 2 Rank 5] [PG 2 Rank 5] ProcessGroupNCCL's watchdog got stuck for 600 seconds without making progress in monitoring enqueued collectives. This typically indicates a NCCL/CUDA API hang blocking the watchdog, and could be triggered by another thread holding the GIL inside a CUDA api, or other deadlock-prone behaviors.If you suspect the watchdog is not actually stuck and a longer timeout would help, you can either increase the timeout (TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC) to a larger value or disable the heartbeat monitor (TORCH_NCCL_ENABLE_MONITORING=0).If either of aforementioned helps, feel free to file an issue to PyTorch about the short timeout or false positive abort; otherwise, please attempt to debug the hang. workMetaList_.size() = 5
[default6]:[rank46]: Traceback (most recent call last):
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default6]:[rank46]:     trainer.train(dataloader)
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default6]:[rank46]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default6]:[rank46]:     outputs = self.pipeline_engine.train_batch_iter(
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default6]:[rank46]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default6]:[rank46]:     output = model(**micro_batch)
[default6]:[rank46]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank46]:     return self._call_impl(*args, **kwargs)
[default6]:[rank46]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank46]:     return forward_call(*args, **kwargs)
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default6]:[rank46]:     sharded_logits = self.model(
[default6]:[rank46]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank46]:     return self._call_impl(*args, **kwargs)
[default6]:[rank46]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank46]:     return forward_call(*args, **kwargs)
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default6]:[rank46]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default6]:[rank46]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default6]:[rank46]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank46]:     return self._call_impl(*args, **kwargs)
[default6]:[rank46]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank46]:     return forward_call(*args, **kwargs)
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default6]:[rank46]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default6]:[rank46]:     pipeline_state.run_communication()
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default6]:[rank46]:     recv_activation_tensor = recv_activation()
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default6]:[rank46]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default6]:[rank46]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default6]:[rank46]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default6]:[rank46]:     dist.recv(
[default6]:[rank46]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default6]:[rank46]:     return func(*args, **kwargs)
[default6]:[rank46]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default6]:[rank46]:     pg.recv([tensor], group_src_rank, tag).wait()
[default6]:[rank46]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default6]:[rank46]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default6]:[rank46]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f3a75bf9897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default6]:[rank46]: frame #1: <unknown function> + 0x5b3a23e (0x7f3aaf71623e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f3aaf710c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f3aaf710f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f3aaf711fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3aaf6c6371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3aaf6c6371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3aaf6c6371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3aaf6c6371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f3a76ed3189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank46]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f3a76eda610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank46]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f3a76ef9978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank46]: frame #12: <unknown function> + 0x5adc309 (0x7f3aaf6b8309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #13: <unknown function> + 0x5ae6f10 (0x7f3aaf6c2f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f3aaf6c2fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #15: <unknown function> + 0x5124446 (0x7f3aaed00446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #16: <unknown function> + 0x1acf4b8 (0x7f3aab6ab4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #17: <unknown function> + 0x5aee004 (0x7f3aaf6ca004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #18: <unknown function> + 0x5af36b5 (0x7f3aaf6cf6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #19: <unknown function> + 0xd2631e (0x7f3ac22b931e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:[rank46]: frame #20: <unknown function> + 0x47def4 (0x7f3ac1a10ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:[rank46]: frame #21: <unknown function> + 0x1445a6 (0x55c44d6395a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55c44d632a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #23: <unknown function> + 0x150866 (0x55c44d645866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55c44d62e142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55c44d639a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #26: PyObject_Call + 0xbc (0x55c44d645f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55c44d62c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55c44d639a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55c44d62a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #30: <unknown function> + 0x150582 (0x55c44d645582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55c44d62a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #32: <unknown function> + 0x150582 (0x55c44d645582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55c44d62a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #34: <unknown function> + 0x150582 (0x55c44d645582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55c44d62a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55c44d631f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55c44d643c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #38: <unknown function> + 0x211239 (0x55c44d706239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55c44d632a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55c44d62e3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55c44d639a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55c44d629c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55c44d639a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55c44d62a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #45: <unknown function> + 0x150582 (0x55c44d645582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #46: PyObject_Call + 0xbc (0x55c44d645f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55c44d62c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #48: <unknown function> + 0x150582 (0x55c44d645582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #49: PyObject_Call + 0xbc (0x55c44d645f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55c44d62c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55c44d639a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55c44d632007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55c44d643c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #54: <unknown function> + 0x211239 (0x55c44d706239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #55: PyObject_Call + 0x207 (0x55c44d646067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55c44d62c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #57: <unknown function> + 0x150582 (0x55c44d645582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55c44d62a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #59: <unknown function> + 0x150582 (0x55c44d645582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #60: PyObject_Call + 0xbc (0x55c44d645f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55c44d62c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #62: <unknown function> + 0x150582 (0x55c44d645582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #63: PyObject_Call + 0xbc (0x55c44d645f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default7]:[rank55]: Traceback (most recent call last):
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default7]:[rank55]:     trainer.train(dataloader)
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default7]:[rank55]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default6]:[rank54]: Traceback (most recent call last):
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default4]:[rank52]: Traceback (most recent call last):
[default5]:[rank53]: Traceback (most recent call last):
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default5]:[rank53]:     trainer.train(dataloader)
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default6]:[rank54]:     trainer.train(dataloader)
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default6]:[rank54]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default7]:[rank55]:     outputs = self.pipeline_engine.train_batch_iter(
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default5]:[rank53]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default4]:[rank52]:     trainer.train(dataloader)
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default6]:[rank54]:     outputs = self.pipeline_engine.train_batch_iter(
[default7]:[rank55]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default7]:[rank55]:     output = model(**micro_batch)
[default5]:[rank53]:     outputs = self.pipeline_engine.train_batch_iter(
[default6]:[rank54]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default7]:[rank55]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default4]:[rank52]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default7]:[rank55]:     return self._call_impl(*args, **kwargs)
[default5]:[rank53]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default6]:[rank54]:     output = model(**micro_batch)
[default6]:[rank54]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default6]:[rank54]:     return self._call_impl(*args, **kwargs)
[default6]:[rank54]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank54]:     return forward_call(*args, **kwargs)
[default5]:[rank53]:     output = model(**micro_batch)
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default7]:[rank55]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default6]:[rank54]:     sharded_logits = self.model(
[default5]:[rank53]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank52]:     outputs = self.pipeline_engine.train_batch_iter(
[default6]:[rank54]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank54]:     return self._call_impl(*args, **kwargs)
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default5]:[rank53]:     return self._call_impl(*args, **kwargs)
[default7]:[rank55]:     return forward_call(*args, **kwargs)
[default6]:[rank54]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank52]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default6]:[rank54]:     return forward_call(*args, **kwargs)
[default5]:[rank53]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default4]:[rank52]:     output = model(**micro_batch)
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default5]:[rank53]:     return forward_call(*args, **kwargs)
[default6]:[rank54]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default4]:[rank52]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default7]:[rank55]:     sharded_logits = self.model(
[default4]:[rank52]:     return self._call_impl(*args, **kwargs)
[default7]:[rank55]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default4]:[rank52]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank53]:     sharded_logits = self.model(
[default5]:[rank53]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank52]:     return forward_call(*args, **kwargs)
[default6]:[rank54]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default7]:[rank55]:     return self._call_impl(*args, **kwargs)
[default5]:[rank53]:     return self._call_impl(*args, **kwargs)
[default6]:[rank54]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank55]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank53]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank54]:     return self._call_impl(*args, **kwargs)
[default7]:[rank55]:     return forward_call(*args, **kwargs)
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default5]:[rank53]:     return forward_call(*args, **kwargs)
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default6]:[rank54]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default4]:[rank52]:     sharded_logits = self.model(
[default5]:[rank53]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default4]:[rank52]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank55]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank54]:     return forward_call(*args, **kwargs)
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default6]:[rank54]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default5]:[rank53]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default4]:[rank52]:     return self._call_impl(*args, **kwargs)
[default5]:[rank53]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default7]:[rank55]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default5]:[rank53]:     return self._call_impl(*args, **kwargs)
[default4]:[rank52]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank52]:     return forward_call(*args, **kwargs)
[default5]:[rank53]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default7]:[rank55]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank53]:     return forward_call(*args, **kwargs)
[default4]:[rank52]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank54]:     pipeline_state.run_communication()
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default5]:[rank53]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default4]:[rank52]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default7]:[rank55]:     return self._call_impl(*args, **kwargs)
[default7]:[rank55]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default6]:[rank54]:     recv_activation_tensor = recv_activation()
[default5]:[rank53]:     pipeline_state.run_communication()
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default6]:[rank54]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default4]:[rank52]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default7]:[rank55]:     return forward_call(*args, **kwargs)
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default5]:[rank53]:     recv_activation_tensor = recv_activation()
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default4]:[rank52]:     return self._call_impl(*args, **kwargs)
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default4]:[rank52]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank55]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default5]:[rank53]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default6]:[rank54]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default5]:[rank53]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default6]:[rank54]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default4]:[rank52]:     return forward_call(*args, **kwargs)
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default5]:[rank53]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default7]:[rank55]:     pipeline_state.run_communication()
[default6]:[rank54]:     dist.recv(
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default7]:[rank55]:     recv_activation_tensor = recv_activation()
[default5]:[rank53]:     dist.recv(
[default6]:[rank54]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default5]:[rank53]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default6]:[rank54]:     return func(*args, **kwargs)
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default6]:[rank54]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default5]:[rank53]:     return func(*args, **kwargs)
[default7]:[rank55]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default6]:[rank54]:     pg.recv([tensor], group_src_rank, tag).wait()
[default4]:[rank52]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default7]:[rank55]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default6]:[rank54]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default5]:[rank53]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default6]:[rank54]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default5]:[rank53]:     pg.recv([tensor], group_src_rank, tag).wait()
[default7]:[rank55]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default4]:[rank52]:     pipeline_state.run_communication()
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default6]:[rank54]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7efbda6f6897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default5]:[rank53]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default6]:[rank54]: frame #1: <unknown function> + 0x5b3a23e (0x7efc1421323e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default4]:[rank52]:     recv_activation_tensor = recv_activation()
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default6]:[rank54]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7efc1420dc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fc56c81e897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default4]:[rank52]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default5]:[rank53]: frame #1: <unknown function> + 0x5b3a23e (0x7fc5a633b23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default6]:[rank54]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7efc1420df82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fc5a6335c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank55]:     dist.recv(
[default5]:[rank53]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fc5a6335f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default7]:[rank55]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default5]:[rank53]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fc5a6336fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7efc1420efd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank55]:     return func(*args, **kwargs)
[default5]:[rank53]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc5a62eb371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7efc141c3371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank55]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default5]:[rank53]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc5a62eb371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank55]:     pg.recv([tensor], group_src_rank, tag).wait()
[default5]:[rank53]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc5a62eb371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7efc141c3371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default5]:[rank53]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc5a62eb371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default5]:[rank53]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fc56daf8189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank55]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default4]:[rank52]:     dist.recv(
[default5]:[rank53]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fc56daff610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank55]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default4]:[rank52]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default6]:[rank54]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7efc141c3371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank55]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f81ab329897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default5]:[rank53]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fc56db1e978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank54]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7efc141c3371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank55]: frame #1: <unknown function> + 0x5b3a23e (0x7f81e4e4623e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #12: <unknown function> + 0x5adc309 (0x7fc5a62dd309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]:     return func(*args, **kwargs)
[default4]:[rank52]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default6]:[rank54]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7efbdb9d0189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank54]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7efbdb9d7610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank52]:     pg.recv([tensor], group_src_rank, tag).wait()
[default5]:[rank53]: frame #13: <unknown function> + 0x5ae6f10 (0x7fc5a62e7f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7efbdb9f6978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank52]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default6]:[rank54]: frame #12: <unknown function> + 0x5adc309 (0x7efc141b5309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]: frame #13: <unknown function> + 0x5ae6f10 (0x7efc141bff10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default4]:[rank52]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fd28ad33897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default7]:[rank55]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f81e4e40c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fc5a62e7fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]: frame #14: <unknown function> + 0x5ae6fa5 (0x7efc141bffa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]: frame #1: <unknown function> + 0x5b3a23e (0x7fd2c485023e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank55]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f81e4e40f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]: frame #15: <unknown function> + 0x5124446 (0x7efc137fd446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fd2c484ac87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #15: <unknown function> + 0x5124446 (0x7fc5a5925446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]: frame #16: <unknown function> + 0x1acf4b8 (0x7efc101a84b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #16: <unknown function> + 0x1acf4b8 (0x7fc5a22d04b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fd2c484af82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]: frame #17: <unknown function> + 0x5aee004 (0x7efc141c7004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #17: <unknown function> + 0x5aee004 (0x7fc5a62ef004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fd2c484bfd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]: frame #18: <unknown function> + 0x5af36b5 (0x7efc141cc6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank55]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f81e4e41fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd2c4800371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]: frame #19: <unknown function> + 0xd2631e (0x7efc26db631e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:[rank52]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd2c4800371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd2c4800371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #18: <unknown function> + 0x5af36b5 (0x7fc5a62f46b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank55]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f81e4df6371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank55]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f81e4df6371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]: frame #20: <unknown function> + 0x47def4 (0x7efc2650def4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:[rank54]: frame #21: <unknown function> + 0x1445a6 (0x564d47a045a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #22: _PyObject_MakeTpCall + 0x26b (0x564d479fda6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #19: <unknown function> + 0xd2631e (0x7fc5b8ede31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:[rank55]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f81e4df6371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd2c4800371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]: frame #23: <unknown function> + 0x150866 (0x564d47a10866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f81e4df6371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #20: <unknown function> + 0x47def4 (0x7fc5b8635ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:[rank54]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x564d479f9142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #25: _PyFunction_Vectorcall + 0x6c (0x564d47a04a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f81ac603189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank53]: frame #21: <unknown function> + 0x1445a6 (0x55877313e5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #26: PyObject_Call + 0xbc (0x564d47a10f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fd28c00d189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank55]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f81ac60a610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank52]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fd28c014610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank54]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x564d479f72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f81ac629978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank53]: frame #22: _PyObject_MakeTpCall + 0x26b (0x558773137a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #28: _PyFunction_Vectorcall + 0x6c (0x564d47a04a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fd28c033978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank53]: frame #23: <unknown function> + 0x150866 (0x55877314a866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x564d479f58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #30: <unknown function> + 0x150582 (0x564d47a10582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #12: <unknown function> + 0x5adc309 (0x7fd2c47f2309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank55]: frame #12: <unknown function> + 0x5adc309 (0x7f81e4de8309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]: frame #13: <unknown function> + 0x5ae6f10 (0x7fd2c47fcf10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x558773133142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55877313ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x564d479f58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #26: PyObject_Call + 0xbc (0x55877314af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #32: <unknown function> + 0x150582 (0x564d47a10582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5587731312b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #13: <unknown function> + 0x5ae6f10 (0x7f81e4df2f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55877313ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x564d479f58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55877312f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fd2c47fcfa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank55]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f81e4df2fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]: frame #34: <unknown function> + 0x150582 (0x564d47a10582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x564d479f58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #15: <unknown function> + 0x5124446 (0x7fd2c3e3a446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank55]: frame #15: <unknown function> + 0x5124446 (0x7f81e4430446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #30: <unknown function> + 0x150582 (0x55877314a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #16: <unknown function> + 0x1acf4b8 (0x7fd2c07e54b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x564d479fcf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55877312f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #17: <unknown function> + 0x5aee004 (0x7fd2c4804004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #32: <unknown function> + 0x150582 (0x55877314a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55877312f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #16: <unknown function> + 0x1acf4b8 (0x7f81e0ddb4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]: frame #37: _PyObject_Call_Prepend + 0x69 (0x564d47a0ec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #38: <unknown function> + 0x211239 (0x564d47ad1239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #18: <unknown function> + 0x5af36b5 (0x7fd2c48096b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank55]: frame #17: <unknown function> + 0x5aee004 (0x7f81e4dfa004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #34: <unknown function> + 0x150582 (0x55877314a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55877312f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #39: _PyObject_MakeTpCall + 0x26b (0x564d479fda6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x564d479f93e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x558773136f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #18: <unknown function> + 0x5af36b5 (0x7f81e4dff6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]: frame #41: _PyFunction_Vectorcall + 0x6c (0x564d47a04a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #19: <unknown function> + 0xd2631e (0x7f81f79e931e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:[rank54]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x564d479f4c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #19: <unknown function> + 0xd2631e (0x7fd2d73f331e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:[rank55]: frame #20: <unknown function> + 0x47def4 (0x7f81f7140ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default5]:[rank53]: frame #37: _PyObject_Call_Prepend + 0x69 (0x558773148c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #20: <unknown function> + 0x47def4 (0x7fd2d6b4aef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:[rank55]: frame #21: <unknown function> + 0x1445a6 (0x5563ae0f15a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #22: _PyObject_MakeTpCall + 0x26b (0x5563ae0eaa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #43: _PyFunction_Vectorcall + 0x6c (0x564d47a04a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #21: <unknown function> + 0x1445a6 (0x55737b75c5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55737b755a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #38: <unknown function> + 0x211239 (0x55877320b239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #23: <unknown function> + 0x150866 (0x5563ae0fd866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #39: _PyObject_MakeTpCall + 0x26b (0x558773137a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x564d479f58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x5563ae0e6142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5587731333e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #45: <unknown function> + 0x150582 (0x564d47a10582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #46: PyObject_Call + 0xbc (0x564d47a10f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #23: <unknown function> + 0x150866 (0x55737b768866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55877313ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55877312ec5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #25: _PyFunction_Vectorcall + 0x6c (0x5563ae0f1a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55737b751142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x564d479f72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #48: <unknown function> + 0x150582 (0x564d47a10582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55737b75ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #26: PyObject_Call + 0xbc (0x5563ae0fdf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55877313ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #26: PyObject_Call + 0xbc (0x55737b768f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5563ae0e42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #28: _PyFunction_Vectorcall + 0x6c (0x5563ae0f1a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55877312f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55737b74f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #49: PyObject_Call + 0xbc (0x564d47a10f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5563ae0e28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55737b75ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x564d479f72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #45: <unknown function> + 0x150582 (0x55877314a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55737b74d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #51: _PyFunction_Vectorcall + 0x6c (0x564d47a04a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x564d479fd007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #46: PyObject_Call + 0xbc (0x55877314af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #53: _PyObject_Call_Prepend + 0x69 (0x564d47a0ec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #30: <unknown function> + 0x150582 (0x55737b768582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55737b74d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #30: <unknown function> + 0x150582 (0x5563ae0fd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #54: <unknown function> + 0x211239 (0x564d47ad1239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5563ae0e28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #55: PyObject_Call + 0x207 (0x564d47a11067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #32: <unknown function> + 0x150582 (0x55737b768582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55737b74d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5587731312b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #32: <unknown function> + 0x150582 (0x5563ae0fd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x564d479f72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #48: <unknown function> + 0x150582 (0x55877314a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #49: PyObject_Call + 0xbc (0x55877314af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #57: <unknown function> + 0x150582 (0x564d47a10582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5587731312b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55877313ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #34: <unknown function> + 0x150582 (0x55737b768582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55737b74d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5563ae0e28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x558773137007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x564d479f58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #53: _PyObject_Call_Prepend + 0x69 (0x558773148c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #34: <unknown function> + 0x150582 (0x5563ae0fd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #59: <unknown function> + 0x150582 (0x564d47a10582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #54: <unknown function> + 0x211239 (0x55877320b239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5563ae0e28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55737b754f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x5563ae0e9f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #55: PyObject_Call + 0x207 (0x55877314b067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55737b766c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #38: <unknown function> + 0x211239 (0x55737b829239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #60: PyObject_Call + 0xbc (0x564d47a10f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5587731312b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55737b755a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x564d479f72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55737b7513e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #62: <unknown function> + 0x150582 (0x564d47a10582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55737b75ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #63: PyObject_Call + 0xbc (0x564d47a10f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #57: <unknown function> + 0x150582 (0x55877314a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55737b74cc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #37: _PyObject_Call_Prepend + 0x69 (0x5563ae0fbc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55877312f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55737b75ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #38: <unknown function> + 0x211239 (0x5563ae1be239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #39: _PyObject_MakeTpCall + 0x26b (0x5563ae0eaa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55737b74d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5563ae0e63e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #59: <unknown function> + 0x150582 (0x55877314a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default7]:[rank55]: frame #41: _PyFunction_Vectorcall + 0x6c (0x5563ae0f1a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #45: <unknown function> + 0x150582 (0x55737b768582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #60: PyObject_Call + 0xbc (0x55877314af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x5563ae0e1c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #46: PyObject_Call + 0xbc (0x55737b768f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55737b74f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5587731312b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #43: _PyFunction_Vectorcall + 0x6c (0x5563ae0f1a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #48: <unknown function> + 0x150582 (0x55737b768582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5563ae0e28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #49: PyObject_Call + 0xbc (0x55737b768f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #45: <unknown function> + 0x150582 (0x5563ae0fd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55737b74f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55737b75ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #62: <unknown function> + 0x150582 (0x55877314a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #46: PyObject_Call + 0xbc (0x5563ae0fdf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55737b755007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5563ae0e42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #48: <unknown function> + 0x150582 (0x5563ae0fd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55737b766c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #63: PyObject_Call + 0xbc (0x55877314af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #49: PyObject_Call + 0xbc (0x5563ae0fdf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #54: <unknown function> + 0x211239 (0x55737b829239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default4]:[rank52]: frame #55: PyObject_Call + 0x207 (0x55737b769067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5563ae0e42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55737b74f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #57: <unknown function> + 0x150582 (0x55737b768582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55737b74d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #59: <unknown function> + 0x150582 (0x55737b768582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #60: PyObject_Call + 0xbc (0x55737b768f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55737b74f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #51: _PyFunction_Vectorcall + 0x6c (0x5563ae0f1a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x5563ae0ea007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #53: _PyObject_Call_Prepend + 0x69 (0x5563ae0fbc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #54: <unknown function> + 0x211239 (0x5563ae1be239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #55: PyObject_Call + 0x207 (0x5563ae0fe067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5563ae0e42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #57: <unknown function> + 0x150582 (0x5563ae0fd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #62: <unknown function> + 0x150582 (0x55737b768582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #63: PyObject_Call + 0xbc (0x55737b768f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5563ae0e28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default7]:[rank55]: frame #59: <unknown function> + 0x150582 (0x5563ae0fd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #60: PyObject_Call + 0xbc (0x5563ae0fdf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5563ae0e42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #62: <unknown function> + 0x150582 (0x5563ae0fd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #63: PyObject_Call + 0xbc (0x5563ae0fdf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default0]:[rank56]: Traceback (most recent call last):
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default0]:[rank56]:     trainer.train(dataloader)
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default0]:[rank56]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default0]:[rank56]:     outputs = self.pipeline_engine.train_batch_iter(
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default0]:[rank56]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default0]:[rank56]:     output = model(**micro_batch)
[default0]:[rank56]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank56]:     return self._call_impl(*args, **kwargs)
[default0]:[rank56]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank56]:     return forward_call(*args, **kwargs)
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default0]:[rank56]:     sharded_logits = self.model(
[default0]:[rank56]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank56]:     return self._call_impl(*args, **kwargs)
[default0]:[rank56]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank56]:     return forward_call(*args, **kwargs)
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default0]:[rank56]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default0]:[rank56]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default0]:[rank56]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank56]:     return self._call_impl(*args, **kwargs)
[default0]:[rank56]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank56]:     return forward_call(*args, **kwargs)
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default0]:[rank56]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default0]:[rank56]:     pipeline_state.run_communication()
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default0]:[rank56]:     recv_activation_tensor = recv_activation()
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default0]:[rank56]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default0]:[rank56]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default0]:[rank56]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default0]:[rank56]:     dist.recv(
[default0]:[rank56]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default1]:[rank57]: Traceback (most recent call last):
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default1]:[rank57]:     trainer.train(dataloader)
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default1]:[rank57]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default0]:[rank56]:     return func(*args, **kwargs)
[default0]:[rank56]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default1]:[rank57]:     outputs = self.pipeline_engine.train_batch_iter(
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default1]:[rank57]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default0]:[rank56]:     pg.recv([tensor], group_src_rank, tag).wait()
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default1]:[rank57]:     output = model(**micro_batch)
[default0]:[rank56]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default0]:[rank56]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default1]:[rank57]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank57]:     return self._call_impl(*args, **kwargs)
[default1]:[rank57]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank57]:     return forward_call(*args, **kwargs)
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default1]:[rank57]:     sharded_logits = self.model(
[default0]:[rank56]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fdfd6744897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default0]:[rank56]: frame #1: <unknown function> + 0x5b3a23e (0x7fe01026123e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank57]:     return self._call_impl(*args, **kwargs)
[default1]:[rank57]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank57]:     return forward_call(*args, **kwargs)
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default1]:[rank57]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank38]: Traceback (most recent call last):
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default6]:[rank38]:     trainer.train(dataloader)
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default6]:[rank38]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default6]:[rank38]:     outputs = self.pipeline_engine.train_batch_iter(
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default6]:[rank38]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanot[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default1]:[rank57]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default0]:[rank56]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fe01025bc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank56]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fe01025bf82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
ron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default1]:[rank57]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank38]:     output = model(**micro_batch)
[default6]:[rank38]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank38]:     return self._call_impl(*args, **kwargs)
[default6]:[rank38]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank38]:     return forward_call(*args, **kwargs)
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default6]:[rank38]:     sharded_logits = self.model(
[default6]:[rank38]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank38]:     return self._call_impl(*args, **kwargs)
[default6]:[rank38]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster[default1]:[rank57]:     return self._call_impl(*args, **kwargs)
[default1]:[rank57]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank57]:     return forward_call(*args, **kwargs)
/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank38]:     return forward_call(*args, **kwargs)
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default6]:[rank38]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default6]:[rank38]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default6]:[rank38]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank38]:     return self._call_impl(*args, **kwargs)
[default6]:[rank38]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in [default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default1]:[rank57]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default1]:[rank57]:     pipeline_state.run_communication()
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
_call_impl
[default6]:[rank38]:     return forward_call(*args, **kwargs)
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default6]:[rank38]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default6]:[rank38]:     pipeline_state.run_communication()
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default6]:[rank38]:     recv_activation_tensor = recv_activation()
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default6]:[rank38]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.fro[default0]:[rank56]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fe01025cfd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
m_rank)[0]
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default6]:[rank38]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default6]:[rank38]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default0]:[rank56]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe010211371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]:     dist.recv(
[default6]:[rank38]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default6]:[rank38]:     return func(*args, **kwargs)
[default6]:[rank38]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default6]:[rank38]:     pg.recv([tensor], group_src_rank, tag).wait()
[default6]:[rank38]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default6]:[rank38]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default6]:[rank38]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f7fb409b897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10[default0]:[rank56]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe010211371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]:     recv_activation_tensor = recv_activation()
[default0]:[rank56]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe010211371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
/site-packages/torch/lib/libc10.so)
[default6]:[rank38]: frame #1: <unknown function> + 0x5b3a23e (0x7f7fedbb823e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f7fedbb2c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f7fedbb2f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f7fedbb3fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7fedb68371 in /fsx/ferdinandmom[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7fedb68371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7fedb68371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7fedb68371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f7fb5375189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank56]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe010211371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f7fb537c610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank38]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f7fb539b978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank38]: frame #12: <unknown function> + 0x5adc309 (0x7f7fedb5a309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #13: <unknown function> + 0x5ae6f10 (0x7f7fedb64f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default6]:[rank38]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f7fedb64fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default6]:[rank38]: frame #15: <unknown function> + 0x5124446 (0x7f7fed1a2446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default6]:[rank38]: frame #16: <unknown function> + 0x1acf4b8 (0x7f7fe9b4d4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #17: <unknown function> + 0x5aee004 (0x7f7fedb6c004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #18: <unknown function> + 0x5af36b5 (0x7f7fedb716b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default1]:[rank57]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default6]:[rank38]: frame #19: <unknown function> + 0xd2631e (0x7f800075b31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:[rank38]: frame #20: <unknown function> + 0x47def4 (0x7f7fffeb2ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default1]:[rank57]:     dist.recv(
[default6]:[rank38]: frame #21: <unknown function> + 0x1445a6 (0x556897db35a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #22: _PyObject_MakeTpCall + 0x26b (0x556897daca6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #23: <unknown function> + 0x150866 (0x556897dbf866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x556897da8142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #25: _PyFunction_Vectorcall + 0x6c (0x556897db3a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #26: PyObject_Call + 0xbc (0x556897dbff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x556897da62b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster[default0]:[rank56]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fdfd7a1e189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank56]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fdfd7a25610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank57]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default0]:[rank56]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fdfd7a44978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
/bin/python3.10)
[default6]:[rank38]: frame #28: _PyFunction_Vectorcall + 0x6c (0x556897db3a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x556897da48fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #30: <unknown function> + 0x150582 (0x556897dbf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x556897da48fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #32: <unknown function> + 0x150582 (0x556897dbf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x556897da48fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]:     return func(*args, **kwargs)
[default6]:[rank38]: frame #34: <unknown function> + 0x150582 (0x556897dbf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x556897da48fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x556897dabf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #37: _PyObject_Call_Prepend + 0x69 (0x556897dbdc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default0]:[rank56]: frame #12: <unknown function> + 0x5adc309 (0x7fe010203309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #38: <unknown function> + 0x211239 (0x556897e80239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #39: _PyObject_MakeTpCall + 0x26b (0x556897daca6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #13: <unknown function> + 0x5ae6f10 (0x7fe01020df10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x556897da83e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #41: _PyFunction_Vectorcall + 0x6c (0x556897db3a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x556897da3c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #43: _PyFunction_Vectorcall + 0x6c (0x556897db3a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x556897da48fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]:     pg.recv([tensor], group_src_rank, tag).wait()
[default6]:[rank38]: frame #45: <unknown function> + 0x150582 (0x556897dbf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #46: PyObject_Call + 0xbc (0x556897dbff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fe01020dfa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x556897da62b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #48: <unknown function> + 0x150582 (0x556897dbf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default6]:[rank38]: frame #49: PyObject_Call + 0xbc (0x556897dbff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x556897da62b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default6]:[rank38]: frame #51: _PyFunction_Vectorcall + 0x6c (0x556897db3a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x556897dac007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #53: _PyObject_Call_Prepend + 0x69 (0x556897dbdc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f0255c22897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default6]:[rank38]: frame #54: <unknown function> + 0x211239 (0x556897e80239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #55: PyObject_Call + 0x207 (0x556897dc0067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #1: <unknown function> + 0x5b3a23e (0x7f028f73f23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x556897da62b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #57: <unknown function> + 0x150582 (0x556897dbf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f028f739c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x556897da48fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #59: <unknown function> + 0x150582 (0x556897dbf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #60: PyObject_Call + 0xbc (0x556897dbff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #15: <unknown function> + 0x5124446 (0x7fe00f84b446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank56]: frame #16: <unknown function> + 0x1acf4b8 (0x7fe00c1f64b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank56]: frame #17: <unknown function> + 0x5aee004 (0x7fe010215004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x556897da62b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #62: <unknown function> + 0x150582 (0x556897dbf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #18: <unknown function> + 0x5af36b5 (0x7fe01021a6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #63: PyObject_Call + 0xbc (0x556897dbff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default0]:[rank56]: frame #19: <unknown function> + 0xd2631e (0x7fe022e0431e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default0]:[rank56]: frame #20: <unknown function> + 0x47def4 (0x7fe02255bef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default0]:[rank56]: frame #21: <unknown function> + 0x1445a6 (0x55e95b6555a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f028f739f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f028f73afd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f028f6ef371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f028f6ef371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f028f6ef371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f028f6ef371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f0256efc189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank57]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f0256f03610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank57]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f0256f22978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank57]: frame #12: <unknown function> + 0x5adc309 (0x7f028f6e1309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]: frame #13: <unknown function> + 0x5ae6f10 (0x7f028f6ebf10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank56]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55e95b64ea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #23: <unknown function> + 0x150866 (0x55e95b661866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55e95b64a142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55e95b655a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #26: PyObject_Call + 0xbc (0x55e95b661f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f028f6ebfa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]: frame #15: <unknown function> + 0x5124446 (0x7f028ed29446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]: frame #16: <unknown function> + 0x1acf4b8 (0x7f028b6d44b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank56]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55e95b6482b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55e95b655a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55e95b6468fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #17: <unknown function> + 0x5aee004 (0x7f028f6f3004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank56]: frame #30: <unknown function> + 0x150582 (0x55e95b661582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55e95b6468fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #32: <unknown function> + 0x150582 (0x55e95b661582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55e95b6468fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #34: <unknown function> + 0x150582 (0x55e95b661582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55e95b6468fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55e95b64df50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #18: <unknown function> + 0x5af36b5 (0x7f028f6f86b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]: frame #19: <unknown function> + 0xd2631e (0x7f02a22e231e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default1]:[rank57]: frame #20: <unknown function> + 0x47def4 (0x7f02a1a39ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default0]:[rank56]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55e95b65fc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #21: <unknown function> + 0x1445a6 (0x562c75f825a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #22: _PyObject_MakeTpCall + 0x26b (0x562c75f7ba6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #23: <unknown function> + 0x150866 (0x562c75f8e866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x562c75f77142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #25: _PyFunction_Vectorcall + 0x6c (0x562c75f82a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #26: PyObject_Call + 0xbc (0x562c75f8ef1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x562c75f752b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #28: _PyFunction_Vectorcall + 0x6c (0x562c75f82a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x562c75f738fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #38: <unknown function> + 0x211239 (0x55e95b722239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #30: <unknown function> + 0x150582 (0x562c75f8e582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x562c75f738fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55e95b64ea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55e95b64a3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55e95b655a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55e95b645c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55e95b655a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55e95b6468fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #45: <unknown function> + 0x150582 (0x55e95b661582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #46: PyObject_Call + 0xbc (0x55e95b661f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55e95b6482b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #48: <unknown function> + 0x150582 (0x55e95b661582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #49: PyObject_Call + 0xbc (0x55e95b661f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55e95b6482b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #32: <unknown function> + 0x150582 (0x562c75f8e582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x562c75f738fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #34: <unknown function> + 0x150582 (0x562c75f8e582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x562c75f738fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x562c75f7af50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #37: _PyObject_Call_Prepend + 0x69 (0x562c75f8cc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55e95b655a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #38: <unknown function> + 0x211239 (0x562c7604f239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #39: _PyObject_MakeTpCall + 0x26b (0x562c75f7ba6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x562c75f773e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55e95b64e007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55e95b65fc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #41: _PyFunction_Vectorcall + 0x6c (0x562c75f82a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x562c75f72c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #54: <unknown function> + 0x211239 (0x55e95b722239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #43: _PyFunction_Vectorcall + 0x6c (0x562c75f82a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x562c75f738fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #45: <unknown function> + 0x150582 (0x562c75f8e582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #46: PyObject_Call + 0xbc (0x562c75f8ef1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #55: PyObject_Call + 0x207 (0x55e95b662067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x562c75f752b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #48: <unknown function> + 0x150582 (0x562c75f8e582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55e95b6482b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #49: PyObject_Call + 0xbc (0x562c75f8ef1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #57: <unknown function> + 0x150582 (0x55e95b661582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55e95b6468fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #59: <unknown function> + 0x150582 (0x55e95b661582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x562c75f752b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #60: PyObject_Call + 0xbc (0x55e95b661f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55e95b6482b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #62: <unknown function> + 0x150582 (0x55e95b661582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #51: _PyFunction_Vectorcall + 0x6c (0x562c75f82a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #63: PyObject_Call + 0xbc (0x55e95b661f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default1]:[rank57]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x562c75f7b007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #53: _PyObject_Call_Prepend + 0x69 (0x562c75f8cc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #54: <unknown function> + 0x211239 (0x562c7604f239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #55: PyObject_Call + 0x207 (0x562c75f8f067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x562c75f752b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #57: <unknown function> + 0x150582 (0x562c75f8e582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x562c75f738fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #59: <unknown function> + 0x150582 (0x562c75f8e582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #60: PyObject_Call + 0xbc (0x562c75f8ef1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x562c75f752b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #62: <unknown function> + 0x150582 (0x562c75f8e582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #63: PyObject_Call + 0xbc (0x562c75f8ef1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default7]:[rank39]: Traceback (most recent call last):
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default7]:[rank39]:     trainer.train(dataloader)
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default7]:[rank39]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default7]:[rank39]:     outputs = self.pipeline_engine.train_batch_iter(
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default7]:[rank39]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default7]:[rank39]:     output = model(**micro_batch)
[default7]:[rank39]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank39]:     return self._call_impl(*args, **kwargs)
[default7]:[rank39]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank39]:     return forward_call(*args, **kwargs)
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default7]:[rank39]:     sharded_logits = self.model(
[default7]:[rank39]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank39]:     return self._call_impl(*args, **kwargs)
[default7]:[rank39]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank39]:     return forward_call(*args, **kwargs)
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default7]:[rank39]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default7]:[rank39]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default7]:[rank39]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank39]:     return self._call_impl(*args, **kwargs)
[default7]:[rank39]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank39]:     return forward_call(*args, **kwargs)
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default7]:[rank39]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default7]:[rank39]:     pipeline_state.run_communication()
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default7]:[rank39]:     recv_activation_tensor = recv_activation()
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default7]:[rank39]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default7]:[rank39]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default7]:[rank39]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default7]:[rank39]:     dist.recv(
[default7]:[rank39]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default7]:[rank39]:     return func(*args, **kwargs)
[default7]:[rank39]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default7]:[rank39]:     pg.recv([tensor], group_src_rank, tag).wait()
[default7]:[rank39]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default7]:[rank39]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default7]:[rank39]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fd3a99fc897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default7]:[rank39]: frame #1: <unknown function> + 0x5b3a23e (0x7fd3e351923e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fd3e3513c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fd3e3513f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fd3e3514fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd3e34c9371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd3e34c9371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd3e34c9371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd3e34c9371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fd3aacd6189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank39]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fd3aacdd610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank39]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fd3aacfc978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank39]: frame #12: <unknown function> + 0x5adc309 (0x7fd3e34bb309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]: frame #13: <unknown function> + 0x5ae6f10 (0x7fd3e34c5f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fd3e34c5fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]: frame #15: <unknown function> + 0x5124446 (0x7fd3e2b03446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]: frame #16: <unknown function> + 0x1acf4b8 (0x7fd3df4ae4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]: frame #17: <unknown function> + 0x5aee004 (0x7fd3e34cd004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]: frame #18: <unknown function> + 0x5af36b5 (0x7fd3e34d26b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]: frame #19: <unknown function> + 0xd2631e (0x7fd3f60bc31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:[rank39]: frame #20: <unknown function> + 0x47def4 (0x7fd3f5813ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:[rank39]: frame #21: <unknown function> + 0x1445a6 (0x559aee9cd5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #22: _PyObject_MakeTpCall + 0x26b (0x559aee9c6a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #23: <unknown function> + 0x150866 (0x559aee9d9866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x559aee9c2142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #25: _PyFunction_Vectorcall + 0x6c (0x559aee9cda2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #26: PyObject_Call + 0xbc (0x559aee9d9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x559aee9c02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #28: _PyFunction_Vectorcall + 0x6c (0x559aee9cda2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x559aee9be8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #30: <unknown function> + 0x150582 (0x559aee9d9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x559aee9be8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #32: <unknown function> + 0x150582 (0x559aee9d9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x559aee9be8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #34: <unknown function> + 0x150582 (0x559aee9d9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x559aee9be8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x559aee9c5f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #37: _PyObject_Call_Prepend + 0x69 (0x559aee9d7c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #38: <unknown function> + 0x211239 (0x559aeea9a239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #39: _PyObject_MakeTpCall + 0x26b (0x559aee9c6a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x559aee9c23e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #41: _PyFunction_Vectorcall + 0x6c (0x559aee9cda2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x559aee9bdc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #43: _PyFunction_Vectorcall + 0x6c (0x559aee9cda2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x559aee9be8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #45: <unknown function> + 0x150582 (0x559aee9d9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #46: PyObject_Call + 0xbc (0x559aee9d9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x559aee9c02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #48: <unknown function> + 0x150582 (0x559aee9d9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #49: PyObject_Call + 0xbc (0x559aee9d9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x559aee9c02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #51: _PyFunction_Vectorcall + 0x6c (0x559aee9cda2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x559aee9c6007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #53: _PyObject_Call_Prepend + 0x69 (0x559aee9d7c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #54: <unknown function> + 0x211239 (0x559aeea9a239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #55: PyObject_Call + 0x207 (0x559aee9da067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x559aee9c02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #57: <unknown function> + 0x150582 (0x559aee9d9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x559aee9be8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #59: <unknown function> + 0x150582 (0x559aee9d9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #60: PyObject_Call + 0xbc (0x559aee9d9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x559aee9c02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #62: <unknown function> + 0x150582 (0x559aee9d9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #63: PyObject_Call + 0xbc (0x559aee9d9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default4]:[rank36]: Traceback (most recent call last):
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default1]:[rank33]: Traceback (most recent call last):
[default4]:[rank36]:     trainer.train(dataloader)
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default4]:[rank36]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default4]:[rank36]:     outputs = self.pipeline_engine.train_batch_iter(
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default4]:[rank36]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default4]:[rank36]:     output = model(**micro_batch)
[default1]:[rank33]:     trainer.train(dataloader)
[default4]:[rank36]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default4]:[rank36]:     return self._call_impl(*args, **kwargs)
[default4]:[rank36]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank33]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default4]:[rank36]:     return forward_call(*args, **kwargs)
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default4]:[rank36]:     sharded_logits = self.model(
[default4]:[rank36]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default4]:[rank36]:     return self._call_impl(*args, **kwargs)
[default1]:[rank33]:     outputs = self.pipeline_engine.train_batch_iter(
[default4]:[rank36]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank36]:     return forward_call(*args, **kwargs)
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default4]:[rank36]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default1]:[rank33]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default4]:[rank36]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default4]:[rank36]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank36]:     return self._call_impl(*args, **kwargs)
[default4]:[rank36]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank36]:     return forward_call(*args, **kwargs)
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default1]:[rank33]:     output = model(**micro_batch)
[default1]:[rank33]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank36]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default1]:[rank33]:     return self._call_impl(*args, **kwargs)
[default1]:[rank33]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank33]:     return forward_call(*args, **kwargs)
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default1]:[rank33]:     sharded_logits = self.model(
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default1]:[rank33]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank49]: Traceback (most recent call last):
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default1]:[rank49]:     trainer.train(dataloader)
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default1]:[rank49]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default1]:[rank49]:     outputs = self.pipeline_engine.train_batch_iter(
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default1]:[rank49]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanot[default4]:[rank36]:     pipeline_state.run_communication()
[default1]:[rank33]:     return self._call_impl(*args, **kwargs)
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default1]:[rank33]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank33]:     return forward_call(*args, **kwargs)
[default4]:[rank36]:     recv_activation_tensor = recv_activation()
ron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default1]:[rank49]:     output = model(**micro_batch)
[default1]:[rank49]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default1]:[rank49]:     return self._call_impl(*args, **kwargs)
[default1]:[rank49]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank49]:     return forward_call(*args, **kwargs)
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default1]:[rank49]:     sharded_logits = self.model(
[default1]:[rank49]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank49]:     return self._call_impl(*args, **kwargs)
[default1]:[rank49]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank49]:     return forward_call(*args, **kwargs)
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotro[default4]:[rank36]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
n/src/nanotron/models/llama.py", line 764, in forward
[default1]:[rank49]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default1]:[rank49]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default1]:[rank49]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank49]:     return self._call_impl(*args, **kwargs)
[default1]:[rank49]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank49]:     return forward_call(*args, **kwargs)
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward

[default1]:[rank49]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default1]:[rank49]:     pipeline_state.run_communication()
[default1]:[rank33]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default1]:[rank49]:     recv_activation_tensor = recv_activation()
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default1]:[rank49]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default1]:[rank49]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default1]:[rank49]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default1]:[rank49]:   Fi[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
le "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default1]:[rank49]:     dist.recv(
[default1]:[rank49]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default1]:[rank49]:     return func(*args, **kwargs)
[default1]:[rank49]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default1]:[rank49]:     pg.recv([tensor], group_src_rank, tag).wait()
[default1]:[rank49]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default1]:[rank33]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default1]:[rank33]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank49]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default1]:[rank49]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fc9342f3897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default1]:[rank49]: frame #1: <unknown function> + 0x5b3a23e (0x7fc96de1023e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]:     return self._call_impl(*args, **kwargs)
[default1]:[rank49]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fc96de0ac87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fc96de0af82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default1]:[rank49]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fc96de0bfd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc96ddc0371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc96ddc0371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc96ddc0371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]:     return forward_call(*args, **kwargs)
[default1]:[rank49]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc96ddc0371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fc9355cd189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank49]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fc9355d4610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank36]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default1]:[rank49]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fc9355f3978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank49]: frame #12: <unknown function> + 0x5adc309 (0x7fc96ddb2309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]: frame #13: <unknown function> + 0x5ae6f10 (0x7fc96ddbcf10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default1]:[rank49]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fc96ddbcfa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]: frame #15: <unknown function> + 0x5124446 (0x7fc96d3fa446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]: frame #16: <unknown function> + 0x1acf4b8 (0x7fc969da54b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]: frame #17: <unknown function> + 0x5aee004 (0x7fc96ddc4004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]: frame #18: <unknown function> + 0x5af36b5 (0x7fc96ddc96b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]: frame #19: <unknown function> + 0xd2631e (0x7fc9809b3[default4]:[rank36]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default4]:[rank36]:     dist.recv(
[default4]:[rank36]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default1]:[rank33]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:[rank36]:     return func(*args, **kwargs)
[default1]:[rank49]: frame #20: <unknown function> + 0x47def4 (0x7fc98010aef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default1]:[rank49]: frame #21: <unknown function> + 0x1445a6 (0x5644043105a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #22: _PyObject_MakeTpCall + 0x26b (0x564404309a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #23: <unknown function> + 0x150866 (0x56440431c866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x564404305142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default1]:[rank49]: frame #25: _PyFunction_Vectorcall + 0x6c (0x564404310a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #26: PyObject_Call + 0xbc (0x56440431cf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5644043032b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]:     pipeline_state.run_communication()
[default4]:[rank36]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default1]:[rank49]: frame #28: _PyFunction_Vectorcall + 0x6c (0x564404310a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5644043018fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #30: <unknown function> + 0x150582 (0x56440431c582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5644043018fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #32: <unknown function> + 0x150582 (0x56440431c582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5644043018fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #34: <unknown function> + 0x150582 (0x56440431c582 in /fsx/ferdinandmom/miniforge3/envs/env[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5644043018fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x564404308f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]:     pg.recv([tensor], group_src_rank, tag).wait()
[default1]:[rank49]: frame #37: _PyObject_Call_Prepend + 0x69 (0x56440431ac39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #38: <unknown function> + 0x211239 (0x5644043dd239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #39: _PyObject_MakeTpCall + 0x26b (0x564404309a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5644043053e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #41: _PyFunction_Vectorcall + 0x6c (0x564404310a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]:     recv_activation_tensor = recv_activation()
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default1]:[rank49]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x564404300c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #43: _PyFunction_Vectorcall + 0x6c (0x564404310a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5644043018fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #45: <unknown function> + 0x150582 (0x56440431c582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default4]:[rank36]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default1]:[rank49]: frame #46: PyObject_Call + 0xbc (0x56440431cf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5644043032b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #48: <unknown function> + 0x150582 (0x56440431c582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #49: PyObject_Call + 0xbc (0x56440431cf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5644043032b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default1]:[rank49]: frame #51: _PyFunction_Vectorcall + 0x6c (0x564404310a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x564404309007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #53: _PyObject_Call_Prepend + 0x69 (0x56440431ac39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #54: <unknown function> + 0x211239 (0x5644043dd239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fad3372a897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default1]:[rank49]: frame #55: PyObject_Call + 0x207 (0x56440431d067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5644043032b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #57: <unknown function> + 0x150582 (0x56440431c582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5644043018fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #1: <unknown function> + 0x5b3a23e (0x7fad6d24723e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]: frame #59: <unknown function> + 0x150582 (0x56440431c582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #60: PyObject_Call + 0xbc (0x56440431cf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5644043032b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #62: <unknown function> + 0x150582 (0x56440431c582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default1]:[rank49]: frame #63: PyObject_Call + 0xbc (0x56440431cf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default4]:[rank36]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fad6d241c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fad6d241f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fad6d242fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fad6d1f7371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default4]:[rank36]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fad6d1f7371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fad6d1f7371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default1]:[rank33]:     dist.recv(
[default1]:[rank33]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default1]:[rank33]:     return func(*args, **kwargs)
[default1]:[rank33]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default4]:[rank36]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fad6d1f7371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]:     pg.recv([tensor], group_src_rank, tag).wait()
[default4]:[rank36]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fad34a04189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank36]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fad34a0b610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank33]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default1]:[rank33]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default1]:[rank33]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f91c2d1a897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default4]:[rank36]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fad34a2a978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank33]: frame #1: <unknown function> + 0x5b3a23e (0x7f91fc83723e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #12: <unknown function> + 0x5adc309 (0x7fad6d1e9309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f91fc831c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #13: <unknown function> + 0x5ae6f10 (0x7fad6d1f3f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f91fc831f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f91fc832fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fad6d1f3fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f91fc7e7371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f91fc7e7371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f91fc7e7371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f91fc7e7371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f91c3ff4189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank36]: frame #15: <unknown function> + 0x5124446 (0x7fad6c831446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #16: <unknown function> + 0x1acf4b8 (0x7fad691dc4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #17: <unknown function> + 0x5aee004 (0x7fad6d1fb004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f91c3ffb610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank36]: frame #18: <unknown function> + 0x5af36b5 (0x7fad6d2006b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f91c401a978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank36]: frame #19: <unknown function> + 0xd2631e (0x7fad7fdea31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default1]:[rank33]: frame #12: <unknown function> + 0x5adc309 (0x7f91fc7d9309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #20: <unknown function> + 0x47def4 (0x7fad7f541ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default1]:[rank33]: frame #13: <unknown function> + 0x5ae6f10 (0x7f91fc7e3f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #21: <unknown function> + 0x1445a6 (0x5650e60965a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #22: _PyObject_MakeTpCall + 0x26b (0x5650e608fa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #23: <unknown function> + 0x150866 (0x5650e60a2866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f91fc7e3fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x5650e608b142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #25: _PyFunction_Vectorcall + 0x6c (0x5650e6096a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #15: <unknown function> + 0x5124446 (0x7f91fbe21446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #26: PyObject_Call + 0xbc (0x5650e60a2f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5650e60892b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #28: _PyFunction_Vectorcall + 0x6c (0x5650e6096a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5650e60878fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #30: <unknown function> + 0x150582 (0x5650e60a2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #16: <unknown function> + 0x1acf4b8 (0x7f91f87cc4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5650e60878fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #32: <unknown function> + 0x150582 (0x5650e60a2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #17: <unknown function> + 0x5aee004 (0x7f91fc7eb004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5650e60878fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #18: <unknown function> + 0x5af36b5 (0x7f91fc7f06b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]: frame #19: <unknown function> + 0xd2631e (0x7f920f3da31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:[rank36]: frame #34: <unknown function> + 0x150582 (0x5650e60a2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #20: <unknown function> + 0x47def4 (0x7f920eb31ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:[rank36]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5650e60878fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x5650e608ef50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #21: <unknown function> + 0x1445a6 (0x55ee363d55a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #37: _PyObject_Call_Prepend + 0x69 (0x5650e60a0c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55ee363cea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #23: <unknown function> + 0x150866 (0x55ee363e1866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #38: <unknown function> + 0x211239 (0x5650e6163239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55ee363ca142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #39: _PyObject_MakeTpCall + 0x26b (0x5650e608fa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5650e608b3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55ee363d5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #41: _PyFunction_Vectorcall + 0x6c (0x5650e6096a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #26: PyObject_Call + 0xbc (0x55ee363e1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55ee363c82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55ee363d5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55ee363c68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #30: <unknown function> + 0x150582 (0x55ee363e1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55ee363c68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x5650e6086c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #32: <unknown function> + 0x150582 (0x55ee363e1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #43: _PyFunction_Vectorcall + 0x6c (0x5650e6096a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55ee363c68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #34: <unknown function> + 0x150582 (0x55ee363e1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55ee363c68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55ee363cdf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55ee363dfc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #38: <unknown function> + 0x211239 (0x55ee364a2239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55ee363cea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5650e60878fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55ee363ca3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55ee363d5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55ee363c5c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55ee363d5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55ee363c68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #45: <unknown function> + 0x150582 (0x55ee363e1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #46: PyObject_Call + 0xbc (0x55ee363e1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55ee363c82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #48: <unknown function> + 0x150582 (0x55ee363e1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #49: PyObject_Call + 0xbc (0x55ee363e1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55ee363c82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #45: <unknown function> + 0x150582 (0x5650e60a2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55ee363d5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #46: PyObject_Call + 0xbc (0x5650e60a2f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55ee363ce007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5650e60892b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55ee363dfc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #48: <unknown function> + 0x150582 (0x5650e60a2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #54: <unknown function> + 0x211239 (0x55ee364a2239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #55: PyObject_Call + 0x207 (0x55ee363e2067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55ee363c82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #49: PyObject_Call + 0xbc (0x5650e60a2f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #57: <unknown function> + 0x150582 (0x55ee363e1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55ee363c68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5650e60892b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #59: <unknown function> + 0x150582 (0x55ee363e1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #51: _PyFunction_Vectorcall + 0x6c (0x5650e6096a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x5650e608f007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #60: PyObject_Call + 0xbc (0x55ee363e1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #53: _PyObject_Call_Prepend + 0x69 (0x5650e60a0c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #54: <unknown function> + 0x211239 (0x5650e6163239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #55: PyObject_Call + 0x207 (0x5650e60a3067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5650e60892b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55ee363c82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #57: <unknown function> + 0x150582 (0x5650e60a2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5650e60878fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #62: <unknown function> + 0x150582 (0x55ee363e1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #59: <unknown function> + 0x150582 (0x5650e60a2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #63: PyObject_Call + 0xbc (0x55ee363e1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default4]:[rank36]: frame #60: PyObject_Call + 0xbc (0x5650e60a2f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5650e60892b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #62: <unknown function> + 0x150582 (0x5650e60a2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #63: PyObject_Call + 0xbc (0x5650e60a2f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default0]:[rank48]: Traceback (most recent call last):
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default0]:[rank48]:     trainer.train(dataloader)
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default0]:[rank48]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default0]:[rank48]:     outputs = self.pipeline_engine.train_batch_iter(
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default0]:[rank48]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default0]:[rank48]:     output = model(**micro_batch)
[default0]:[rank48]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank48]:     return self._call_impl(*args, **kwargs)
[default0]:[rank48]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank48]:     return forward_call(*args, **kwargs)
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default0]:[rank48]:     sharded_logits = self.model(
[default0]:[rank48]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank48]:     return self._call_impl(*args, **kwargs)
[default0]:[rank48]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank48]:     return forward_call(*args, **kwargs)
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default0]:[rank48]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default0]:[rank48]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default0]:[rank48]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank48]:     return self._call_impl(*args, **kwargs)
[default0]:[rank48]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank48]:     return forward_call(*args, **kwargs)
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default0]:[rank48]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default0]:[rank48]:     pipeline_state.run_communication()
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default0]:[rank48]:     recv_activation_tensor = recv_activation()
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default0]:[rank48]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default0]:[rank48]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default0]:[rank48]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default0]:[rank48]:     dist.recv(
[default0]:[rank48]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default0]:[rank48]:     return func(*args, **kwargs)
[default0]:[rank48]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default0]:[rank48]:     pg.recv([tensor], group_src_rank, tag).wait()
[default0]:[rank48]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default0]:[rank48]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default0]:[rank48]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f13d9806897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default0]:[rank48]: frame #1: <unknown function> + 0x5b3a23e (0x7f141332323e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f141331dc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f141331df82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f141331efd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f14132d3371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f14132d3371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f14132d3371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f14132d3371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f13daae0189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank48]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f13daae7610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank48]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f13dab06978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank48]: frame #12: <unknown function> + 0x5adc309 (0x7f14132c5309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: frame #13: <unknown function> + 0x5ae6f10 (0x7f14132cff10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f14132cffa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: frame #15: <unknown function> + 0x5124446 (0x7f141290d446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: frame #16: <unknown function> + 0x1acf4b8 (0x7f140f2b84b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: frame #17: <unknown function> + 0x5aee004 (0x7f14132d7004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: frame #18: <unknown function> + 0x5af36b5 (0x7f14132dc6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: frame #19: <unknown function> + 0xd2631e (0x7f1425ec631e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default0]:[rank48]: frame #20: <unknown function> + 0x47def4 (0x7f142561def4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default0]:[rank48]: frame #21: <unknown function> + 0x1445a6 (0x563e6676d5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #22: _PyObject_MakeTpCall + 0x26b (0x563e66766a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #23: <unknown function> + 0x150866 (0x563e66779866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x563e66762142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #25: _PyFunction_Vectorcall + 0x6c (0x563e6676da2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #26: PyObject_Call + 0xbc (0x563e66779f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x563e667602b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #28: _PyFunction_Vectorcall + 0x6c (0x563e6676da2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x563e6675e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #30: <unknown function> + 0x150582 (0x563e66779582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x563e6675e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #32: <unknown function> + 0x150582 (0x563e66779582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x563e6675e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #34: <unknown function> + 0x150582 (0x563e66779582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x563e6675e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x563e66765f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #37: _PyObject_Call_Prepend + 0x69 (0x563e66777c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #38: <unknown function> + 0x211239 (0x563e6683a239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #39: _PyObject_MakeTpCall + 0x26b (0x563e66766a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x563e667623e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #41: _PyFunction_Vectorcall + 0x6c (0x563e6676da2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x563e6675dc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #43: _PyFunction_Vectorcall + 0x6c (0x563e6676da2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x563e6675e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #45: <unknown function> + 0x150582 (0x563e66779582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #46: PyObject_Call + 0xbc (0x563e66779f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x563e667602b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #48: <unknown function> + 0x150582 (0x563e66779582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #49: PyObject_Call + 0xbc (0x563e66779f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x563e667602b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #51: _PyFunction_Vectorcall + 0x6c (0x563e6676da2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x563e66766007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #53: _PyObject_Call_Prepend + 0x69 (0x563e66777c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #54: <unknown function> + 0x211239 (0x563e6683a239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #55: PyObject_Call + 0x207 (0x563e6677a067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x563e667602b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #57: <unknown function> + 0x150582 (0x563e66779582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x563e6675e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #59: <unknown function> + 0x150582 (0x563e66779582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #60: PyObject_Call + 0xbc (0x563e66779f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x563e667602b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #62: <unknown function> + 0x150582 (0x563e66779582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #63: PyObject_Call + 0xbc (0x563e66779f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default3]:[rank51]: Traceback (most recent call last):
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default3]:[rank51]:     trainer.train(dataloader)
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default2]:[rank50]: Traceback (most recent call last):
[default3]:[rank51]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default2]:[rank50]:     trainer.train(dataloader)
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default2]:[rank50]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default2]:[rank50]:     outputs = self.pipeline_engine.train_batch_iter(
[default3]:[rank51]:     outputs = self.pipeline_engine.train_batch_iter(
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default2]:[rank50]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default2]:[rank50]:     output = model(**micro_batch)
[default2]:[rank50]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank51]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default2]:[rank50]:     return self._call_impl(*args, **kwargs)
[default2]:[rank50]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default3]:[rank51]:     output = model(**micro_batch)
[default2]:[rank50]:     return forward_call(*args, **kwargs)
[default3]:[rank51]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank51]:     return self._call_impl(*args, **kwargs)
[default3]:[rank51]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank51]:     return forward_call(*args, **kwargs)
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default2]:[rank50]:     sharded_logits = self.model(
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default2]:[rank50]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank50]:     return self._call_impl(*args, **kwargs)
[default2]:[rank50]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank50]:     return forward_call(*args, **kwargs)
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default2]:[rank50]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default2]:[rank50]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default3]:[rank51]:     sharded_logits = self.model(
[default2]:[rank50]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank51]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank51]:     return self._call_impl(*args, **kwargs)
[default3]:[rank51]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank50]:     return self._call_impl(*args, **kwargs)
[default2]:[rank50]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank50]:     return forward_call(*args, **kwargs)
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default2]:[rank50]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default3]:[rank51]:     return forward_call(*args, **kwargs)
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default3]:[rank51]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default3]:[rank51]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default3]:[rank51]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank51]:     return self._call_impl(*args, **kwargs)
[default3]:[rank51]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank50]:     pipeline_state.run_communication()
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default2]:[rank50]:     recv_activation_tensor = recv_activation()
[default3]:[rank51]:     return forward_call(*args, **kwargs)
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default2]:[rank50]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default2]:[rank50]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default2]:[rank50]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default2]:[rank50]:     dist.recv(
[default2]:[rank50]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default2]:[rank50]:     return func(*args, **kwargs)
[default2]:[rank50]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default2]:[rank50]:     pg.recv([tensor], group_src_rank, tag).wait()
[default3]:[rank51]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default2]:[rank50]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default2]:[rank50]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default2]:[rank50]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f66cd479897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default3]:[rank51]:     pipeline_state.run_communication()
[default2]:[rank50]: frame #1: <unknown function> + 0x5b3a23e (0x7f6706f9623e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default3]:[rank51]:     recv_activation_tensor = recv_activation()
[default2]:[rank50]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f6706f90c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f6706f90f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f6706f91fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default2]:[rank50]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6706f46371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6706f46371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default2]:[rank50]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6706f46371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default3]:[rank51]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default3]:[rank51]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default2]:[rank50]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6706f46371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f66ce753189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank50]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f66ce75a610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank50]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f66ce779978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:[rank51]:     dist.recv(
[default2]:[rank50]: frame #12: <unknown function> + 0x5adc309 (0x7f6706f38309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default2]:[rank50]: frame #13: <unknown function> + 0x5ae6f10 (0x7f6706f42f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]:     return func(*args, **kwargs)
[default3]:[rank51]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default3]:[rank51]:     pg.recv([tensor], group_src_rank, tag).wait()
[default3]:[rank51]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default3]:[rank51]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default3]:[rank51]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f5cd061e897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default3]:[rank51]: frame #1: <unknown function> + 0x5b3a23e (0x7f5d0a13b23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f5d0a135c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f5d0a135f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f5d0a136fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f6706f42fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f5d0a0eb371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]: frame #15: <unknown function> + 0x5124446 (0x7f6706580446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]: frame #16: <unknown function> + 0x1acf4b8 (0x7f6702f2b4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]: frame #17: <unknown function> + 0x5aee004 (0x7f6706f4a004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]: frame #18: <unknown function> + 0x5af36b5 (0x7f6706f4f6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]: frame #19: <unknown function> + 0xd2631e (0x7f6719b3931e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default2]:[rank50]: frame #20: <unknown function> + 0x47def4 (0x7f6719290ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default2]:[rank50]: frame #21: <unknown function> + 0x1445a6 (0x55ba1a0045a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55ba19ffda6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #23: <unknown function> + 0x150866 (0x55ba1a010866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55ba19ff9142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f5d0a0eb371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55ba1a004a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #26: PyObject_Call + 0xbc (0x55ba1a010f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f5d0a0eb371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f5d0a0eb371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f5cd18f8189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank50]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55ba19ff72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55ba1a004a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55ba19ff58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f5cd18ff610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank50]: frame #30: <unknown function> + 0x150582 (0x55ba1a010582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55ba19ff58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #32: <unknown function> + 0x150582 (0x55ba1a010582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55ba19ff58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f5cd191e978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank50]: frame #34: <unknown function> + 0x150582 (0x55ba1a010582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55ba19ff58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #12: <unknown function> + 0x5adc309 (0x7f5d0a0dd309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55ba19ffcf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #13: <unknown function> + 0x5ae6f10 (0x7f5d0a0e7f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55ba1a00ec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #38: <unknown function> + 0x211239 (0x55ba1a0d1239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55ba19ffda6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55ba19ff93e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55ba1a004a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f5d0a0e7fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]: frame #15: <unknown function> + 0x5124446 (0x7f5d09725446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]: frame #16: <unknown function> + 0x1acf4b8 (0x7f5d060d04b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]: frame #17: <unknown function> + 0x5aee004 (0x7f5d0a0ef004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: Traceback (most recent call last):
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default0]:[rank32]:     trainer.train(dataloader)
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default0]:[rank32]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default0]:[rank32]:     outputs = self.pipeline_engine.train_batch_iter(
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default0]:[rank32]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanot[default2]:[rank50]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55ba19ff4c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55ba1a004a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55ba19ff58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #45: <unknown function> + 0x150582 (0x55ba1a010582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #46: PyObject_Call + 0xbc (0x55ba1a010f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55ba19ff72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #18: <unknown function> + 0x5af36b5 (0x7f5d0a0f46b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
ron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default0]:[rank32]:     output = model(**micro_batch)
[default0]:[rank32]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank32]:     return self._call_impl(*args, **kwargs)
[default0]:[rank32]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank32]:     return forward_call(*args, **kwargs)
[default3]:[rank51]: frame #19: <unknown function> + 0xd2631e (0x7f5d1ccde31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default3]:[rank51]: frame #20: <unknown function> + 0x47def4 (0x7f5d1c435ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default0]:[rank32]:     sharded_logits = self.model(
[default0]:[rank32]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank32]:     return self._call_impl(*args, **kwargs)
[default0]:[rank32]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank32]:     return forward_call(*args, **kwargs)
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default0]:[rank32]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in[default3]:[rank51]: frame #21: <unknown function> + 0x1445a6 (0x560c1bcbd5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #22: _PyObject_MakeTpCall + 0x26b (0x560c1bcb6a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
 forward_with_hidden_states
[default0]:[rank32]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default0]:[rank32]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank32]:     return self._call_impl(*args, **kwargs)
[default0]:[rank32]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank32]:     return forward_call(*args, **kwargs)
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default0]:[rank32]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default0]:[rank32]:     pipe[default3]:[rank51]: frame #23: <unknown function> + 0x150866 (0x560c1bcc9866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
line_state.run_communication()
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default0]:[rank32]:     recv_activation_tensor = recv_activation()
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default0]:[rank32]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default0]:[rank32]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default2]:[rank50]: frame #48: <unknown function> + 0x150582 (0x55ba1a010582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #49: PyObject_Call + 0xbc (0x55ba1a010f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55ba19ff72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default0]:[rank32]:     dist.recv(
[default0]:[rank32]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default0]:[rank32]:     return func(*args, **kwargs)
[default0]:[rank32]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default0]:[rank32]:     pg.recv([tensor], group_src_rank, tag).wait()
[default0]:[rank32]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default0]:[rank32]: Exception raised from recvBytes at ../torch/csr[default2]:[rank50]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55ba1a004a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55ba19ffd007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
c/distributed/c10d/Utils.hpp:672 (most recent call first):
[default0]:[rank32]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f41069a4897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default0]:[rank32]: frame #1: <unknown function> + 0x5b3a23e (0x7f41404c123e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f41404bbc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f41404bbf82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f41404[default3]:[rank51]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x560c1bcb2142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #25: _PyFunction_Vectorcall + 0x6c (0x560c1bcbda2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
bcfd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4140471371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4140471371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4140471371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4140471371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLI[default3]:[rank51]: frame #26: PyObject_Call + 0xbc (0x560c1bcc9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x560c1bcb02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
D(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f4107c7e189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank32]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f4107c85610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:[rank51]: frame #28: _PyFunction_Vectorcall + 0x6c (0x560c1bcbda2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f4107ca4978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank32]: frame #12: <unknown function> + 0x5adc309 (0x7f4140463309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #13: <unknown function> + 0x5ae6f10 (0x7f414046df10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55ba1a00ec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #54: <unknown function> + 0x211239 (0x55ba1a0d1239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x560c1bcae8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #55: PyObject_Call + 0x207 (0x55ba1a011067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f414046dfa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]: frame #30: <unknown function> + 0x150582 (0x560c1bcc9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #15: <unknown function> + 0x5124446 (0x7f413faab446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #16: <unknown function> + 0x1acf4b8 (0x7f413c4564b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #17: <unknown function> + 0x5aee004 (0x7f4140475004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #18: <unknown function> + 0x5af36b5 (0x7f414047a6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55ba19ff72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #19: <unknown function> + 0xd2631e (0x7f415306431e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default0]:[rank32]: frame #20: <unknown function> + 0x47def4 (0x7f41527bbef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default0]:[rank32]: frame #21: <unknown function> + 0x1445a6 (0x557d896c35a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x560c1bcae8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #22: _PyObject_MakeTpCall + 0x26b (0x557d896bca6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #23: <unknown function> + 0x150866 (0x557d896cf866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x557d896b8142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #25: _PyFunction_Vectorcall + 0x6c (0x557d896c3a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #26: PyObject_Call + 0xbc (0x557d896cff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x557d896b62b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #28: _PyFunction_Vectorcall + 0x6c (0x557d896c3a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x557d896b48fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #30: <unknown function> + 0x150582 (0x557d896cf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x557d896b48fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #32: <unknown function> + 0x150582 (0x557d896cf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x557d896b48fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #34: <unknown function> + 0x150582 (0x557d896cf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x557d896b48fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x557d896bbf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #37: _PyObject_Call_Prepend + 0x69 (0x557d896cdc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #38: <unknown function> + 0x211239 (0x557d89790239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #39: _PyObject_MakeTpCall + 0x26b (0x557d896bca6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x557d896b83e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #41: _PyFunction_Vectorcall + 0x6c (0x557d896c3a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x557d896b3c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #43: _PyFunction_Vectorcall + 0x6c (0x557d896c3a2c in /fsx/f[default3]:[rank51]: frame #32: <unknown function> + 0x150582 (0x560c1bcc9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
erdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x557d896b48fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x560c1bcae8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #34: <unknown function> + 0x150582 (0x560c1bcc9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #45: <unknown function> + 0x150582 (0x557d896cf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #46: PyObject_Call + 0xbc (0x557d896cff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x557d896b62b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #48: <unknown function> + 0x150582 (0x557d896cf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #49: PyObject_Call + 0xbc (0x557d896cff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x557d896b62b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #51: _PyFunction_Vectorcall + 0x6c (0x557d896c3a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/pyt[default2]:[rank50]: frame #57: <unknown function> + 0x150582 (0x55ba1a010582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55ba19ff58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
hon3.10)
[default3]:[rank51]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x560c1bcae8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x560c1bcb5f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x557d896bc007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #53: _PyObject_Call_Prepend + 0x69 (0x557d896cdc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #54: <unknown function> + 0x211239 (0x557d89790239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #55: PyObject_Call + 0x207 (0x557d896d0067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x557d896b62b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #57: <unknown function> + 0x150582 (0x557d896cf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #37: _PyObject_Call_Prepend + 0x69 (0x560c1bcc7c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #38: <unknown function> + 0x211239 (0x560c1bd8a239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x557d896b48fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #59: <unknown function> + 0x150582 (0x557d896cf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #60: PyObject_Call + 0xbc (0x557d896cff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x557d896b62b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #39: _PyObject_MakeTpCall + 0x26b (0x560c1bcb6a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #62: <unknown function> + 0x150582 (0x557d896cf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #63: PyObject_Call + 0xbc (0x557d896cff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default2]:[rank50]: frame #59: <unknown function> + 0x150582 (0x55ba1a010582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: Traceback (most recent call last):
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default2]:[rank34]:     trainer.train(dataloader)
[default3]:[rank51]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x560c1bcb23e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default2]:[rank34]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default2]:[rank50]: frame #60: PyObject_Call + 0xbc (0x55ba1a010f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]:     outputs = self.pipeline_engine.train_batch_iter(
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default2]:[rank34]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default3]:[rank51]: frame #41: _PyFunction_Vectorcall + 0x6c (0x560c1bcbda2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]:     output = model(**micro_batch)
[default2]:[rank34]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank34]:     return self._call_impl(*args, **kwargs)
[default2]:[rank34]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank50]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55ba19ff72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]:     return forward_call(*args, **kwargs)
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default2]:[rank34]:     sharded_logits = self.model(
[default2]:[rank34]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank34]:     return self._call_impl(*args, **kwargs)
[default2]:[rank34]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank34]:     return forward_call(*args, **kwargs)
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default2]:[rank34]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/benc[default3]:[rank51]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x560c1bcadc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
h_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default2]:[rank34]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default2]:[rank34]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank34]:     return self._call_impl(*args, **kwargs)
[default2]:[rank34]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank50]: frame #62: <unknown function> + 0x150582 (0x55ba1a010582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]:     return forward_call(*args, **kwargs)
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default2]:[rank34]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default2]:[rank34]:     pipeline_state.run_communication()
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default3]:[rank51]: frame #43: _PyFunction_Vectorcall + 0x6c (0x560c1bcbda2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]:     recv_activation_tensor = recv_activation()
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default2]:[rank34]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default2]:[rank50]: frame #63: PyObject_Call + 0xbc (0x55ba1a010f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default2]:[rank34]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default2]:[rank34]:     dist.recv(
[default3]:[rank51]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x560c1bcae8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default2]:[rank34]:     return func(*args, **kwargs)
[default2]:[rank34]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default2]:[rank34]:     pg.recv([tensor], group_src_rank, tag).wait()
[default2]:[rank34]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default2]:[rank34]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default2]:[rank34]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fc91c129897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default2]:[rank50]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default2]:[rank34]: frame #1: <unknown function> + 0x5b3a23e (0x7fc955c4623e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fc955c40c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fc955c40f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fc955c41fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc955bf6371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/l[default3]:[rank51]: frame #45: <unknown function> + 0x150582 (0x560c1bcc9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #46: PyObject_Call + 0xbc (0x560c1bcc9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x560c1bcb02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
ib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc955bf6371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc955bf6371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc955bf6371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fc91d403189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank34]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&,[default3]:[rank51]: frame #48: <unknown function> + 0x150582 (0x560c1bcc9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #49: PyObject_Call + 0xbc (0x560c1bcc9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x560c1bcb02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
 c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fc91d40a610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank34]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fc91d429978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank34]: frame #12: <unknown function> + 0x5adc309 (0x7fc955be8309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]: frame #13: <unknown function> + 0x5ae6f10 (0x7fc955bf2f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]: frame #51: _PyFunction_Vectorcall + 0x6c (0x560c1bcbda2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x560c1bcb6007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #53: _PyObject_Call_Prepend + 0x69 (0x560c1bcc7c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fc955bf2fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]: frame #15: <unknown function> + 0x5124446 (0x7fc955230446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]: frame #16: <unknown function> + 0x1acf4b8 (0x7fc951bdb4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]: frame #17: <unknown function> + 0x5aee004 (0x7fc955bfa004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]: frame #54: <unknown function> + 0x211239 (0x560c1bd8a239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #55: PyObject_Call + 0x207 (0x560c1bcca067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x560c1bcb02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #57: <unknown function> + 0x150582 (0x560c1bcc9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #18: <unknown function> + 0x5af36b5 (0x7fc955bff6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]: frame #19: <unknown function> + 0xd2631e (0x7fc9687e931e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default2]:[rank34]: frame #20: <unknown function> + 0x47def4 (0x7fc967f40ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default2]:[rank34]: frame #21: <unknown function> + 0x1445a6 (0x558a9b2b65a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x560c1bcae8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #59: <unknown function> + 0x150582 (0x560c1bcc9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #60: PyObject_Call + 0xbc (0x560c1bcc9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x560c1bcb02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #62: <unknown function> + 0x150582 (0x560c1bcc9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #63: PyObject_Call + 0xbc (0x560c1bcc9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #22: _PyObject_MakeTpCall + 0x26b (0x558a9b2afa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #23: <unknown function> + 0x150866 (0x558a9b2c2866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x558a9b2ab142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #25: _PyFunction_Vectorcall + 0x6c (0x558a9b2b6a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #26: PyObject_Call + 0xbc (0x558a9b2c2f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default2]:[rank34]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x558a9b2a92b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #28: _PyFunction_Vectorcall + 0x6c (0x558a9b2b6a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x558a9b2a78fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #30: <unknown function> + 0x150582 (0x558a9b2c2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x558a9b2a78fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #32: <unknown function> + 0x150582 (0x558a9b2c2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x558a9b2a78fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #34: <unknown function> + 0x150582 (0x558a9b2c2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x558a9b2a78fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x558a9b2aef50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #37: _PyObject_Call_Prepend + 0x69 (0x558a9b2c0c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #38: <unknown function> + 0x211239 (0x558a9b383239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #39: _PyObject_MakeTpCall + 0x26b (0x558a9b2afa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x558a9b2ab3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #41: _PyFunction_Vectorcall + 0x6c (0x558a9b2b6a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x558a9b2a6c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #43: _PyFunction_Vectorcall + 0x6c (0x558a9b2b6a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x558a9b2a78fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #45: <unknown function> + 0x150582 (0x558a9b2c2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #46: PyObject_Call + 0xbc (0x558a9b2c2f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x558a9b2a92b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #48: <unknown function> + 0x150582 (0x558a9b2c2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #49: PyObject_Call + 0xbc (0x558a9b2c2f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: Traceback (most recent call last):
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default2]:[rank34]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x558a9b2a92b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]:     trainer.train(dataloader)
[default2]:[rank34]: frame #51: _PyFunction_Vectorcall + 0x6c (0x558a9b2b6a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x558a9b2af007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default2]:[rank34]: frame #53: _PyObject_Call_Prepend + 0x69 (0x558a9b2c0c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default2]:[rank34]: frame #54: <unknown function> + 0x211239 (0x558a9b383239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]:     outputs = self.pipeline_engine.train_batch_iter(
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default2]:[rank34]: frame #55: PyObject_Call + 0x207 (0x558a9b2c3067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x558a9b2a92b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default2]:[rank34]: frame #57: <unknown function> + 0x150582 (0x558a9b2c2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]:     output = model(**micro_batch)
[default2]:[rank34]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x558a9b2a78fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank34]: frame #59: <unknown function> + 0x150582 (0x558a9b2c2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]:     return self._call_impl(*args, **kwargs)
[default2]:[rank34]: frame #60: PyObject_Call + 0xbc (0x558a9b2c2f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x558a9b2a92b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #62: <unknown function> + 0x150582 (0x558a9b2c2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #63: PyObject_Call + 0xbc (0x558a9b2c2f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank35]:     return forward_call(*args, **kwargs)
[default2]:[rank34]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default3]:[rank35]:     sharded_logits = self.model(
[default3]:[rank35]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank35]:     return self._call_impl(*args, **kwargs)
[default3]:[rank35]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank35]:     return forward_call(*args, **kwargs)
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default3]:[rank35]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default3]:[rank35]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default3]:[rank35]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank35]:     return self._call_impl(*args, **kwargs)
[default3]:[rank35]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank35]:     return forward_call(*args, **kwargs)
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default3]:[rank35]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default3]:[rank35]:     pipeline_state.run_communication()
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default3]:[rank35]:     recv_activation_tensor = recv_activation()
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default3]:[rank35]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default3]:[rank35]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default3]:[rank35]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default3]:[rank35]:     dist.recv(
[default3]:[rank35]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default3]:[rank35]:     return func(*args, **kwargs)
[default3]:[rank35]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default3]:[rank35]:     pg.recv([tensor], group_src_rank, tag).wait()
[default3]:[rank35]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default3]:[rank35]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default3]:[rank35]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fd0d3267897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default3]:[rank35]: frame #1: <unknown function> + 0x5b3a23e (0x7fd10cd8423e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fd10cd7ec87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fd10cd7ef82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fd10cd7ffd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd10cd34371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd10cd34371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd10cd34371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd10cd34371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fd0d4541189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:[rank35]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fd0d4548610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:[rank35]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fd0d4567978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:[rank35]: frame #12: <unknown function> + 0x5adc309 (0x7fd10cd26309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #13: <unknown function> + 0x5ae6f10 (0x7fd10cd30f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fd10cd30fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #15: <unknown function> + 0x5124446 (0x7fd10c36e446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #16: <unknown function> + 0x1acf4b8 (0x7fd108d194b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #17: <unknown function> + 0x5aee004 (0x7fd10cd38004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #18: <unknown function> + 0x5af36b5 (0x7fd10cd3d6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #19: <unknown function> + 0xd2631e (0x7fd11f92731e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default3]:[rank35]: frame #20: <unknown function> + 0x47def4 (0x7fd11f07eef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default3]:[rank35]: frame #21: <unknown function> + 0x1445a6 (0x555fb852d5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #22: _PyObject_MakeTpCall + 0x26b (0x555fb8526a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #23: <unknown function> + 0x150866 (0x555fb8539866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x555fb8522142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #25: _PyFunction_Vectorcall + 0x6c (0x555fb852da2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #26: PyObject_Call + 0xbc (0x555fb8539f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x555fb85202b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #28: _PyFunction_Vectorcall + 0x6c (0x555fb852da2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x555fb851e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #30: <unknown function> + 0x150582 (0x555fb8539582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x555fb851e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #32: <unknown function> + 0x150582 (0x555fb8539582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x555fb851e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #34: <unknown function> + 0x150582 (0x555fb8539582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x555fb851e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x555fb8525f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #37: _PyObject_Call_Prepend + 0x69 (0x555fb8537c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #38: <unknown function> + 0x211239 (0x555fb85fa239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #39: _PyObject_MakeTpCall + 0x26b (0x555fb8526a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x555fb85223e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #41: _PyFunction_Vectorcall + 0x6c (0x555fb852da2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x555fb851dc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #43: _PyFunction_Vectorcall + 0x6c (0x555fb852da2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x555fb851e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #45: <unknown function> + 0x150582 (0x555fb8539582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #46: PyObject_Call + 0xbc (0x555fb8539f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x555fb85202b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #48: <unknown function> + 0x150582 (0x555fb8539582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #49: PyObject_Call + 0xbc (0x555fb8539f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x555fb85202b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #51: _PyFunction_Vectorcall + 0x6c (0x555fb852da2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x555fb8526007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #53: _PyObject_Call_Prepend + 0x69 (0x555fb8537c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #54: <unknown function> + 0x211239 (0x555fb85fa239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #55: PyObject_Call + 0x207 (0x555fb853a067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x555fb85202b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #57: <unknown function> + 0x150582 (0x555fb8539582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x555fb851e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #59: <unknown function> + 0x150582 (0x555fb8539582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #60: PyObject_Call + 0xbc (0x555fb8539f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x555fb85202b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #62: <unknown function> + 0x150582 (0x555fb8539582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #63: PyObject_Call + 0xbc (0x555fb8539f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default1]:[rank41]: Traceback (most recent call last):
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default1]:[rank41]:     trainer.train(dataloader)
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default1]:[rank41]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default1]:[rank41]:     outputs = self.pipeline_engine.train_batch_iter(
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default1]:[rank41]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default1]:[rank41]:     output = model(**micro_batch)
[default1]:[rank41]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank41]:     return self._call_impl(*args, **kwargs)
[default1]:[rank41]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank41]:     return forward_call(*args, **kwargs)
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default1]:[rank41]:     sharded_logits = self.model(
[default1]:[rank41]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank41]:     return self._call_impl(*args, **kwargs)
[default1]:[rank41]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank41]:     return forward_call(*args, **kwargs)
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default1]:[rank41]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default1]:[rank41]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default1]:[rank41]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank41]:     return self._call_impl(*args, **kwargs)
[default1]:[rank41]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank41]:     return forward_call(*args, **kwargs)
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default1]:[rank41]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default1]:[rank41]:     pipeline_state.run_communication()
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default1]:[rank41]:     recv_activation_tensor = recv_activation()
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default1]:[rank41]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default1]:[rank41]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default1]:[rank41]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default1]:[rank41]:     dist.recv(
[default1]:[rank41]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default1]:[rank41]:     return func(*args, **kwargs)
[default1]:[rank41]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default1]:[rank41]:     pg.recv([tensor], group_src_rank, tag).wait()
[default1]:[rank41]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default1]:[rank41]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default1]:[rank41]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fc270c66897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default1]:[rank41]: frame #1: <unknown function> + 0x5b3a23e (0x7fc2aa78323e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fc2aa77dc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fc2aa77df82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fc2aa77efd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc2aa733371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc2aa733371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc2aa733371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc2aa733371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fc271f40189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank41]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fc271f47610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank41]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fc271f66978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank41]: frame #12: <unknown function> + 0x5adc309 (0x7fc2aa725309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #13: <unknown function> + 0x5ae6f10 (0x7fc2aa72ff10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fc2aa72ffa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #15: <unknown function> + 0x5124446 (0x7fc2a9d6d446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #16: <unknown function> + 0x1acf4b8 (0x7fc2a67184b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #17: <unknown function> + 0x5aee004 (0x7fc2aa737004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #18: <unknown function> + 0x5af36b5 (0x7fc2aa73c6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #19: <unknown function> + 0xd2631e (0x7fc2bd32631e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default1]:[rank41]: frame #20: <unknown function> + 0x47def4 (0x7fc2bca7def4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default1]:[rank41]: frame #21: <unknown function> + 0x1445a6 (0x563e61b0a5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #22: _PyObject_MakeTpCall + 0x26b (0x563e61b03a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #23: <unknown function> + 0x150866 (0x563e61b16866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x563e61aff142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #25: _PyFunction_Vectorcall + 0x6c (0x563e61b0aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #26: PyObject_Call + 0xbc (0x563e61b16f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x563e61afd2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #28: _PyFunction_Vectorcall + 0x6c (0x563e61b0aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x563e61afb8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #30: <unknown function> + 0x150582 (0x563e61b16582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x563e61afb8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #32: <unknown function> + 0x150582 (0x563e61b16582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x563e61afb8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #34: <unknown function> + 0x150582 (0x563e61b16582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x563e61afb8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x563e61b02f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #37: _PyObject_Call_Prepend + 0x69 (0x563e61b14c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #38: <unknown function> + 0x211239 (0x563e61bd7239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #39: _PyObject_MakeTpCall + 0x26b (0x563e61b03a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x563e61aff3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #41: _PyFunction_Vectorcall + 0x6c (0x563e61b0aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x563e61afac5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #43: _PyFunction_Vectorcall + 0x6c (0x563e61b0aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x563e61afb8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #45: <unknown function> + 0x150582 (0x563e61b16582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #46: PyObject_Call + 0xbc (0x563e61b16f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x563e61afd2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #48: <unknown function> + 0x150582 (0x563e61b16582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #49: PyObject_Call + 0xbc (0x563e61b16f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x563e61afd2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #51: _PyFunction_Vectorcall + 0x6c (0x563e61b0aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x563e61b03007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #53: _PyObject_Call_Prepend + 0x69 (0x563e61b14c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #54: <unknown function> + 0x211239 (0x563e61bd7239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #55: PyObject_Call + 0x207 (0x563e61b17067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x563e61afd2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #57: <unknown function> + 0x150582 (0x563e61b16582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x563e61afb8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #59: <unknown function> + 0x150582 (0x563e61b16582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #60: PyObject_Call + 0xbc (0x563e61b16f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x563e61afd2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #62: <unknown function> + 0x150582 (0x563e61b16582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #63: PyObject_Call + 0xbc (0x563e61b16f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default5]:[rank61]: Traceback (most recent call last):
[default2]:[rank58]: Traceback (most recent call last):
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default2]:[rank58]:     trainer.train(dataloader)
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default2]:[rank58]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default5]:[rank61]:     trainer.train(dataloader)
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default2]:[rank58]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default2]:[rank58]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default5]:[rank61]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default2]:[rank58]:     output = model(**micro_batch)
[default2]:[rank58]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default2]:[rank58]:     return self._call_impl(*args, **kwargs)
[default5]:[rank61]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default5]:[rank61]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default5]:[rank61]:     output = model(**micro_batch)
[default5]:[rank61]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank61]:     return self._call_impl(*args, **kwargs)
[default5]:[rank61]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank61]:     return forward_call(*args, **kwargs)
[default2]:[rank58]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default2]:[rank58]:     return forward_call(*args, **kwargs)
[default5]:[rank61]:     sharded_logits = self.model(
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default2]:[rank58]:     sharded_logits = self.model(
[default2]:[rank58]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank58]:     return self._call_impl(*args, **kwargs)
[default2]:[rank58]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank58]:     return forward_call(*args, **kwargs)
[default5]:[rank61]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default2]:[rank58]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default5]:[rank61]:     return self._call_impl(*args, **kwargs)
[default5]:[rank61]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank61]:     return forward_call(*args, **kwargs)
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default2]:[rank58]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default2]:[rank58]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default5]:[rank61]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default2]:[rank58]:     return self._call_impl(*args, **kwargs)
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default5]:[rank61]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default2]:[rank58]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank58]:     return forward_call(*args, **kwargs)
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default5]:[rank61]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank61]:     return self._call_impl(*args, **kwargs)
[default5]:[rank61]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank61]:     return forward_call(*args, **kwargs)
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default5]:[rank61]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default5]:[rank61]:     pipeline_state.run_communication()
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default5]:[rank61]:     recv_activation_tensor = recv_activation()
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default5]:[rank61]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default5]:[rank61]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default2]:[rank58]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default2]:[rank58]:     pipeline_state.run_communication()
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default2]:[rank58]:     recv_activation_tensor = recv_activation()
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default5]:[rank61]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default2]:[rank58]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default5]:[rank61]:     dist.recv(
[default5]:[rank61]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default2]:[rank58]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default5]:[rank61]:     return func(*args, **kwargs)
[default5]:[rank61]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default2]:[rank58]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default2]:[rank58]:     dist.recv(
[default2]:[rank58]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default2]:[rank58]:     return func(*args, **kwargs)
[default5]:[rank61]:     pg.recv([tensor], group_src_rank, tag).wait()
[default5]:[rank61]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default5]:[rank61]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default5]:[rank61]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f4200a76897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default5]:[rank61]: frame #1: <unknown function> + 0x5b3a23e (0x7f423a59323e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default5]:[rank61]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f423a58dc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f423a58df82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f423a58efd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f423a543371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f423a543371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]:     pg.recv([tensor], group_src_rank, tag).wait()
[default5]:[rank61]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f423a543371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f423a543371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default2]:[rank58]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default2]:[rank58]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f84a33df897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default2]:[rank58]: frame #1: <unknown function> + 0x5b3a23e (0x7f84dcefc23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f84dcef6c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f84dcef6f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f84dcef7fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f84dceac371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f4201d50189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank61]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f4201d57610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank61]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f4201d76978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank61]: frame #12: <unknown function> + 0x5adc309 (0x7f423a535309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #13: <unknown function> + 0x5ae6f10 (0x7f423a53ff10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f84dceac371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f84dceac371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f84dceac371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f84a46b9189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank58]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f84a46c0610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank58]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f84a46df978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank58]: frame #12: <unknown function> + 0x5adc309 (0x7f84dce9e309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f423a53ffa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: frame #13: <unknown function> + 0x5ae6f10 (0x7f84dcea8f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: Traceback (most recent call last):
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default2]:[rank58]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f84dcea8fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: frame #15: <unknown function> + 0x5124446 (0x7f84dc4e6446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: frame #16: <unknown function> + 0x1acf4b8 (0x7f84d8e914b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #15: <unknown function> + 0x5124446 (0x7f4239b7d446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #16: <unknown function> + 0x1acf4b8 (0x7f42365284b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]:     trainer.train(dataloader)
[default2]:[rank58]: frame #17: <unknown function> + 0x5aee004 (0x7f84dceb0004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default5]:[rank61]: frame #17: <unknown function> + 0x5aee004 (0x7f423a547004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #18: <unknown function> + 0x5af36b5 (0x7f423a54c6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #19: <unknown function> + 0xd2631e (0x7f424d13631e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default5]:[rank61]: frame #20: <unknown function> + 0x47def4 (0x7f424c88def4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:[rank63]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default2]:[rank58]: frame #18: <unknown function> + 0x5af36b5 (0x7f84dceb56b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: frame #19: <unknown function> + 0xd2631e (0x7f84efa9f31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default2]:[rank58]: frame #20: <unknown function> + 0x47def4 (0x7f84ef1f6ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default7]:[rank63]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank61]: frame #21: <unknown function> + 0x1445a6 (0x5602d74455a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default2]:[rank58]: frame #21: <unknown function> + 0x1445a6 (0x55e37d0895a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #22: _PyObject_MakeTpCall + 0x26b (0x5602d743ea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default5]:[rank61]: frame #23: <unknown function> + 0x150866 (0x5602d7451866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55e37d082a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #23: <unknown function> + 0x150866 (0x55e37d095866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default5]:[rank61]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x5602d743a142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55e37d07e142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:     output = model(**micro_batch)
[default7]:[rank63]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank58]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55e37d089a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #25: _PyFunction_Vectorcall + 0x6c (0x5602d7445a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:     return self._call_impl(*args, **kwargs)
[default5]:[rank61]: frame #26: PyObject_Call + 0xbc (0x5602d7451f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5602d74382b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #28: _PyFunction_Vectorcall + 0x6c (0x5602d7445a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5602d74368fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #26: PyObject_Call + 0xbc (0x55e37d095f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #30: <unknown function> + 0x150582 (0x5602d7451582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55e37d07c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55e37d089a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank61]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5602d74368fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #32: <unknown function> + 0x150582 (0x5602d7451582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55e37d07a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:     return forward_call(*args, **kwargs)
[default2]:[rank58]: frame #30: <unknown function> + 0x150582 (0x55e37d095582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5602d74368fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55e37d07a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #32: <unknown function> + 0x150582 (0x55e37d095582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default2]:[rank58]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55e37d07a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #34: <unknown function> + 0x150582 (0x5602d7451582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:     sharded_logits = self.model(
[default2]:[rank58]: frame #34: <unknown function> + 0x150582 (0x55e37d095582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank63]:     return self._call_impl(*args, **kwargs)
[default2]:[rank58]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55e37d07a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5602d74368fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55e37d081f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank58]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55e37d093c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:     return forward_call(*args, **kwargs)
[default2]:[rank58]: frame #38: <unknown function> + 0x211239 (0x55e37d156239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default2]:[rank58]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55e37d082a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default5]:[rank61]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x5602d743df50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55e37d07e3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55e37d089a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55e37d079c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default5]:[rank61]: frame #37: _PyObject_Call_Prepend + 0x69 (0x5602d744fc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank61]: frame #38: <unknown function> + 0x211239 (0x5602d7512239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:     return self._call_impl(*args, **kwargs)
[default2]:[rank58]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55e37d089a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55e37d07a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank63]:     return forward_call(*args, **kwargs)
[default2]:[rank58]: frame #45: <unknown function> + 0x150582 (0x55e37d095582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #39: _PyObject_MakeTpCall + 0x26b (0x5602d743ea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5602d743a3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default2]:[rank58]: frame #46: PyObject_Call + 0xbc (0x55e37d095f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #41: _PyFunction_Vectorcall + 0x6c (0x5602d7445a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x5602d7435c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default2]:[rank58]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55e37d07c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #43: _PyFunction_Vectorcall + 0x6c (0x5602d7445a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #48: <unknown function> + 0x150582 (0x55e37d095582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #49: PyObject_Call + 0xbc (0x55e37d095f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5602d74368fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55e37d07c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55e37d089a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55e37d082007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #45: <unknown function> + 0x150582 (0x5602d7451582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #46: PyObject_Call + 0xbc (0x5602d7451f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default7]:[rank63]:     pipeline_state.run_communication()
[default5]:[rank61]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5602d74382b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55e37d093c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default2]:[rank58]: frame #54: <unknown function> + 0x211239 (0x55e37d156239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:     recv_activation_tensor = recv_activation()
[default5]:[rank61]: frame #48: <unknown function> + 0x150582 (0x5602d7451582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #49: PyObject_Call + 0xbc (0x5602d7451f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default7]:[rank63]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default5]:[rank61]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5602d74382b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #51: _PyFunction_Vectorcall + 0x6c (0x5602d7445a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default2]:[rank58]: frame #55: PyObject_Call + 0x207 (0x55e37d096067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default5]:[rank61]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x5602d743e007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default7]:[rank63]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default2]:[rank58]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55e37d07c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #57: <unknown function> + 0x150582 (0x55e37d095582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55e37d07a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default7]:[rank63]:     dist.recv(
[default3]:[rank59]: Traceback (most recent call last):
[default2]:[rank58]: frame #59: <unknown function> + 0x150582 (0x55e37d095582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #60: PyObject_Call + 0xbc (0x55e37d095f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #53: _PyObject_Call_Prepend + 0x69 (0x5602d744fc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default2]:[rank58]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55e37d07c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #54: <unknown function> + 0x211239 (0x5602d7512239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default2]:[rank58]: frame #62: <unknown function> + 0x150582 (0x55e37d095582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:     return func(*args, **kwargs)
[default3]:[rank59]:     trainer.train(dataloader)
[default2]:[rank58]: frame #63: PyObject_Call + 0xbc (0x55e37d095f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default5]:[rank61]: frame #55: PyObject_Call + 0x207 (0x5602d7452067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default5]:[rank61]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5602d74382b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default5]:[rank61]: frame #57: <unknown function> + 0x150582 (0x5602d7451582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank61]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5602d74368fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:     pg.recv([tensor], group_src_rank, tag).wait()
[default5]:[rank61]: frame #59: <unknown function> + 0x150582 (0x5602d7451582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default3]:[rank59]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default5]:[rank61]: frame #60: PyObject_Call + 0xbc (0x5602d7451f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default5]:[rank61]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5602d74382b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default7]:[rank63]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f6d1cf51897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default7]:[rank63]: frame #1: <unknown function> + 0x5b3a23e (0x7f6d56a6e23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default3]:[rank59]:     output = model(**micro_batch)
[default7]:[rank63]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f6d56a68c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #62: <unknown function> + 0x150582 (0x5602d7451582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #63: PyObject_Call + 0xbc (0x5602d7451f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank59]:     return self._call_impl(*args, **kwargs)
[default5]:[rank61]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default7]:[rank63]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f6d56a68f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f6d56a69fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank59]:     return forward_call(*args, **kwargs)
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default2]:[rank42]: Traceback (most recent call last):
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default2]:[rank42]:     trainer.train(dataloader)
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default2]:[rank42]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default2]:[rank42]:     outputs = self.pipeline_engine.train_batch_iter(
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default2]:[rank42]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanot[default3]:[rank59]:     sharded_logits = self.model(
[default3]:[rank59]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank59]:     return self._call_impl(*args, **kwargs)
[default3]:[rank59]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank59]:     return forward_call(*args, **kwargs)
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default3]:[rank59]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
ron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default2]:[rank42]:     output = model(**micro_batch)
[default2]:[rank42]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank63]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6d56a1e371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]:     return self._call_impl(*args, **kwargs)
[default2]:[rank42]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank42]:     return forward_call(*args, **kwargs)
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default2]:[rank42]:     sharded_logits = self.model(
[default2]:[rank42]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank42]:     return self._call_impl(*args, **kwargs)
[default2]:[rank42]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank42]:     return forward_call(*args, **kwargs)
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotro[default7]:[rank63]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6d56a1e371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
n/src/nanotron/models/llama.py", line 764, in forward
[default2]:[rank42]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default2]:[rank42]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default2]:[rank42]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank42]:     return self._call_impl(*args, **kwargs)
[default2]:[rank42]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank42]:     return forward_call(*args, **kwargs)
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states

[default2]:[rank42]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default7]:[rank63]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6d56a1e371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6d56a1e371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]:     pipeline_state.run_communication()
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default2]:[rank42]:     recv_activation_tensor = recv_activation()
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default2]:[rank42]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default2]:[rank42]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default2]:[rank42]:     meta = self._recv_[default3]:[rank59]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
meta(from_rank=from_rank, tag=tag)
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default2]:[rank42]:     dist.recv(
[default2]:[rank42]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default2]:[rank42]:     return func(*args, **kwargs)
[default2]:[rank42]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default2]:[rank42]:     pg.recv([tensor], group_src_rank, tag).wait()
[default2]:[rank42]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default2]:[rank42]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most rece[default7]:[rank63]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f6d1e22b189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
nt call first):
[default2]:[rank42]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f2c2a613897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default2]:[rank42]: frame #1: <unknown function> + 0x5b3a23e (0x7f2c6413023e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank59]:     return self._call_impl(*args, **kwargs)
[default2]:[rank42]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f2c6412ac87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f2c6412af82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f6d1e232610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank42]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f2c6412bfd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2c640e0371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank42]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2c640e0371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2c640e0371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2c640e0371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f6d1e251978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank63]: frame #12: <unknown function> + 0x5adc309 (0x7f6d56a10309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f2c2b8ed189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank42]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f2c2b8f4610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank42]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f2c2b913978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:[rank59]:     return forward_call(*args, **kwargs)
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default2]:[rank42]: frame #12: <unknown function> + 0x5adc309 (0x7f2c640d2309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #13: <unknown function> + 0x5ae6f10 (0x7f2c640dcf10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default7]:[rank63]: frame #13: <unknown function> + 0x5ae6f10 (0x7f6d56a1af10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f2c640dcfa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #15: <unknown function> + 0x5124446 (0x7f2c6371a446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #16: <unknown function> + 0x1acf4b8 (0x7f2c600c54b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #17: <unknown function> + 0x5aee004 (0x7f2c640e4004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default2]:[rank42]: frame #18: <unknown function> + 0x5af36b5 (0x7f2c640e96b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #19: <unknown function> + 0xd2631e (0x7f2c76cd331e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default2]:[rank42]: frame #20: <unknown function> + 0x47def4 (0x7f2c7642aef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default3]:[rank59]:     pipeline_state.run_communication()
[default2]:[rank42]: frame #21: <unknown function> + 0x1445a6 (0x56018ace25a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #22: _PyObject_MakeTpCall + 0x26b (0x56018acdba6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #23: <unknown function> + 0x150866 (0x56018acee866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f6d56a1afa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x56018acd7142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #25: _PyFunction_Vectorcall + 0x6c (0x56018ace2a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #26: PyObject_Call + 0xbc (0x56018aceef1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x56018acd52b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #28: _PyFunction_Vectorcall + 0x6c (0x56018ace2a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x56018acd38fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #15: <unknown function> + 0x5124446 (0x7f6d56058446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #16: <unknown function> + 0x1acf4b8 (0x7f6d52a034b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #17: <unknown function> + 0x5aee004 (0x7f6d56a22004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #30: <unknown function> + 0x150582 (0x56018acee582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x56018acd38fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #32: <unknown function> + 0x150582 (0x56018acee582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default2]:[rank42]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x56018acd38fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #34: <unknown function> + 0x150582 (0x56018acee582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x56018acd38fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #18: <unknown function> + 0x5af36b5 (0x7f6d56a276b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #19: <unknown function> + 0xd2631e (0x7f6d6961131e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:[rank63]: frame #20: <unknown function> + 0x47def4 (0x7f6d68d68ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default2]:[rank42]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x56018acdaf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #37: _PyObject_Call_Prepend + 0x69 (0x56018acecc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #21: <unknown function> + 0x1445a6 (0x557741fd85a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #38: <unknown function> + 0x211239 (0x56018adaf239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #39: _PyObject_MakeTpCall + 0x26b (0x56018acdba6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x56018acd73e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]:     recv_activation_tensor = recv_activation()
[default2]:[rank42]: frame #41: _PyFunction_Vectorcall + 0x6c (0x56018ace2a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x56018acd2c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #43: _PyFunction_Vectorcall + 0x6c (0x56018ace2a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x56018acd38fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default2]:[rank42]: frame #45: <unknown function> + 0x150582 (0x56018acee582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #46: PyObject_Call + 0xbc (0x56018aceef1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x56018acd52b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #22: _PyObject_MakeTpCall + 0x26b (0x557741fd1a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #23: <unknown function> + 0x150866 (0x557741fe4866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #48: <unknown function> + 0x150582 (0x56018acee582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #49: PyObject_Call + 0xbc (0x56018aceef1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x56018acd52b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x557741fcd142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #51: _PyFunction_Vectorcall + 0x6c (0x56018ace2a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x56018acdb007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #53: _PyObject_Call_Prepend + 0x69 (0x56018acecc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default2]:[rank42]: frame #54: <unknown function> + 0x211239 (0x56018adaf239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #55: PyObject_Call + 0x207 (0x56018acef067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x56018acd52b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #57: <unknown function> + 0x150582 (0x56018acee582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x56018acd38fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #59: <unknown function> + 0x150582 (0x56018acee582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #60: PyObject_Call + 0xbc (0x56018aceef1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/py[default7]:[rank63]: frame #25: _PyFunction_Vectorcall + 0x6c (0x557741fd8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
thon3.10)
[default2]:[rank42]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x56018acd52b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #62: <unknown function> + 0x150582 (0x56018acee582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #63: PyObject_Call + 0xbc (0x56018aceef1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default3]:[rank59]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default7]:[rank63]: frame #26: PyObject_Call + 0xbc (0x557741fe4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default7]:[rank63]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x557741fcb2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #28: _PyFunction_Vectorcall + 0x6c (0x557741fd8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default3]:[rank59]:     dist.recv(
[default3]:[rank59]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default3]:[rank59]:     return func(*args, **kwargs)
[default3]:[rank59]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default3]:[rank59]:     pg.recv([tensor], group_src_rank, tag).wait()
[default3]:[rank59]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default7]:[rank63]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x557741fc98fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default7]:[rank63]: frame #30: <unknown function> + 0x150582 (0x557741fe4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f77f9c18897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default7]:[rank63]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x557741fc98fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #32: <unknown function> + 0x150582 (0x557741fe4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x557741fc98fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #1: <unknown function> + 0x5b3a23e (0x7f783373523e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #34: <unknown function> + 0x150582 (0x557741fe4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x557741fc98fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x557741fd0f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #37: _PyObject_Call_Prepend + 0x69 (0x557741fe2c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #38: <unknown function> + 0x211239 (0x5577420a5239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f783372fc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f783372ff82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f7833730fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f78336e5371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f78336e5371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f78336e5371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #39: _PyObject_MakeTpCall + 0x26b (0x557741fd1a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f78336e5371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f77faef2189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank63]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x557741fcd3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f77faef9610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:[rank59]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f77faf18978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:[rank59]: frame #12: <unknown function> + 0x5adc309 (0x7f78336d7309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]: frame #13: <unknown function> + 0x5ae6f10 (0x7f78336e1f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #41: _PyFunction_Vectorcall + 0x6c (0x557741fd8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f78336e1fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x557741fc8c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #15: <unknown function> + 0x5124446 (0x7f7832d1f446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]: frame #16: <unknown function> + 0x1acf4b8 (0x7f782f6ca4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #43: _PyFunction_Vectorcall + 0x6c (0x557741fd8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x557741fc98fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #17: <unknown function> + 0x5aee004 (0x7f78336e9004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #45: <unknown function> + 0x150582 (0x557741fe4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #46: PyObject_Call + 0xbc (0x557741fe4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #18: <unknown function> + 0x5af36b5 (0x7f78336ee6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x557741fcb2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #19: <unknown function> + 0xd2631e (0x7f78462d831e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:[rank63]: frame #48: <unknown function> + 0x150582 (0x557741fe4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #20: <unknown function> + 0x47def4 (0x7f7845a2fef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:[rank63]: frame #49: PyObject_Call + 0xbc (0x557741fe4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #21: <unknown function> + 0x1445a6 (0x556d6b1565a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x557741fcb2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #51: _PyFunction_Vectorcall + 0x6c (0x557741fd8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #22: _PyObject_MakeTpCall + 0x26b (0x556d6b14fa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x557741fd1007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #53: _PyObject_Call_Prepend + 0x69 (0x557741fe2c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #54: <unknown function> + 0x211239 (0x5577420a5239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #55: PyObject_Call + 0x207 (0x557741fe5067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x557741fcb2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #57: <unknown function> + 0x150582 (0x557741fe4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x557741fc98fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #59: <unknown function> + 0x150582 (0x557741fe4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #60: PyObject_Call + 0xbc (0x557741fe4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x557741fcb2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #62: <unknown function> + 0x150582 (0x557741fe4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #23: <unknown function> + 0x150866 (0x556d6b162866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #63: PyObject_Call + 0xbc (0x557741fe4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default3]:[rank59]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x556d6b14b142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #25: _PyFunction_Vectorcall + 0x6c (0x556d6b156a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #26: PyObject_Call + 0xbc (0x556d6b162f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x556d6b1492b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #28: _PyFunction_Vectorcall + 0x6c (0x556d6b156a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x556d6b1478fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #30: <unknown function> + 0x150582 (0x556d6b162582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x556d6b1478fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #32: <unknown function> + 0x150582 (0x556d6b162582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x556d6b1478fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #34: <unknown function> + 0x150582 (0x556d6b162582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x556d6b1478fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x556d6b14ef50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #37: _PyObject_Call_Prepend + 0x69 (0x556d6b160c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #38: <unknown function> + 0x211239 (0x556d6b223239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #39: _PyObject_MakeTpCall + 0x26b (0x556d6b14fa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x556d6b14b3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #41: _PyFunction_Vectorcall + 0x6c (0x556d6b156a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x556d6b146c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #43: _PyFunction_Vectorcall + 0x6c (0x556d6b156a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x556d6b1478fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #45: <unknown function> + 0x150582 (0x556d6b162582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #46: PyObject_Call + 0xbc (0x556d6b162f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x556d6b1492b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #48: <unknown function> + 0x150582 (0x556d6b162582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #49: PyObject_Call + 0xbc (0x556d6b162f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x556d6b1492b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #51: _PyFunction_Vectorcall + 0x6c (0x556d6b156a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x556d6b14f007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #53: _PyObject_Call_Prepend + 0x69 (0x556d6b160c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #54: <unknown function> + 0x211239 (0x556d6b223239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #55: PyObject_Call + 0x207 (0x556d6b163067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x556d6b1492b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #57: <unknown function> + 0x150582 (0x556d6b162582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x556d6b1478fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #59: <unknown function> + 0x150582 (0x556d6b162582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #60: PyObject_Call + 0xbc (0x556d6b162f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x556d6b1492b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #62: <unknown function> + 0x150582 (0x556d6b162582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #63: PyObject_Call + 0xbc (0x556d6b162f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default4]:[rank60]: Traceback (most recent call last):
[default6]:[rank62]: Traceback (most recent call last):
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default6]:[rank62]:     trainer.train(dataloader)
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default6]:[rank62]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default6]:[rank62]:     outputs = self.pipeline_engine.train_batch_iter(
[default4]:[rank60]:     trainer.train(dataloader)
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default6]:[rank62]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default6]:[rank62]:     output = model(**micro_batch)
[default4]:[rank60]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default6]:[rank62]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank62]:     return self._call_impl(*args, **kwargs)
[default6]:[rank62]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank62]:     return forward_call(*args, **kwargs)
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default4]:[rank60]:     outputs = self.pipeline_engine.train_batch_iter(
[default6]:[rank62]:     sharded_logits = self.model(
[default6]:[rank62]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank62]:     return self._call_impl(*args, **kwargs)
[default6]:[rank62]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank62]:     return forward_call(*args, **kwargs)
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default3]:[rank43]: Traceback (most recent call last):
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default6]:[rank62]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default6]:[rank62]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default4]:[rank60]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default6]:[rank62]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank62]:     return self._call_impl(*args, **kwargs)
[default6]:[rank62]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank62]:     return forward_call(*args, **kwargs)
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default6]:[rank62]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default3]:[rank43]:     trainer.train(dataloader)
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default0]:[rank40]: Traceback (most recent call last):
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default3]:[rank43]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default4]:[rank60]:     output = model(**micro_batch)
[default6]:[rank62]:     pipeline_state.run_communication()
[default4]:[rank60]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default6]:[rank62]:     recv_activation_tensor = recv_activation()
[default4]:[rank60]:     return self._call_impl(*args, **kwargs)
[default4]:[rank60]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank60]:     return forward_call(*args, **kwargs)
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default0]:[rank40]:     trainer.train(dataloader)
[default7]:[rank47]: Traceback (most recent call last):
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default3]:[rank43]:     outputs = self.pipeline_engine.train_batch_iter(
[default4]:[rank44]: Traceback (most recent call last):
[default7]:[rank47]:     trainer.train(dataloader)
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default6]:[rank62]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default4]:[rank60]:     sharded_logits = self.model(
[default4]:[rank60]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank60]:     return self._call_impl(*args, **kwargs)
[default4]:[rank60]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank60]:     return forward_call(*args, **kwargs)
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default6]:[rank62]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default4]:[rank60]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank62]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default4]:[rank60]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default4]:[rank60]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default0]:[rank40]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default3]:[rank43]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default4]:[rank44]:     trainer.train(dataloader)
[default6]:[rank62]:     dist.recv(
[default4]:[rank60]:     return self._call_impl(*args, **kwargs)
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default0]:[rank40]:     outputs = self.pipeline_engine.train_batch_iter(
[default6]:[rank62]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default6]:[rank62]:     return func(*args, **kwargs)
[default6]:[rank62]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default6]:[rank62]:     pg.recv([tensor], group_src_rank, tag).wait()
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default0]:[rank40]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default4]:[rank60]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank43]:     output = model(**micro_batch)
[default6]:[rank62]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default4]:[rank60]:     return forward_call(*args, **kwargs)
[default6]:[rank62]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default6]:[rank62]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fa24ff8d897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default7]:[rank47]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default6]:[rank62]: frame #1: <unknown function> + 0x5b3a23e (0x7fa289aaa23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default6]:[rank62]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fa289aa4c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank60]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default6]:[rank62]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fa289aa4f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]:     output = model(**micro_batch)
[default6]:[rank62]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fa289aa5fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default3]:[rank43]:     return self._call_impl(*args, **kwargs)
[default4]:[rank44]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default6]:[rank62]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa289a5a371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa289a5a371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank60]:     pipeline_state.run_communication()
[default7]:[rank47]:     outputs = self.pipeline_engine.train_batch_iter(
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default6]:[rank62]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa289a5a371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa289a5a371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default6]:[rank62]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fa251267189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank44]:     outputs = self.pipeline_engine.train_batch_iter(
[default4]:[rank60]:     recv_activation_tensor = recv_activation()
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default3]:[rank43]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank60]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default6]:[rank62]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fa25126e610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank62]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fa25128d978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank62]: frame #12: <unknown function> + 0x5adc309 (0x7fa289a4c309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #13: <unknown function> + 0x5ae6f10 (0x7fa289a56f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank60]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default6]:[rank62]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fa289a56fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default4]:[rank60]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default4]:[rank44]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default3]:[rank43]:     return forward_call(*args, **kwargs)
[default6]:[rank62]: frame #15: <unknown function> + 0x5124446 (0x7fa289094446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #16: <unknown function> + 0x1acf4b8 (0x7fa285a3f4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default6]:[rank62]: frame #17: <unknown function> + 0x5aee004 (0x7fa289a5e004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]:     return self._call_impl(*args, **kwargs)
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default4]:[rank60]:     dist.recv(
[default6]:[rank62]: frame #18: <unknown function> + 0x5af36b5 (0x7fa289a636b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default4]:[rank60]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default0]:[rank40]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank62]: frame #19: <unknown function> + 0xd2631e (0x7fa29c64d31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default3]:[rank43]:     sharded_logits = self.model(
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default6]:[rank62]: frame #20: <unknown function> + 0x47def4 (0x7fa29bda4ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:[rank60]:     return func(*args, **kwargs)
[default0]:[rank40]:     return forward_call(*args, **kwargs)
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default6]:[rank62]: frame #21: <unknown function> + 0x1445a6 (0x5646d09715a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #22: _PyObject_MakeTpCall + 0x26b (0x5646d096aa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default6]:[rank62]: frame #23: <unknown function> + 0x150866 (0x5646d097d866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x5646d0966142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #25: _PyFunction_Vectorcall + 0x6c (0x5646d0971a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #26: PyObject_Call + 0xbc (0x5646d097df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5646d09642b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #28: _PyFunction_Vectorcall + 0x6c (0x5646d0971a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default4]:[rank60]:     pg.recv([tensor], group_src_rank, tag).wait()
[default6]:[rank62]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5646d09628fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #30: <unknown function> + 0x150582 (0x5646d097d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5646d09628fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #32: <unknown function> + 0x150582 (0x5646d097d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]:     sharded_logits = self.model(
[default4]:[rank60]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default3]:[rank43]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank60]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default4]:[rank60]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f17a72bf897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default7]:[rank47]:     output = model(**micro_batch)
[default4]:[rank60]: frame #1: <unknown function> + 0x5b3a23e (0x7f17e0ddc23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank60]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f17e0dd6c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5646d09628fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]:     return self._call_impl(*args, **kwargs)
[default4]:[rank60]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f17e0dd6f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank60]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f17e0dd7fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank47]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank40]:     return self._call_impl(*args, **kwargs)
[default4]:[rank44]:     output = model(**micro_batch)
[default6]:[rank62]: frame #34: <unknown function> + 0x150582 (0x5646d097d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]:     return self._call_impl(*args, **kwargs)
[default6]:[rank62]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5646d09628fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f17e0d8c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]:     return forward_call(*args, **kwargs)
[default6]:[rank62]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x5646d0969f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #37: _PyObject_Call_Prepend + 0x69 (0x5646d097bc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank47]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank60]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f17e0d8c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default6]:[rank62]: frame #38: <unknown function> + 0x211239 (0x5646d0a3e239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank60]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f17e0d8c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #39: _PyObject_MakeTpCall + 0x26b (0x5646d096aa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5646d09663e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #41: _PyFunction_Vectorcall + 0x6c (0x5646d0971a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x5646d0961c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f17e0d8c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank60]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f17a8599189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank62]: frame #43: _PyFunction_Vectorcall + 0x6c (0x5646d0971a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5646d09628fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #45: <unknown function> + 0x150582 (0x5646d097d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #46: PyObject_Call + 0xbc (0x5646d097df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f17a85a0610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank60]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f17a85bf978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank44]:     return self._call_impl(*args, **kwargs)
[default4]:[rank60]: frame #12: <unknown function> + 0x5adc309 (0x7f17e0d7e309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5646d09642b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #13: <unknown function> + 0x5ae6f10 (0x7f17e0d88f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank43]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank62]: frame #48: <unknown function> + 0x150582 (0x5646d097d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]:     return forward_call(*args, **kwargs)
[default6]:[rank62]: frame #49: PyObject_Call + 0xbc (0x5646d097df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]:     return forward_call(*args, **kwargs)
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default4]:[rank60]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f17e0d88fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default6]:[rank62]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5646d09642b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default6]:[rank62]: frame #51: _PyFunction_Vectorcall + 0x6c (0x5646d0971a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #15: <unknown function> + 0x5124446 (0x7f17e03c6446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank62]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x5646d096a007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default4]:[rank60]: frame #16: <unknown function> + 0x1acf4b8 (0x7f17dcd714b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank62]: frame #53: _PyObject_Call_Prepend + 0x69 (0x5646d097bc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #54: <unknown function> + 0x211239 (0x5646d0a3e239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]:     sharded_logits = self.model(
[default7]:[rank47]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default4]:[rank44]:     return forward_call(*args, **kwargs)
[default7]:[rank47]:     return self._call_impl(*args, **kwargs)
[default0]:[rank40]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default3]:[rank43]:     return self._call_impl(*args, **kwargs)
[default6]:[rank62]: frame #55: PyObject_Call + 0x207 (0x5646d097e067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5646d09642b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]:     sharded_logits = self.model(
[default6]:[rank62]: frame #57: <unknown function> + 0x150582 (0x5646d097d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5646d09628fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank62]: frame #59: <unknown function> + 0x150582 (0x5646d097d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #60: PyObject_Call + 0xbc (0x5646d097df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5646d09642b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #17: <unknown function> + 0x5aee004 (0x7f17e0d90004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank62]: frame #62: <unknown function> + 0x150582 (0x5646d097d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank62]: frame #63: PyObject_Call + 0xbc (0x5646d097df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]:     return forward_call(*args, **kwargs)
[default0]:[rank40]:     return self._call_impl(*args, **kwargs)
[default0]:[rank40]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank62]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default4]:[rank60]: frame #18: <unknown function> + 0x5af36b5 (0x7f17e0d956b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank60]: frame #19: <unknown function> + 0xd2631e (0x7f17f397f31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:[rank44]:     return self._call_impl(*args, **kwargs)
[default0]:[rank40]:     return forward_call(*args, **kwargs)
[default3]:[rank43]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank47]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default4]:[rank60]: frame #20: <unknown function> + 0x47def4 (0x7f17f30d6ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:[rank60]: frame #21: <unknown function> + 0x1445a6 (0x55b5a95d75a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55b5a95d0a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #23: <unknown function> + 0x150866 (0x55b5a95e3866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55b5a95cc142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55b5a95d7a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #26: PyObject_Call + 0xbc (0x55b5a95e3f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55b5a95ca2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55b5a95d7a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55b5a95c88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #30: <unknown function> + 0x150582 (0x55b5a95e3582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55b5a95c88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #32: <unknown function> + 0x150582 (0x55b5a95e3582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55b5a95c88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #34: <unknown function> + 0x150582 (0x55b5a95e3582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55b5a95c88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55b5a95cff50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55b5a95e1c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #38: <unknown function> + 0x211239 (0x55b5a96a4239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default4]:[rank60]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55b5a95d0a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55b5a95cc3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55b5a95d7a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55b5a95c7c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default4]:[rank60]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55b5a95d7a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55b5a95c88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #45: <unknown function> + 0x150582 (0x55b5a95e3582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #46: PyObject_Call + 0xbc (0x55b5a95e3f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default0]:[rank40]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default3]:[rank43]:     return forward_call(*args, **kwargs)
[default4]:[rank60]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55b5a95ca2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #48: <unknown function> + 0x150582 (0x55b5a95e3582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #49: PyObject_Call + 0xbc (0x55b5a95e3f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank60]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55b5a95ca2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55b5a95d7a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55b5a95d0007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55b5a95e1c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default7]:[rank47]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank60]: frame #54: <unknown function> + 0x211239 (0x55b5a96a4239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #55: PyObject_Call + 0x207 (0x55b5a95e4067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55b5a95ca2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]:     return self._call_impl(*args, **kwargs)
[default0]:[rank40]:     pipeline_state.run_communication()
[default4]:[rank60]: frame #57: <unknown function> + 0x150582 (0x55b5a95e3582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55b5a95c88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #59: <unknown function> + 0x150582 (0x55b5a95e3582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #60: PyObject_Call + 0xbc (0x55b5a95e3f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55b5a95ca2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #62: <unknown function> + 0x150582 (0x55b5a95e3582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #63: PyObject_Call + 0xbc (0x55b5a95e3f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/pyt[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default7]:[rank47]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
hon3.10)
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default3]:[rank43]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default4]:[rank60]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default7]:[rank47]:     return forward_call(*args, **kwargs)
[default4]:[rank44]:     return forward_call(*args, **kwargs)
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default7]:[rank47]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default4]:[rank44]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default0]:[rank40]:     recv_activation_tensor = recv_activation()
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default3]:[rank43]:     pipeline_state.run_communication()
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default0]:[rank40]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default7]:[rank47]:     pipeline_state.run_communication()
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default4]:[rank44]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default0]:[rank40]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default4]:[rank44]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default4]:[rank44]:     return self._call_impl(*args, **kwargs)
[default7]:[rank47]:     recv_activation_tensor = recv_activation()
[default0]:[rank40]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default3]:[rank43]:     recv_activation_tensor = recv_activation()
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default4]:[rank44]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank47]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default3]:[rank43]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default4]:[rank44]:     return forward_call(*args, **kwargs)
[default0]:[rank40]:     dist.recv(
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default3]:[rank43]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default4]:[rank44]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default0]:[rank40]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default0]:[rank40]:     return func(*args, **kwargs)
[default7]:[rank47]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default4]:[rank44]:     pipeline_state.run_communication()
[default0]:[rank40]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default3]:[rank43]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default0]:[rank40]:     pg.recv([tensor], group_src_rank, tag).wait()
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default7]:[rank47]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default4]:[rank44]:     recv_activation_tensor = recv_activation()
[default3]:[rank43]:     dist.recv(
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default7]:[rank47]:     dist.recv(
[default3]:[rank43]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default0]:[rank40]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default7]:[rank47]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default3]:[rank43]:     return func(*args, **kwargs)
[default4]:[rank44]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default7]:[rank47]:     return func(*args, **kwargs)
[default3]:[rank43]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default7]:[rank47]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default3]:[rank43]:     pg.recv([tensor], group_src_rank, tag).wait()
[default4]:[rank44]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default3]:[rank43]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default7]:[rank47]:     pg.recv([tensor], group_src_rank, tag).wait()
[default3]:[rank43]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default3]:[rank43]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f618ddaa897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default7]:[rank47]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default4]:[rank44]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default3]:[rank43]: frame #1: <unknown function> + 0x5b3a23e (0x7f61c78c723e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default7]:[rank47]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default3]:[rank43]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f61c78c1c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f0852fa2897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default4]:[rank44]:     dist.recv(
[default3]:[rank43]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f61c78c1f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #1: <unknown function> + 0x5b3a23e (0x7f088cabf23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f7a0d442897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default4]:[rank44]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default7]:[rank47]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f088cab9c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #1: <unknown function> + 0x5b3a23e (0x7f7a46f5f23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f7a46f59c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f088cab9f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]:     return func(*args, **kwargs)
[default0]:[rank40]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f7a46f59f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f088cabafd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f61c78c2fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f7a46f5afd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f088ca6f371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default3]:[rank43]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f61c7877371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f088ca6f371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]:     pg.recv([tensor], group_src_rank, tag).wait()
[default0]:[rank40]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7a46f0f371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f088ca6f371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f088ca6f371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default0]:[rank40]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7a46f0f371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f085427c189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:[rank43]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f61c7877371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7a46f0f371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f0854283610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank44]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default3]:[rank43]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f61c7877371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7a46f0f371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fa2fa9c7897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default3]:[rank43]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f61c7877371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f7a0e71c189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank44]: frame #1: <unknown function> + 0x5b3a23e (0x7fa3344e423e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f08542a2978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:[rank43]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f618f084189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank44]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fa3344dec87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f7a0e723610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank47]: frame #12: <unknown function> + 0x5adc309 (0x7f088ca61309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f618f08b610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank40]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f7a0e742978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank47]: frame #13: <unknown function> + 0x5ae6f10 (0x7f088ca6bf10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f618f0aa978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank44]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fa3344def82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #12: <unknown function> + 0x5adc309 (0x7f7a46f01309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f088ca6bfa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fa3344dffd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #12: <unknown function> + 0x5adc309 (0x7f61c7869309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #15: <unknown function> + 0x5124446 (0x7f088c0a9446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa334494371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #13: <unknown function> + 0x5ae6f10 (0x7f61c7873f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #13: <unknown function> + 0x5ae6f10 (0x7f7a46f0bf10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #16: <unknown function> + 0x1acf4b8 (0x7f0888a544b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f61c7873fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa334494371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f7a46f0bfa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #15: <unknown function> + 0x5124446 (0x7f61c6eb1446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #17: <unknown function> + 0x5aee004 (0x7f088ca73004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #15: <unknown function> + 0x5124446 (0x7f7a46549446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa334494371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #18: <unknown function> + 0x5af36b5 (0x7f088ca786b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #16: <unknown function> + 0x1acf4b8 (0x7f7a42ef44b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #16: <unknown function> + 0x1acf4b8 (0x7f61c385c4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #19: <unknown function> + 0xd2631e (0x7f089f66231e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default3]:[rank43]: frame #17: <unknown function> + 0x5aee004 (0x7f61c787b004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #20: <unknown function> + 0x47def4 (0x7f089edb9ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:[rank44]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa334494371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #18: <unknown function> + 0x5af36b5 (0x7f61c78806b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #17: <unknown function> + 0x5aee004 (0x7f7a46f13004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fa2fbca1189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank40]: frame #18: <unknown function> + 0x5af36b5 (0x7f7a46f186b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #19: <unknown function> + 0xd2631e (0x7f61da46a31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:[rank47]: frame #21: <unknown function> + 0x1445a6 (0x56538ab505a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #19: <unknown function> + 0xd2631e (0x7f7a59b0231e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default3]:[rank43]: frame #20: <unknown function> + 0x47def4 (0x7f61d9bc1ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:[rank47]: frame #22: _PyObject_MakeTpCall + 0x26b (0x56538ab49a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #20: <unknown function> + 0x47def4 (0x7f7a59259ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default0]:[rank40]: frame #21: <unknown function> + 0x1445a6 (0x55a86cfc35a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55a86cfbca6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #23: <unknown function> + 0x150866 (0x56538ab5c866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fa2fbca8610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:[rank43]: frame #21: <unknown function> + 0x1445a6 (0x55ffae5175a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x56538ab45142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fa2fbcc7978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank47]: frame #25: _PyFunction_Vectorcall + 0x6c (0x56538ab50a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #12: <unknown function> + 0x5adc309 (0x7fa334486309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55ffae510a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #23: <unknown function> + 0x150866 (0x55a86cfcf866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #13: <unknown function> + 0x5ae6f10 (0x7fa334490f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #26: PyObject_Call + 0xbc (0x56538ab5cf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x56538ab432b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55a86cfb8142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55a86cfc3a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #28: _PyFunction_Vectorcall + 0x6c (0x56538ab50a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #26: PyObject_Call + 0xbc (0x55a86cfcff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x56538ab418fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #30: <unknown function> + 0x150582 (0x56538ab5c582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #23: <unknown function> + 0x150866 (0x55ffae523866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55a86cfb62b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55ffae50c142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55a86cfc3a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fa334490fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55ffae517a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x56538ab418fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #15: <unknown function> + 0x5124446 (0x7fa333ace446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #26: PyObject_Call + 0xbc (0x55ffae523f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #32: <unknown function> + 0x150582 (0x56538ab5c582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #16: <unknown function> + 0x1acf4b8 (0x7fa3304794b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55ffae50a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x56538ab418fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #17: <unknown function> + 0x5aee004 (0x7fa334498004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55a86cfb48fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #34: <unknown function> + 0x150582 (0x56538ab5c582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55ffae517a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x56538ab418fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #30: <unknown function> + 0x150582 (0x55a86cfcf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55ffae5088fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x56538ab48f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55a86cfb48fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #30: <unknown function> + 0x150582 (0x55ffae523582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #32: <unknown function> + 0x150582 (0x55a86cfcf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #37: _PyObject_Call_Prepend + 0x69 (0x56538ab5ac39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #38: <unknown function> + 0x211239 (0x56538ac1d239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55ffae5088fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #32: <unknown function> + 0x150582 (0x55ffae523582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #39: _PyObject_MakeTpCall + 0x26b (0x56538ab49a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55a86cfb48fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #18: <unknown function> + 0x5af36b5 (0x7fa33449d6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55ffae5088fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #34: <unknown function> + 0x150582 (0x55a86cfcf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #19: <unknown function> + 0xd2631e (0x7fa34708731e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:[rank47]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x56538ab453e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #34: <unknown function> + 0x150582 (0x55ffae523582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #41: _PyFunction_Vectorcall + 0x6c (0x56538ab50a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #20: <unknown function> + 0x47def4 (0x7fa3467deef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default3]:[rank43]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55ffae5088fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x56538ab40c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #43: _PyFunction_Vectorcall + 0x6c (0x56538ab50a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #21: <unknown function> + 0x1445a6 (0x556080a105a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #22: _PyObject_MakeTpCall + 0x26b (0x556080a09a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x56538ab418fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55a86cfb48fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55ffae50ff50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55ffae521c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #45: <unknown function> + 0x150582 (0x56538ab5c582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #23: <unknown function> + 0x150866 (0x556080a1c866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55a86cfbbf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #38: <unknown function> + 0x211239 (0x55ffae5e4239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x556080a05142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55a86cfcdc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55ffae510a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #25: _PyFunction_Vectorcall + 0x6c (0x556080a10a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #38: <unknown function> + 0x211239 (0x55a86d090239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55ffae50c3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #26: PyObject_Call + 0xbc (0x556080a1cf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55a86cfbca6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55a86cfb83e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #46: PyObject_Call + 0xbc (0x56538ab5cf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55ffae517a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x556080a032b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55a86cfc3a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55ffae507c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x56538ab432b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55a86cfb3c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55ffae517a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55ffae5088fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #48: <unknown function> + 0x150582 (0x56538ab5c582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #28: _PyFunction_Vectorcall + 0x6c (0x556080a10a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55a86cfc3a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #45: <unknown function> + 0x150582 (0x55ffae523582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x556080a018fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #49: PyObject_Call + 0xbc (0x56538ab5cf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #30: <unknown function> + 0x150582 (0x556080a1c582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x556080a018fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55a86cfb48fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x56538ab432b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #46: PyObject_Call + 0xbc (0x55ffae523f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #45: <unknown function> + 0x150582 (0x55a86cfcf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #51: _PyFunction_Vectorcall + 0x6c (0x56538ab50a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x56538ab49007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #32: <unknown function> + 0x150582 (0x556080a1c582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #46: PyObject_Call + 0xbc (0x55a86cfcff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #53: _PyObject_Call_Prepend + 0x69 (0x56538ab5ac39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x556080a018fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55a86cfb62b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55ffae50a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #34: <unknown function> + 0x150582 (0x556080a1c582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #48: <unknown function> + 0x150582 (0x55a86cfcf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #48: <unknown function> + 0x150582 (0x55ffae523582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #54: <unknown function> + 0x211239 (0x56538ac1d239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x556080a018fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x556080a08f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #49: PyObject_Call + 0xbc (0x55ffae523f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #55: PyObject_Call + 0x207 (0x56538ab5d067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x56538ab432b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #37: _PyObject_Call_Prepend + 0x69 (0x556080a1ac39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #49: PyObject_Call + 0xbc (0x55a86cfcff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55ffae50a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #38: <unknown function> + 0x211239 (0x556080add239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55a86cfb62b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55ffae517a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #39: _PyObject_MakeTpCall + 0x26b (0x556080a09a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55a86cfc3a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #57: <unknown function> + 0x150582 (0x56538ab5c582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x556080a053e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55a86cfbc007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55a86cfcdc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x56538ab418fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #41: _PyFunction_Vectorcall + 0x6c (0x556080a10a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #54: <unknown function> + 0x211239 (0x55a86d090239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x556080a00c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #59: <unknown function> + 0x150582 (0x56538ab5c582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #55: PyObject_Call + 0x207 (0x55a86cfd0067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #60: PyObject_Call + 0xbc (0x56538ab5cf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55a86cfb62b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55ffae510007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #43: _PyFunction_Vectorcall + 0x6c (0x556080a10a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #57: <unknown function> + 0x150582 (0x55a86cfcf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55ffae521c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x56538ab432b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #54: <unknown function> + 0x211239 (0x55ffae5e4239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x556080a018fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #62: <unknown function> + 0x150582 (0x56538ab5c582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55a86cfb48fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #45: <unknown function> + 0x150582 (0x556080a1c582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #59: <unknown function> + 0x150582 (0x55a86cfcf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #46: PyObject_Call + 0xbc (0x556080a1cf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #55: PyObject_Call + 0x207 (0x55ffae524067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #60: PyObject_Call + 0xbc (0x55a86cfcff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #63: PyObject_Call + 0xbc (0x56538ab5cf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55ffae50a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #57: <unknown function> + 0x150582 (0x55ffae523582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default0]:[rank40]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55a86cfb62b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55ffae5088fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x556080a032b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #62: <unknown function> + 0x150582 (0x55a86cfcf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #59: <unknown function> + 0x150582 (0x55ffae523582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #63: PyObject_Call + 0xbc (0x55a86cfcff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default3]:[rank43]: frame #60: PyObject_Call + 0xbc (0x55ffae523f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #48: <unknown function> + 0x150582 (0x556080a1c582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #49: PyObject_Call + 0xbc (0x556080a1cf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x556080a032b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55ffae50a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #62: <unknown function> + 0x150582 (0x55ffae523582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #63: PyObject_Call + 0xbc (0x55ffae523f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default4]:[rank44]: frame #51: _PyFunction_Vectorcall + 0x6c (0x556080a10a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x556080a09007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #53: _PyObject_Call_Prepend + 0x69 (0x556080a1ac39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #54: <unknown function> + 0x211239 (0x556080add239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #55: PyObject_Call + 0x207 (0x556080a1d067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x556080a032b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #57: <unknown function> + 0x150582 (0x556080a1c582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x556080a018fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #59: <unknown function> + 0x150582 (0x556080a1c582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #60: PyObject_Call + 0xbc (0x556080a1cf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x556080a032b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #62: <unknown function> + 0x150582 (0x556080a1c582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #63: PyObject_Call + 0xbc (0x556080a1cf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default5]:[rank45]: Traceback (most recent call last):
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default5]:[rank45]:     trainer.train(dataloader)
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default5]:[rank45]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default5]:[rank45]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default5]:[rank45]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default5]:[rank45]:     output = model(**micro_batch)
[default5]:[rank45]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank45]:     return self._call_impl(*args, **kwargs)
[default5]:[rank45]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank45]:     return forward_call(*args, **kwargs)
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default5]:[rank45]:     sharded_logits = self.model(
[default5]:[rank45]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank45]:     return self._call_impl(*args, **kwargs)
[default5]:[rank45]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank45]:     return forward_call(*args, **kwargs)
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default5]:[rank45]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default5]:[rank45]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default5]:[rank45]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank45]:     return self._call_impl(*args, **kwargs)
[default5]:[rank45]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank45]:     return forward_call(*args, **kwargs)
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default5]:[rank45]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default5]:[rank45]:     pipeline_state.run_communication()
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default5]:[rank45]:     recv_activation_tensor = recv_activation()
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default5]:[rank45]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default5]:[rank45]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default5]:[rank45]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default5]:[rank45]:     dist.recv(
[default5]:[rank45]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default5]:[rank45]:     return func(*args, **kwargs)
[default5]:[rank45]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default5]:[rank45]:     pg.recv([tensor], group_src_rank, tag).wait()
[default5]:[rank45]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default5]:[rank45]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default5]:[rank45]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f371456d897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default5]:[rank45]: frame #1: <unknown function> + 0x5b3a23e (0x7f374e08a23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f374e084c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f374e084f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f374e085fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f374e03a371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f374e03a371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f374e03a371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f374e03a371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f3715847189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank45]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f371584e610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank45]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f371586d978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank45]: frame #12: <unknown function> + 0x5adc309 (0x7f374e02c309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #13: <unknown function> + 0x5ae6f10 (0x7f374e036f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f374e036fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #15: <unknown function> + 0x5124446 (0x7f374d674446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #16: <unknown function> + 0x1acf4b8 (0x7f374a01f4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #17: <unknown function> + 0x5aee004 (0x7f374e03e004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #18: <unknown function> + 0x5af36b5 (0x7f374e0436b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #19: <unknown function> + 0xd2631e (0x7f3760c2d31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default5]:[rank45]: frame #20: <unknown function> + 0x47def4 (0x7f3760384ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default5]:[rank45]: frame #21: <unknown function> + 0x1445a6 (0x56243426a5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #22: _PyObject_MakeTpCall + 0x26b (0x562434263a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #23: <unknown function> + 0x150866 (0x562434276866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x56243425f142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #25: _PyFunction_Vectorcall + 0x6c (0x56243426aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #26: PyObject_Call + 0xbc (0x562434276f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x56243425d2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #28: _PyFunction_Vectorcall + 0x6c (0x56243426aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x56243425b8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #30: <unknown function> + 0x150582 (0x562434276582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x56243425b8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #32: <unknown function> + 0x150582 (0x562434276582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x56243425b8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #34: <unknown function> + 0x150582 (0x562434276582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x56243425b8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x562434262f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #37: _PyObject_Call_Prepend + 0x69 (0x562434274c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #38: <unknown function> + 0x211239 (0x562434337239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #39: _PyObject_MakeTpCall + 0x26b (0x562434263a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x56243425f3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #41: _PyFunction_Vectorcall + 0x6c (0x56243426aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x56243425ac5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #43: _PyFunction_Vectorcall + 0x6c (0x56243426aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x56243425b8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #45: <unknown function> + 0x150582 (0x562434276582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #46: PyObject_Call + 0xbc (0x562434276f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x56243425d2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #48: <unknown function> + 0x150582 (0x562434276582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #49: PyObject_Call + 0xbc (0x562434276f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x56243425d2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #51: _PyFunction_Vectorcall + 0x6c (0x56243426aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x562434263007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #53: _PyObject_Call_Prepend + 0x69 (0x562434274c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #54: <unknown function> + 0x211239 (0x562434337239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #55: PyObject_Call + 0x207 (0x562434277067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x56243425d2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #57: <unknown function> + 0x150582 (0x562434276582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x56243425b8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #59: <unknown function> + 0x150582 (0x562434276582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #60: PyObject_Call + 0xbc (0x562434276f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x56243425d2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #62: <unknown function> + 0x150582 (0x562434276582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #63: PyObject_Call + 0xbc (0x562434276f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default5]:[rank37]: Traceback (most recent call last):
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default5]:[rank37]:     trainer.train(dataloader)
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default5]:[rank37]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default5]:[rank37]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default5]:[rank37]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default5]:[rank37]:     output = model(**micro_batch)
[default5]:[rank37]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank37]:     return self._call_impl(*args, **kwargs)
[default5]:[rank37]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank37]:     return forward_call(*args, **kwargs)
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default5]:[rank37]:     sharded_logits = self.model(
[default5]:[rank37]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank37]:     return self._call_impl(*args, **kwargs)
[default5]:[rank37]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank37]:     return forward_call(*args, **kwargs)
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default5]:[rank37]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default5]:[rank37]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default5]:[rank37]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank37]:     return self._call_impl(*args, **kwargs)
[default5]:[rank37]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank37]:     return forward_call(*args, **kwargs)
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default5]:[rank37]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default5]:[rank37]:     pipeline_state.run_communication()
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default5]:[rank37]:     recv_activation_tensor = recv_activation()
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default5]:[rank37]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default5]:[rank37]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default5]:[rank37]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default5]:[rank37]:     dist.recv(
[default5]:[rank37]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default5]:[rank37]:     return func(*args, **kwargs)
[default5]:[rank37]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default5]:[rank37]:     pg.recv([tensor], group_src_rank, tag).wait()
[default5]:[rank37]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default5]:[rank37]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default5]:[rank37]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f6833cea897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default5]:[rank37]: frame #1: <unknown function> + 0x5b3a23e (0x7f686d80723e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f686d801c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f686d801f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f686d802fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f686d7b7371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f686d7b7371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f686d7b7371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f686d7b7371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f6834fc4189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank37]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f6834fcb610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank37]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f6834fea978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank37]: frame #12: <unknown function> + 0x5adc309 (0x7f686d7a9309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #13: <unknown function> + 0x5ae6f10 (0x7f686d7b3f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f686d7b3fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #15: <unknown function> + 0x5124446 (0x7f686cdf1446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #16: <unknown function> + 0x1acf4b8 (0x7f686979c4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #17: <unknown function> + 0x5aee004 (0x7f686d7bb004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #18: <unknown function> + 0x5af36b5 (0x7f686d7c06b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #19: <unknown function> + 0xd2631e (0x7f68803aa31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default5]:[rank37]: frame #20: <unknown function> + 0x47def4 (0x7f687fb01ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default5]:[rank37]: frame #21: <unknown function> + 0x1445a6 (0x55a84168f5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55a841688a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #23: <unknown function> + 0x150866 (0x55a84169b866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55a841684142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55a84168fa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #26: PyObject_Call + 0xbc (0x55a84169bf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55a8416822b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55a84168fa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55a8416808fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #30: <unknown function> + 0x150582 (0x55a84169b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55a8416808fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #32: <unknown function> + 0x150582 (0x55a84169b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55a8416808fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #34: <unknown function> + 0x150582 (0x55a84169b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55a8416808fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55a841687f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55a841699c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #38: <unknown function> + 0x211239 (0x55a84175c239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55a841688a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55a8416843e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55a84168fa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55a84167fc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55a84168fa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55a8416808fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #45: <unknown function> + 0x150582 (0x55a84169b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #46: PyObject_Call + 0xbc (0x55a84169bf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55a8416822b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #48: <unknown function> + 0x150582 (0x55a84169b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #49: PyObject_Call + 0xbc (0x55a84169bf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55a8416822b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55a84168fa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55a841688007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55a841699c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #54: <unknown function> + 0x211239 (0x55a84175c239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #55: PyObject_Call + 0x207 (0x55a84169c067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55a8416822b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #57: <unknown function> + 0x150582 (0x55a84169b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55a8416808fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #59: <unknown function> + 0x150582 (0x55a84169b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #60: PyObject_Call + 0xbc (0x55a84169bf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55a8416822b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #62: <unknown function> + 0x150582 (0x55a84169b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #63: PyObject_Call + 0xbc (0x55a84169bf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: . This may indicate a possible application crash on rank 0 or a network set up issue.
W0703 00:33:30.405000 140629515450176 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1108347 closing signal SIGTERM
W0703 00:33:30.405000 140629515450176 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1108348 closing signal SIGTERM
W0703 00:33:30.405000 140629515450176 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1108350 closing signal SIGTERM
E0703 00:33:31.466000 140629515450176 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: -6) local_rank: 0 (pid: 1108345) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-07-03_00:33:30
  host      : ip-26-0-160-192.ec2.internal
  rank      : 1 (local_rank: 1)
  exitcode  : -6 (pid: 1108346)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 1108346
[2]:
  time      : 2024-07-03_00:33:30
  host      : ip-26-0-160-192.ec2.internal
  rank      : 4 (local_rank: 4)
  exitcode  : -6 (pid: 1108349)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 1108349
[3]:
  time      : 2024-07-03_00:33:30
  host      : ip-26-0-160-192.ec2.internal
  rank      : 6 (local_rank: 6)
  exitcode  : -6 (pid: 1108351)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 1108351
[4]:
  time      : 2024-07-03_00:33:30
  host      : ip-26-0-160-192.ec2.internal
  rank      : 7 (local_rank: 7)
  exitcode  : -6 (pid: 1108352)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 1108352
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-03_00:33:30
  host      : ip-26-0-160-192.ec2.internal
  rank      : 0 (local_rank: 0)
  exitcode  : -6 (pid: 1108345)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 1108345
============================================================
srun: error: ip-26-0-160-192: task 0: Exited with exit code 1
W0703 00:33:34.132000 139844076955392 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-172-73.ec2.internal_873734_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 00:33:34.304000 139711063140096 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-168-238.ec2.internal_1832456_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 00:33:34.837000 140439604508416 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-169-86.ec2.internal_1804798_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 00:33:34.877000 139980002621184 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-172-57.ec2.internal_1032079_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 00:33:35.429000 139716723873600 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1832530 closing signal SIGTERM
W0703 00:33:35.429000 139716723873600 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1832531 closing signal SIGTERM
W0703 00:33:35.429000 139716723873600 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1832532 closing signal SIGTERM
W0703 00:33:35.429000 139716723873600 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1832533 closing signal SIGTERM
W0703 00:33:35.430000 139716723873600 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1832534 closing signal SIGTERM
W0703 00:33:35.430000 139716723873600 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1832535 closing signal SIGTERM
W0703 00:33:35.433000 139716723873600 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1832536 closing signal SIGTERM
W0703 00:33:35.434000 139716723873600 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1832537 closing signal SIGTERM
W0703 00:33:35.437000 139985663354688 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1032152 closing signal SIGTERM
W0703 00:33:35.437000 139985663354688 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1032153 closing signal SIGTERM
W0703 00:33:35.437000 139985663354688 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1032154 closing signal SIGTERM
W0703 00:33:35.438000 139985663354688 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1032155 closing signal SIGTERM
W0703 00:33:35.440000 139985663354688 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1032156 closing signal SIGTERM
W0703 00:33:35.440000 139985663354688 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1032157 closing signal SIGTERM
W0703 00:33:35.440000 139985663354688 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1032158 closing signal SIGTERM
W0703 00:33:35.441000 139985663354688 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1032159 closing signal SIGTERM
W0703 00:33:35.491000 140445265241920 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1804872 closing signal SIGTERM
W0703 00:33:35.491000 140445265241920 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1804873 closing signal SIGTERM
W0703 00:33:35.491000 140445265241920 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1804874 closing signal SIGTERM
W0703 00:33:35.493000 140445265241920 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1804875 closing signal SIGTERM
W0703 00:33:35.493000 140445265241920 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1804876 closing signal SIGTERM
W0703 00:33:35.493000 140445265241920 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1804877 closing signal SIGTERM
W0703 00:33:35.493000 140445265241920 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1804878 closing signal SIGTERM
W0703 00:33:35.496000 140445265241920 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1804879 closing signal SIGTERM
W0703 00:33:35.503000 139849737688896 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 873807 closing signal SIGTERM
W0703 00:33:35.503000 139849737688896 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 873808 closing signal SIGTERM
W0703 00:33:35.504000 139849737688896 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 873809 closing signal SIGTERM
W0703 00:33:35.505000 139849737688896 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 873810 closing signal SIGTERM
W0703 00:33:35.505000 139849737688896 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 873811 closing signal SIGTERM
W0703 00:33:35.506000 139849737688896 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 873812 closing signal SIGTERM
W0703 00:33:35.506000 139849737688896 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 873813 closing signal SIGTERM
W0703 00:33:35.506000 139849737688896 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 873814 closing signal SIGTERM
W0703 00:33:39.137000 139844076955392 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-172-73.ec2.internal_873734_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 00:33:39.309000 139711063140096 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-168-238.ec2.internal_1832456_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 00:33:39.645000 139849737688896 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-172-73.ec2.internal_873734_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 00:33:39.659000 139849737688896 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-172-73.ec2.internal_873734_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 113, in _call_store
    return getattr(self._store, store_op)(*args, **kwargs)
torch.distributed.DistNetworkError: Broken pipe

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 254, in launch_agent
    result = agent.run()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 123, in wrapper
    result = f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 733, in run
    result = self._invoke_run(role)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 908, in _invoke_run
    num_nodes_waiting = rdzv_handler.num_nodes_waiting()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 1174, in num_nodes_waiting
    self._state_holder.sync()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 419, in sync
    get_response = self._backend.get_state()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 73, in get_state
    base64_state: bytes = self._call_store("get", self._key)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 115, in _call_store
    raise RendezvousConnectionError(
torch.distributed.elastic.rendezvous.api.RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details.
W0703 00:33:39.841000 140439604508416 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-169-86.ec2.internal_1804798_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 00:33:39.882000 139980002621184 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-172-57.ec2.internal_1032079_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
srun: error: ip-26-0-172-73: task 7: Exited with exit code 1
W0703 00:33:42.960000 139985663354688 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-172-57.ec2.internal_1032079_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 00:33:42.976000 139985663354688 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-172-57.ec2.internal_1032079_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 113, in _call_store
    return getattr(self._store, store_op)(*args, **kwargs)
torch.distributed.DistNetworkError: Broken pipe

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 254, in launch_agent
    result = agent.run()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 123, in wrapper
    result = f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 733, in run
    result = self._invoke_run(role)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 908, in _invoke_run
    num_nodes_waiting = rdzv_handler.num_nodes_waiting()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 1174, in num_nodes_waiting
    self._state_holder.sync()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 419, in sync
    get_response = self._backend.get_state()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 73, in get_state
    base64_state: bytes = self._call_store("get", self._key)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 115, in _call_store
    raise RendezvousConnectionError(
torch.distributed.elastic.rendezvous.api.RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details.
W0703 00:33:43.535000 140445265241920 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-169-86.ec2.internal_1804798_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 00:33:43.548000 140445265241920 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-169-86.ec2.internal_1804798_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 113, in _call_store
    return getattr(self._store, store_op)(*args, **kwargs)
torch.distributed.DistNetworkError: Broken pipe

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 254, in launch_agent
    result = agent.run()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 123, in wrapper
    result = f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 733, in run
    result = self._invoke_run(role)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 908, in _invoke_run
    num_nodes_waiting = rdzv_handler.num_nodes_waiting()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 1174, in num_nodes_waiting
    self._state_holder.sync()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 419, in sync
    get_response = self._backend.get_state()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 73, in get_state
    base64_state: bytes = self._call_store("get", self._key)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 115, in _call_store
    raise RendezvousConnectionError(
torch.distributed.elastic.rendezvous.api.RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details.
W0703 00:33:43.564000 139716723873600 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-168-238.ec2.internal_1832456_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0703 00:33:43.582000 139716723873600 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-168-238.ec2.internal_1832456_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 113, in _call_store
    return getattr(self._store, store_op)(*args, **kwargs)
torch.distributed.DistNetworkError: Broken pipe

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 254, in launch_agent
    result = agent.run()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 123, in wrapper
    result = f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 733, in run
    result = self._invoke_run(role)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 908, in _invoke_run
    num_nodes_waiting = rdzv_handler.num_nodes_waiting()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 1174, in num_nodes_waiting
    self._state_holder.sync()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 419, in sync
    get_response = self._backend.get_state()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 73, in get_state
    base64_state: bytes = self._call_store("get", self._key)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 115, in _call_store
    raise RendezvousConnectionError(
torch.distributed.elastic.rendezvous.api.RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details.
srun: error: ip-26-0-172-57: task 6: Exited with exit code 1
srun: error: ip-26-0-168-238: task 4: Exited with exit code 1
srun: error: ip-26-0-169-86: task 5: Exited with exit code 1
Consider using `hf_transfer` for faster uploads. This solution comes with some limitations. See https://huggingface.co/docs/huggingface_hub/hf_transfer for more details.