========================
START TIME: Tue Jul  2 21:44:57 UTC 2024
python3 version = Python 3.10.14
========================
The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well.
Token is valid (permission: write).
Your token has been saved to /admin/home/ferdinand_mom/.cache/huggingface/token
Login successful
Already on 'bench_cluster'
M	examples/config_tiny_llama.py
M	examples/config_tiny_llama.yaml
M	examples/train_tiny_llama.sh
M	src/nanotron/models/llama.py
M	src/nanotron/trainer.py
Your branch is up to date with 'origin/bench_cluster'.
Job status: RUNNING
W0702 21:45:04.281000 140397136058176 torch/distributed/run.py:757] 
W0702 21:45:04.281000 140397136058176 torch/distributed/run.py:757] *****************************************
W0702 21:45:04.281000 140397136058176 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0702 21:45:04.281000 140397136058176 torch/distributed/run.py:757] *****************************************
W0702 21:45:04.284000 139776129410880 torch/distributed/run.py:757] 
W0702 21:45:04.284000 139776129410880 torch/distributed/run.py:757] *****************************************
W0702 21:45:04.284000 139776129410880 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0702 21:45:04.284000 139776129410880 torch/distributed/run.py:757] *****************************************
W0702 21:45:04.386000 140046963582784 torch/distributed/run.py:757] 
W0702 21:45:04.386000 140046963582784 torch/distributed/run.py:757] *****************************************
W0702 21:45:04.386000 140046963582784 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0702 21:45:04.386000 140046963582784 torch/distributed/run.py:757] *****************************************
W0702 21:45:04.422000 140469081098048 torch/distributed/run.py:757] 
W0702 21:45:04.422000 140469081098048 torch/distributed/run.py:757] *****************************************
W0702 21:45:04.422000 140469081098048 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0702 21:45:04.422000 140469081098048 torch/distributed/run.py:757] *****************************************
W0702 21:45:04.607000 139990385469248 torch/distributed/run.py:757] 
W0702 21:45:04.607000 139990385469248 torch/distributed/run.py:757] *****************************************
W0702 21:45:04.607000 139990385469248 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0702 21:45:04.607000 139990385469248 torch/distributed/run.py:757] *****************************************
W0702 21:45:04.634000 140106095568704 torch/distributed/run.py:757] 
W0702 21:45:04.634000 140106095568704 torch/distributed/run.py:757] *****************************************
W0702 21:45:04.634000 140106095568704 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0702 21:45:04.634000 140106095568704 torch/distributed/run.py:757] *****************************************
W0702 21:45:04.851000 139678711797568 torch/distributed/run.py:757] 
W0702 21:45:04.851000 139678711797568 torch/distributed/run.py:757] *****************************************
W0702 21:45:04.851000 139678711797568 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0702 21:45:04.851000 139678711797568 torch/distributed/run.py:757] *****************************************
W0702 21:45:05.792000 139896234325824 torch/distributed/run.py:757] 
W0702 21:45:05.792000 139896234325824 torch/distributed/run.py:757] *****************************************
W0702 21:45:05.792000 139896234325824 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0702 21:45:05.792000 139896234325824 torch/distributed/run.py:757] *****************************************
[default0]:07/02/2024 21:45:31 [WARNING|DP=0|PP=0|TP=0|ip-26-0-160-225]: [Vocab Size Padding] Padded vocab (size: 50257) with 15 dummy tokens (new size: 50272)
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Config:
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Config(general=GeneralArgs(project='bench_cluster',
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                            run='%date_%jobid',
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                            seed=42,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                            step=None,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                            consumed_train_samples=None,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                            benchmark_csv_path=None,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                            ignore_sanity_checks=True),
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:        parallelism=ParallelismArgs(dp=1,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                    pp=2,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                    tp=32,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                    pp_engine=<nanotron.parallel.pipeline_parallel.engine.OneForwardOneBackwardPipelineEngine object at 0x7fa8915848e0>,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                    tp_mode=<TensorParallelLinearMode.REDUCE_SCATTER: 2>,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                    tp_linear_async_communication=False,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                    expert_parallel_size=1),
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:        model=ModelArgs(model_config=LlamaConfig(bos_token_id=1,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 eos_token_id=2,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 hidden_act='silu',
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 hidden_size=2048,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 initializer_range=0.02,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 intermediate_size=4096,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 is_llama_config=True,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 max_position_embeddings=4096,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 num_attention_heads=32,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 num_hidden_layers=24,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 num_key_value_heads=32,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 pad_token_id=None,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 pretraining_tp=1,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 rms_norm_eps=1e-05,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 rope_scaling=None,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 rope_theta=10000.0,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 tie_word_embeddings=True,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 use_cache=True,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                 vocab_size=50272),
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                        init_method=RandomInit(std=0.025),
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                        dtype=torch.bfloat16,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                        make_vocab_size_divisible_by=1,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                        ddp_bucket_cap_mb=25),
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:        tokenizer=TokenizerArgs(tokenizer_name_or_path='openai-community/gpt2',
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                tokenizer_revision=None,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                tokenizer_max_length=None),
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:        checkpoints=CheckpointsArgs(checkpoints_path=Path('/dev/null'),
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                    checkpoint_interval=100000,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                    save_initial_state=False,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                    resume_checkpoint_path=None,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                    checkpoints_path_is_shared_file_system=False),
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:        logging=LoggingArgs(log_level='info',
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                            log_level_replica='info',
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                            iteration_step_info_interval=1),
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:        tokens=TokensArgs(sequence_length=4096,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                          train_steps=20,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                          micro_batch_size=1024,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                          batch_accumulation_per_replica=1,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                          val_check_interval=-1,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                          limit_val_batches=0,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                          limit_test_batches=0),
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:        optimizer=OptimizerArgs(optimizer_factory=AdamWOptimizerArgs(adam_eps=1e-08,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                     adam_beta1=0.9,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                     adam_beta2=0.95,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                     torch_adam_is_fused=True,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                     name='adamW'),
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                zero_stage=1,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                weight_decay=0.01,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                clip_grad=1.0,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                accumulate_grad_in_fp32=True,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                learning_rate_scheduler=LRSchedulerArgs(learning_rate=0.0001,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                        lr_warmup_steps=1,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                        lr_warmup_style='linear',
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                        lr_decay_style='linear',
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                        lr_decay_steps=19,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                        lr_decay_starting_step=None,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                        min_decay_lr=1e-05)),
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:        data_stages=[DatasetStageArgs(name='Training Stage',
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                      start_training_step=1,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                      data=DataArgs(dataset=PretrainDatasetsArgs(hf_dataset_or_datasets='roneneldan/TinyStories',
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                                 hf_dataset_splits='train',
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                                 hf_dataset_config_name=None,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                                 dataset_processing_num_proc_per_process=64,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                                 dataset_overwrite_cache=False,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                                                 text_column_name='text'),
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                    seed=42,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:                                                    num_loading_workers=0))],
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:        profiler=ProfilerArgs(profiler_export_path=Path('/fsx/ferdinandmom/ferdinand-hf/bench_cluster/results/llama-1B/64_GPUS/dp-1_tp-32_pp-2_mbz-1024')),
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:        lighteval=None)
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Model Config:
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: LlamaConfig(bos_token_id=1,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             eos_token_id=2,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             hidden_act='silu',
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             hidden_size=2048,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             initializer_range=0.02,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             intermediate_size=4096,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             is_llama_config=True,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             max_position_embeddings=4096,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             num_attention_heads=32,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             num_hidden_layers=24,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             num_key_value_heads=32,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             pad_token_id=None,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             pretraining_tp=1,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             rms_norm_eps=1e-05,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             rope_scaling=None,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             rope_theta=10000.0,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             tie_word_embeddings=True,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             use_cache=True,
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:             vocab_size=50272)
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Building model..
[default0]:07/02/2024 21:45:31 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Setting PP block ranks...
[default3]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=11|ip-26-0-171-102]: Local number of parameters: 16.4M (31.22MiB)
[default3]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=11|ip-26-0-171-102]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default3]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=11|ip-26-0-171-102]: No checkpoint path provided.
[default6]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=14|ip-26-0-171-102]: Local number of parameters: 16.4M (31.22MiB)
[default6]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=14|ip-26-0-171-102]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default6]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=14|ip-26-0-171-102]: No checkpoint path provided.
[default1]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=9|ip-26-0-171-102]: Local number of parameters: 16.4M (31.22MiB)
[default1]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=9|ip-26-0-171-102]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default1]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=9|ip-26-0-171-102]: No checkpoint path provided.
[default7]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=15|ip-26-0-171-102]: Local number of parameters: 16.4M (31.22MiB)
[default7]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=15|ip-26-0-171-102]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default7]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=15|ip-26-0-171-102]: No checkpoint path provided.
[default4]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=12|ip-26-0-171-102]: Local number of parameters: 16.4M (31.22MiB)
[default4]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=12|ip-26-0-171-102]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default4]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=12|ip-26-0-171-102]: No checkpoint path provided.
[default4]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=28|ip-26-0-161-78]: Local number of parameters: 21.6M (41.25MiB)
[default2]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=26|ip-26-0-161-78]: Local number of parameters: 21.6M (41.25MiB)
[default2]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=26|ip-26-0-161-78]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default4]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=28|ip-26-0-161-78]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default4]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=4|ip-26-0-160-225]: Local number of parameters: 21.6M (41.25MiB)
[default4]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=4|ip-26-0-160-225]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default1]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=1|ip-26-0-160-225]: Local number of parameters: 21.6M (41.25MiB)
[default1]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=1|ip-26-0-160-225]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default1]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=1|ip-26-0-160-225]: No checkpoint path provided.
[default4]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=4|ip-26-0-160-225]: No checkpoint path provided.
[default4]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=28|ip-26-0-161-78]: No checkpoint path provided.
[default2]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=26|ip-26-0-161-78]: No checkpoint path provided.
[default2]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=2|ip-26-0-160-225]: Local number of parameters: 21.6M (41.25MiB)
[default2]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=2|ip-26-0-160-225]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default2]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=2|ip-26-0-160-225]: No checkpoint path provided.
[default5]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=5|ip-26-0-160-225]: Local number of parameters: 21.6M (41.25MiB)
[default5]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=5|ip-26-0-160-225]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default5]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=5|ip-26-0-160-225]: No checkpoint path provided.
[default2]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=10|ip-26-0-171-102]: Local number of parameters: 16.4M (31.22MiB)
[default2]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=10|ip-26-0-171-102]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default2]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=10|ip-26-0-171-102]: No checkpoint path provided.
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=8|ip-26-0-171-102]: Local number of parameters: 16.4M (31.22MiB)
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=8|ip-26-0-171-102]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=8|ip-26-0-171-102]: No checkpoint path provided.
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Total number of parameters: 1.22G (2318.88MiB)
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Local number of parameters: 21.6M (41.25MiB)
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: No checkpoint path provided.
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Parametrizing model parameters using StandardParametrizator
[default3]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=3|ip-26-0-160-225]: Local number of parameters: 21.6M (41.25MiB)
[default3]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=3|ip-26-0-160-225]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default3]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=3|ip-26-0-160-225]: No checkpoint path provided.
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=24|ip-26-0-161-78]: Local number of parameters: 21.6M (41.25MiB)
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=24|ip-26-0-161-78]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=24|ip-26-0-161-78]: No checkpoint path provided.
[default1]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=25|ip-26-0-161-78]: Local number of parameters: 21.6M (41.25MiB)
[default1]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=25|ip-26-0-161-78]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default1]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=25|ip-26-0-161-78]: No checkpoint path provided.
[default5]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=29|ip-26-0-161-78]: Local number of parameters: 21.6M (41.25MiB)
[default5]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=29|ip-26-0-161-78]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default5]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=29|ip-26-0-161-78]: No checkpoint path provided.
[default3]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=27|ip-26-0-161-78]: Local number of parameters: 21.6M (41.25MiB)
[default3]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=27|ip-26-0-161-78]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default3]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=27|ip-26-0-161-78]: No checkpoint path provided.
[default7]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=31|ip-26-0-161-78]: Local number of parameters: 21.6M (41.25MiB)
[default7]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=31|ip-26-0-161-78]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=0|ip-26-0-162-233]: Local number of parameters: 16.4M (31.22MiB)
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=0|ip-26-0-162-233]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=0|ip-26-0-162-233]: No checkpoint path provided.
[default7]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=31|ip-26-0-161-78]: No checkpoint path provided.
[default6]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=30|ip-26-0-161-78]: Local number of parameters: 21.6M (41.25MiB)
[default6]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=30|ip-26-0-161-78]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default6]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=30|ip-26-0-161-78]: No checkpoint path provided.
[default5]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=13|ip-26-0-171-102]: Local number of parameters: 16.4M (31.22MiB)
[default5]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=13|ip-26-0-171-102]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default5]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=13|ip-26-0-171-102]: No checkpoint path provided.
[default7]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=23|ip-26-0-161-153]: Local number of parameters: 21.6M (41.25MiB)
[default7]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=23|ip-26-0-161-153]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default6]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=6|ip-26-0-160-225]: Local number of parameters: 21.6M (41.25MiB)
[default6]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=6|ip-26-0-160-225]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default6]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=6|ip-26-0-160-225]: No checkpoint path provided.
[default7]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=23|ip-26-0-161-153]: No checkpoint path provided.
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=16|ip-26-0-161-153]: Local number of parameters: 21.6M (41.25MiB)
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=16|ip-26-0-161-153]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=16|ip-26-0-161-153]: No checkpoint path provided.
[default5]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=21|ip-26-0-161-153]: Local number of parameters: 21.6M (41.25MiB)
[default5]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=21|ip-26-0-161-153]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default5]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=21|ip-26-0-161-153]: No checkpoint path provided.
[default7]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=7|ip-26-0-160-225]: Local number of parameters: 21.6M (41.25MiB)
[default7]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=7|ip-26-0-160-225]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default7]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=7|ip-26-0-160-225]: No checkpoint path provided.
[default7]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=15|ip-26-0-161-103]: Local number of parameters: 21.6M (41.25MiB)
[default7]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=15|ip-26-0-161-103]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default7]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=15|ip-26-0-161-103]: No checkpoint path provided.
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=8|ip-26-0-161-103]: Local number of parameters: 21.6M (41.25MiB)
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=8|ip-26-0-161-103]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=8|ip-26-0-161-103]: No checkpoint path provided.
[default3]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=19|ip-26-0-161-153]: Local number of parameters: 21.6M (41.25MiB)
[default3]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=19|ip-26-0-161-153]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default3]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=19|ip-26-0-161-153]: No checkpoint path provided.
[default1]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=17|ip-26-0-161-153]: Local number of parameters: 21.6M (41.25MiB)
[default1]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=17|ip-26-0-161-153]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default1]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=17|ip-26-0-161-153]: No checkpoint path provided.
[default4]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=20|ip-26-0-161-153]: Local number of parameters: 21.6M (41.25MiB)
[default4]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=20|ip-26-0-161-153]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default4]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=20|ip-26-0-161-153]: No checkpoint path provided.
[default4]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=28|ip-26-0-171-88]: Local number of parameters: 16.4M (31.22MiB)
[default4]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=28|ip-26-0-171-88]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default4]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=28|ip-26-0-171-88]: No checkpoint path provided.
[default5]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=29|ip-26-0-171-88]: Local number of parameters: 16.4M (31.22MiB)
[default5]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=29|ip-26-0-171-88]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default5]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=29|ip-26-0-171-88]: No checkpoint path provided.
[default3]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=27|ip-26-0-171-88]: Local number of parameters: 16.4M (31.22MiB)
[default3]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=27|ip-26-0-171-88]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default3]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=27|ip-26-0-171-88]: No checkpoint path provided.
[default6]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=14|ip-26-0-161-103]: Local number of parameters: 21.6M (41.25MiB)
[default3]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=3|ip-26-0-162-233]: Local number of parameters: 16.4M (31.22MiB)
[default3]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=11|ip-26-0-161-103]: Local number of parameters: 21.6M (41.25MiB)
[default3]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=3|ip-26-0-162-233]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default3]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=3|ip-26-0-162-233]: No checkpoint path provided.
[default3]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=11|ip-26-0-161-103]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default3]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=11|ip-26-0-161-103]: No checkpoint path provided.
[default1]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=1|ip-26-0-162-233]: Local number of parameters: 16.4M (31.22MiB)
[default2]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=2|ip-26-0-162-233]: Local number of parameters: 16.4M (31.22MiB)
[default6]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=14|ip-26-0-161-103]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default6]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=14|ip-26-0-161-103]: No checkpoint path provided.
[default2]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=2|ip-26-0-162-233]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default1]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=1|ip-26-0-162-233]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default2]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=2|ip-26-0-162-233]: No checkpoint path provided.
[default1]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=1|ip-26-0-162-233]: No checkpoint path provided.
[default5]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=13|ip-26-0-161-103]: Local number of parameters: 21.6M (41.25MiB)
[default5]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=13|ip-26-0-161-103]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default4]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=12|ip-26-0-161-103]: Local number of parameters: 21.6M (41.25MiB)
[default4]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=12|ip-26-0-161-103]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default4]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=12|ip-26-0-161-103]: No checkpoint path provided.
[default5]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=13|ip-26-0-161-103]: No checkpoint path provided.
[default2]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=18|ip-26-0-161-153]: Local number of parameters: 21.6M (41.25MiB)
[default2]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=18|ip-26-0-161-153]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default2]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=18|ip-26-0-161-153]: No checkpoint path provided.
[default6]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=22|ip-26-0-161-153]: Local number of parameters: 21.6M (41.25MiB)
[default6]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=22|ip-26-0-161-153]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default6]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=22|ip-26-0-161-153]: No checkpoint path provided.
[default6]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=30|ip-26-0-171-88]: Local number of parameters: 16.4M (31.22MiB)
[default6]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=30|ip-26-0-171-88]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default6]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=30|ip-26-0-171-88]: No checkpoint path provided.
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=24|ip-26-0-171-88]: Local number of parameters: 16.4M (31.22MiB)
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=24|ip-26-0-171-88]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=24|ip-26-0-171-88]: No checkpoint path provided.
[default7]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=31|ip-26-0-171-88]: Local number of parameters: 16.4M (31.22MiB)
[default7]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=31|ip-26-0-171-88]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default2]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=26|ip-26-0-171-88]: Local number of parameters: 16.4M (31.22MiB)
[default2]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=26|ip-26-0-171-88]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default7]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=31|ip-26-0-171-88]: No checkpoint path provided.
[default1]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=25|ip-26-0-171-88]: Local number of parameters: 16.4M (31.22MiB)
[default2]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=26|ip-26-0-171-88]: No checkpoint path provided.
[default1]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=25|ip-26-0-171-88]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default1]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=25|ip-26-0-171-88]: No checkpoint path provided.
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=16|ip-26-0-171-62]: Local number of parameters: 16.4M (31.22MiB)
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=16|ip-26-0-171-62]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default0]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=16|ip-26-0-171-62]: No checkpoint path provided.
[default5]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=21|ip-26-0-171-62]: Local number of parameters: 16.4M (31.22MiB)
[default2]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=10|ip-26-0-161-103]: Local number of parameters: 21.6M (41.25MiB)
[default2]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=10|ip-26-0-161-103]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default2]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=10|ip-26-0-161-103]: No checkpoint path provided.
[default5]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=21|ip-26-0-171-62]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default7]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=23|ip-26-0-171-62]: Local number of parameters: 16.4M (31.22MiB)
[default7]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=23|ip-26-0-171-62]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default3]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=19|ip-26-0-171-62]: Local number of parameters: 16.4M (31.22MiB)
[default3]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=19|ip-26-0-171-62]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default3]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=19|ip-26-0-171-62]: No checkpoint path provided.
[default5]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=21|ip-26-0-171-62]: No checkpoint path provided.
[default1]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=9|ip-26-0-161-103]: Local number of parameters: 21.6M (41.25MiB)
[default1]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=9|ip-26-0-161-103]: [After model building] Memory usage: 55.26MiB. Peak allocated: 57.29MiB Peak reserved: 72.00MiB
[default5]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=5|ip-26-0-162-233]: Local number of parameters: 16.4M (31.22MiB)
[default7]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=23|ip-26-0-171-62]: No checkpoint path provided.
[default1]:07/02/2024 21:45:50 [INFO|DP=0|PP=0|TP=9|ip-26-0-161-103]: No checkpoint path provided.
[default5]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=5|ip-26-0-162-233]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default5]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=5|ip-26-0-162-233]: No checkpoint path provided.
[default4]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=4|ip-26-0-162-233]: Local number of parameters: 16.4M (31.22MiB)
[default4]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=4|ip-26-0-162-233]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default4]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=4|ip-26-0-162-233]: No checkpoint path provided.
[default1]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=17|ip-26-0-171-62]: Local number of parameters: 16.4M (31.22MiB)
[default7]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=7|ip-26-0-162-233]: Local number of parameters: 16.4M (31.22MiB)
[default7]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=7|ip-26-0-162-233]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default7]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=7|ip-26-0-162-233]: No checkpoint path provided.
[default1]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=17|ip-26-0-171-62]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default1]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=17|ip-26-0-171-62]: No checkpoint path provided.
[default4]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=20|ip-26-0-171-62]: Local number of parameters: 16.4M (31.22MiB)
[default6]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=22|ip-26-0-171-62]: Local number of parameters: 16.4M (31.22MiB)
[default6]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=22|ip-26-0-171-62]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default6]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=22|ip-26-0-171-62]: No checkpoint path provided.
[default4]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=20|ip-26-0-171-62]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default4]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=20|ip-26-0-171-62]: No checkpoint path provided.
[default2]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=18|ip-26-0-171-62]: Local number of parameters: 16.4M (31.22MiB)
[default2]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=18|ip-26-0-171-62]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default2]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=18|ip-26-0-171-62]: No checkpoint path provided.
[default6]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=6|ip-26-0-162-233]: Local number of parameters: 16.4M (31.22MiB)
[default6]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=6|ip-26-0-162-233]: [After model building] Memory usage: 41.23MiB. Peak allocated: 43.26MiB Peak reserved: 58.00MiB
[default6]:07/02/2024 21:45:50 [INFO|DP=0|PP=1|TP=6|ip-26-0-162-233]: No checkpoint path provided.
[default0]:07/02/2024 21:45:51 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [Optimizer Building] Using LearningRateForSP as learning rate
[default0]:07/02/2024 21:45:51 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [ZeRO sharding] Size of optimizer params per rank:
[default0]:07/02/2024 21:45:51 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [ZeRO sharding] DP Rank 0 has 21.6M out of 21.6M (100.00%) params' optimizer states
[default0]:07/02/2024 21:45:53 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [Training Plan] Stage Training Stage has 19 remaining training steps and has consumed 0 samples
[default0]:07/02/2024 21:45:53 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Using `datasets` library
[default0]:07/02/2024 21:45:53 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Loading tokenizer from openai-community/gpt2 and transformers/hf_hub versions ('4.41.2', '0.23.4')
[default0]:07/02/2024 21:45:53 [WARNING|DP=0|PP=0|TP=0|ip-26-0-160-225]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/02/2024 21:45:55 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [Training Plan] There are 1 training stages 
[default0]:07/02/2024 21:45:55 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [Stage Training Stage] start from step 1 
[default0]:07/02/2024 21:45:55 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: 
[default0]:07/02/2024 21:45:55 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [Start training] datetime: 2024-07-02 21:45:55.822622 | mbs: 1024 | grad_accum: 1 | global_batch_size: 1024 | sequence_length: 4096 | train_steps: 20 | start_iteration_step: 0 | consumed_train_samples: 0
[default0]:07/02/2024 21:45:55 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Resuming training from stage Training Stage, it has trained for 0 samples and has 19 remaining train steps
[default0]:07/02/2024 21:45:55 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]:  Memory usage: 220.25MiB. Peak allocated 220.25MiB. Peak reserved: 240.00MiB
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=9|ip-26-0-171-102]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=28|ip-26-0-161-78]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/02/2024 21:45:55 [WARNING|DP=0|PP=1|TP=12|ip-26-0-171-102]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=26|ip-26-0-161-78]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=10|ip-26-0-171-102]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=5|ip-26-0-160-225]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=4|ip-26-0-160-225]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/02/2024 21:45:55 [WARNING|DP=0|PP=0|TP=3|ip-26-0-160-225]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=8|ip-26-0-171-102]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=25|ip-26-0-161-78]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=27|ip-26-0-161-78]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=29|ip-26-0-161-78]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=31|ip-26-0-161-78]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=7|ip-26-0-160-225]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=23|ip-26-0-161-153]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=21|ip-26-0-161-153]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=16|ip-26-0-161-153]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=15|ip-26-0-161-103]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/02/2024 21:45:55 [WARNING|DP=0|PP=0|TP=8|ip-26-0-161-103]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=17|ip-26-0-161-153]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=20|ip-26-0-161-153]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=3|ip-26-0-162-233]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=27|ip-26-0-171-88]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=13|ip-26-0-161-103]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=11|ip-26-0-161-103]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=12|ip-26-0-161-103]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=14|ip-26-0-161-103]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=30|ip-26-0-171-88]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=26|ip-26-0-171-88]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=16|ip-26-0-171-62]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=9|ip-26-0-161-103]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=7|ip-26-0-162-233]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=5|ip-26-0-162-233]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=11|ip-26-0-171-102]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=14|ip-26-0-171-102]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=15|ip-26-0-171-102]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=2|ip-26-0-160-225]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=24|ip-26-0-161-78]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=30|ip-26-0-161-78]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=13|ip-26-0-171-102]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=0|ip-26-0-162-233]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=6|ip-26-0-160-225]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=19|ip-26-0-161-153]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=29|ip-26-0-171-88]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=28|ip-26-0-171-88]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=1|ip-26-0-162-233]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=2|ip-26-0-162-233]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=24|ip-26-0-171-88]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=31|ip-26-0-171-88]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=25|ip-26-0-171-88]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=18|ip-26-0-161-153]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=4|ip-26-0-162-233]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=22|ip-26-0-171-62]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=22|ip-26-0-161-153]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=20|ip-26-0-171-62]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=17|ip-26-0-171-62]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=6|ip-26-0-162-233]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=1|ip-26-0-160-225]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=21|ip-26-0-171-62]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=23|ip-26-0-171-62]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=18|ip-26-0-171-62]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/02/2024 21:45:56 [WARNING|DP=0|PP=1|TP=19|ip-26-0-171-62]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/02/2024 21:45:56 [WARNING|DP=0|PP=0|TP=10|ip-26-0-161-103]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:[rank21]: Traceback (most recent call last):
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default5]:[rank21]:     trainer.train(dataloader)
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default5]:[rank21]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default5]:[rank21]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default5]:[rank21]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default5]:[rank21]:     output = model(**micro_batch)
[default5]:[rank21]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank21]:     return self._call_impl(*args, **kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank21]:     return forward_call(*args, **kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default5]:[rank21]:     sharded_logits = self.model(
[default5]:[rank21]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank21]:     return self._call_impl(*args, **kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank21]:     return forward_call(*args, **kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default5]:[rank21]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default5]:[rank21]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default5]:[rank21]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank21]:     return self._call_impl(*args, **kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank21]:     return forward_call(*args, **kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default5]:[rank21]:     output = self.pp_block(**new_kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank21]:     return self._call_impl(*args, **kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank21]:     return forward_call(*args, **kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default5]:[rank21]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default5]:[rank21]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank21]:     return self._call_impl(*args, **kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank21]:     return forward_call(*args, **kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default5]:[rank21]:     output = self.o_proj(attention_output)
[default5]:[rank21]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank21]:     return self._call_impl(*args, **kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank21]:     return forward_call(*args, **kwargs)
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default5]:[rank21]:     return row_linear(
[default5]:[rank21]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default5]:[rank21]:     out = F.linear(input, weight, bias)
[default5]:[rank21]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU  has a total capacity of 79.33 GiB of which 5.06 GiB is free. Including non-PyTorch memory, this process has 74.26 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default0]:[rank16]: Traceback (most recent call last):
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default0]:[rank16]:     trainer.train(dataloader)
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default0]:[rank16]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default0]:[rank16]:     outputs = self.pipeline_engine.train_batch_iter(
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default0]:[rank16]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default0]:[rank16]:     output = model(**micro_batch)
[default0]:[rank16]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank16]:     return self._call_impl(*args, **kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank16]:     return forward_call(*args, **kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default0]:[rank16]:     sharded_logits = self.model(
[default0]:[rank16]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank16]:     return self._call_impl(*args, **kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank16]:     return forward_call(*args, **kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default0]:[rank16]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default0]:[rank16]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default0]:[rank16]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank16]:     return self._call_impl(*args, **kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank16]:     return forward_call(*args, **kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default0]:[rank16]:     output = self.pp_block(**new_kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank16]:     return self._call_impl(*args, **kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank16]:     return forward_call(*args, **kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default0]:[rank16]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default0]:[rank16]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank16]:     return self._call_impl(*args, **kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank16]:     return forward_call(*args, **kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default0]:[rank16]:     output = self.o_proj(attention_output)
[default0]:[rank16]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank16]:     return self._call_impl(*args, **kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank16]:     return forward_call(*args, **kwargs)
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default0]:[rank16]:     return row_linear(
[default0]:[rank16]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default0]:[rank16]:     out = F.linear(input, weight, bias)
[default0]:[rank16]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 
[default1]:[rank17]: Traceback (most recent call last):
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default4]:[rank20]: Traceback (most recent call last):
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default1]:[rank17]:     trainer.train(dataloader)
[default3]:[rank19]: Traceback (most recent call last):
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default4]:[rank20]:     trainer.train(dataloader)
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default1]:[rank17]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default3]:[rank19]:     trainer.train(dataloader)
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default4]:[rank20]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default3]:[rank19]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default4]:[rank20]:     outputs = self.pipeline_engine.train_batch_iter(
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default1]:[rank17]:     outputs = self.pipeline_engine.train_batch_iter(
[default3]:[rank19]:     outputs = self.pipeline_engine.train_batch_iter(
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default4]:[rank20]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default1]:[rank17]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default3]:[rank19]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default1]:[rank17]:     output = model(**micro_batch)
[default4]:[rank20]:     output = model(**micro_batch)
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default1]:[rank17]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank19]:     output = model(**micro_batch)
[default4]:[rank20]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank17]:     return self._call_impl(*args, **kwargs)
[default1]:[rank17]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank20]:     return self._call_impl(*args, **kwargs)
[default4]:[rank20]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank20]:     return forward_call(*args, **kwargs)
[default3]:[rank19]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank17]:     return forward_call(*args, **kwargs)
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default1]:[rank17]:     sharded_logits = self.model(
[default3]:[rank19]:     return self._call_impl(*args, **kwargs)
[default4]:[rank20]:     sharded_logits = self.model(
[default1]:[rank17]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank17]:     return self._call_impl(*args, **kwargs)
[default4]:[rank20]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank20]:     return self._call_impl(*args, **kwargs)
[default1]:[rank17]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank20]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank20]:     return forward_call(*args, **kwargs)
[default3]:[rank19]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default1]:[rank17]:     return forward_call(*args, **kwargs)
[default3]:[rank19]:     return forward_call(*args, **kwargs)
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default4]:[rank20]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default4]:[rank20]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default3]:[rank19]:     sharded_logits = self.model(
[default1]:[rank17]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default4]:[rank20]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default1]:[rank17]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default4]:[rank20]:     return self._call_impl(*args, **kwargs)
[default1]:[rank17]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank17]:     return self._call_impl(*args, **kwargs)
[default4]:[rank20]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank20]:     return forward_call(*args, **kwargs)
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default1]:[rank17]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank20]:     output = self.pp_block(**new_kwargs)
[default3]:[rank19]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank20]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank19]:     return self._call_impl(*args, **kwargs)
[default4]:[rank20]:     return self._call_impl(*args, **kwargs)
[default1]:[rank17]:     return forward_call(*args, **kwargs)
[default3]:[rank19]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default4]:[rank20]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank19]:     return forward_call(*args, **kwargs)
[default1]:[rank17]:     output = self.pp_block(**new_kwargs)
[default4]:[rank20]:     return forward_call(*args, **kwargs)
[default1]:[rank17]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default3]:[rank19]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default1]:[rank17]:     return self._call_impl(*args, **kwargs)
[default4]:[rank20]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default4]:[rank20]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank17]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank20]:     return self._call_impl(*args, **kwargs)
[default1]:[rank17]:     return forward_call(*args, **kwargs)
[default4]:[rank20]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default3]:[rank19]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default4]:[rank20]:     return forward_call(*args, **kwargs)
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default3]:[rank19]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank17]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default3]:[rank19]:     return self._call_impl(*args, **kwargs)
[default1]:[rank17]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank19]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank20]:     output = self.o_proj(attention_output)
[default3]:[rank19]:     return forward_call(*args, **kwargs)
[default1]:[rank17]:     return self._call_impl(*args, **kwargs)
[default4]:[rank20]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank17]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default1]:[rank17]:     return forward_call(*args, **kwargs)
[default4]:[rank20]:     return self._call_impl(*args, **kwargs)
[default4]:[rank20]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank20]:     return forward_call(*args, **kwargs)
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default3]:[rank19]:     output = self.pp_block(**new_kwargs)
[default1]:[rank17]:     output = self.o_proj(attention_output)
[default4]:[rank20]:     return row_linear(
[default3]:[rank19]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank17]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank19]:     return self._call_impl(*args, **kwargs)
[default1]:[rank17]:     return self._call_impl(*args, **kwargs)
[default4]:[rank20]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default1]:[rank17]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank20]:     out = F.linear(input, weight, bias)
[default3]:[rank19]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank17]:     return forward_call(*args, **kwargs)
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default4]:[rank20]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU  has a total capacity of 79.33 GiB of which 5.17 GiB is free. Including non-PyTorch memory, this process has 74.15 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default1]:[rank17]:     return row_linear(
[default1]:[rank17]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default1]:[rank17]:     out = F.linear(input, weight, bias)
[default3]:[rank19]:     return forward_call(*args, **kwargs)
[default1]:[rank17]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 
 has a total capacity of 79.33 GiB of which 5.06 GiB is free. Including non-PyTorch memory, this process has 74.26 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default3]:[rank19]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default3]:[rank19]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank19]:     return self._call_impl(*args, **kwargs)
[default3]:[rank19]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank19]:     return forward_call(*args, **kwargs)
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default3]:[rank19]:     output = self.o_proj(attention_output)
[default3]:[rank19]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank19]:     return self._call_impl(*args, **kwargs)
[default3]:[rank19]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank19]:     return forward_call(*args, **kwargs)
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default3]:[rank19]:     return row_linear(
[default3]:[rank19]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default3]:[rank19]:     out = F.linear(input, weight, bias)
[default3]:[rank19]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU  has a total capacity of 79.33 GiB of which 5.06 GiB is free. Including non-PyTorch memory, this process has 74.26 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default5]:[rank29]: Traceback (most recent call last):
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default5]:[rank29]:     trainer.train(dataloader)
[default0]:[rank24]: Traceback (most recent call last):
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default0]:[rank24]:     trainer.train(dataloader)
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default5]:[rank29]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default0]:[rank24]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default5]:[rank29]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default5]:[rank29]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default0]:[rank24]:     outputs = self.pipeline_engine.train_batch_iter(
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default0]:[rank24]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default0]:[rank24]:     output = model(**micro_batch)
[default0]:[rank24]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank24]:     return self._call_impl(*args, **kwargs)
[default5]:[rank29]:     output = model(**micro_batch)
[default5]:[rank29]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank29]:     return self._call_impl(*args, **kwargs)
[default0]:[rank24]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank24]:     return forward_call(*args, **kwargs)
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default0]:[rank24]:     sharded_logits = self.model(
[default5]:[rank29]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank29]:     return forward_call(*args, **kwargs)
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default5]:[rank29]:     sharded_logits = self.model(
[default0]:[rank24]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank29]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank29]:     return self._call_impl(*args, **kwargs)
[default5]:[rank29]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank29]:     return forward_call(*args, **kwargs)
[default0]:[rank24]:     return self._call_impl(*args, **kwargs)
[default0]:[rank24]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank24]:     return forward_call(*args, **kwargs)
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default5]:[rank29]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default0]:[rank24]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default5]:[rank29]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default5]:[rank29]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank24]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default5]:[rank29]:     return self._call_impl(*args, **kwargs)
[default0]:[rank24]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank24]:     return self._call_impl(*args, **kwargs)
[default0]:[rank24]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank29]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank29]:     return forward_call(*args, **kwargs)
[default0]:[rank24]:     return forward_call(*args, **kwargs)
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default0]:[rank24]:     output = self.pp_block(**new_kwargs)
[default5]:[rank29]:     output = self.pp_block(**new_kwargs)
[default0]:[rank24]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank29]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank24]:     return self._call_impl(*args, **kwargs)
[default5]:[rank29]:     return self._call_impl(*args, **kwargs)
[default0]:[rank24]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank24]:     return forward_call(*args, **kwargs)
[default5]:[rank29]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank29]:     return forward_call(*args, **kwargs)
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default5]:[rank29]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default5]:[rank29]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default5]:[rank29]:     return self._call_impl(*args, **kwargs)
[default5]:[rank29]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank29]:     return forward_call(*args, **kwargs)
[default0]:[rank24]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default0]:[rank24]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank24]:     return self._call_impl(*args, **kwargs)
[default0]:[rank24]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank24]:     return forward_call(*args, **kwargs)
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default5]:[rank29]:     output = self.o_proj(attention_output)
[default5]:[rank29]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank29]:     return self._call_impl(*args, **kwargs)
[default7]:[rank31]: Traceback (most recent call last):
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default0]:[rank24]:     output = self.o_proj(attention_output)
[default0]:[rank24]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank31]:     trainer.train(dataloader)
[default5]:[rank29]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank24]:     return self._call_impl(*args, **kwargs)
[default0]:[rank24]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank24]:     return forward_call(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default5]:[rank29]:     return forward_call(*args, **kwargs)
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default7]:[rank31]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default5]:[rank29]:     return row_linear(
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default0]:[rank24]:     return row_linear(
[default7]:[rank31]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank29]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default5]:[rank29]:     out = F.linear(input, weight, bias)
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default7]:[rank31]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default0]:[rank24]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default5]:[rank29]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU  has a total capacity of 79.33 GiB of which 5.42 GiB is free. Including non-PyTorch memory, this process has 73.90 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default0]:[rank24]:     out = F.linear(input, weight, bias)
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default0]:[rank24]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 
[default7]:[rank31]:     output = model(**micro_batch)
[default7]:[rank31]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank31]:     return self._call_impl(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank31]:     return forward_call(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default7]:[rank31]:     sharded_logits = self.model(
[default7]:[rank31]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank31]:     return self._call_impl(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank31]:     return forward_call(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default7]:[rank31]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default7]:[rank31]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default7]:[rank31]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank31]:     return self._call_impl(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank31]:     return forward_call(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default7]:[rank31]:     output = self.pp_block(**new_kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank31]:     return self._call_impl(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank31]:     return forward_call(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default7]:[rank31]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default7]:[rank31]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank31]:     return self._call_impl(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank31]:     return forward_call(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default7]:[rank31]:     output = self.o_proj(attention_output)
[default7]:[rank31]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank31]:     return self._call_impl(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank31]:     return forward_call(*args, **kwargs)
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default7]:[rank31]:     return row_linear(
[default7]:[rank31]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default7]:[rank31]:     out = F.linear(input, weight, bias)
[default7]:[rank31]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU  has a total capacity of 79.33 GiB of which 5.42 GiB is free. Including non-PyTorch memory, this process has 73.90 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default4]:[rank28]: Traceback (most recent call last):
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default4]:[rank28]:     trainer.train(dataloader)
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default4]:[rank28]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default4]:[rank28]:     outputs = self.pipeline_engine.train_batch_iter(
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default4]:[rank28]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default4]:[rank28]:     output = model(**micro_batch)
[default4]:[rank28]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank28]:     return self._call_impl(*args, **kwargs)
[default4]:[rank28]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank28]:     return forward_call(*args, **kwargs)
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default4]:[rank28]:     sharded_logits = self.model(
[default4]:[rank28]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank28]:     return self._call_impl(*args, **kwargs)
[default4]:[rank28]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank28]:     return forward_call(*args, **kwargs)
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default4]:[rank28]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default4]:[rank28]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default4]:[rank28]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank28]:     return self._call_impl(*args, **kwargs)
[default4]:[rank28]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank28]:     return forward_call(*args, **kwargs)
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default4]:[rank28]:     output = self.pp_block(**new_kwargs)
[default4]:[rank28]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank28]:     return self._call_impl(*args, **kwargs)
[default4]:[rank28]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank28]:     return forward_call(*args, **kwargs)
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default4]:[rank28]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default4]:[rank28]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank28]:     return self._call_impl(*args, **kwargs)
[default4]:[rank28]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank28]:     return forward_call(*args, **kwargs)
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default4]:[rank28]:     output = self.o_proj(attention_output)
[default4]:[rank28]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank28]:     return self._call_impl(*args, **kwargs)
[default4]:[rank28]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank28]:     return forward_call(*args, **kwargs)
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default4]:[rank28]:     return row_linear(
[default4]:[rank28]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default4]:[rank28]:     out = F.linear(input, weight, bias)
[default4]:[rank28]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU  has a total capacity of 79.33 GiB of which 5.24 GiB is free. Including non-PyTorch memory, this process has 74.08 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default1]:[rank25]: Traceback (most recent call last):
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default1]:[rank25]:     trainer.train(dataloader)
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default1]:[rank25]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default1]:[rank25]:     outputs = self.pipeline_engine.train_batch_iter(
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default1]:[rank25]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default1]:[rank25]:     output = model(**micro_batch)
[default1]:[rank25]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank25]:     return self._call_impl(*args, **kwargs)
[default1]:[rank25]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank25]:     return forward_call(*args, **kwargs)
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default1]:[rank25]:     sharded_logits = self.model(
[default1]:[rank25]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank25]:     return self._call_impl(*args, **kwargs)
[default1]:[rank25]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank25]:     return forward_call(*args, **kwargs)
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default1]:[rank25]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default1]:[rank25]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default1]:[rank25]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank25]:     return self._call_impl(*args, **kwargs)
[default1]:[rank25]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank25]:     return forward_call(*args, **kwargs)
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default1]:[rank25]:     output = self.pp_block(**new_kwargs)
[default1]:[rank25]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank25]:     return self._call_impl(*args, **kwargs)
[default1]:[rank25]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank25]:     return forward_call(*args, **kwargs)
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in[default7]:[rank7]: Traceback (most recent call last):
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default7]:[rank7]:     trainer.train(dataloader)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default7]:[rank7]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default7]:[rank7]:     outputs = self.pipeline_engine.train_batch_iter(
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default7]:[rank7]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/na forward
[default1]:[rank25]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default1]:[rank25]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank25]:     return self._call_impl(*args, **kwargs)
[default1]:[rank25]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank25]:     return forward_call(*args, **kwargs)
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default1]:[rank25]:     output = self.o_proj(attention_output)
[default1]:[rank25]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank25]:     return self._call_impl(*args, **kwargs)
[default1]:[ranotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default7]:[rank7]:     output = model(**micro_batch)
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank7]:     return self._call_impl(*args, **kwargs)
nk25]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank25]:     return forward_call(*args, **kwargs)
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default1]:[rank25]:     return row_linear(
[default1]:[rank25]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default1]:[rank25]:     out = F.linear(input, weight, bias)
[default1]:[rank25]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 
 has a total capacity of 79.33 GiB of which 5.42 GiB is free. Including non-PyTorch memory, this process has 73.90 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large tr[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank7]:     return forward_call(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default7]:[rank7]:     sharded_logits = self.model(
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank7]:     return self._call_impl(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank7]:     return forward_call(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default7]:[rank7]: y setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
    return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default7]:[rank7]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank7]:     return self._call_impl(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank7]:     return forward_call(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default7]:[rank7]:     output = self.pp_block(**new_kwargs)
[default7]:[rank7]: [default2]:[rank26]: Traceback (most recent call last):
[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default2]:[rank26]:     trainer.train(dataloader)
[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default2]:[rank26]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank7]:     return self._call_impl(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank7]:     return forward_call(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default7]:[rank7]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank7]:     return self._call_impl(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default2]:[rank26]:     outputs = self.pipeline_engine.train_batch_iter(
[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default2]:[rank26]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default2]:[rank26]:     output = model(**micro_batch)
nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank7]:     return forward_call(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default7]:[rank7]:     output = self.o_proj(attention_output)
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank7]:     return self._call_impl(*args, **kwargs)
[default2]:[rank26]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank26]:     return self._call_impl(*args, **kwargs)
[default2]:[rank26]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank7]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank7]:     return forward_call(*args, **kwargs)
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default7]:[rank7]:     return row_linear(
[default7]:[rank7]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default7]:[rank7]:     out = F.linear(input, weight, bias)
[default7]:[rank7]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU  has a total capacity of 79.33 GiB of which 5.94 GiB is free. Including non-PyTorch memory, this process has 73.38 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is l[default2]:[rank26]:     return forward_call(*args, **kwargs)
[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default2]:[rank26]:     sharded_logits = self.model(
[default2]:[rank26]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank26]:     return self._call_impl(*args, **kwargs)
arge try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default2]:[rank26]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank26]:     return forward_call(*args, **kwargs)
[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default2]:[rank26]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default2]:[rank26]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default2]:[rank26]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank26]:     return self._call_impl(*args, **kwargs)
[default2]:[rank26]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank26]:     return forward_call(*args, **kwargs)
[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default2]:[rank26]:     output = self.pp_block(**new_kwargs)
[default2]:[rank26]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank26]:     return self._call_impl(*args, **kwargs)
[default2]:[rank26]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank26]:     return forward_call(*args, **kwargs)
[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default2]:[rank26]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default2]:[rank26]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank26]:     return self._call_impl(*args, **kwargs)
[default2]:[rank26]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank26]:     return forward_call(*args, **kwargs)
[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default2]:[rank26]:     output = self.o_proj(attention_output)
[default2]:[rank26]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank26]:     return self._call_impl(*args, **kwargs)
[default2]:[rank26]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank26]:     return forward_call(*args, **kwargs)
[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default2]:[rank26]:     return row_linear(
[default2]:[rank26]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default2]:[rank26]:     out = F.linear(input, weight, bias)
[default2]:[rank26]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU  has a total capacity of 79.33 GiB of which 5.24 GiB is free. Including non-PyTorch memory, this process has 74.08 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default6]:[rank30]: Traceback (most recent call last):
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default6]:[rank30]:     trainer.train(dataloader)
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default6]:[rank30]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default6]:[rank30]:     outputs = self.pipeline_engine.train_batch_iter(
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default6]:[rank30]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default6]:[rank30]:     output = model(**micro_batch)
[default6]:[rank30]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank30]:     return self._call_impl(*args, **kwargs)
[default6]:[rank30]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank30]:     return forward_call(*args, **kwargs)
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default6]:[rank30]:     sharded_logits = self.model(
[default6]:[rank30]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank30]:     return self._call_impl(*args, **kwargs)
[default6]:[rank30]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank30]:     return forward_call(*args, **kwargs)
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default6]:[rank30]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default6]:[rank30]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default6]:[rank30]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank30]:     return self._call_impl(*args, **kwargs)
[default6]:[rank30]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank30]:     return forward_call(*args, **kwargs)
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default6]:[rank30]:     output = self.pp_block(**new_kwargs)
[default6]:[rank30]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank30]:     return self._call_impl(*args, **kwargs)
[default6]:[rank30]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank30]:     return forward_call(*args, **kwargs)
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default6]:[rank30]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default6]:[rank30]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank30]:     return self._call_impl(*args, **kwargs)
[default6]:[rank30]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank30]:     return forward_call(*args, **kwargs)
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default6]:[rank30]:     output = self.o_proj(attention_output)
[default6]:[rank30]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank30]:     return self._call_impl(*args, **kwargs)
[default6]:[rank30]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank30]:     return forward_call(*args, **kwargs)
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default6]:[rank30]:     return row_linear(
[default6]:[rank30]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default6]:[rank30]:     out = F.linear(input, weight, bias)
[default6]:[rank30]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU  has a total capacity of 79.33 GiB of which 5.24 GiB is free. Including non-PyTorch memory, this process has 74.08 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default0]:[rank0]: Traceback (most recent call last):
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default0]:[rank0]:     trainer.train(dataloader)
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default0]:[rank0]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default0]:[rank0]:     outputs = self.pipeline_engine.train_batch_iter(
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default0]:[rank0]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default0]:[rank0]:     output = model(**micro_batch)
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank0]:     return self._call_impl(*args, **kwargs)
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank0]:     return forward_call(*args, **kwargs)
[default5]:[rank13]: Traceback (most recent call last):
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default5]:[rank13]:     trainer.train(dataloader)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default5]:[rank13]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default5]:[rank13]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default5]:[rank13]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanot[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default0]:[rank0]:     sharded_logits = self.model(
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank0]:     return self._call_impl(*args, **kwargs)
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank0]:     return forward_call(*args, **kwargs)
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default0]:[rank0]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_ron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default5]:[rank13]:     output = model(**micro_batch)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank13]:     return self._call_impl(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank13]:     return forward_call(*args, **kwargs)
with_hidden_states
[default0]:[rank0]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank0]:     return self._call_impl(*args, **kwargs)
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank0]:     return forward_call(*args, **kwargs)
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default0]:[rank0]:     output = self.pp_block(**new_kwargs)
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank0]:     return self._call_impl(*args, **kwargs)
[default0[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default5]:[rank13]:     sharded_logits = self.model(
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank13]:     return self._call_impl(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank13]:     return forward_call(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default5]:[rank13]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank0]:     return forward_call(*args, **kwargs)
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default0]:[rank0]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank0]:     return self._call_impl(*args, **kwargs)
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank0]:     return forward_call(*args, **kwargs)
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", li forward_with_hidden_states
[default5]:[rank13]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank13]:     return self._call_impl(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank13]:     return forward_call(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default5]:[rank13]:     output = self.pp_block(**new_kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank13]:     return self._call_impl(*args, *ne 598, in forward
[default0]:[rank0]:     output = self.o_proj(attention_output)
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank0]:     return self._call_impl(*args, **kwargs)
[default0]:[rank0]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank0]:     return forward_call(*args, **kwargs)
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
*kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank13]:     return forward_call(*args, **kwargs)
[default0]:[rank0]:     return row_linear(
[default0]:[rank0]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default0]:[rank0]:     out = F.linear(input, weight, bias)
[default0]:[rank0]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 
[default3]:[rank27]: Traceback (most recent call last):
[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default3]:[rank27]:     trainer.train(dataloader)
[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default3]:[rank27]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default3]:[rank27]:     outputs = self.pipeline_engine.train_batch_iter(
[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default3]:[rank27]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanot[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default5]:[rank13]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank13]:     return self._call_impl(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank13]:     return forward_call(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default5]:[rank13]:     output = self.o_proj(attention_output)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank13]:     return self._call_impl(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank13]:     return forward_call(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default5]:[rank13]:     return row_linear(
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default5]:[rank13]:     out = F.linear(input, weight, bias)
[default5]:[rank13]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU  has a total capacity of 79.33 GiB of which 5.17 GiB is free. Including non-PyTorch memory, this process has 74.15 GiB memory in use. Of the allocated memory 62.85 Giron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
B is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default3]:[rank27]:     output = model(**micro_batch)
[default3]:[rank27]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank27]:     return self._call_impl(*args, **kwargs)
[default3]:[rank27]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank27]:     return forward_call(*args, **kwargs)
[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default3]:[rank27]:     sharded_logits = self.model(
[default3]:[rank27]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank27]:     return self._call_impl(*args, **kwargs)
[default3]:[rank27]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank27]:     return forward_call(*args, **kwargs)
[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default3]:[rank27]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default3]:[rank27]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default3]:[rank27]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank27]:     return self._call_impl(*args, **kwargs)
[default3]:[rank27]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank27]:     return forward_call(*args, **kwargs)
[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default3]:[rank27]:     output = self.pp_block(**new_kwargs)
[default3]:[rank27]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank27]:     return self._call_impl(*args, **kwargs)
[default3]:[rank27]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank27]:     return forward_call(*args, **kwargs)
[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default3]:[rank27]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default3]:[rank27]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank27]:     return self._call_impl(*args, **kwargs)
[default3]:[rank27]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank27]:     return forward_call(*args, **kwargs)
[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default3]:[rank27]:     output = self.o_proj(attention_output)
[default3]:[rank27]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank27]:     return self._call_impl(*args, **kwargs)
[default3]:[rank27]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank27]:     return forward_call(*args, **kwargs)
[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default3]:[rank27]:     return row_linear(
[default3]:[rank27]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default3]:[rank27]:     out = F.linear(input, weight, bias)
[default3]:[rank27]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU  has a total capacity of 79.33 GiB of which 5.42 GiB is free. Including non-PyTorch memory, this process has 73.90 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default4]:[rank12]: Traceback (most recent call last):
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default4]:[rank12]:     trainer.train(dataloader)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default4]:[rank12]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default4]:[rank12]:     outputs = self.pipeline_engine.train_batch_iter(
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default4]:[rank12]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default4]:[rank12]:     output = model(**micro_batch)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank12]:     return self._call_impl(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank12]:     return forward_call(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default4]:[rank12]:     sharded_logits = self.model(
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank12]:     return self._call_impl(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank12]:     return forward_call(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default4]:[rank12]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default4]:[rank12]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank12]:     return self._call_impl(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank12]:     return forward_call(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default4]:[rank12]:     output = self.pp_block(**new_kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank12]:     return self._call_impl(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank12]:     return forward_call(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default4]:[rank12]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank12]:     return self._call_impl(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank12]:     return forward_call(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default4]:[rank12]:     output = self.o_proj(attention_output)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank12]:     return self._call_impl(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank12]:     return forward_call(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default4]:[rank12]:     return row_linear(
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default4]:[rank12]:     out = F.linear(input, weight, bias)
[default4]:[rank12]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU  has a total capacity of 79.33 GiB of which 5.06 GiB is free. Including non-PyTorch memory, this process has 74.26 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default1]:[rank9]: Traceback (most recent call last):
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default1]:[rank9]:     trainer.train(dataloader)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default1]:[rank9]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default1]:[rank9]:     outputs = self.pipeline_engine.train_batch_iter(
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default1]:[rank9]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default1]:[rank9]:     output = model(**micro_batch)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank9]:     return self._call_impl(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank9]:     return forward_call(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default1]:[rank9]:     sharded_logits = self.model(
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank9]:     return self._call_impl(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank9]:     return forward_call(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default1]:[rank9]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default1]:[rank9]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank9]:     return self._call_impl(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank9]:     return forward_call(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default1]:[rank9]:     output = self.pp_block(**new_kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank9]:     return self._call_impl(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank9]:     return forward_call(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default1]:[rank9]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank9]:     return self._call_impl(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank9]:     return forward_call(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default1]:[rank9]:     output = self.o_proj(attention_output)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank9]:     return self._call_impl(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank9]:     return forward_call(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default1]:[rank9]:     return row_linear(
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default1]:[rank9]:     out = F.linear(input, weight, bias)
[default1]:[rank9]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 
 has a total capacity of 79.33 GiB of which 5.17 GiB is free. Including non-PyTorch memory, this process has 74.15 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default0]:[rank8]: Traceback (most recent call last):
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default0]:[rank8]:     trainer.train(dataloader)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default0]:[rank8]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default0]:[rank8]:     outputs = self.pipeline_engine.train_batch_iter(
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default0]:[rank8]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default0]:[rank8]:     output = model(**micro_batch)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank8]:     return self._call_impl(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank8]:     return forward_call(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default0]:[rank8]:     sharded_logits = self.model(
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank8]:     return self._call_impl(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank8]:     return forward_call(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default0]:[rank8]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default0]:[rank8]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank8]:     return self._call_impl(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank8]:     return forward_call(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default0]:[rank8]:     output = self.pp_block(**new_kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank8]:     return self._call_impl(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank8]:     return forward_call(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default0]:[rank8]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank8]:     return self._call_impl(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank8]:     return forward_call(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default0]:[rank8]:     output = self.o_proj(attention_output)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank8]:     return self._call_impl(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank8]:     return forward_call(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default0]:[rank8]:     return row_linear(
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default0]:[rank8]:     out = F.linear(input, weight, bias)
[default0]:[rank8]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 
[default7]:[rank15]: Traceback (most recent call last):
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default7]:[rank15]:     trainer.train(dataloader)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default7]:[rank15]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default7]:[rank15]:     outputs = self.pipeline_engine.train_batch_iter(
[default3]:[rank11]: Traceback (most recent call last):
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default7]:[rank15]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default7]:[rank15]:     output = model(**micro_batch)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank15]:     return self._call_impl(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank15]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default3]:[rank11]:     trainer.train(dataloader)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default7]:[rank15]:     sharded_logits = self.model(
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank15]:     return self._call_impl(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank15]:     return forward_call(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default7]:[rank15]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default7]:[rank15]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank15]:     return self._call_impl(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank15]:     return forward_call(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default7]:[rank15]:     output = self.pp_block(**new_kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank15]:     return self._call_impl(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank11]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default7]:[rank15]:     return forward_call(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default7]:[rank15]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default3]:[rank11]:     outputs = self.pipeline_engine.train_batch_iter(
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default3]:[rank11]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default3]:[rank11]:     output = model(**micro_batch)
[default7]:[rank15]:     return self._call_impl(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank15]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default3]:[rank11]:     return self._call_impl(*args, **kwargs)
[default7]:[rank15]:     output = self.o_proj(attention_output)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank11]:     return forward_call(*args, **kwargs)
[default7]:[rank15]:     return self._call_impl(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank11]:     sharded_logits = self.model(
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank11]:     return self._call_impl(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank11]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default7]:[rank15]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default7]:[rank15]:     return row_linear(
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default3]:[rank11]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank15]:     out = F.linear(input, weight, bias)
[default7]:[rank15]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU  has a total capacity of 79.33 GiB of which 5.17 GiB is free. Including non-PyTorch memory, this process has 74.15 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default3]:[rank11]:     return self._call_impl(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank11]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default3]:[rank11]:     output = self.pp_block(**new_kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank11]:     return self._call_impl(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank11]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default3]:[rank11]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank11]:     return self._call_impl(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank11]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default3]:[rank11]:     output = self.o_proj(attention_output)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank11]:     return self._call_impl(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank11]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default3]:[rank11]:     return row_linear(
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default3]:[rank11]:     out = F.linear(input, weight, bias)
[default3]:[rank11]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU  has a total capacity of 79.33 GiB of which 5.17 GiB is free. Including non-PyTorch memory, this process has 74.15 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default1]:[rank1]: Traceback (most recent call last):
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default2]:[rank2]: Traceback (most recent call last):
[default6]:[rank6]: Traceback (most recent call last):
[default1]:[rank1]:     trainer.train(dataloader)
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default1]:[rank1]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default1]:[rank1]:     outputs = self.pipeline_engine.train_batch_iter(
[default2]:[rank2]:     trainer.train(dataloader)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default2]:[rank2]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default1]:[rank1]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default1]:[rank1]:     output = model(**micro_batch)
[default1]:[rank1]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank6]:     trainer.train(dataloader)
[default1]:[rank1]:     return self._call_impl(*args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default2]:[rank2]:     outputs = self.pipeline_engine.train_batch_iter(
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default1]:[rank1]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank1]:     return forward_call(*args, **kwargs)
[default2]:[rank2]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default1]:[rank1]:     sharded_logits = self.model(
[default6]:[rank6]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default1]:[rank1]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank2]:     output = model(**micro_batch)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default1]:[rank1]:     return self._call_impl(*args, **kwargs)
[default6]:[rank6]:     outputs = self.pipeline_engine.train_batch_iter(
[default1]:[rank1]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default2]:[rank2]:     return self._call_impl(*args, **kwargs)
[default1]:[rank1]:     return forward_call(*args, **kwargs)
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank1]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default2]:[rank2]:     return forward_call(*args, **kwargs)
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default1]:[rank1]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default2]:[rank2]:     sharded_logits = self.model(
[default6]:[rank6]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default1]:[rank1]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank1]:     return self._call_impl(*args, **kwargs)
[default2]:[rank2]:     return self._call_impl(*args, **kwargs)
[default1]:[rank1]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank1]:     return forward_call(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default2]:[rank2]:     return forward_call(*args, **kwargs)
[default6]:[rank6]:     output = model(**micro_batch)
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default6]:[rank6]:     return self._call_impl(*args, **kwargs)
[default2]:[rank2]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank1]:     output = self.pp_block(**new_kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default6]:[rank6]:     return forward_call(*args, **kwargs)
[default1]:[rank1]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank1]:     return self._call_impl(*args, **kwargs)
[default2]:[rank2]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default1]:[rank1]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank2]:     return self._call_impl(*args, **kwargs)
[default1]:[rank1]:     return forward_call(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank1]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default2]:[rank2]:     return forward_call(*args, **kwargs)
[default1]:[rank1]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank6]:     sharded_logits = self.model(
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default1]:[rank1]:     return self._call_impl(*args, **kwargs)
[default2]:[rank2]:     output = self.pp_block(**new_kwargs)
[default1]:[rank1]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank1]:     return forward_call(*args, **kwargs)
[default2]:[rank2]:     return self._call_impl(*args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default1]:[rank1]:     output = self.o_proj(attention_output)
[default2]:[rank2]:     return forward_call(*args, **kwargs)
[default6]:[rank6]:     return self._call_impl(*args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default1]:[rank1]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank2]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank6]:     return forward_call(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default2]:[rank2]:     return self._call_impl(*args, **kwargs)
[default6]:[rank6]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default1]:[rank1]:     return self._call_impl(*args, **kwargs)
[default6]:[rank6]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank1]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank6]:     return self._call_impl(*args, **kwargs)
[default2]:[rank2]:     return forward_call(*args, **kwargs)
[default1]:[rank1]:     return forward_call(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default6]:[rank6]:     return forward_call(*args, **kwargs)
[default1]:[rank1]:     return row_linear(
[default1]:[rank1]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default1]:[rank1]:     out = F.linear(input, weight, bias)
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default6]:[rank6]:     output = self.pp_block(**new_kwargs)
[default1]:[rank1]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 
 has a total capacity of 79.33 GiB of which 5.94 GiB is free. Including non-PyTorch memory, this process has 73.38 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default2]:[rank2]:     output = self.o_proj(attention_output)
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank6]:     return self._call_impl(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank2]:     return self._call_impl(*args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank2]:     return forward_call(*args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default2]:[rank2]:     return row_linear(
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default2]:[rank2]:     out = F.linear(input, weight, bias)
[default2]:[rank2]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU  has a total capacity of 79.33 GiB of which 6.05 GiB is free. Including non-PyTorch memory, this process has 73.27 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default3]:[rank3]: Traceback (most recent call last):
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default3]:[rank3]:     trainer.train(dataloader)
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default3]:[rank3]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default3]:[rank3]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank5]: Traceback (most recent call last):
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default5]:[rank5]:     trainer.train(dataloader)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default5]:[rank5]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default5]:[rank5]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default3]:[rank3]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default3]:[rank3]:     output = model(**micro_batch)
[default3]:[rank3]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank3]:     return self._call_impl(*args, **kwargs)
[default3]:[rank3]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank5]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default3]:[rank3]:     return forward_call(*args, **kwargs)
[default5]:[rank5]:     output = model(**micro_batch)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default5]:[rank5]:     return self._call_impl(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank3]:     sharded_logits = self.model(
[default5]:[rank5]:     return forward_call(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default3]:[rank3]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank3]:     return self._call_impl(*args, **kwargs)
[default5]:[rank5]:     sharded_logits = self.model(
[default3]:[rank3]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank3]:     return forward_call(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default3]:[rank3]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default3]:[rank3]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default3]:[rank3]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank5]:     return self._call_impl(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank3]:     return self._call_impl(*args, **kwargs)
[default3]:[rank3]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank5]:     return forward_call(*args, **kwargs)
[default3]:[rank3]:     return forward_call(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default3]:[rank3]:     output = self.pp_block(**new_kwargs)
[default3]:[rank3]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank5]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default3]:[rank3]:     return self._call_impl(*args, **kwargs)
[default5]:[rank5]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default3]:[rank3]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank3]:     return forward_call(*args, **kwargs)
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default3]:[rank3]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default3]:[rank3]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank3]:     return self._call_impl(*args, **kwargs)
[default3]:[rank3]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank3]:     return forward_call(*args, **kwargs)
[default5]:[rank5]:     return self._call_impl(*args, **kwargs)
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank3]:     output = self.o_proj(attention_output)
[default3]:[rank3]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank3]:     return self._call_impl(*args, **kwargs)
[default3]:[rank3]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank3]:     return forward_call(*args, **kwargs)
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default3]:[rank3]:     return row_linear(
[default3]:[rank3]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default3]:[rank3]:     out = F.linear(input, weight, bias)
[default5]:[rank5]:     return forward_call(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default3]:[rank3]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU  has a total capacity of 79.33 GiB of which 5.94 GiB is free. Including non-PyTorch memory, this process has 73.38 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default5]:[rank5]:     output = self.pp_block(**new_kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank5]:     return self._call_impl(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank5]:     return forward_call(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default5]:[rank5]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank5]:     return self._call_impl(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank5]:     return forward_call(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default5]:[rank5]:     output = self.o_proj(attention_output)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank5]:     return self._call_impl(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank5]:     return forward_call(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default7]:[rank23]: Traceback (most recent call last):
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default7]:[rank23]:     trainer.train(dataloader)
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default7]:[rank23]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default7]:[rank23]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank5]:     return row_linear(
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default2]:[rank10]: Traceback (most recent call last):
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default5]:[rank5]:     out = F.linear(input, weight, bias)
[default5]:[rank5]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU  has a total capacity of 79.33 GiB of which 5.94 GiB is free. Including non-PyTorch memory, this process has 73.38 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default7]:[rank23]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default7]:[rank23]:     output = model(**micro_batch)
[default7]:[rank23]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank23]:     return self._call_impl(*args, **kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank23]:     return forward_call(*args, **kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_clust[default6]:[rank6]:     return forward_call(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default6]:[rank6]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
er/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default7]:[rank23]:     sharded_logits = self.model(
[default7]:[rank23]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank23]:     return self._call_impl(*args, **kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank23]:     return forward_call(*args, **kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default7]:[rank23]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default7]:[rank23]:     hidden_encoder_sta[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank6]:     return self._call_impl(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank6]:     return forward_call(*args, **kwargs)
tes = encoder_block(**hidden_encoder_states)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default6]:[rank6]:     output = self.o_proj(attention_output)
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank6]:     return self._call_impl(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank23]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank23]:     return self._call_impl(*args, **kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank23]:     return forward_call(*args, **kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default7]:[rank23]:     output = self.pp_block(**new_kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank23]:     return self._call_impl(*args, **kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packag[default2]:[rank10]:     trainer.train(dataloader)
[default6]:[rank14]: Traceback (most recent call last):
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default2]:[rank10]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default2]:[rank10]:     outputs = self.pipeline_engine.train_batch_iter(
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default2]:[rank10]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default6]:[rank6]:     return forward_call(*args, **kwargs)
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
es/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank23]:     return forward_call(*args, **kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default7]:[rank23]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default7]:[rank23]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank23]:     return self._call_impl(*args, **kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank23]:     return forward_call(*args, **kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default7]:[rank23]:     output = self.o_proj(attention_output)
[defau[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default6]:[rank14]:     trainer.train(dataloader)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default6]:[rank14]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default6]:[rank6]:     return row_linear(
[default6]:[rank6]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default6]:[rank6]:     out = F.linear(input, weight, bias)
lt7]:[rank23]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank23]:     return self._call_impl(*args, **kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default6]:[rank14]:     outputs = self.pipeline_engine.train_batch_iter(
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default6]:[rank6]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU  has a total capacity of 79.33 GiB of which 6.05 GiB is free. Including non-PyTorch memory, this process has 73.27 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default7]:[rank23]:     return forward_call(*args, **kwargs)
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default7]:[rank23]:     return row_linear(
[default7]:[rank23]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default7]:[rank23]:     out = F.linear(input, weight, bias)
[default2]:[rank10]:     output = model(**micro_batch)
[default7]:[rank23]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU  has a total capacity of 79.33 GiB of which 5.06 GiB is free. Including non-PyTorch memory, this process has 74.26 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default6]:[rank14]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank10]:     return self._call_impl(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank10]:     return forward_call(*args, **kwargs)
[default6]:[rank14]:     output = model(**micro_batch)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default2]:[rank10]:     sharded_logits = self.model(
[default4]:[rank4]: Traceback (most recent call last):
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default4]:[rank4]:     trainer.train(dataloader)
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default4]:[rank4]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank10]:     return self._call_impl(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default4]:[rank4]:     outputs = self.pipeline_engine.train_batch_iter(
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default4]:[rank4]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default4]:[rank4]:     output = model(**micro_batch)
[default4]:[rank4]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank4]:     return self._call_impl(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank4]:     return forward_call(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default4]:[rank4]:     sharded_logits = self.model(
[default4]:[rank4]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank4]:     return self._call_impl(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank4]:     return forward_call(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nano[default2]:[rank10]:     return forward_call(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
tron/models/llama.py", line 764, in forward
[default4]:[rank4]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default4]:[rank4]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default4]:[rank4]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank4]:     return self._call_impl(*args, **kwargs)
[default6]:[rank14]:     return self._call_impl(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank4]:     return forward_call(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default4]:[rank4]:     output = self.pp_block(**new_kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank4]:     return self._call_impl(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank10]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default4]:[rank4]:     return forward_call(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default4]:[rank4]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default4]:[rank4]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank4]:     return self._call_impl(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank4]:     return forward_call(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default4]:[rank4]:     output = self.o_proj(attention_output)
[default4]:[rank4]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank4]:     return self._call_impl(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank4]:     return forward_call(*args, **kwargs)
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default4]:[rank4]:     return row_linear(
[default4]:[rank4]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default4]:[rank4]:     out = F.linear(input, weight, bias)
[default4]:[rank4]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU  has a total capacity of 79.33 GiB of which 6.05 GiB is free. Including non-PyTorch memory, this process has 73.27 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default2]:[rank10]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank14]:     return forward_call(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default2]:[rank10]:     return self._call_impl(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank14]:     sharded_logits = self.model(
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank14]:     return self._call_impl(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank14]:     return forward_call(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default6]:[rank14]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default2]:[rank10]:     return forward_call(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default6]:[rank22]: Traceback (most recent call last):
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default6]:[rank22]:     trainer.train(dataloader)
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default6]:[rank22]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default6]:[rank22]:     outputs = self.pipeline_engine.train_batch_iter(
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default6]:[rank22]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default6]:[rank14]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank14]:     return self._call_impl(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank14]:     return forward_call(*args, **kwargs)
[default2]:[rank10]:     output = self.pp_block(**new_kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default6]:[rank22]:     output = model(**micro_batch)
[default6]:[rank22]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank22]:     return self._call_impl(*args, **kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank22]:     return forward_call(*args, **kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default6]:[rank22]:     sharded_logits = self.model(
[default6]:[rank22]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
call_impl
[default6]:[rank22]:     return self._call_impl(*args, **kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank22]:     return forward_call(*args, **kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default6]:[rank22]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank14]:     output = self.pp_block(**new_kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank10]:     return self._call_impl(*args, **kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default6]:[rank22]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default6]:[rank22]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank22]:     return self._call_impl(*args, **kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank22]:     return forward_call(*args, **kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default6]:[rank22]:     output = self.pp_block(**new_kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-p[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
ackages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank22]:     return self._call_impl(*args, **kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank22]:     return forward_call(*args, **kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default6]:[rank22]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default6]:[rank22]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank22]:     return self._call_impl(*args, **kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank10]:     return forward_call(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default6]:[rank22]:     return forward_call(*args, **kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default6]:[rank22]:     output = self.o_proj(attention_output)
[default6]:[rank22]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank22]:     return self._call_impl(*args, **kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank10]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default6]:[rank22]:     return forward_call(*args, **kwargs)
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default6]:[rank22]:     return row_linear(
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank22]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default6]:[rank22]:     out = F.linear(input, weight, bias)
[default2]:[rank10]:     return self._call_impl(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank22]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU  has a total capacity of 79.33 GiB of which 5.17 GiB is free. Including non-PyTorch memory, this process has 74.15 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default6]:[rank14]:     return self._call_impl(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank10]:     return forward_call(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default2]:[rank10]:     output = self.o_proj(attention_output)
[default6]:[rank14]:     return forward_call(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default6]:[rank14]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank14]:     return self._call_impl(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank10]:     return self._call_impl(*args, **kwargs)
[default6]:[rank14]:     return forward_call(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank10]:     return forward_call(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default2]:[rank10]:     return row_linear(
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default6]:[rank14]:     output = self.o_proj(attention_output)
[default2]:[rank10]:     out = F.linear(input, weight, bias)
[default2]:[rank10]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU  has a total capacity of 79.33 GiB of which 5.06 GiB is free. Including non-PyTorch memory, this process has 74.26 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank14]:     return self._call_impl(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank14]:     return forward_call(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default6]:[rank14]:     return row_linear(
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default6]:[rank14]:     out = F.linear(input, weight, bias)
[default6]:[rank14]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU  has a total capacity of 79.33 GiB of which 5.06 GiB is free. Including non-PyTorch memory, this process has 74.26 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default2]:[rank18]: Traceback (most recent call last):
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default2]:[rank18]:     trainer.train(dataloader)
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default2]:[rank18]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default2]:[rank18]:     outputs = self.pipeline_engine.train_batch_iter(
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default2]:[rank18]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default2]:[rank18]:     output = model(**micro_batch)
[default2]:[rank18]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank18]:     return self._call_impl(*args, **kwargs)
[default2]:[rank18]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank18]:     return forward_call(*args, **kwargs)
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default2]:[rank18]:     sharded_logits = self.model(
[default2]:[rank18]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank18]:     return self._call_impl(*args, **kwargs)
[default2]:[rank18]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank18]:     return forward_call(*args, **kwargs)
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default2]:[rank18]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default2]:[rank18]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default2]:[rank18]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank18]:     return self._call_impl(*args, **kwargs)
[default2]:[rank18]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank18]:     return forward_call(*args, **kwargs)
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default2]:[rank18]:     output = self.pp_block(**new_kwargs)
[default2]:[rank18]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank18]:     return self._call_impl(*args, **kwargs)
[default2]:[rank18]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank18]:     return forward_call(*args, **kwargs)
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default2]:[rank18]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default2]:[rank18]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank18]:     return self._call_impl(*args, **kwargs)
[default2]:[rank18]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank18]:     return forward_call(*args, **kwargs)
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 598, in forward
[default2]:[rank18]:     output = self.o_proj(attention_output)
[default2]:[rank18]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank18]:     return self._call_impl(*args, **kwargs)
[default2]:[rank18]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank18]:     return forward_call(*args, **kwargs)
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 159, in forward
[default2]:[rank18]:     return row_linear(
[default2]:[rank18]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 474, in row_linear
[default2]:[rank18]:     out = F.linear(input, weight, bias)
[default2]:[rank18]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU  has a total capacity of 79.33 GiB of which 5.17 GiB is free. Including non-PyTorch memory, this process has 74.15 GiB memory in use. Of the allocated memory 62.85 GiB is allocated by PyTorch, and 1.50 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[default0]:[rank40]: Traceback (most recent call last):
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default0]:[rank40]:     trainer.train(dataloader)
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default1]:[rank41]: Traceback (most recent call last):
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default0]:[rank40]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default1]:[rank41]:     trainer.train(dataloader)
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default1]:[rank41]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default1]:[rank41]:     outputs = self.pipeline_engine.train_batch_iter(
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default1]:[rank41]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default0]:[rank40]:     outputs = self.pipeline_engine.train_batch_iter(
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default0]:[rank40]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default0]:[rank40]:     output = model(**micro_batch)
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default1]:[rank41]:     output = model(**micro_batch)
[default0]:[rank40]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank40]:     return self._call_impl(*args, **kwargs)
[default0]:[rank40]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank41]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank41]:     return self._call_impl(*args, **kwargs)
[default0]:[rank40]:     return forward_call(*args, **kwargs)
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default1]:[rank41]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank40]:     sharded_logits = self.model(
[default0]:[rank40]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank41]:     return forward_call(*args, **kwargs)
[default0]:[rank40]:     return self._call_impl(*args, **kwargs)
[default0]:[rank40]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default0]:[rank40]:     return forward_call(*args, **kwargs)
[default1]:[rank41]:     sharded_logits = self.model(
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default0]:[rank40]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default1]:[rank41]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default0]:[rank40]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default0]:[rank40]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank40]:     return self._call_impl(*args, **kwargs)
[default1]:[rank41]:     return self._call_impl(*args, **kwargs)
[default1]:[rank41]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank40]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank41]:     return forward_call(*args, **kwargs)
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default1]:[rank41]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default1]:[rank41]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default1]:[rank41]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank41]:     return self._call_impl(*args, **kwargs)
[default0]:[rank40]:     return forward_call(*args, **kwargs)
[default1]:[rank41]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default1]:[rank41]:     return forward_call(*args, **kwargs)
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default0]:[rank40]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default1]:[rank41]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default1]:[rank41]:     pipeline_state.run_communication()
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default1]:[rank41]:     recv_activation_tensor = recv_activation()
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default0]:[rank40]:     pipeline_state.run_communication()
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default0]:[rank40]:     recv_activation_tensor = recv_activation()
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default0]:[rank40]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default0]:[rank40]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default1]:[rank41]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default1]:[rank41]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default0]:[rank40]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default0]:[rank40]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default1]:[rank41]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default0]:[rank40]:     dist.recv(
[default0]:[rank40]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default0]:[rank40]:     return func(*args, **kwargs)
[default0]:[rank40]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default0]:[rank40]:     pg.recv([tensor], group_src_rank, tag).wait()
[default0]:[rank40]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default1]:[rank41]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default0]:[rank40]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default1]:[rank41]:     dist.recv(
[default0]:[rank40]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7ff4dee46897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default0]:[rank40]: frame #1: <unknown function> + 0x5b3a23e (0x7ff51896323e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7ff51895dc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7ff51895df82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7ff51895efd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff518913371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff518913371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff518913371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default0]:[rank40]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff518913371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]:     return func(*args, **kwargs)
[default1]:[rank41]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default0]:[rank40]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7ff4e0120189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank40]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7ff4e0127610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank40]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7ff4e0146978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank41]:     pg.recv([tensor], group_src_rank, tag).wait()
[default1]:[rank41]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default1]:[rank41]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default1]:[rank41]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f88fe95d897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default1]:[rank41]: frame #1: <unknown function> + 0x5b3a23e (0x7f893847a23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #12: <unknown function> + 0x5adc309 (0x7ff518905309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f8938474c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f8938474f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f8938475fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f893842a371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #13: <unknown function> + 0x5ae6f10 (0x7ff51890ff10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f893842a371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f893842a371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f893842a371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #14: <unknown function> + 0x5ae6fa5 (0x7ff51890ffa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f88ffc37189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank40]: frame #15: <unknown function> + 0x5124446 (0x7ff517f4d446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f88ffc3e610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank41]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f88ffc5d978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank41]: frame #12: <unknown function> + 0x5adc309 (0x7f893841c309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #16: <unknown function> + 0x1acf4b8 (0x7ff5148f84b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #17: <unknown function> + 0x5aee004 (0x7ff518917004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #13: <unknown function> + 0x5ae6f10 (0x7f8938426f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #18: <unknown function> + 0x5af36b5 (0x7ff51891c6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #19: <unknown function> + 0xd2631e (0x7ff52b50631e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default0]:[rank40]: frame #20: <unknown function> + 0x47def4 (0x7ff52ac5def4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default0]:[rank40]: frame #21: <unknown function> + 0x1445a6 (0x55777de395a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55777de32a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f8938426fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #23: <unknown function> + 0x150866 (0x55777de45866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55777de2e142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #15: <unknown function> + 0x5124446 (0x7f8937a64446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #16: <unknown function> + 0x1acf4b8 (0x7f893440f4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank41]: frame #17: <unknown function> + 0x5aee004 (0x7f893842e004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55777de39a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #18: <unknown function> + 0x5af36b5 (0x7f89384336b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank40]: frame #26: PyObject_Call + 0xbc (0x55777de45f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #19: <unknown function> + 0xd2631e (0x7f894b01d31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default0]:[rank40]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55777de2c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #20: <unknown function> + 0x47def4 (0x7f894a774ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default0]:[rank40]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55777de39a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55777de2a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #21: <unknown function> + 0x1445a6 (0x5640def945a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #30: <unknown function> + 0x150582 (0x55777de45582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55777de2a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #32: <unknown function> + 0x150582 (0x55777de45582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55777de2a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #22: _PyObject_MakeTpCall + 0x26b (0x5640def8da6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #23: <unknown function> + 0x150866 (0x5640defa0866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #34: <unknown function> + 0x150582 (0x55777de45582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55777de2a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55777de31f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x5640def89142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #25: _PyFunction_Vectorcall + 0x6c (0x5640def94a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #26: PyObject_Call + 0xbc (0x5640defa0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5640def872b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #28: _PyFunction_Vectorcall + 0x6c (0x5640def94a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5640def858fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #30: <unknown function> + 0x150582 (0x5640defa0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55777de43c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #38: <unknown function> + 0x211239 (0x55777df06239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5640def858fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55777de32a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #32: <unknown function> + 0x150582 (0x5640defa0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5640def858fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #34: <unknown function> + 0x150582 (0x5640defa0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55777de2e3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55777de39a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5640def858fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x5640def8cf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55777de29c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55777de39a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55777de2a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #45: <unknown function> + 0x150582 (0x55777de45582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #37: _PyObject_Call_Prepend + 0x69 (0x5640def9ec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #38: <unknown function> + 0x211239 (0x5640df061239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #46: PyObject_Call + 0xbc (0x55777de45f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #39: _PyObject_MakeTpCall + 0x26b (0x5640def8da6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5640def893e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #41: _PyFunction_Vectorcall + 0x6c (0x5640def94a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x5640def84c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #43: _PyFunction_Vectorcall + 0x6c (0x5640def94a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5640def858fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #45: <unknown function> + 0x150582 (0x5640defa0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55777de2c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #48: <unknown function> + 0x150582 (0x55777de45582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #46: PyObject_Call + 0xbc (0x5640defa0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #49: PyObject_Call + 0xbc (0x55777de45f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5640def872b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #48: <unknown function> + 0x150582 (0x5640defa0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55777de2c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55777de39a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55777de32007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #49: PyObject_Call + 0xbc (0x5640defa0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5640def872b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #51: _PyFunction_Vectorcall + 0x6c (0x5640def94a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55777de43c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #54: <unknown function> + 0x211239 (0x55777df06239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #55: PyObject_Call + 0x207 (0x55777de46067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x5640def8d007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55777de2c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #57: <unknown function> + 0x150582 (0x55777de45582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #53: _PyObject_Call_Prepend + 0x69 (0x5640def9ec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55777de2a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #54: <unknown function> + 0x211239 (0x5640df061239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #55: PyObject_Call + 0x207 (0x5640defa1067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #59: <unknown function> + 0x150582 (0x55777de45582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5640def872b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #60: PyObject_Call + 0xbc (0x55777de45f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55777de2c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #62: <unknown function> + 0x150582 (0x55777de45582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: frame #63: PyObject_Call + 0xbc (0x55777de45f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank40]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default1]:[rank41]: frame #57: <unknown function> + 0x150582 (0x5640defa0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5640def858fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #59: <unknown function> + 0x150582 (0x5640defa0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #60: PyObject_Call + 0xbc (0x5640defa0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5640def872b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #62: <unknown function> + 0x150582 (0x5640defa0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: frame #63: PyObject_Call + 0xbc (0x5640defa0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank41]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default2]:[rank42]: Traceback (most recent call last):
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default2]:[rank42]:     trainer.train(dataloader)
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default2]:[rank42]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default2]:[rank42]:     outputs = self.pipeline_engine.train_batch_iter(
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default2]:[rank42]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default2]:[rank42]:     output = model(**micro_batch)
[default2]:[rank42]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank42]:     return self._call_impl(*args, **kwargs)
[default2]:[rank42]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank42]:     return forward_call(*args, **kwargs)
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default2]:[rank42]:     sharded_logits = self.model(
[default2]:[rank42]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank42]:     return self._call_impl(*args, **kwargs)
[default2]:[rank42]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank42]:     return forward_call(*args, **kwargs)
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default2]:[rank42]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default2]:[rank42]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default2]:[rank42]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank42]:     return self._call_impl(*args, **kwargs)
[default2]:[rank42]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank42]:     return forward_call(*args, **kwargs)
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default2]:[rank42]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default2]:[rank42]:     pipeline_state.run_communication()
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default2]:[rank42]:     recv_activation_tensor = recv_activation()
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default2]:[rank42]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default2]:[rank42]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default2]:[rank42]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default2]:[rank42]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default2]:[rank42]:     dist.recv(
[default2]:[rank42]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default2]:[rank42]:     return func(*args, **kwargs)
[default2]:[rank42]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default2]:[rank42]:     pg.recv([tensor], group_src_rank, tag).wait()
[default2]:[rank42]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default2]:[rank42]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default2]:[rank42]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fa2975aa897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default2]:[rank42]: frame #1: <unknown function> + 0x5b3a23e (0x7fa2d10c723e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fa2d10c1c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fa2d10c1f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fa2d10c2fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa2d1077371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa2d1077371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa2d1077371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa2d1077371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fa298884189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank42]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fa29888b610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank42]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fa2988aa978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank42]: frame #12: <unknown function> + 0x5adc309 (0x7fa2d1069309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #13: <unknown function> + 0x5ae6f10 (0x7fa2d1073f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fa2d1073fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #15: <unknown function> + 0x5124446 (0x7fa2d06b1446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #16: <unknown function> + 0x1acf4b8 (0x7fa2cd05c4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #17: <unknown function> + 0x5aee004 (0x7fa2d107b004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #18: <unknown function> + 0x5af36b5 (0x7fa2d10806b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank42]: frame #19: <unknown function> + 0xd2631e (0x7fa2e3c6a31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default2]:[rank42]: frame #20: <unknown function> + 0x47def4 (0x7fa2e33c1ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default2]:[rank42]: frame #21: <unknown function> + 0x1445a6 (0x56035b6075a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #22: _PyObject_MakeTpCall + 0x26b (0x56035b600a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #23: <unknown function> + 0x150866 (0x56035b613866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x56035b5fc142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #25: _PyFunction_Vectorcall + 0x6c (0x56035b607a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #26: PyObject_Call + 0xbc (0x56035b613f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x56035b5fa2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #28: _PyFunction_Vectorcall + 0x6c (0x56035b607a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x56035b5f88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #30: <unknown function> + 0x150582 (0x56035b613582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x56035b5f88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #32: <unknown function> + 0x150582 (0x56035b613582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: Traceback (most recent call last):
[default2]:[rank42]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x56035b5f88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #34: <unknown function> + 0x150582 (0x56035b613582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x56035b5f88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x56035b5fff50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #37: _PyObject_Call_Prepend + 0x69 (0x56035b611c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #38: <unknown function> + 0x211239 (0x56035b6d4239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #39: _PyObject_MakeTpCall + 0x26b (0x56035b600a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x56035b5fc3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #41: _PyFunction_Vectorcall + 0x6c (0x56035b607a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x56035b5f7c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #43: _PyFunction_Vectorcall + 0x6c (0x56035b607a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x56035b5f88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #45: <unknown function> + 0x150582 (0x56035b613582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #46: PyObject_Call + 0xbc (0x56035b613f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x56035b5fa2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #48: <unknown function> + 0x150582 (0x56035b613582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #49: PyObject_Call + 0xbc (0x56035b613f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x56035b5fa2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #51: _PyFunction_Vectorcall + 0x6c (0x56035b607a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x56035b600007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default1]:[rank33]:     trainer.train(dataloader)
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default7]:[rank39]: Traceback (most recent call last):
[default2]:[rank42]: frame #53: _PyObject_Call_Prepend + 0x69 (0x56035b611c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default7]:[rank39]:     trainer.train(dataloader)
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default1]:[rank33]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default2]:[rank42]: frame #54: <unknown function> + 0x211239 (0x56035b6d4239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default2]:[rank42]: frame #55: PyObject_Call + 0x207 (0x56035b614067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x56035b5fa2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #57: <unknown function> + 0x150582 (0x56035b613582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x56035b5f88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #59: <unknown function> + 0x150582 (0x56035b613582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default2]:[rank42]: frame #60: PyObject_Call + 0xbc (0x56035b613f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x56035b5fa2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #62: <unknown function> + 0x150582 (0x56035b613582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank42]: frame #63: PyObject_Call + 0xbc (0x56035b613f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]:     outputs = self.pipeline_engine.train_batch_iter(
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default2]:[rank42]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default1]:[rank33]:     outputs = self.pipeline_engine.train_batch_iter(
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default1]:[rank33]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default1]:[rank33]:     output = model(**micro_batch)
[default1]:[rank33]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default7]:[rank39]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default2]:[rank34]: Traceback (most recent call last):
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default1]:[rank33]:     return self._call_impl(*args, **kwargs)
[default7]:[rank39]:     output = model(**micro_batch)
[default1]:[rank33]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank39]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank33]:     return forward_call(*args, **kwargs)
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default2]:[rank34]:     trainer.train(dataloader)
[default7]:[rank39]:     return self._call_impl(*args, **kwargs)
[default7]:[rank39]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank39]:     return forward_call(*args, **kwargs)
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default2]:[rank34]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default1]:[rank33]:     sharded_logits = self.model(
[default1]:[rank33]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default2]:[rank34]:     outputs = self.pipeline_engine.train_batch_iter(
[default1]:[rank33]:     return self._call_impl(*args, **kwargs)
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default7]:[rank39]:     sharded_logits = self.model(
[default7]:[rank39]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank39]:     return self._call_impl(*args, **kwargs)
[default1]:[rank33]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank39]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank39]:     return forward_call(*args, **kwargs)
[default2]:[rank34]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default1]:[rank33]:     return forward_call(*args, **kwargs)
[default7]:[rank39]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default2]:[rank34]:     output = model(**micro_batch)
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default2]:[rank34]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank33]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank46]: Traceback (most recent call last):
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default2]:[rank34]:     return self._call_impl(*args, **kwargs)
[default7]:[rank39]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default2]:[rank34]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default1]:[rank33]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default2]:[rank34]:     return forward_call(*args, **kwargs)
[default1]:[rank33]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default7]:[rank39]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank33]:     return self._call_impl(*args, **kwargs)
[default2]:[rank34]:     sharded_logits = self.model(
[default7]:[rank39]:     return self._call_impl(*args, **kwargs)
[default6]:[rank46]:     trainer.train(dataloader)
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default6]:[rank46]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default2]:[rank34]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default6]:[rank46]:     outputs = self.pipeline_engine.train_batch_iter(
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default6]:[rank46]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default6]:[rank46]:     output = model(**micro_batch)
[default6]:[rank46]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank46]:     return self._call_impl(*args, **kwargs)
[default6]:[rank46]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/py[default7]:[rank39]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
thon3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank46]:     return forward_call(*args, **kwargs)
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default6]:[rank46]:     sharded_logits = self.model(
[default6]:[rank46]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank46]:     return self._call_impl(*args, **kwargs)
[default6]:[rank46]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank46]:     return forward_call(*args, **kwargs)
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default6]:[rank46]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=in[default7]:[rank39]:     return forward_call(*args, **kwargs)
[default1]:[rank33]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank34]:     return self._call_impl(*args, **kwargs)
put_mask)[0]
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default6]:[rank46]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default6]:[rank46]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank46]:     return self._call_impl(*args, **kwargs)
[default6]:[rank46]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank46]:     return forward_call(*args, **kwargs)
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default6]:[rank46]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default1]:[rank33]:     return forward_call(*args, **kwargs)
[default2]:[rank34]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank39]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default6]:[rank46]:     pipeline_state.run_communication()
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default6]:[rank46]:     recv_activation_tensor = recv_activation()
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default6]:[rank46]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default6]:[rank46]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default6]:[rank46]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default6]:[rank46]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default6]:[rank46]:     dist.recv(
[default6]:[rank46]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default2]:[rank34]:     return forward_call(*args, **kwargs)
[default6]:[rank46]:     return func(*args, **kwargs)
[default6]:[rank46]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default6]:[rank46]:     pg.recv([tensor], group_src_rank, tag).wait()
[default1]:[rank33]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default6]:[rank46]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default6]:[rank46]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default7]:[rank39]:     pipeline_state.run_communication()
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default6]:[rank46]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f93fd8a2897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default6]:[rank46]: frame #1: <unknown function> + 0x5b3a23e (0x7f94373bf23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]:     pipeline_state.run_communication()
[default6]:[rank46]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f94373b9c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f94373b9f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank46]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f94373bafd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f943736f371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default6]:[rank46]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f943736f371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f943736f371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default1]:[rank33]:     recv_activation_tensor = recv_activation()
[default7]:[rank39]:     recv_activation_tensor = recv_activation()
[default6]:[rank46]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f943736f371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f93feb7c189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank46]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f93feb83610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default6]:[rank46]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f93feba2978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank46]: frame #12: <unknown function> + 0x5adc309 (0x7f9437361309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #13: <unknown function> + 0x5ae6f10 (0x7f943736bf10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default6]:[rank46]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f943736bfa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #15: <unknown function> + 0x5124446 (0x7f94369a9446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default1]:[rank33]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default2]:[rank34]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default7]:[rank39]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default6]:[rank46]: frame #16: <unknown function> + 0x1acf4b8 (0x7f94333544b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default6]:[rank46]: frame #17: <unknown function> + 0x5aee004 (0x7f9437373004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank46]: frame #18: <unknown function> + 0x5af36b5 (0x7f94373786b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #19: <unknown function> + 0xd2631e (0x7f9449f6231e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default1]:[rank33]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default6]:[rank46]: frame #20: <unknown function> + 0x47def4 (0x7f94496b9ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default2]:[rank34]:     return self._call_impl(*args, **kwargs)
[default6]:[rank46]: frame #21: <unknown function> + 0x1445a6 (0x55ce31ea65a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default6]:[rank46]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55ce31e9fa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #23: <unknown function> + 0x150866 (0x55ce31eb2866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default7]:[rank47]: Traceback (most recent call last):
[default7]:[rank39]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default6]:[rank46]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55ce31e9b142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default1]:[rank33]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default7]:[rank47]:     trainer.train(dataloader)
[default7]:[rank39]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default1]:[rank33]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default7]:[rank47]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default7]:[rank39]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default1]:[rank33]:     dist.recv(
[default7]:[rank47]:     outputs = self.pipeline_engine.train_batch_iter(
[default2]:[rank34]:     return forward_call(*args, **kwargs)
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default1]:[rank33]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default6]:[rank46]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55ce31ea6a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]:     return func(*args, **kwargs)
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default2]:[rank34]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default7]:[rank47]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default1]:[rank33]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default2]:[rank34]:     pipeline_state.run_communication()
[default6]:[rank46]: frame #26: PyObject_Call + 0xbc (0x55ce31eb2f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]:     dist.recv(
[default6]:[rank46]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55ce31e992b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]:     output = model(**micro_batch)
[default1]:[rank33]:     pg.recv([tensor], group_src_rank, tag).wait()
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default6]:[rank46]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55ce31ea6a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default7]:[rank47]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank34]:     recv_activation_tensor = recv_activation()
[default7]:[rank47]:     return self._call_impl(*args, **kwargs)
[default6]:[rank46]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55ce31e978fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #30: <unknown function> + 0x150582 (0x55ce31eb2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]:     return func(*args, **kwargs)
[default1]:[rank33]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default7]:[rank47]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank39]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default7]:[rank47]:     return forward_call(*args, **kwargs)
[default2]:[rank34]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default6]:[rank46]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55ce31e978fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]:     pg.recv([tensor], group_src_rank, tag).wait()
[default7]:[rank39]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default1]:[rank33]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default7]:[rank39]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default6]:[rank46]: frame #32: <unknown function> + 0x150582 (0x55ce31eb2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default6]:[rank46]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55ce31e978fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]:     sharded_logits = self.model(
[default1]:[rank33]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fecf1b81897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default6]:[rank46]: frame #34: <unknown function> + 0x150582 (0x55ce31eb2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7ff3eb3ac897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default7]:[rank47]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank33]: frame #1: <unknown function> + 0x5b3a23e (0x7fed2b69e23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]:     return self._call_impl(*args, **kwargs)
[default6]:[rank46]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55ce31e978fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55ce31e9ef50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default6]:[rank46]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55ce31eb0c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #1: <unknown function> + 0x5b3a23e (0x7ff424ec923e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fed2b698c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank47]:     return forward_call(*args, **kwargs)
[default7]:[rank39]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7ff424ec3c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default7]:[rank47]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default1]:[rank33]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fed2b698f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default7]:[rank39]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7ff424ec3f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default2]:[rank34]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default7]:[rank47]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank47]:     return self._call_impl(*args, **kwargs)
[default7]:[rank47]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank47]:     return forward_call(*args, **kwargs)
[default6]:[rank46]: frame #38: <unknown function> + 0x211239 (0x55ce31f73239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fed2b699fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55ce31e9fa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default7]:[rank47]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default1]:[rank33]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fed2b64e371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default2]:[rank34]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default6]:[rank46]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55ce31e9b3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fed2b64e371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7ff424ec4fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]:     dist.recv(
[default7]:[rank47]:     pipeline_state.run_communication()
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default1]:[rank33]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fed2b64e371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55ce31ea6a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff424e79371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default1]:[rank33]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fed2b64e371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55ce31e96c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]:     recv_activation_tensor = recv_activation()
[default2]:[rank34]:     return func(*args, **kwargs)
[default6]:[rank46]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55ce31ea6a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fecf2e5b189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default7]:[rank39]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff424e79371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default6]:[rank46]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55ce31e978fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default6]:[rank46]: frame #45: <unknown function> + 0x150582 (0x55ce31eb2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #46: PyObject_Call + 0xbc (0x55ce31eb2f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]:     pg.recv([tensor], group_src_rank, tag).wait()
[default7]:[rank39]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff424e79371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fecf2e62610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank46]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55ce31e992b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #48: <unknown function> + 0x150582 (0x55ce31eb2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff424e79371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #49: PyObject_Call + 0xbc (0x55ce31eb2f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fecf2e81978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank33]: frame #12: <unknown function> + 0x5adc309 (0x7fed2b640309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default7]:[rank39]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7ff3ec686189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank47]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default1]:[rank33]: frame #13: <unknown function> + 0x5ae6f10 (0x7fed2b64af10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default7]:[rank47]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default2]:[rank34]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default7]:[rank47]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default2]:[rank34]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default6]:[rank46]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55ce31e992b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fed2b64afa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55ce31ea6a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]:     dist.recv(
[default7]:[rank39]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7ff3ec68d610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank34]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fe2221f4897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default7]:[rank47]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default7]:[rank39]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7ff3ec6ac978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank46]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55ce31e9f007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #1: <unknown function> + 0x5b3a23e (0x7fe25bd1123e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]:     return func(*args, **kwargs)
[default1]:[rank33]: frame #15: <unknown function> + 0x5124446 (0x7fed2ac88446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]: frame #12: <unknown function> + 0x5adc309 (0x7ff424e6b309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default2]:[rank34]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fe25bd0bc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]: frame #13: <unknown function> + 0x5ae6f10 (0x7ff424e75f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]:     pg.recv([tensor], group_src_rank, tag).wait()
[default2]:[rank34]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fe25bd0bf82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55ce31eb0c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fe25bd0cfd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe25bcc1371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #54: <unknown function> + 0x211239 (0x55ce31f73239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe25bcc1371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #55: PyObject_Call + 0x207 (0x55ce31eb3067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55ce31e992b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #16: <unknown function> + 0x1acf4b8 (0x7fed276334b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]: frame #14: <unknown function> + 0x5ae6fa5 (0x7ff424e75fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #57: <unknown function> + 0x150582 (0x55ce31eb2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe25bcc1371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default2]:[rank34]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe25bcc1371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]: frame #17: <unknown function> + 0x5aee004 (0x7fed2b652004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55ce31e978fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #18: <unknown function> + 0x5af36b5 (0x7fed2b6576b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #59: <unknown function> + 0x150582 (0x55ce31eb2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #60: PyObject_Call + 0xbc (0x55ce31eb2f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #15: <unknown function> + 0x5124446 (0x7ff4244b3446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank46]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55ce31e992b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fe2234ce189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank46]: frame #62: <unknown function> + 0x150582 (0x55ce31eb2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank46]: frame #63: PyObject_Call + 0xbc (0x55ce31eb2f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #16: <unknown function> + 0x1acf4b8 (0x7ff420e5e4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default2]:[rank34]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fe2234d5610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank33]: frame #19: <unknown function> + 0xd2631e (0x7fed3e24131e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:[rank46]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default2]:[rank34]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fe2234f4978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank47]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f8f7b3ec897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default7]:[rank47]: frame #1: <unknown function> + 0x5b3a23e (0x7f8fb4f0923e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f8fb4f03c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]: frame #20: <unknown function> + 0x47def4 (0x7fed3d998ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:[rank47]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f8fb4f03f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f8fb4f04fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f8fb4eb9371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]: frame #17: <unknown function> + 0x5aee004 (0x7ff424e7d004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]: frame #18: <unknown function> + 0x5af36b5 (0x7ff424e826b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f8fb4eb9371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f8fb4eb9371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f8fb4eb9371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]: frame #21: <unknown function> + 0x1445a6 (0x55a937f1e5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f8f7c6c6189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank47]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f8f7c6cd610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank47]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f8f7c6ec978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank33]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55a937f17a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #12: <unknown function> + 0x5adc309 (0x7f8fb4eab309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #13: <unknown function> + 0x5ae6f10 (0x7f8fb4eb5f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]: frame #12: <unknown function> + 0x5adc309 (0x7fe25bcb3309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f8fb4eb5fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]: frame #19: <unknown function> + 0xd2631e (0x7ff437a6c31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default1]:[rank33]: frame #23: <unknown function> + 0x150866 (0x55a937f2a866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #15: <unknown function> + 0x5124446 (0x7f8fb44f3446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #16: <unknown function> + 0x1acf4b8 (0x7f8fb0e9e4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #17: <unknown function> + 0x5aee004 (0x7f8fb4ebd004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]: frame #13: <unknown function> + 0x5ae6f10 (0x7fe25bcbdf10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #18: <unknown function> + 0x5af36b5 (0x7f8fb4ec26b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #19: <unknown function> + 0xd2631e (0x7f8fc7aac31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:[rank47]: frame #20: <unknown function> + 0x47def4 (0x7f8fc7203ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:[rank47]: frame #21: <unknown function> + 0x1445a6 (0x5644df0c85a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #20: <unknown function> + 0x47def4 (0x7ff4371c3ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:[rank47]: frame #22: _PyObject_MakeTpCall + 0x26b (0x5644df0c1a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #23: <unknown function> + 0x150866 (0x5644df0d4866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x5644df0bd142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #25: _PyFunction_Vectorcall + 0x6c (0x5644df0c8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #21: <unknown function> + 0x1445a6 (0x5567c04d55a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #22: _PyObject_MakeTpCall + 0x26b (0x5567c04cea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #26: PyObject_Call + 0xbc (0x5644df0d4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5644df0bb2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #28: _PyFunction_Vectorcall + 0x6c (0x5644df0c8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5644df0b98fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #23: <unknown function> + 0x150866 (0x5567c04e1866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #30: <unknown function> + 0x150582 (0x5644df0d4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5644df0b98fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #32: <unknown function> + 0x150582 (0x5644df0d4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55a937f13142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5644df0b98fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #34: <unknown function> + 0x150582 (0x5644df0d4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5644df0b98fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x5644df0c0f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55a937f1ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #37: _PyObject_Call_Prepend + 0x69 (0x5644df0d2c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #38: <unknown function> + 0x211239 (0x5644df195239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #39: _PyObject_MakeTpCall + 0x26b (0x5644df0c1a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fe25bcbdfa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5644df0bd3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #41: _PyFunction_Vectorcall + 0x6c (0x5644df0c8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x5644df0b8c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #43: _PyFunction_Vectorcall + 0x6c (0x5644df0c8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x5567c04ca142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5644df0b98fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #45: <unknown function> + 0x150582 (0x5644df0d4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #46: PyObject_Call + 0xbc (0x5644df0d4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5644df0bb2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #25: _PyFunction_Vectorcall + 0x6c (0x5567c04d5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #26: PyObject_Call + 0xbc (0x55a937f2af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #15: <unknown function> + 0x5124446 (0x7fe25b2fb446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank47]: frame #48: <unknown function> + 0x150582 (0x5644df0d4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #49: PyObject_Call + 0xbc (0x5644df0d4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5644df0bb2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #26: PyObject_Call + 0xbc (0x5567c04e1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #51: _PyFunction_Vectorcall + 0x6c (0x5644df0c8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x5644df0c1007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #53: _PyObject_Call_Prepend + 0x69 (0x5644df0d2c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #54: <unknown function> + 0x211239 (0x5644df195239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #16: <unknown function> + 0x1acf4b8 (0x7fe257ca64b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5567c04c82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55a937f112b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #17: <unknown function> + 0x5aee004 (0x7fe25bcc5004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank39]: frame #28: _PyFunction_Vectorcall + 0x6c (0x5567c04d5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #18: <unknown function> + 0x5af36b5 (0x7fe25bcca6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]: frame #19: <unknown function> + 0xd2631e (0x7fe26e8b431e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:[rank39]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5567c04c68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55a937f1ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #55: PyObject_Call + 0x207 (0x5644df0d5067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5644df0bb2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #57: <unknown function> + 0x150582 (0x5644df0d4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #20: <unknown function> + 0x47def4 (0x7fe26e00bef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:[rank39]: frame #30: <unknown function> + 0x150582 (0x5567c04e1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5644df0b98fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #59: <unknown function> + 0x150582 (0x5644df0d4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #60: PyObject_Call + 0xbc (0x5644df0d4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5644df0bb2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #62: <unknown function> + 0x150582 (0x5644df0d4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: frame #63: PyObject_Call + 0xbc (0x5644df0d4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank47]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default1]:[rank33]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55a937f0f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5567c04c68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #21: <unknown function> + 0x1445a6 (0x55ab037185a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55ab03711a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #30: <unknown function> + 0x150582 (0x55a937f2a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #32: <unknown function> + 0x150582 (0x5567c04e1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #23: <unknown function> + 0x150866 (0x55ab03724866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55a937f0f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55ab0370d142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55ab03718a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5567c04c68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #34: <unknown function> + 0x150582 (0x5567c04e1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #32: <unknown function> + 0x150582 (0x55a937f2a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #26: PyObject_Call + 0xbc (0x55ab03724f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: Traceback (most recent call last):
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default3]:[rank43]:     trainer.train(dataloader)
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default3]:[rank43]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default3]:[rank43]:     outputs = self.pipeline_engine.train_batch_iter(
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default3]:[rank43]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanot[default1]:[rank33]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55a937f0f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55ab0370b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55ab03718a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5567c04c68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x5567c04cdf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #37: _PyObject_Call_Prepend + 0x69 (0x5567c04dfc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55ab037098fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #34: <unknown function> + 0x150582 (0x55a937f2a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #38: <unknown function> + 0x211239 (0x5567c05a2239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #30: <unknown function> + 0x150582 (0x55ab03724582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #39: _PyObject_MakeTpCall + 0x26b (0x5567c04cea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
ron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default3]:[rank43]:     output = model(**micro_batch)
[default3]:[rank43]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank43]:     return self._call_impl(*args, **kwargs)
[default3]:[rank43]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank43]:     return forward_call(*args, **kwargs)
[default1]:[rank33]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55a937f0f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default3]:[rank43]:     sharded_logits = self.model(
[default3]:[rank43]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank43]:     return self._call_impl(*args, **kwargs)
[default3]:[rank43]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank43]:     return forward_call(*args, **kwargs)
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default3]:[rank43]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in[default7]:[rank39]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5567c04ca3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
 forward_with_hidden_states
[default3]:[rank43]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default3]:[rank43]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank43]:     return self._call_impl(*args, **kwargs)
[default3]:[rank43]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank43]:     return forward_call(*args, **kwargs)
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default3]:[rank43]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default3]:[rank43]:     pipe[default2]:[rank34]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55ab037098fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
line_state.run_communication()
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default3]:[rank43]:     recv_activation_tensor = recv_activation()
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default3]:[rank43]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default1]:[rank33]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55a937f16f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #41: _PyFunction_Vectorcall + 0x6c (0x5567c04d5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default3]:[rank43]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default3]:[rank43]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default3]:[rank43]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default3]:[rank43]:     dist.recv(
[default3]:[rank43]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default3]:[rank43]:     return func(*args, **kwargs)
[default3]:[rank43]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site[default2]:[rank34]: frame #32: <unknown function> + 0x150582 (0x55ab03724582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x5567c04c5c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default3]:[rank43]:     pg.recv([tensor], group_src_rank, tag).wait()
[default3]:[rank43]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default3]:[rank43]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default3]:[rank43]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f12eb74d897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default3]:[rank43]: frame #1: <unknown function> + 0x5b3a23e (0x7f132526a23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l[default2]:[rank34]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55ab037098fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #34: <unknown function> + 0x150582 (0x55ab03724582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
> >) + 0x2c7 (0x7f1325264c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f1325264f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f1325265fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f132521a371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f132521a371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #7: c10d::PrefixStore::get(st[default1]:[rank33]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55a937f28c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55ab037098fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
d::string const&) + 0x31 (0x7f132521a371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f132521a371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f12eca27189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:[rank43]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f12eca2e610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank34]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55ab03710f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f12eca4d978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:[rank43]: frame #12: <unknown function> + 0x5adc309 (0x7f132520c309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #13: <unknown function> + 0x5ae6f10 (0x7f1325216f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]: frame #38: <unknown function> + 0x211239 (0x55a937feb239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f1325216fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #15: <unknown function> + 0x5124446 (0x7f1324854446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #16: <unknown function> + 0x1acf4b8 (0x7f13211ff4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #17: <unknown function> + 0x5aee004 (0x7f132521e004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank43]: frame #18: <unknown function> + 0x5af36b5 (0x7f13252236b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55ab03722c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #19: <unknown function> + 0xd2631e (0x7f1337e0d31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default3]:[rank43]: frame #20: <unknown function> + 0x47def4 (0x7f1337564ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default3]:[rank43]: frame #21: <unknown function> + 0x1445a6 (0x55c4f4c245a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55a937f17a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55c4f4c1da6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #23: <unknown function> + 0x150866 (0x55c4f4c30866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55c4f4c19142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55c4f4c24a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #38: <unknown function> + 0x211239 (0x55ab037e5239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #26: PyObject_Call + 0xbc (0x55c4f4c30f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55c4f4c172b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55c4f4c24a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55c4f4c158fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #30: <unknown function> + 0x150582 (0x55c4f4c30582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55c4f4c158fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #32: <unknown function> + 0x150582 (0x55c4f4c30582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cl[default7]:[rank39]: frame #43: _PyFunction_Vectorcall + 0x6c (0x5567c04d5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
uster/bin/python3.10)
[default3]:[rank43]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55c4f4c158fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #34: <unknown function> + 0x150582 (0x55c4f4c30582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55c4f4c158fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55c4f4c1cf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55c4f4c2ec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #38: <unknown function> + 0x211239 (0x55c4f4cf1239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55c4f4c1da6b in /fsx/ferdinand[default1]:[rank33]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55a937f133e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
mom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55c4f4c193e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55c4f4c24a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55c4f4c14c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55c4f4c24a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55ab03711a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5567c04c68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55c4f4c158fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #45: <unknown function> + 0x150582 (0x55c4f4c30582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #46: PyObject_Call + 0xbc (0x55c4f4c30f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55c4f4c172b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #48: <unknown function> + 0x150582 (0x55c4f4c30582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #49: PyObject_Call + 0xbc (0x55c4f4c30f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55c4f4c172b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin[default1]:[rank33]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55a937f1ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55ab0370d3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
/python3.10)
[default7]:[rank39]: frame #45: <unknown function> + 0x150582 (0x5567c04e1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #46: PyObject_Call + 0xbc (0x5567c04e1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55c4f4c24a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55c4f4c1d007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55c4f4c2ec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #54: <unknown function> + 0x211239 (0x55c4f4cf1239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #55: PyObject_Call + 0x207 (0x55c4f4c31067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55ab03718a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55ab03708c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55c4f4c172b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #57: <unknown function> + 0x150582 (0x55c4f4c30582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55c4f4c158fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #59: <unknown function> + 0x150582 (0x55c4f4c30582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5567c04c82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #60: PyObject_Call + 0xbc (0x55c4f4c30f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55c4f4c172b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #62: <unknown function> + 0x150582 (0x55c4f4c30582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55ab03718a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55ab037098fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: frame #63: PyObject_Call + 0xbc (0x55c4f4c30f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank43]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default7]:[rank39]: frame #48: <unknown function> + 0x150582 (0x5567c04e1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #45: <unknown function> + 0x150582 (0x55ab03724582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55a937f0ec5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #46: PyObject_Call + 0xbc (0x55ab03724f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55a937f1ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55a937f0f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55ab0370b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #49: PyObject_Call + 0xbc (0x5567c04e1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5567c04c82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #48: <unknown function> + 0x150582 (0x55ab03724582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #45: <unknown function> + 0x150582 (0x55a937f2a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #51: _PyFunction_Vectorcall + 0x6c (0x5567c04d5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: Traceback (most recent call last):
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default7]:[rank39]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x5567c04ce007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #46: PyObject_Call + 0xbc (0x55a937f2af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #49: PyObject_Call + 0xbc (0x55ab03724f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]:     trainer.train(dataloader)
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default5]:[rank45]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default1]:[rank33]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55a937f112b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default5]:[rank45]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default5]:[rank45]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default2]:[rank34]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55ab0370b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default5]:[rank45]:     output = model(**micro_batch)
[default5]:[rank45]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank39]: frame #53: _PyObject_Call_Prepend + 0x69 (0x5567c04dfc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]:     return self._call_impl(*args, **kwargs)
[default5]:[rank45]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank45]:     return forward_call(*args, **kwargs)
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default2]:[rank34]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55ab03718a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]:     sharded_logits = self.model(
[default5]:[rank45]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank45]:     return self._call_impl(*args, **kwargs)
[default7]:[rank39]: frame #54: <unknown function> + 0x211239 (0x5567c05a2239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #55: PyObject_Call + 0x207 (0x5567c04e2067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank45]:     return forward_call(*args, **kwargs)
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default1]:[rank33]: frame #48: <unknown function> + 0x150582 (0x55a937f2a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55ab03711007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default5]:[rank45]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default1]:[rank33]: frame #49: PyObject_Call + 0xbc (0x55a937f2af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5567c04c82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #57: <unknown function> + 0x150582 (0x5567c04e1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank45]:     return self._call_impl(*args, **kwargs)
[default5]:[rank45]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank45]:     return forward_call(*args, **kwargs)
[default2]:[rank34]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55ab03722c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default5]:[rank45]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default2]:[rank34]: frame #54: <unknown function> + 0x211239 (0x55ab037e5239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]:     pipeline_state.run_communication()
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default5]:[rank45]:     recv_activation_tensor = recv_activation()
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default7]:[rank39]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5567c04c68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55a937f112b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default5]:[rank45]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default5]:[rank45]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default5]:[rank45]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default2]:[rank34]: frame #55: PyObject_Call + 0x207 (0x55ab03725067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55ab0370b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]:     dist.recv(
[default5]:[rank45]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default5]:[rank45]:     return func(*args, **kwargs)
[default7]:[rank39]: frame #59: <unknown function> + 0x150582 (0x5567c04e1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #60: PyObject_Call + 0xbc (0x5567c04e1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default5]:[rank45]:     pg.recv([tensor], group_src_rank, tag).wait()
[default5]:[rank45]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default1]:[rank33]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55a937f1ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default5]:[rank45]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f50c3294897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default5]:[rank45]: frame #1: <unknown function> + 0x5b3a23e (0x7f50fcdb123e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f50fcdabc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]: frame #57: <unknown function> + 0x150582 (0x55ab03724582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5567c04c82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f50fcdabf82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f50fcdacfd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f50fcd61371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55a937f17007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank39]: frame #62: <unknown function> + 0x150582 (0x5567c04e1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f50fcd61371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f50fcd61371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f50fcd61371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank34]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55ab037098fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f50c456e189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank45]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f50c4575610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank45]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f50c4594978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank39]: frame #63: PyObject_Call + 0xbc (0x5567c04e1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #12: <unknown function> + 0x5adc309 (0x7f50fcd53309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #13: <unknown function> + 0x5ae6f10 (0x7f50fcd5df10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank33]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55a937f28c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f50fcd5dfa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #15: <unknown function> + 0x5124446 (0x7f50fc39b446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #16: <unknown function> + 0x1acf4b8 (0x7f50f8d464b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #17: <unknown function> + 0x5aee004 (0x7f50fcd65004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #18: <unknown function> + 0x5af36b5 (0x7f50fcd6a6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank45]: frame #19: <unknown function> + 0xd2631e (0x7f510f954[default7]:[rank39]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default2]:[rank34]: frame #59: <unknown function> + 0x150582 (0x55ab03724582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default5]:[rank45]: frame #20: <unknown function> + 0x47def4 (0x7f510f0abef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default5]:[rank45]: frame #21: <unknown function> + 0x1445a6 (0x5639dd83e5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #22: _PyObject_MakeTpCall + 0x26b (0x5639dd837a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #23: <unknown function> + 0x150866 (0x5639dd84a866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #54: <unknown function> + 0x211239 (0x55a937feb239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x5639dd833142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #25: _PyFunction_Vectorcall + 0x6c (0x5639dd83ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #26: PyObject_Call + 0xbc (0x5639dd84af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5639dd8312b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #60: PyObject_Call + 0xbc (0x55ab03724f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #28: _PyFunction_Vectorcall + 0x6c (0x5639dd83ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5639dd82f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #30: <unknown function> + 0x150582 (0x5639dd84a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5639dd82f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #55: PyObject_Call + 0x207 (0x55a937f2b067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #32: <unknown function> + 0x150582 (0x5639dd84a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5639dd82f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #34: <unknown function> + 0x150582 (0x5639dd84a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55ab0370b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #62: <unknown function> + 0x150582 (0x55ab03724582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5639dd82f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x5639dd836f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #37: _PyObject_Call_Prepend + 0x69 (0x5639dd848c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #38: <unknown function> + 0x211239 (0x5639dd90b239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: frame #63: PyObject_Call + 0xbc (0x55ab03724f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #39: _PyObject_MakeTpCall + 0x26b (0x5639dd837a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5639dd8333e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #41: _PyFunction_Vectorcall + 0x6c (0x5639dd83ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x5639dd82ec5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55a937f112b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #43: _PyFunction_Vectorcall + 0x6c (0x5639dd83ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5639dd82f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #45: <unknown function> + 0x150582 (0x5639dd84a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #46: PyObject_Call + 0xbc (0x5639dd84af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5639dd8312b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #48: <unknown function> + 0x150582 (0x5639dd84a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #49: PyObject_Call + 0xbc (0x5639dd84af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/pyt[default1]:[rank33]: frame #57: <unknown function> + 0x150582 (0x55a937f2a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank34]: . This may indicate a possible application crash on rank 0 or a network set up issue.
hon3.10)
[default5]:[rank45]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5639dd8312b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #51: _PyFunction_Vectorcall + 0x6c (0x5639dd83ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55a937f0f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x5639dd837007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #53: _PyObject_Call_Prepend + 0x69 (0x5639dd848c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #54: <unknown function> + 0x211239 (0x5639dd90b239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #55: PyObject_Call + 0x207 (0x5639dd84b067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5639dd8312b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #59: <unknown function> + 0x150582 (0x55a937f2a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #60: PyObject_Call + 0xbc (0x55a937f2af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #57: <unknown function> + 0x150582 (0x5639dd84a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5639dd82f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #59: <unknown function> + 0x150582 (0x5639dd84a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55a937f112b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #62: <unknown function> + 0x150582 (0x55a937f2a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank33]: frame #63: PyObject_Call + 0xbc (0x55a937f2af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #60: PyObject_Call + 0xbc (0x5639dd84af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5639dd8312b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #62: <unknown function> + 0x150582 (0x5639dd84a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: frame #63: PyObject_Call + 0xbc (0x5639dd84af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank45]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default1]:[rank33]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default4]:[rank44]: Traceback (most recent call last):
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default4]:[rank44]:     trainer.train(dataloader)
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default4]:[rank44]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default4]:[rank44]:     outputs = self.pipeline_engine.train_batch_iter(
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default4]:[rank44]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default4]:[rank44]:     output = model(**micro_batch)
[default4]:[rank44]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank44]:     return self._call_impl(*args, **kwargs)
[default4]:[rank44]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank44]:     return forward_call(*args, **kwargs)
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default4]:[rank44]:     sharded_logits = self.model(
[default4]:[rank44]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank44]:     return self._call_impl(*args, **kwargs)
[default4]:[rank44]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank44]:     return forward_call(*args, **kwargs)
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default4]:[rank44]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default4]:[rank44]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default4]:[rank44]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank44]:     return self._call_impl(*args, **kwargs)
[default4]:[rank44]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank44]:     return forward_call(*args, **kwargs)
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default4]:[rank44]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default4]:[rank44]:     pipeline_state.run_communication()
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default4]:[rank44]:     recv_activation_tensor = recv_activation()
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default4]:[rank44]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default4]:[rank44]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default4]:[rank44]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default4]:[rank44]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default4]:[rank44]:     dist.recv(
[default4]:[rank44]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default4]:[rank44]:     return func(*args, **kwargs)
[default4]:[rank44]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default4]:[rank44]:     pg.recv([tensor], group_src_rank, tag).wait()
[default4]:[rank44]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default4]:[rank44]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default4]:[rank44]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f9f496d0897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default4]:[rank44]: frame #1: <unknown function> + 0x5b3a23e (0x7f9f831ed23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f9f831e7c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f9f831e7f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f9f831e8fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9f8319d371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9f8319d371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9f8319d371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9f8319d371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f9f4a9aa189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank44]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f9f4a9b1610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank44]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f9f4a9d0978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank44]: frame #12: <unknown function> + 0x5adc309 (0x7f9f8318f309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]: frame #13: <unknown function> + 0x5ae6f10 (0x7f9f83199f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f9f83199fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]: frame #15: <unknown function> + 0x5124446 (0x7f9f827d7446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]: frame #16: <unknown function> + 0x1acf4b8 (0x7f9f7f1824b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]: frame #17: <unknown function> + 0x5aee004 (0x7f9f831a1004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]: frame #18: <unknown function> + 0x5af36b5 (0x7f9f831a66b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank44]: frame #19: <unknown function> + 0xd2631e (0x7f9f95d9031e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:[rank44]: frame #20: <unknown function> + 0x47def4 (0x7f9f954e7ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:[rank44]: frame #21: <unknown function> + 0x1445a6 (0x559e65c695a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #22: _PyObject_MakeTpCall + 0x26b (0x559e65c62a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #23: <unknown function> + 0x150866 (0x559e65c75866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x559e65c5e142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #25: _PyFunction_Vectorcall + 0x6c (0x559e65c69a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #26: PyObject_Call + 0xbc (0x559e65c75f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x559e65c5c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #28: _PyFunction_Vectorcall + 0x6c (0x559e65c69a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x559e65c5a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #30: <unknown function> + 0x150582 (0x559e65c75582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x559e65c5a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #32: <unknown function> + 0x150582 (0x559e65c75582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x559e65c5a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #34: <unknown function> + 0x150582 (0x559e65c75582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x559e65c5a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x559e65c61f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #37: _PyObject_Call_Prepend + 0x69 (0x559e65c73c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #38: <unknown function> + 0x211239 (0x559e65d36239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #39: _PyObject_MakeTpCall + 0x26b (0x559e65c62a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x559e65c5e3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #41: _PyFunction_Vectorcall + 0x6c (0x559e65c69a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x559e65c59c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #43: _PyFunction_Vectorcall + 0x6c (0x559e65c69a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x559e65c5a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #45: <unknown function> + 0x150582 (0x559e65c75582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #46: PyObject_Call + 0xbc (0x559e65c75f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x559e65c5c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #48: <unknown function> + 0x150582 (0x559e65c75582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #49: PyObject_Call + 0xbc (0x559e65c75f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x559e65c5c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #51: _PyFunction_Vectorcall + 0x6c (0x559e65c69a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x559e65c62007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #53: _PyObject_Call_Prepend + 0x69 (0x559e65c73c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #54: <unknown function> + 0x211239 (0x559e65d36239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #55: PyObject_Call + 0x207 (0x559e65c76067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x559e65c5c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #57: <unknown function> + 0x150582 (0x559e65c75582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x559e65c5a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #59: <unknown function> + 0x150582 (0x559e65c75582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #60: PyObject_Call + 0xbc (0x559e65c75f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x559e65c5c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #62: <unknown function> + 0x150582 (0x559e65c75582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: frame #63: PyObject_Call + 0xbc (0x559e65c75f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank44]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default4]:[rank36]: Traceback (most recent call last):
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default4]:[rank36]:     trainer.train(dataloader)
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default4]:[rank36]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default4]:[rank36]:     outputs = self.pipeline_engine.train_batch_iter(
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default4]:[rank36]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default4]:[rank36]:     output = model(**micro_batch)
[default0]:[rank56]: Traceback (most recent call last):
[default4]:[rank36]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank36]:     return self._call_impl(*args, **kwargs)
[default4]:[rank36]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank36]:     return forward_call(*args, **kwargs)
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default4]:[rank36]:     sharded_logits = self.model(
[default4]:[rank36]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank36]:     return self._call_impl(*args, **kwargs)
[default4]:[rank36]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank36]:     return forward_call(*args, **kwargs)
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default4]:[rank36]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default4]:[rank36]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default4]:[rank36]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank36]:     return self._call_impl(*args, **kwargs)
[default4]:[rank36]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank36]:     return forward_call(*args, **kwargs)
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default4]:[rank36]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default4]:[rank36]:     pipeline_state.run_communication()
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default4]:[rank36]:     recv_activation_tensor = recv_activation()
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default4]:[rank36]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default4]:[rank36]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default4]:[rank36]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default4]:[rank36]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default4]:[rank36]:   Fi[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default0]:[rank56]:     trainer.train(dataloader)
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default0]:[rank56]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default6]:[rank62]: Traceback (most recent call last):
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
le "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default4]:[rank36]:     dist.recv(
[default4]:[rank36]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default4]:[rank36]:     return func(*args, **kwargs)
[default4]:[rank36]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default4]:[rank36]:     pg.recv([tensor], group_src_rank, tag).wait()
[default6]:[rank62]:     trainer.train(dataloader)
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default0]:[rank56]:     outputs = self.pipeline_engine.train_batch_iter(
[default4]:[rank36]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default4]:[rank36]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default4]:[rank36]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f566a79f897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default4]:[rank36]: frame #1: <unknown function> + 0x5b3a23e (0x7f56a42bc23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f56a42b6c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f56a42b6f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default4]:[rank36]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f56a42b7fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f56a426c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f56a426c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f56a426c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default4]:[rank36]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f56a426c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f566ba79189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank36]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f566ba80610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank57]: Traceback (most recent call last):
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default4]:[rank36]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f566ba9f978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank36]: frame #12: <unknown function> + 0x5adc309 (0x7f56a425e309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #13: <unknown function> + 0x5ae6f10 (0x7f56a4268f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default6]:[rank62]:     outputs = self.pipeline_engine.train_batch_iter(
[default1]:[rank57]:     trainer.train(dataloader)
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default0]:[rank56]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default0]:[rank56]:     output = model(**micro_batch)
[default4]:[rank36]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f56a4268fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #15: <unknown function> + 0x5124446 (0x7f56a38a6446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #16: <unknown function> + 0x1acf4b8 (0x7f56a02514b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default4]:[rank36]: frame #17: <unknown function> + 0x5aee004 (0x7f56a4270004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #18: <unknown function> + 0x5af36b5 (0x7f56a42756b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank36]: frame #19: <unknown function> + 0xd2631e (0x7f56b6e5f31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default0]:[rank56]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank56]:     return self._call_impl(*args, **kwargs)
[default4]:[rank36]: frame #20: <unknown function> + 0x47def4 (0x7f56b65b6ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:[rank36]: frame #21: <unknown function> + 0x1445a6 (0x559a03f8f5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #22: _PyObject_MakeTpCall + 0x26b (0x559a03f88a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #23: <unknown function> + 0x150866 (0x559a03f9b866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x559a03f84142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #25: _PyFunction_Vectorcall + 0x6c (0x559a03f8fa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #26: PyObject_Call + 0xbc (0x559a03f9bf1c in /fsx/ferdina[default6]:[rank62]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default0]:[rank56]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
ndmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]:     outputs = self.pipeline_engine.train_batch_iter(
[default4]:[rank36]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x559a03f822b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #28: _PyFunction_Vectorcall + 0x6c (0x559a03f8fa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x559a03f808fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #30: <unknown function> + 0x150582 (0x559a03f9b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x559a03f808fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default0]:[rank56]:     return forward_call(*args, **kwargs)
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default4]:[rank36]: frame #32: <unknown function> + 0x150582 (0x559a03f9b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x559a03f808fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #34: <unknown function> + 0x150582 (0x559a03f9b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default4]:[rank36]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x559a03f808fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x559a03f87f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #37: _PyObject_Call_Prepend + 0x69 (0x559a03f99c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #38: <unknown function> + 0x211239 (0x559a0405c239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #39: _PyObject_MakeTpCall + 0x26b (0x559a03f88a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]:     sharded_logits = self.model(
[default6]:[rank62]:     output = model(**micro_batch)
[default4]:[rank36]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x559a03f843e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #41: _PyFunction_Vectorcall + 0x6c (0x559a03f8fa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x559a03f7fc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default4]:[rank36]: frame #43: _PyFunction_Vectorcall + 0x6c (0x559a03f8fa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x559a03f808fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #45: <unknown function> + 0x150582 (0x559a03f9b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #46: PyObject_Call + 0xbc (0x559a03f9bf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank36]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x559a03f822b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #48: <unknown function> + 0x150582 (0x559a03f9b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #49: PyObject_Call + 0xbc (0x559a03f9bf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x559a03f822b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default4]:[rank36]: frame #51: _PyFunction_Vectorcall + 0x6c (0x559a03f8fa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x559a03f88007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #53: _PyObject_Call_Prepend + 0x69 (0x559a03f99c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]:     return self._call_impl(*args, **kwargs)
[default1]:[rank57]:     output = model(**micro_batch)
[default4]:[rank36]: frame #54: <unknown function> + 0x211239 (0x559a0405c239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #55: PyObject_Call + 0x207 (0x559a03f9c067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x559a03f822b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #57: <unknown function> + 0x150582 (0x559a03f9b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank36]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x559a03f808fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #59: <unknown function> + 0x150582 (0x559a03f9b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #60: PyObject_Call + 0xbc (0x559a03f9bf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x559a03f822b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank36]: frame #62: <unknown function> + 0x150582 (0x559a03f9b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: frame #63: PyObject_Call + 0xbc (0x559a03f9bf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank36]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default0]:[rank56]:     return self._call_impl(*args, **kwargs)
[default1]:[rank57]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank56]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank57]:     return self._call_impl(*args, **kwargs)
[default6]:[rank62]:     return forward_call(*args, **kwargs)
[default1]:[rank57]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank57]:     return forward_call(*args, **kwargs)
[default0]:[rank56]:     return forward_call(*args, **kwargs)
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default1]:[rank57]:     sharded_logits = self.model(
[default1]:[rank57]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank57]:     return self._call_impl(*args, **kwargs)
[default6]:[rank62]:     sharded_logits = self.model(
[default6]:[rank62]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank56]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default1]:[rank57]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank56]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default0]:[rank56]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank62]:     return self._call_impl(*args, **kwargs)
[default1]:[rank57]:     return forward_call(*args, **kwargs)
[default0]:[rank56]:     return self._call_impl(*args, **kwargs)
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default6]:[rank62]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank56]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank62]:     return forward_call(*args, **kwargs)
[default0]:[rank56]:     return forward_call(*args, **kwargs)
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default1]:[rank57]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank62]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default6]:[rank62]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default0]:[rank56]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default6]:[rank62]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default1]:[rank57]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default6]:[rank62]:     return self._call_impl(*args, **kwargs)
[default6]:[rank62]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank57]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank56]:     pipeline_state.run_communication()
[default6]:[rank62]:     return forward_call(*args, **kwargs)
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default1]:[rank57]:     return self._call_impl(*args, **kwargs)
[default0]:[rank56]:     recv_activation_tensor = recv_activation()
[default6]:[rank62]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default1]:[rank57]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default1]:[rank57]:     return forward_call(*args, **kwargs)
[default0]:[rank56]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default1]:[rank57]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default6]:[rank62]:     pipeline_state.run_communication()
[default0]:[rank56]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default1]:[rank57]:     pipeline_state.run_communication()
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default1]:[rank57]:     recv_activation_tensor = recv_activation()
[default0]:[rank56]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default6]:[rank62]:     recv_activation_tensor = recv_activation()
[default0]:[rank56]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default1]:[rank57]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default0]:[rank56]:     dist.recv(
[default6]:[rank62]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default0]:[rank56]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default0]:[rank56]:     return func(*args, **kwargs)
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default1]:[rank57]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default0]:[rank56]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default0]:[rank56]:     pg.recv([tensor], group_src_rank, tag).wait()
[default1]:[rank57]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default0]:[rank56]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default1]:[rank57]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default6]:[rank62]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default0]:[rank56]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default1]:[rank57]:     dist.recv(
[default6]:[rank62]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default0]:[rank56]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f41ec4ef897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default6]:[rank62]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default0]:[rank56]: frame #1: <unknown function> + 0x5b3a23e (0x7f422600c23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default6]:[rank62]:     dist.recv(
[default1]:[rank57]:     return func(*args, **kwargs)
[default1]:[rank57]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default1]:[rank57]:     pg.recv([tensor], group_src_rank, tag).wait()
[default0]:[rank56]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f4226006c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default1]:[rank57]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default6]:[rank62]:     return func(*args, **kwargs)
[default6]:[rank62]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default0]:[rank56]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f4226006f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default6]:[rank62]:     pg.recv([tensor], group_src_rank, tag).wait()
[default1]:[rank57]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f456fddc897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default6]:[rank62]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default6]:[rank62]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default1]:[rank57]: frame #1: <unknown function> + 0x5b3a23e (0x7f45a98f923e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fcacc57c897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default0]:[rank56]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f4226007fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #1: <unknown function> + 0x5b3a23e (0x7fcb0609923e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f45a98f3c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank56]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4225fbc371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fcb06093c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank56]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4225fbc371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f45a98f3f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank56]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4225fbc371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fcb06093f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fcb06094fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f45a98f4fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fcb06049371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f45a98a9371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fcb06049371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fcb06049371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f45a98a9371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank56]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4225fbc371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fcb06049371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f45a98a9371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank56]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f41ed7c9189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank62]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fcacd856189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank56]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f41ed7d0610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank57]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f45a98a9371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fcacd85d610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank56]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f41ed7ef978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank62]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fcacd87c978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank57]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f45710b6189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank62]: frame #12: <unknown function> + 0x5adc309 (0x7fcb0603b309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f45710bd610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank62]: frame #13: <unknown function> + 0x5ae6f10 (0x7fcb06045f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f45710dc978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank57]: frame #12: <unknown function> + 0x5adc309 (0x7f45a989b309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fcb06045fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank56]: frame #12: <unknown function> + 0x5adc309 (0x7f4225fae309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank56]: frame #13: <unknown function> + 0x5ae6f10 (0x7f4225fb8f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #15: <unknown function> + 0x5124446 (0x7fcb05683446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #16: <unknown function> + 0x1acf4b8 (0x7fcb0202e4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank56]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f4225fb8fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #17: <unknown function> + 0x5aee004 (0x7fcb0604d004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]: frame #13: <unknown function> + 0x5ae6f10 (0x7f45a98a5f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank56]: frame #15: <unknown function> + 0x5124446 (0x7f42255f6446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #18: <unknown function> + 0x5af36b5 (0x7fcb060526b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #19: <unknown function> + 0xd2631e (0x7fcb18c3c31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:[rank62]: frame #20: <unknown function> + 0x47def4 (0x7fcb18393ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default0]:[rank56]: frame #16: <unknown function> + 0x1acf4b8 (0x7f4221fa14b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f45a98a5fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #21: <unknown function> + 0x1445a6 (0x55a18ee4d5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #17: <unknown function> + 0x5aee004 (0x7f4225fc0004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]: frame #15: <unknown function> + 0x5124446 (0x7f45a8ee3446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank56]: frame #18: <unknown function> + 0x5af36b5 (0x7f4225fc56b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank57]: frame #16: <unknown function> + 0x1acf4b8 (0x7f45a588e4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank56]: frame #19: <unknown function> + 0xd2631e (0x7f4238baf31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default0]:[rank56]: frame #20: <unknown function> + 0x47def4 (0x7f4238306ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:[rank62]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55a18ee46a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #23: <unknown function> + 0x150866 (0x55a18ee59866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #21: <unknown function> + 0x1445a6 (0x557695fb45a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #17: <unknown function> + 0x5aee004 (0x7f45a98ad004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank56]: frame #22: _PyObject_MakeTpCall + 0x26b (0x557695fada6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #23: <unknown function> + 0x150866 (0x557695fc0866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #18: <unknown function> + 0x5af36b5 (0x7f45a98b26b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank62]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55a18ee42142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #19: <unknown function> + 0xd2631e (0x7f45bc49c31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:[rank62]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55a18ee4da2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x557695fa9142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #20: <unknown function> + 0x47def4 (0x7f45bbbf3ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:[rank62]: frame #26: PyObject_Call + 0xbc (0x55a18ee59f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #25: _PyFunction_Vectorcall + 0x6c (0x557695fb4a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #21: <unknown function> + 0x1445a6 (0x55cea371f5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #26: PyObject_Call + 0xbc (0x557695fc0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55a18ee402b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55cea3718a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x557695fa72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55a18ee4da2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #23: <unknown function> + 0x150866 (0x55cea372b866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #28: _PyFunction_Vectorcall + 0x6c (0x557695fb4a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55cea3714142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55cea371fa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55a18ee3e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x557695fa58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #30: <unknown function> + 0x150582 (0x557695fc0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #26: PyObject_Call + 0xbc (0x55cea372bf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #30: <unknown function> + 0x150582 (0x55a18ee59582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55a18ee3e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x557695fa58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #32: <unknown function> + 0x150582 (0x55a18ee59582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55cea37122b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55a18ee3e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #32: <unknown function> + 0x150582 (0x557695fc0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55cea371fa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #34: <unknown function> + 0x150582 (0x55a18ee59582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x557695fa58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55cea37108fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #34: <unknown function> + 0x150582 (0x557695fc0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x557695fa58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #30: <unknown function> + 0x150582 (0x55cea372b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55a18ee3e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x557695facf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #37: _PyObject_Call_Prepend + 0x69 (0x557695fbec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55a18ee45f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55cea37108fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #38: <unknown function> + 0x211239 (0x557696081239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #32: <unknown function> + 0x150582 (0x55cea372b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55a18ee57c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #39: _PyObject_MakeTpCall + 0x26b (0x557695fada6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x557695fa93e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55cea37108fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #34: <unknown function> + 0x150582 (0x55cea372b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55cea37108fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #38: <unknown function> + 0x211239 (0x55a18ef1a239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55a18ee46a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55cea3717f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55a18ee423e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55cea3729c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #41: _PyFunction_Vectorcall + 0x6c (0x557695fb4a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55a18ee4da2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #38: <unknown function> + 0x211239 (0x55cea37ec239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55a18ee3dc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x557695fa4c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55cea3718a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55cea37143e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #43: _PyFunction_Vectorcall + 0x6c (0x557695fb4a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x557695fa58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55a18ee4da2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55cea371fa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55a18ee3e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #45: <unknown function> + 0x150582 (0x557695fc0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55cea370fc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #45: <unknown function> + 0x150582 (0x55a18ee59582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #46: PyObject_Call + 0xbc (0x557695fc0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #46: PyObject_Call + 0xbc (0x55a18ee59f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55cea371fa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x557695fa72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55a18ee402b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55cea37108fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #48: <unknown function> + 0x150582 (0x55a18ee59582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #48: <unknown function> + 0x150582 (0x557695fc0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #45: <unknown function> + 0x150582 (0x55cea372b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #49: PyObject_Call + 0xbc (0x55a18ee59f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #46: PyObject_Call + 0xbc (0x55cea372bf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55a18ee402b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55cea37122b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55a18ee4da2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #48: <unknown function> + 0x150582 (0x55cea372b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #49: PyObject_Call + 0xbc (0x557695fc0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55a18ee46007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #49: PyObject_Call + 0xbc (0x55cea372bf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x557695fa72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55a18ee57c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55cea37122b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55cea371fa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #54: <unknown function> + 0x211239 (0x55a18ef1a239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #51: _PyFunction_Vectorcall + 0x6c (0x557695fb4a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #55: PyObject_Call + 0x207 (0x55a18ee5a067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55a18ee402b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x557695fad007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #57: <unknown function> + 0x150582 (0x55a18ee59582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55cea3718007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55cea3729c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55a18ee3e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #59: <unknown function> + 0x150582 (0x55a18ee59582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #53: _PyObject_Call_Prepend + 0x69 (0x557695fbec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #60: PyObject_Call + 0xbc (0x55a18ee59f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55a18ee402b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #54: <unknown function> + 0x211239 (0x55cea37ec239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #54: <unknown function> + 0x211239 (0x557696081239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #55: PyObject_Call + 0x207 (0x55cea372c067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #62: <unknown function> + 0x150582 (0x55a18ee59582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #55: PyObject_Call + 0x207 (0x557695fc1067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55cea37122b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: frame #63: PyObject_Call + 0xbc (0x55a18ee59f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x557695fa72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #57: <unknown function> + 0x150582 (0x55cea372b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank62]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default0]:[rank56]: frame #57: <unknown function> + 0x150582 (0x557695fc0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55cea37108fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x557695fa58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #59: <unknown function> + 0x150582 (0x55cea372b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #60: PyObject_Call + 0xbc (0x55cea372bf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55cea37122b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #59: <unknown function> + 0x150582 (0x557695fc0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #60: PyObject_Call + 0xbc (0x557695fc0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #62: <unknown function> + 0x150582 (0x55cea372b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: frame #63: PyObject_Call + 0xbc (0x55cea372bf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank57]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default0]:[rank56]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x557695fa72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #62: <unknown function> + 0x150582 (0x557695fc0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: frame #63: PyObject_Call + 0xbc (0x557695fc0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank56]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default4]:[rank60]: Traceback (most recent call last):
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default4]:[rank60]:     trainer.train(dataloader)
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default4]:[rank60]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default4]:[rank60]:     outputs = self.pipeline_engine.train_batch_iter(
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default4]:[rank60]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default4]:[rank60]:     output = model(**micro_batch)
[default4]:[rank60]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank60]:     return self._call_impl(*args, **kwargs)
[default4]:[rank60]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank60]:     return forward_call(*args, **kwargs)
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default4]:[rank60]:     sharded_logits = self.model(
[default4]:[rank60]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank60]:     return self._call_impl(*args, **kwargs)
[default4]:[rank60]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank60]:     return forward_call(*args, **kwargs)
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default4]:[rank60]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default4]:[rank60]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default4]:[rank60]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank60]:     return self._call_impl(*args, **kwargs)
[default4]:[rank60]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank60]:     return forward_call(*args, **kwargs)
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default4]:[rank60]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default4]:[rank60]:     pipeline_state.run_communication()
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default4]:[rank60]:     recv_activation_tensor = recv_activation()
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default4]:[rank60]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default4]:[rank60]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default4]:[rank60]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default4]:[rank60]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default4]:[rank60]:     dist.recv(
[default4]:[rank60]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default4]:[rank60]:     return func(*args, **kwargs)
[default4]:[rank60]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default4]:[rank60]:     pg.recv([tensor], group_src_rank, tag).wait()
[default4]:[rank60]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default4]:[rank60]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default4]:[rank60]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fb3a5074897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default4]:[rank60]: frame #1: <unknown function> + 0x5b3a23e (0x7fb3deb9123e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank60]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fb3deb8bc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank60]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fb3deb8bf82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank60]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fb3deb8cfd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank60]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb3deb41371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank60]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb3deb41371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank60]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb3deb41371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank60]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb3deb41371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank60]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fb3a634e189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank60]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fb3a6355610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank60]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fb3a6374978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank60]: frame #12: <unknown function> + 0x5adc309 (0x7fb3deb33309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank60]: frame #13: <unknown function> + 0x5ae6f10 (0x7fb3deb3df10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank60]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fb3deb3dfa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank60]: frame #15: <unknown function> + 0x5124446 (0x7fb3de17b446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank60]: frame #16: <unknown function> + 0x1acf4b8 (0x7fb3dab264b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank60]: frame #17: <unknown function> + 0x5aee004 (0x7fb3deb45004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank60]: frame #18: <unknown function> + 0x5af36b5 (0x7fb3deb4a6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank60]: frame #19: <unknown function> + 0xd2631e (0x7fb3f173431e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:[rank60]: frame #20: <unknown function> + 0x47def4 (0x7fb3f0e8bef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:[rank60]: frame #21: <unknown function> + 0x1445a6 (0x557a9691c5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #22: _PyObject_MakeTpCall + 0x26b (0x557a96915a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #23: <unknown function> + 0x150866 (0x557a96928866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x557a96911142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #25: _PyFunction_Vectorcall + 0x6c (0x557a9691ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #26: PyObject_Call + 0xbc (0x557a96928f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x557a9690f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #28: _PyFunction_Vectorcall + 0x6c (0x557a9691ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x557a9690d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #30: <unknown function> + 0x150582 (0x557a96928582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x557a9690d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #32: <unknown function> + 0x150582 (0x557a96928582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x557a9690d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #34: <unknown function> + 0x150582 (0x557a96928582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x557a9690d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x557a96914f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #37: _PyObject_Call_Prepend + 0x69 (0x557a96926c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #38: <unknown function> + 0x211239 (0x557a969e9239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #39: _PyObject_MakeTpCall + 0x26b (0x557a96915a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x557a969113e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #41: _PyFunction_Vectorcall + 0x6c (0x557a9691ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x557a9690cc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #43: _PyFunction_Vectorcall + 0x6c (0x557a9691ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x557a9690d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #45: <unknown function> + 0x150582 (0x557a96928582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #46: PyObject_Call + 0xbc (0x557a96928f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x557a9690f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #48: <unknown function> + 0x150582 (0x557a96928582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #49: PyObject_Call + 0xbc (0x557a96928f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x557a9690f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #51: _PyFunction_Vectorcall + 0x6c (0x557a9691ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x557a96915007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #53: _PyObject_Call_Prepend + 0x69 (0x557a96926c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #54: <unknown function> + 0x211239 (0x557a969e9239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #55: PyObject_Call + 0x207 (0x557a96929067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x557a9690f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #57: <unknown function> + 0x150582 (0x557a96928582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x557a9690d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #59: <unknown function> + 0x150582 (0x557a96928582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #60: PyObject_Call + 0xbc (0x557a96928f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x557a9690f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #62: <unknown function> + 0x150582 (0x557a96928582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: frame #63: PyObject_Call + 0xbc (0x557a96928f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank60]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default6]:[rank38]: Traceback (most recent call last):
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default6]:[rank38]:     trainer.train(dataloader)
[default0]:[rank32]: Traceback (most recent call last):
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default0]:[rank32]:     trainer.train(dataloader)
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default0]:[rank32]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default6]:[rank38]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default6]:[rank38]:     outputs = self.pipeline_engine.train_batch_iter(
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default6]:[rank38]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default0]:[rank32]:     outputs = self.pipeline_engine.train_batch_iter(
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default6]:[rank38]:     output = model(**micro_batch)
[default6]:[rank38]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank38]:     return self._call_impl(*args, **kwargs)
[default6]:[rank38]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank38]:     return forward_call(*args, **kwargs)
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default0]:[rank32]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default0]:[rank32]:     output = model(**micro_batch)
[default0]:[rank32]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank32]:     return self._call_impl(*args, **kwargs)
[default0]:[rank32]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank32]:     return forward_call(*args, **kwargs)
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default0]:[rank32]:     sharded_logits = self.model(
[default0]:[rank32]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank32]:     return self._call_impl(*args, **kwargs)
[default0]:[rank32]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank32]:     return forward_call(*args, **kwargs)
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default0]:[rank32]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank38]:     sharded_logits = self.model(
[default6]:[rank38]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank38]:     return self._call_impl(*args, **kwargs)
[default6]:[rank38]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank38]:     return forward_call(*args, **kwargs)
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default6]:[rank38]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default6]:[rank38]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default6]:[rank38]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank38]:     return self._call_impl(*args, **kwargs)
[default6]:[rank38]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank32]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default6]:[rank38]:     return forward_call(*args, **kwargs)
[default0]:[rank32]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank32]:     return self._call_impl(*args, **kwargs)
[default0]:[rank32]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default0]:[rank32]:     return forward_call(*args, **kwargs)
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default0]:[rank32]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default6]:[rank38]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default6]:[rank38]:     pipeline_state.run_communication()
[default0]:[rank32]:     pipeline_state.run_communication()
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default0]:[rank32]:     recv_activation_tensor = recv_activation()
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default0]:[rank32]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default6]:[rank38]:     recv_activation_tensor = recv_activation()
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default6]:[rank38]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default6]:[rank38]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default6]:[rank38]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default6]:[rank38]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default0]:[rank32]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default6]:[rank38]:     dist.recv(
[default6]:[rank38]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default6]:[rank38]:     return func(*args, **kwargs)
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default6]:[rank38]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default6]:[rank38]:     pg.recv([tensor], group_src_rank, tag).wait()
[default0]:[rank32]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default0]:[rank32]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default6]:[rank38]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default6]:[rank38]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default6]:[rank38]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f3bc7aa0897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default6]:[rank38]: frame #1: <unknown function> + 0x5b3a23e (0x7f3c015bd23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]:     dist.recv(
[default0]:[rank32]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default6]:[rank38]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f3c015b7c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f3c015b7f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f3c015b8fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3c0156d371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3c0156d371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3c0156d371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]:     return func(*args, **kwargs)
[default6]:[rank38]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3c0156d371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default0]:[rank32]:     pg.recv([tensor], group_src_rank, tag).wait()
[default6]:[rank38]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f3bc8d7a189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank38]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f3bc8d81610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank38]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f3bc8da0978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank32]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default6]:[rank38]: frame #12: <unknown function> + 0x5adc309 (0x7f3c0155f309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #13: <unknown function> + 0x5ae6f10 (0x7f3c01569f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default0]:[rank32]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f9f35a17897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default0]:[rank32]: frame #1: <unknown function> + 0x5b3a23e (0x7f9f6f53423e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f9f6f52ec87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f9f6f52ef82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f3c01569fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f9f6f52ffd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9f6f4e4371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9f6f4e4371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9f6f4e4371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9f6f4e4371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f9f36cf1189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank32]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f9f36cf8610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank32]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f9f36d17978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank32]: frame #12: <unknown function> + 0x5adc309 (0x7f9f6f4d6309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #13: <unknown function> + 0x5ae6f10 (0x7f9f6f4e0f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #15: <unknown function> + 0x5124446 (0x7f3c00ba7446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f9f6f4e0fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #16: <unknown function> + 0x1acf4b8 (0x7f3bfd5524b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #17: <unknown function> + 0x5aee004 (0x7f3c01571004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #18: <unknown function> + 0x5af36b5 (0x7f3c015766b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #15: <unknown function> + 0x5124446 (0x7f9f6eb1e446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #19: <unknown function> + 0xd2631e (0x7f3c1416031e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:[rank38]: frame #20: <unknown function> + 0x47def4 (0x7f3c138b7ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:[rank38]: frame #21: <unknown function> + 0x1445a6 (0x55c1ac0545a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #16: <unknown function> + 0x1acf4b8 (0x7f9f6b4c94b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55c1ac04da6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #17: <unknown function> + 0x5aee004 (0x7f9f6f4e8004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #18: <unknown function> + 0x5af36b5 (0x7f9f6f4ed6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #19: <unknown function> + 0xd2631e (0x7f9f820d731e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default0]:[rank32]: frame #20: <unknown function> + 0x47def4 (0x7f9f8182eef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:[rank38]: frame #23: <unknown function> + 0x150866 (0x55c1ac060866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #21: <unknown function> + 0x1445a6 (0x56374abaf5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55c1ac049142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55c1ac054a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #22: _PyObject_MakeTpCall + 0x26b (0x56374aba8a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #26: PyObject_Call + 0xbc (0x55c1ac060f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #23: <unknown function> + 0x150866 (0x56374abbb866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55c1ac0472b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55c1ac054a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x56374aba4142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55c1ac0458fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #30: <unknown function> + 0x150582 (0x55c1ac060582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55c1ac0458fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #32: <unknown function> + 0x150582 (0x55c1ac060582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55c1ac0458fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #25: _PyFunction_Vectorcall + 0x6c (0x56374abafa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #34: <unknown function> + 0x150582 (0x55c1ac060582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #26: PyObject_Call + 0xbc (0x56374abbbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: Traceback (most recent call last):
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default3]:[rank51]:     trainer.train(dataloader)
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default3]:[rank51]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default3]:[rank51]:     outputs = self.pipeline_engine.train_batch_iter(
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default3]:[rank51]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanot[default0]:[rank32]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x56374aba22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #28: _PyFunction_Vectorcall + 0x6c (0x56374abafa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55c1ac0458fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x56374aba08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #30: <unknown function> + 0x150582 (0x56374abbb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
ron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default3]:[rank51]:     output = model(**micro_batch)
[default3]:[rank51]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank32]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x56374aba08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #32: <unknown function> + 0x150582 (0x56374abbb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x56374aba08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]:     return self._call_impl(*args, **kwargs)
[default3]:[rank51]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank51]:     return forward_call(*args, **kwargs)
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default3]:[rank51]:     sharded_logits = self.model(
[default3]:[rank51]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank51]:     return self._call_impl(*args, **kwargs)
[default3]:[rank51]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank51]:     return forward_call(*args, **kwargs)
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotro[default0]:[rank32]: frame #34: <unknown function> + 0x150582 (0x56374abbb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55c1ac04cf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55c1ac05ec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
n/src/nanotron/models/llama.py", line 764, in forward
[default3]:[rank51]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default3]:[rank51]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default3]:[rank51]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank51]:     return self._call_impl(*args, **kwargs)
[default3]:[rank51]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank32]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x56374aba08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x56374aba7f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]:     return forward_call(*args, **kwargs)
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default3]:[rank51]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default3]:[rank51]:     pipeline_state.run_communication()
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default3]:[rank51]:     recv_activation_tensor = recv_activation()
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default3]:[rank51]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default6]:[rank38]: frame #38: <unknown function> + 0x211239 (0x55c1ac121239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default3]:[rank51]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default3]:[rank51]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default3]:[rank51]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default3]:[rank51]:     dist.recv(
[default3]:[rank51]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default3]:[rank51]:     return func(*args, **kwargs)
[default3]:[rank51]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site[default6]:[rank38]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55c1ac04da6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55c1ac0493e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default3]:[rank51]:     pg.recv([tensor], group_src_rank, tag).wait()
[default0]:[rank32]: frame #37: _PyObject_Call_Prepend + 0x69 (0x56374abb9c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default3]:[rank51]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default3]:[rank51]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7ff00df8a897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default6]:[rank38]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55c1ac054a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #1: <unknown function> + 0x5b3a23e (0x7ff047aa723e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7ff047aa1c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #38: <unknown function> + 0x211239 (0x56374ac7c239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7ff047aa1f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7ff047aa2fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55c1ac044c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff047a57371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff047a57371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff047a57371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff047a57371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55c1ac054a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #39: _PyObject_MakeTpCall + 0x26b (0x56374aba8a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x56374aba43e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7ff00f264189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:[rank51]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7ff00f26b610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank32]: frame #41: _PyFunction_Vectorcall + 0x6c (0x56374abafa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x56374ab9fc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7ff00f28a978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:[rank51]: frame #12: <unknown function> + 0x5adc309 (0x7ff047a49309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55c1ac0458fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #13: <unknown function> + 0x5ae6f10 (0x7ff047a53f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #43: _PyFunction_Vectorcall + 0x6c (0x56374abafa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #14: <unknown function> + 0x5ae6fa5 (0x7ff047a53fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]: frame #15: <unknown function> + 0x5124446 (0x7ff047091446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank38]: frame #45: <unknown function> + 0x150582 (0x55c1ac060582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #16: <unknown function> + 0x1acf4b8 (0x7ff043a3c4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]: frame #17: <unknown function> + 0x5aee004 (0x7ff047a5b004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank32]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x56374aba08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #45: <unknown function> + 0x150582 (0x56374abbb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #18: <unknown function> + 0x5af36b5 (0x7ff047a606b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank51]: frame #19: <unknown function> + 0xd2631e (0x7ff05a64a31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default3]:[rank51]: frame #20: <unknown function> + 0x47def4 (0x7ff059da1ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default3]:[rank51]: frame #21: <unknown function> + 0x1445a6 (0x557dafb055a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #22: _PyObject_MakeTpCall + 0x26b (0x557dafafea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #23: <unknown function> + 0x150866 (0x557dafb11866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[def[default0]:[rank32]: frame #46: PyObject_Call + 0xbc (0x56374abbbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x56374aba22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
ault3]:[rank51]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x557dafafa142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #25: _PyFunction_Vectorcall + 0x6c (0x557dafb05a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #48: <unknown function> + 0x150582 (0x56374abbb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #26: PyObject_Call + 0xbc (0x557dafb11f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x557dafaf82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #28: _PyFunction_Vectorcall + 0x6c (0x557dafb05a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x557dafaf68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #30: <unknown function> + 0x150582 (0x557dafb11582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x557dafaf68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #46: PyObject_Call + 0xbc (0x55c1ac060f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #32: <unknown function> + 0x150582 (0x557dafb11582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x557dafaf68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #34: <unknown function> + 0x150582 (0x557dafb11582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #49: PyObject_Call + 0xbc (0x56374abbbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x56374aba22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x557dafaf68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x557dafafdf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #37: _PyObject_Call_Prepend + 0x69 (0x557dafb0fc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #38: <unknown function> + 0x211239 (0x557dafbd2239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #51: _PyFunction_Vectorcall + 0x6c (0x56374abafa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x56374aba8007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #39: _PyObject_MakeTpCall + 0x26b (0x557dafafea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x557dafafa3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #41: _PyFunction_Vectorcall + 0x6c (0x557dafb05a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #53: _PyObject_Call_Prepend + 0x69 (0x56374abb9c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #54: <unknown function> + 0x211239 (0x56374ac7c239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x557dafaf5c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #43: _PyFunction_Vectorcall + 0x6c (0x557dafb05a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #55: PyObject_Call + 0x207 (0x56374abbc067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x557dafaf68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #45: <unknown function> + 0x150582 (0x557dafb11582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #46: PyObject_Call + 0xbc (0x557dafb11f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55c1ac0472b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x557dafaf82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #48: <unknown function> + 0x150582 (0x557dafb11582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #49: PyObject_Call + 0xbc (0x557dafb11f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x557dafaf82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #51: _PyFunction_Vectorcall + 0x6c (0x557dafb05a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x557dafafe007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #53: _PyObject_Call_Prepend + 0x69 (0x557dafb0fc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench[default6]:[rank38]: frame #48: <unknown function> + 0x150582 (0x55c1ac060582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x56374aba22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
-cluster/bin/python3.10)
[default3]:[rank51]: frame #54: <unknown function> + 0x211239 (0x557dafbd2239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #55: PyObject_Call + 0x207 (0x557dafb12067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x557dafaf82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #57: <unknown function> + 0x150582 (0x557dafb11582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x557dafaf68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #59: <unknown function> + 0x150582 (0x557dafb11582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #49: PyObject_Call + 0xbc (0x55c1ac060f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #60: PyObject_Call + 0xbc (0x557dafb11f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x557dafaf82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #62: <unknown function> + 0x150582 (0x557dafb11582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #57: <unknown function> + 0x150582 (0x56374abbb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: frame #63: PyObject_Call + 0xbc (0x557dafb11f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank51]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default6]:[rank38]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55c1ac0472b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55c1ac054a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55c1ac04d007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55c1ac05ec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x56374aba08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #54: <unknown function> + 0x211239 (0x55c1ac121239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #55: PyObject_Call + 0x207 (0x55c1ac061067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55c1ac0472b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #57: <unknown function> + 0x150582 (0x55c1ac060582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55c1ac0458fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #59: <unknown function> + 0x150582 (0x55c1ac060582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #60: PyObject_Call + 0xbc (0x55c1ac060f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #59: <unknown function> + 0x150582 (0x56374abbb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55c1ac0472b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #62: <unknown function> + 0x150582 (0x55c1ac060582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: frame #63: PyObject_Call + 0xbc (0x55c1ac060f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank38]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default0]:[rank32]: frame #60: PyObject_Call + 0xbc (0x56374abbbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x56374aba22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #62: <unknown function> + 0x150582 (0x56374abbb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: frame #63: PyObject_Call + 0xbc (0x56374abbbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank32]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default5]:[rank37]: Traceback (most recent call last):
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default5]:[rank37]:     trainer.train(dataloader)
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default5]:[rank37]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default5]:[rank37]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default5]:[rank37]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default5]:[rank37]:     output = model(**micro_batch)
[default5]:[rank37]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank37]:     return self._call_impl(*args, **kwargs)
[default5]:[rank37]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank37]:     return forward_call(*args, **kwargs)
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default5]:[rank37]:     sharded_logits = self.model(
[default5]:[rank37]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank37]:     return self._call_impl(*args, **kwargs)
[default5]:[rank37]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank37]:     return forward_call(*args, **kwargs)
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default5]:[rank37]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default5]:[rank37]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default5]:[rank37]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank37]:     return self._call_impl(*args, **kwargs)
[default5]:[rank37]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank37]:     return forward_call(*args, **kwargs)
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default5]:[rank37]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default5]:[rank37]:     pipeline_state.run_communication()
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default5]:[rank37]:     recv_activation_tensor = recv_activation()
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default5]:[rank37]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default5]:[rank37]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default5]:[rank37]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default5]:[rank37]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default5]:[rank37]:     dist.recv(
[default5]:[rank37]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default5]:[rank37]:     return func(*args, **kwargs)
[default5]:[rank37]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default5]:[rank37]:     pg.recv([tensor], group_src_rank, tag).wait()
[default5]:[rank37]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default5]:[rank37]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default5]:[rank37]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f342de24897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default5]:[rank37]: frame #1: <unknown function> + 0x5b3a23e (0x7f346794123e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f346793bc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f346793bf82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f346793cfd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f34678f1371 in /fsx/ferdinandmom[default7]:[rank55]: Traceback (most recent call last):
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default7]:[rank55]:     trainer.train(dataloader)
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default7]:[rank55]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default7]:[rank55]:     outputs = self.pipeline_engine.train_batch_iter(
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default7]:[rank55]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanot/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f34678f1371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f34678f1371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f34678f1371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f342f0fe189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank37]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f342f105610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank37]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f342f124978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank37]: frame #12: <unknown function> + 0x5adc309 (0x7f34678e3309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #13: <unknown function> + 0x5ae6f10 (0x7f34678edf10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
ron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default7]:[rank55]:     output = model(**micro_batch)
[default5]:[rank37]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f34678edfa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank55]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank55]:     return self._call_impl(*args, **kwargs)
[default7]:[rank55]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank55]:     return forward_call(*args, **kwargs)
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default7]:[rank55]:     sharded_logits = self.model(
[default7]:[rank55]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank55]:     return self._call_impl(*args, **kwargs)
[default7]:[rank55]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.p[default5]:[rank37]: frame #15: <unknown function> + 0x5124446 (0x7f3466f2b446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #16: <unknown function> + 0x1acf4b8 (0x7f34638d64b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #17: <unknown function> + 0x5aee004 (0x7f34678f5004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank37]: frame #18: <unknown function> + 0x5af36b5 (0x7f34678fa6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
y", line 1541, in _call_impl
[default7]:[rank55]:     return forward_call(*args, **kwargs)
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default7]:[rank55]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default7]:[rank55]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default7]:[rank55]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank55]:     return self._call_impl(*args, **kwargs)
[default7]:[rank55]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank37]: frame #19: <unknown function> + 0xd2631e (0x7f347a4e431e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default5]:[rank37]: frame #20: <unknown function> + 0x47def4 (0x7f3479c3bef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default5]:[rank37]: frame #21: <unknown function> + 0x1445a6 (0x560a7fbad5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #22: _PyObject_MakeTpCall + 0x26b (0x560a7fba6a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]:     return forward_call(*args, **kwargs)
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default7]:[rank55]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default7]:[rank55]:     pipeline_state.run_communication()
[default5]:[rank37]: frame #23: <unknown function> + 0x150866 (0x560a7fbb9866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x560a7fba2142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #25: _PyFunction_Vectorcall + 0x6c (0x560a7fbada2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default7]:[rank55]:     recv_activation_tensor = recv_activation()
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default5]:[rank37]: frame #26: PyObject_Call + 0xbc (0x560a7fbb9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x560a7fba02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #28: _PyFunction_Vectorcall + 0x6c (0x560a7fbada2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x560a7fb9e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default7]:[rank55]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default5]:[rank37]: frame #30: <unknown function> + 0x150582 (0x560a7fbb9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x560a7fb9e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #32: <unknown function> + 0x150582 (0x560a7fbb9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x560a7fb9e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default7]:[rank55]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default7]:[rank55]:     dist.recv(
[default5]:[rank37]: frame #34: <unknown function> + 0x150582 (0x560a7fbb9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x560a7fb9e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x560a7fba5f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #37: _PyObject_Call_Prepend + 0x69 (0x560a7fbb7c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default7]:[rank55]:     return func(*args, **kwargs)
[default7]:[rank55]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default3]:[rank59]: Traceback (most recent call last):
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default5]:[rank37]: frame #38: <unknown function> + 0x211239 (0x560a7fc7a239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #39: _PyObject_MakeTpCall + 0x26b (0x560a7fba6a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x560a7fba23e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #41: _PyFunction_Vectorcall + 0x6c (0x560a7fbada2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: Traceback (most recent call last):
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default4]:[rank52]:     trainer.train(dataloader)
[default5]:[rank37]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x560a7fb9dc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #43: _PyFunction_Vectorcall + 0x6c (0x560a7fbada2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x560a7fb9e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #45: <unknown function> + 0x150582 (0x560a7fbb9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]:     pg.recv([tensor], group_src_rank, tag).wait()
[default5]:[rank37]: frame #46: PyObject_Call + 0xbc (0x560a7fbb9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x560a7fba02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #48: <unknown function> + 0x150582 (0x560a7fbb9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #49: PyObject_Call + 0xbc (0x560a7fbb9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x560a7fba02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #51: _PyFunction_Vectorcall + 0x6c (0x560a7fbada2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x560a7fba6007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
bin/python3.10)
[default5]:[rank37]: frame #53: _PyObject_Call_Prepend + 0x69 (0x560a7fbb7c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #54: <unknown function> + 0x211239 (0x560a7fc7a239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default5]:[rank37]: frame #55: PyObject_Call + 0x207 (0x560a7fbba067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x560a7fba02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #57: <unknown function> + 0x150582 (0x560a7fbb9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x560a7fb9e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #59: <unknown function> + 0x150582 (0x560a7fbb9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default4]:[rank52]:     outputs = self.pipeline_engine.train_batch_iter(
[default7]:[rank63]: Traceback (most recent call last):
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default7]:[rank63]:     trainer.train(dataloader)
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default5]:[rank37]: frame #60: PyObject_Call + 0xbc (0x560a7fbb9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x560a7fba02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: frame #62: <unknown function> + 0x150582 (0x560a7fbb9582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default3]:[rank59]:     trainer.train(dataloader)
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default5]:[rank37]: frame #63: PyObject_Call + 0xbc (0x560a7fbb9f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank37]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default3]:[rank59]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default7]:[rank63]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default7]:[rank55]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default2]:[rank58]: Traceback (most recent call last):
[default4]:[rank52]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default7]:[rank63]:     outputs = self.pipeline_engine.train_batch_iter(
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default5]:[rank53]: Traceback (most recent call last):
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default4]:[rank52]:     output = model(**micro_batch)
[default3]:[rank59]:     outputs = self.pipeline_engine.train_batch_iter(
[default4]:[rank52]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank63]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default5]:[rank53]:     trainer.train(dataloader)
[default2]:[rank58]:     trainer.train(dataloader)
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default5]:[rank53]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default4]:[rank52]:     return self._call_impl(*args, **kwargs)
[default4]:[rank52]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank52]:     return forward_call(*args, **kwargs)
[default7]:[rank55]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f18022c9897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default7]:[rank55]: frame #1: <unknown function> + 0x5b3a23e (0x7f183bde623e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default2]:[rank58]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default5]:[rank53]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank61]: Traceback (most recent call last):
[default7]:[rank55]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f183bde0c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default3]:[rank59]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default7]:[rank55]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f183bde0f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank55]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f183bde1fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default7]:[rank55]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f183bd96371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]:     output = model(**micro_batch)
[default7]:[rank63]:     output = model(**micro_batch)
[default0]:[rank48]: Traceback (most recent call last):
[default5]:[rank61]:     trainer.train(dataloader)
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default0]:[rank48]:     trainer.train(dataloader)
[default7]:[rank63]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default3]:[rank59]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank59]:     return self._call_impl(*args, **kwargs)
[default7]:[rank63]:     return self._call_impl(*args, **kwargs)
[default7]:[rank63]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank59]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank63]:     return forward_call(*args, **kwargs)
[default3]:[rank59]:     return forward_call(*args, **kwargs)
[default7]:[rank55]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f183bd96371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank55]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f183bd96371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default2]:[rank58]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank53]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default5]:[rank53]:     output = model(**micro_batch)
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default7]:[rank55]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f183bd96371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank55]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f18035a3189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default5]:[rank53]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank53]:     return self._call_impl(*args, **kwargs)
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default2]:[rank58]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default0]:[rank48]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default3]:[rank59]:     sharded_logits = self.model(
[default7]:[rank63]:     sharded_logits = self.model(
[default6]:[rank54]: Traceback (most recent call last):
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default1]:[rank49]: Traceback (most recent call last):
[default7]:[rank55]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f18035aa610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default7]:[rank55]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f18035c9978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default4]:[rank52]:     sharded_logits = self.model(
[default3]:[rank59]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank54]:     trainer.train(dataloader)
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default5]:[rank53]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank63]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank61]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default7]:[rank55]: frame #12: <unknown function> + 0x5adc309 (0x7f183bd88309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default2]:[rank58]:     output = model(**micro_batch)
[default6]:[rank54]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default3]:[rank59]:     return self._call_impl(*args, **kwargs)
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default7]:[rank55]: frame #13: <unknown function> + 0x5ae6f10 (0x7f183bd92f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default0]:[rank48]:     outputs = self.pipeline_engine.train_batch_iter(
[default4]:[rank52]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank58]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank55]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f183bd92fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank55]: frame #15: <unknown function> + 0x5124446 (0x7f183b3d0446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]:     return self._call_impl(*args, **kwargs)
[default3]:[rank59]:     return forward_call(*args, **kwargs)
[default7]:[rank55]: frame #16: <unknown function> + 0x1acf4b8 (0x7f1837d7b4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default0]:[rank48]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default5]:[rank53]:     return forward_call(*args, **kwargs)
[default2]:[rank58]:     return self._call_impl(*args, **kwargs)
[default4]:[rank52]:     return self._call_impl(*args, **kwargs)
[default4]:[rank52]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank52]:     return forward_call(*args, **kwargs)
[default2]:[rank58]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank61]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default7]:[rank63]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank54]:     outputs = self.pipeline_engine.train_batch_iter(
[default2]:[rank58]:     return forward_call(*args, **kwargs)
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default5]:[rank53]:     sharded_logits = self.model(
[default3]:[rank59]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default7]:[rank63]:     return forward_call(*args, **kwargs)
[default5]:[rank53]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default6]:[rank54]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default1]:[rank49]:     trainer.train(dataloader)
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default0]:[rank48]:     output = model(**micro_batch)
[default3]:[rank59]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default0]:[rank48]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default5]:[rank61]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default3]:[rank59]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default3]:[rank59]:     return self._call_impl(*args, **kwargs)
[default5]:[rank61]:     output = model(**micro_batch)
[default7]:[rank63]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default2]:[rank58]:     sharded_logits = self.model(
[default0]:[rank48]:     return self._call_impl(*args, **kwargs)
[default0]:[rank48]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank53]:     return self._call_impl(*args, **kwargs)
[default4]:[rank52]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default5]:[rank61]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default7]:[rank55]: frame #17: <unknown function> + 0x5aee004 (0x7f183bd9a004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default1]:[rank49]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default7]:[rank55]: frame #18: <unknown function> + 0x5af36b5 (0x7f183bd9f6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]:     output = model(**micro_batch)
[default0]:[rank48]:     return forward_call(*args, **kwargs)
[default5]:[rank61]:     return self._call_impl(*args, **kwargs)
[default3]:[rank59]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank58]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank53]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank61]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank55]: frame #19: <unknown function> + 0xd2631e (0x7f184e98931e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:[rank55]: frame #20: <unknown function> + 0x47def4 (0x7f184e0e0ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default2]:[rank58]:     return self._call_impl(*args, **kwargs)
[default7]:[rank55]: frame #21: <unknown function> + 0x1445a6 (0x5579918215a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]:     return forward_call(*args, **kwargs)
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default7]:[rank63]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default6]:[rank54]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank52]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default5]:[rank61]:     return forward_call(*args, **kwargs)
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default1]:[rank49]:     outputs = self.pipeline_engine.train_batch_iter(
[default3]:[rank59]:     return forward_call(*args, **kwargs)
[default6]:[rank54]:     return self._call_impl(*args, **kwargs)
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default7]:[rank55]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55799181aa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank58]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default4]:[rank52]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank52]:     return self._call_impl(*args, **kwargs)
[default0]:[rank48]:     sharded_logits = self.model(
[default5]:[rank61]:     sharded_logits = self.model(
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default1]:[rank49]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default7]:[rank55]: frame #23: <unknown function> + 0x150866 (0x55799182d866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]:     return forward_call(*args, **kwargs)
[default0]:[rank48]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank48]:     return self._call_impl(*args, **kwargs)
[default5]:[rank61]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank49]:     output = model(**micro_batch)
[default3]:[rank59]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default0]:[rank48]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank58]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default7]:[rank63]:     return self._call_impl(*args, **kwargs)
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default1]:[rank49]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank54]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank54]:     return forward_call(*args, **kwargs)
[default5]:[rank61]:     return self._call_impl(*args, **kwargs)
[default4]:[rank52]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank52]:     return forward_call(*args, **kwargs)
[default3]:[rank59]:     pipeline_state.run_communication()
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default5]:[rank61]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default7]:[rank63]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank61]:     return forward_call(*args, **kwargs)
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default7]:[rank63]:     return forward_call(*args, **kwargs)
[default3]:[rank59]:     recv_activation_tensor = recv_activation()
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default2]:[rank58]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default5]:[rank53]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default2]:[rank58]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank59]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default6]:[rank54]:     sharded_logits = self.model(
[default2]:[rank58]:     return self._call_impl(*args, **kwargs)
[default4]:[rank52]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default5]:[rank53]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default0]:[rank48]:     return forward_call(*args, **kwargs)
[default7]:[rank63]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default7]:[rank55]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x557991816142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #25: _PyFunction_Vectorcall + 0x6c (0x557991821a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank49]:     return self._call_impl(*args, **kwargs)
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default4]:[rank52]:     pipeline_state.run_communication()
[default5]:[rank53]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default2]:[rank58]:     return forward_call(*args, **kwargs)
[default1]:[rank49]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank59]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default1]:[rank49]:     return forward_call(*args, **kwargs)
[default5]:[rank61]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank54]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank54]:     return self._call_impl(*args, **kwargs)
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default1]:[rank49]:     sharded_logits = self.model(
[default6]:[rank54]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default0]:[rank48]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default5]:[rank61]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default7]:[rank55]: frame #26: PyObject_Call + 0xbc (0x55799182df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]:     return self._call_impl(*args, **kwargs)
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default7]:[rank55]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5579918142b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default3]:[rank59]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default5]:[rank61]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default2]:[rank58]:     pipeline_state.run_communication()
[default3]:[rank59]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default7]:[rank63]:     pipeline_state.run_communication()
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default5]:[rank61]:     return self._call_impl(*args, **kwargs)
[default4]:[rank52]:     recv_activation_tensor = recv_activation()
[default3]:[rank59]:     dist.recv(
[default5]:[rank53]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default5]:[rank61]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank58]:     recv_activation_tensor = recv_activation()
[default6]:[rank54]:     return forward_call(*args, **kwargs)
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default7]:[rank63]:     recv_activation_tensor = recv_activation()
[default1]:[rank49]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank61]:     return forward_call(*args, **kwargs)
[default5]:[rank53]:     return forward_call(*args, **kwargs)
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default7]:[rank55]: frame #28: _PyFunction_Vectorcall + 0x6c (0x557991821a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5579918128fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default3]:[rank59]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default0]:[rank48]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default0]:[rank48]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank59]:     return func(*args, **kwargs)
[default6]:[rank54]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default2]:[rank58]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default3]:[rank35]: Traceback (most recent call last):
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default3]:[rank35]:     trainer.train(dataloader)
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default3]:[rank35]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default3]:[rank35]:     outputs = self.pipeline_engine.train_batch_iter(
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default3]:[rank35]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanot[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default3]:[rank59]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default1]:[rank49]:     return self._call_impl(*args, **kwargs)
[default5]:[rank61]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default1]:[rank49]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank55]: frame #30: <unknown function> + 0x150582 (0x55799182d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default7]:[rank63]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default1]:[rank49]:     return forward_call(*args, **kwargs)
[default6]:[rank54]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default0]:[rank48]:     return self._call_impl(*args, **kwargs)
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default0]:[rank48]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank55]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5579918128fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
ron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default3]:[rank35]:     output = model(**micro_batch)
[default3]:[rank35]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank35]:     return self._call_impl(*args, **kwargs)
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default3]:[rank59]:     pg.recv([tensor], group_src_rank, tag).wait()
[default3]:[rank35]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank35]:     return forward_call(*args, **kwargs)
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default3]:[rank35]:     sharded_logits = self.model(
[default3]:[rank35]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank35]:     return self._call_impl(*args, **kwargs)
[default3]:[rank35]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank35]:     return forward_call(*args, **kwargs)
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default3]:[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default7]:[rank55]: frame #32: <unknown function> + 0x150582 (0x55799182d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]:     pipeline_state.run_communication()
[rank35]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default3]:[rank35]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default3]:[rank35]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank35]:     return self._call_impl(*args, **kwargs)
[default3]:[rank35]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank35]:     return forward_call(*args, **kwargs)
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default3]:[rank35]:     new_kwargs[name] = recv_from_pipeline_s[default7]:[rank63]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
tate_buffer(
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default3]:[rank35]:     pipeline_state.run_communication()
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default3]:[rank35]:     recv_activation_tensor = recv_activation()
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default3]:[rank35]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default3]:[rank59]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default3]:[rank59]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default3]:[rank35]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default3]:[rank35]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default3]:[rank35]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default3]:[rank35]:     dist.recv(
[default3]:[rank35]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default3]:[rank35]:     return func(*args, **kwargs)
[default3]:[rank35]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default3]:[rank35]:     pg.recv([tensor], group_src_rank, tag).wait()
[default3]:[rank35]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default3]:[rank35]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default3]:[rank35]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fe60a745897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default3]:[rank35]: frame #1: <unknown function> + 0x5b3a23e (0x7fe64426223e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
> >) + 0x2c7 (0x7fe64425cc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default3]:[rank59]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fb5b1821897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default7]:[rank63]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default3]:[rank35]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fe64425cf82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fe64425dfd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe644212371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe644212371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe644212371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libto[default5]:[rank53]:     pipeline_state.run_communication()
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
rch_cpu.so)
[default3]:[rank35]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe644212371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default4]:[rank52]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default2]:[rank58]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default3]:[rank35]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fe60ba1f189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:[rank35]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fe60ba26610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:[rank35]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fe60ba45978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:[rank35]: frame #12: <unknown function> + 0x5adc309 (0x7fe644204309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #13: <unknown function> [default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default5]:[rank61]:     recv_activation_tensor = recv_activation()
+ 0x5ae6f10 (0x7fe64420ef10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank55]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5579918128fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #34: <unknown function> + 0x150582 (0x55799182d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #1: <unknown function> + 0x5b3a23e (0x7fb5eb33e23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]:     dist.recv(
[default3]:[rank35]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fe64420efa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]:     recv_activation_tensor = recv_activation()
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default3]:[rank35]: frame #15: <unknown function> + 0x5124446 (0x7fe64384c446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #16: <unknown function> + 0x1acf4b8 (0x7fe6401f74b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #17: <unknown function> + 0x5aee004 (0x7fe644216004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #18: <unknown function> + 0x5af36b5 (0x7fe64421b6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default7]:[rank55]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5579918128fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default3]:[rank35]: frame #19: <unknown function> + 0xd2631e (0x7fe656e0531e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default3]:[rank35]: frame #20: <unknown function> + 0x47def4 (0x7fe65655cef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:[rank54]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default5]:[rank53]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default5]:[rank61]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default3]:[rank35]: frame #21: <unknown function> + 0x1445a6 (0x56033bca35a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #22: _PyObject_MakeTpCall + 0x26b (0x56033bc9ca6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #23: <unknown function> + 0x150866 (0x56033bcaf866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x56033bc98142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fb5eb338c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #25: _PyFunction_Vectorcall + 0x6c (0x56033bca3a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #26: PyObject_Call + 0xbc (0x56033bcaff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x56033bc962b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #28: _PyFunction_Vectorcall + 0x6c (0x56033bca3a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x56033bc948fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #30: <unknown function> + 0x150582 (0x56033bcaf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x56033bc948fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cl[default2]:[rank58]:     return func(*args, **kwargs)
uster/bin/python3.10)
[default3]:[rank35]: frame #32: <unknown function> + 0x150582 (0x56033bcaf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default7]:[rank63]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default3]:[rank35]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x56033bc948fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #34: <unknown function> + 0x150582 (0x56033bcaf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x56033bc948fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x56033bc9bf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #37: _PyObject_Call_Prepend + 0x69 (0x56033bcadc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default3]:[rank35]: frame #38: <unknown function> + 0x211239 (0x56033bd70239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #39: _PyObject_MakeTpCall + 0x26b (0x56033bc9ca6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x56033bc983e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #41: _PyFunction_Vectorcall + 0x6c (0x56033bca3a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: Traceback (most recent call last):
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default1]:[rank49]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default7]:[rank55]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x557991819f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default3]:[rank59]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fb5eb338f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x56033bc93c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #43: _PyFunction_Vectorcall + 0x6c (0x56033bca3a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x56033bc948fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #45: <unknown function> + 0x150582 (0x56033bcaf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #46: PyObject_Call + 0xbc (0x56033bcaff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x56033bc962b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #48: <unknown function> + 0x150582 (0x56033bcaf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-clu[default5]:[rank53]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default2]:[rank50]:     trainer.train(dataloader)
[default0]:[rank48]:     return forward_call(*args, **kwargs)
[default7]:[rank55]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55799182bc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #38: <unknown function> + 0x211239 (0x5579918ee239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]:     dist.recv(
[default2]:[rank58]:     pg.recv([tensor], group_src_rank, tag).wait()
ster/bin/python3.10)
[default3]:[rank35]: frame #49: PyObject_Call + 0xbc (0x56033bcaff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x56033bc962b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #51: _PyFunction_Vectorcall + 0x6c (0x56033bca3a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x56033bc9c007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #53: _PyObject_Call_Prepend + 0x69 (0x56033bcadc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #54: <unknown function> + 0x211239 (0x56033bd70239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default3]:[rank59]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fb5eb339fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank35]: frame #55: PyObject_Call + 0x207 (0x56033bcb0067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x56033bc962b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #57: <unknown function> + 0x150582 (0x56033bcaf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x56033bc948fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #59: <unknown function> + 0x150582 (0x56033bcaf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default7]:[rank55]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55799181aa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default3]:[rank35]: frame #60: PyObject_Call + 0xbc (0x56033bcaff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x56033bc962b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #62: <unknown function> + 0x150582 (0x56033bcaf582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: frame #63: PyObject_Call + 0xbc (0x56033bcaff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank35]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default3]:[rank59]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb5eb2ee371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default7]:[rank63]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default6]:[rank54]:     return self._call_impl(*args, **kwargs)
[default2]:[rank58]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default1]:[rank49]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank58]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default1]:[rank49]:     return self._call_impl(*args, **kwargs)
[default7]:[rank63]:     return func(*args, **kwargs)
[default5]:[rank61]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default3]:[rank59]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb5eb2ee371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default7]:[rank63]:     pg.recv([tensor], group_src_rank, tag).wait()
[default2]:[rank58]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f419ad43897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default3]:[rank59]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb5eb2ee371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default2]:[rank58]: frame #1: <unknown function> + 0x5b3a23e (0x7f41d486023e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default2]:[rank50]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default6]:[rank54]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank54]:     return forward_call(*args, **kwargs)
[default5]:[rank61]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default5]:[rank53]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default3]:[rank59]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb5eb2ee371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]:     dist.recv(
[default2]:[rank58]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f41d485ac87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank55]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5579918163e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]:     dist.recv(
[default3]:[rank59]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fb5b2afb189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank55]: frame #41: _PyFunction_Vectorcall + 0x6c (0x557991821a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f41d485af82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default5]:[rank53]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default5]:[rank53]:     return func(*args, **kwargs)
[default0]:[rank48]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default5]:[rank61]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default2]:[rank58]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f41d485bfd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank59]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fb5b2b02610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank49]:     return forward_call(*args, **kwargs)
[default7]:[rank55]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x557991811c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]:     return func(*args, **kwargs)
[default7]:[rank55]: frame #43: _PyFunction_Vectorcall + 0x6c (0x557991821a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f507984a897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default2]:[rank58]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f41d4810371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default5]:[rank61]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default6]:[rank54]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default7]:[rank63]: frame #1: <unknown function> + 0x5b3a23e (0x7f50b336723e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]:     pg.recv([tensor], group_src_rank, tag).wait()
[default3]:[rank59]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fb5b2b21978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank58]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f41d4810371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default7]:[rank63]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f50b3361c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]: frame #12: <unknown function> + 0x5adc309 (0x7fb5eb2e0309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f41d4810371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default7]:[rank63]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f50b3361f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]:     outputs = self.pipeline_engine.train_batch_iter(
[default2]:[rank58]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f41d4810371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]: frame #13: <unknown function> + 0x5ae6f10 (0x7fb5eb2eaf10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default4]:[rank52]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default7]:[rank63]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f50b3362fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f50b3317371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]:     dist.recv(
[default2]:[rank58]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f419c01d189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank49]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default7]:[rank63]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f50b3317371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fb5eb2eafa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default0]:[rank48]:     pipeline_state.run_communication()
[default4]:[rank52]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default2]:[rank58]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f419c024610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank58]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f419c043978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default5]:[rank61]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default6]:[rank54]:     pipeline_state.run_communication()
[default7]:[rank63]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f50b3317371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default4]:[rank52]:     return func(*args, **kwargs)
[default3]:[rank59]: frame #15: <unknown function> + 0x5124446 (0x7fb5ea928446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]: frame #16: <unknown function> + 0x1acf4b8 (0x7fb5e72d34b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default5]:[rank53]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default5]:[rank61]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f6ac0f06897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default0]:[rank48]:     recv_activation_tensor = recv_activation()
[default2]:[rank58]: frame #12: <unknown function> + 0x5adc309 (0x7f41d4802309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank55]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5579918128fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #45: <unknown function> + 0x150582 (0x55799182d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #17: <unknown function> + 0x5aee004 (0x7fb5eb2f2004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default5]:[rank61]: frame #1: <unknown function> + 0x5b3a23e (0x7f6afaa2323e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: frame #13: <unknown function> + 0x5ae6f10 (0x7f41d480cf10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f50b3317371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f507ab24189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:[rank59]: frame #18: <unknown function> + 0x5af36b5 (0x7fb5eb2f76b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f507ab2b610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank52]:     pg.recv([tensor], group_src_rank, tag).wait()
[default5]:[rank53]:     pg.recv([tensor], group_src_rank, tag).wait()
[default5]:[rank61]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f6afaa1dc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]:     output = model(**micro_batch)
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default2]:[rank58]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f41d480cfa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: frame #15: <unknown function> + 0x5124446 (0x7f41d3e4a446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default7]:[rank63]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f507ab4a978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank54]:     recv_activation_tensor = recv_activation()
[default5]:[rank61]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f6afaa1df82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default1]:[rank49]:     pipeline_state.run_communication()
[default2]:[rank58]: frame #16: <unknown function> + 0x1acf4b8 (0x7f41d07f54b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default7]:[rank63]: frame #12: <unknown function> + 0x5adc309 (0x7f50b3309309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]: frame #19: <unknown function> + 0xd2631e (0x7fb5fdee131e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:[rank52]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default2]:[rank58]: frame #17: <unknown function> + 0x5aee004 (0x7f41d4814004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default6]:[rank54]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default5]:[rank61]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f6afaa1efd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default2]:[rank58]: frame #18: <unknown function> + 0x5af36b5 (0x7f41d48196b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f6b82784897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default7]:[rank63]: frame #13: <unknown function> + 0x5ae6f10 (0x7f50b3313f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6afa9d3371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: frame #19: <unknown function> + 0xd2631e (0x7f41e740331e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default3]:[rank59]: frame #20: <unknown function> + 0x47def4 (0x7fb5fd638ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default3]:[rank59]: frame #21: <unknown function> + 0x1445a6 (0x5593130465a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #20: <unknown function> + 0x47def4 (0x7f41e6b5aef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default5]:[rank61]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6afa9d3371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6afa9d3371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]:     return self._call_impl(*args, **kwargs)
[default0]:[rank48]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default5]:[rank61]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6afa9d3371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default0]:[rank48]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default2]:[rank58]: frame #21: <unknown function> + 0x1445a6 (0x55dc64d9e5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55dc64d97a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #46: PyObject_Call + 0xbc (0x55799182df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f6ac21e0189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank55]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5579918142b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f6ac21e7610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank48]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default3]:[rank59]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55931303fa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default1]:[rank49]:     recv_activation_tensor = recv_activation()
[default7]:[rank55]: frame #48: <unknown function> + 0x150582 (0x55799182d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #23: <unknown function> + 0x150866 (0x55dc64daa866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #49: PyObject_Call + 0xbc (0x55799182df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55dc64d93142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f50b3313fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank53]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default3]:[rank59]: frame #23: <unknown function> + 0x150866 (0x559313052866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default5]:[rank61]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f6ac2206978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank58]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55dc64d9ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]:     dist.recv(
[default3]:[rank59]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55931303b142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]:     return forward_call(*args, **kwargs)
[default7]:[rank63]: frame #15: <unknown function> + 0x5124446 (0x7f50b2951446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f5c23b27897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default5]:[rank61]: frame #12: <unknown function> + 0x5adc309 (0x7f6afa9c5309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank55]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5579918142b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #51: _PyFunction_Vectorcall + 0x6c (0x557991821a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #26: PyObject_Call + 0xbc (0x55dc64daaf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #25: _PyFunction_Vectorcall + 0x6c (0x559313046a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default7]:[rank63]: frame #16: <unknown function> + 0x1acf4b8 (0x7f50af2fc4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]: frame #26: PyObject_Call + 0xbc (0x559313052f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #13: <unknown function> + 0x5ae6f10 (0x7f6afa9cff10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5593130392b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #28: _PyFunction_Vectorcall + 0x6c (0x559313046a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5593130378fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #17: <unknown function> + 0x5aee004 (0x7f50b331b004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #18: <unknown function> + 0x5af36b5 (0x7f50b33206b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55dc64d912b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #1: <unknown function> + 0x5b3a23e (0x7f5c5d64423e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55dc64d9ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default2]:[rank58]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55dc64d8f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default7]:[rank63]: frame #19: <unknown function> + 0xd2631e (0x7f50c5f0a31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default5]:[rank53]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f5c5d63ec87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]: frame #30: <unknown function> + 0x150582 (0x559313052582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5593130378fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]:     sharded_logits = self.model(
[default2]:[rank50]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank63]: frame #20: <unknown function> + 0x47def4 (0x7f50c5661ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:[rank52]: frame #1: <unknown function> + 0x5b3a23e (0x7f6bbc2a123e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f6afa9cffa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f6bbc29bc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]: frame #32: <unknown function> + 0x150582 (0x559313052582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55799181a007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #21: <unknown function> + 0x1445a6 (0x5633c3c875a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #30: <unknown function> + 0x150582 (0x55dc64daa582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]:     return self._call_impl(*args, **kwargs)
[default3]:[rank59]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5593130378fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f5c5d63ef82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #15: <unknown function> + 0x5124446 (0x7f6afa00d446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default3]:[rank59]: frame #34: <unknown function> + 0x150582 (0x559313052582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank50]:     return forward_call(*args, **kwargs)
[default2]:[rank58]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55dc64d8f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f5c5d63ffd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #22: _PyObject_MakeTpCall + 0x26b (0x5633c3c80a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5593130378fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #32: <unknown function> + 0x150582 (0x55dc64daa582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55dc64d8f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #23: <unknown function> + 0x150866 (0x5633c3c93866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55931303ef50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #34: <unknown function> + 0x150582 (0x55dc64daa582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x5633c3c7c142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default1]:[rank49]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default7]:[rank55]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55799182bc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #37: _PyObject_Call_Prepend + 0x69 (0x559313050c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #54: <unknown function> + 0x211239 (0x5579918ee239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #38: <unknown function> + 0x211239 (0x559313113239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55dc64d8f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55dc64d96f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default3]:[rank59]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55931303fa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]:     dist.recv(
[default7]:[rank63]: frame #25: _PyFunction_Vectorcall + 0x6c (0x5633c3c87a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55dc64da8c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #16: <unknown function> + 0x1acf4b8 (0x7f6af69b84b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #17: <unknown function> + 0x5aee004 (0x7f6afa9d7004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f5c5d5f4371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f5c5d5f4371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: frame #38: <unknown function> + 0x211239 (0x55dc64e6b239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default4]:[rank52]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f6bbc29bf82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default1]:[rank49]:     return func(*args, **kwargs)
[default5]:[rank61]: frame #18: <unknown function> + 0x5af36b5 (0x7f6afa9dc6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default2]:[rank58]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55dc64d97a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #55: PyObject_Call + 0x207 (0x55799182e067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55dc64d933e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #26: PyObject_Call + 0xbc (0x5633c3c93f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default2]:[rank50]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default3]:[rank59]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55931303b3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f6bbc29cfd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6bbc251371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55dc64d9ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default0]:[rank48]:     return func(*args, **kwargs)
[default7]:[rank55]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5579918142b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #57: <unknown function> + 0x150582 (0x55799182d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5633c3c7a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5579918128fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default7]:[rank63]: frame #28: _PyFunction_Vectorcall + 0x6c (0x5633c3c87a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f5c5d5f4371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f5c5d5f4371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]: frame #41: _PyFunction_Vectorcall + 0x6c (0x559313046a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #59: <unknown function> + 0x150582 (0x55799182d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55dc64d8ec5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default5]:[rank61]: frame #19: <unknown function> + 0xd2631e (0x7f6b0d5c631e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:[rank54]:     dist.recv(
[default2]:[rank58]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55dc64d9ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default2]:[rank58]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55dc64d8f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5633c3c788fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x559313036c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6bbc251371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6bbc251371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #20: <unknown function> + 0x47def4 (0x7f6b0cd1def4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:[rank54]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default0]:[rank48]:     pg.recv([tensor], group_src_rank, tag).wait()
[default3]:[rank59]: frame #43: _PyFunction_Vectorcall + 0x6c (0x559313046a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f5c24e01189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank61]: frame #21: <unknown function> + 0x1445a6 (0x5640607835a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6bbc251371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: frame #45: <unknown function> + 0x150582 (0x55dc64daa582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5593130378fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #30: <unknown function> + 0x150582 (0x5633c3c93582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #45: <unknown function> + 0x150582 (0x559313052582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #46: PyObject_Call + 0xbc (0x55dc64daaf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5633c3c788fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #32: <unknown function> + 0x150582 (0x5633c3c93582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #46: PyObject_Call + 0xbc (0x559313052f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55dc64d912b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #22: _PyObject_MakeTpCall + 0x26b (0x56406077ca6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5633c3c788fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #34: <unknown function> + 0x150582 (0x5633c3c93582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #48: <unknown function> + 0x150582 (0x55dc64daa582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5593130392b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #23: <unknown function> + 0x150866 (0x56406078f866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x564060778142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #48: <unknown function> + 0x150582 (0x559313052582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #49: PyObject_Call + 0xbc (0x55dc64daaf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5633c3c788fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #49: PyObject_Call + 0xbc (0x559313052f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5593130392b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x5633c3c7ff50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55dc64d912b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #51: _PyFunction_Vectorcall + 0x6c (0x559313046a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]:     return func(*args, **kwargs)
[default6]:[rank54]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default7]:[rank63]: frame #37: _PyObject_Call_Prepend + 0x69 (0x5633c3c91c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f5c24e08610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank58]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55dc64d9ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f5c24e27978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:[rank59]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55931303f007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default2]:[rank50]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank63]: frame #38: <unknown function> + 0x211239 (0x5633c3d54239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f6b83a5e189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank52]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f6b83a65610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank58]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55dc64d97007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #60: PyObject_Call + 0xbc (0x55799182df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #53: _PyObject_Call_Prepend + 0x69 (0x559313050c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #25: _PyFunction_Vectorcall + 0x6c (0x564060783a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #39: _PyObject_MakeTpCall + 0x26b (0x5633c3c80a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55dc64da8c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5579918142b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5633c3c7c3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #12: <unknown function> + 0x5adc309 (0x7f5c5d5e6309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #13: <unknown function> + 0x5ae6f10 (0x7f5c5d5f0f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #26: PyObject_Call + 0xbc (0x56406078ff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]:     pg.recv([tensor], group_src_rank, tag).wait()
[default7]:[rank63]: frame #41: _PyFunction_Vectorcall + 0x6c (0x5633c3c87a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]:     pg.recv([tensor], group_src_rank, tag).wait()
[default3]:[rank59]: frame #54: <unknown function> + 0x211239 (0x559313113239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default2]:[rank50]:     return self._call_impl(*args, **kwargs)
[default2]:[rank58]: frame #54: <unknown function> + 0x211239 (0x55dc64e6b239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x5633c3c77c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5640607762b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank55]: frame #62: <unknown function> + 0x150582 (0x55799182d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default2]:[rank58]: frame #55: PyObject_Call + 0x207 (0x55dc64dab067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default0]:[rank48]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default5]:[rank53]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f5c5d5f0fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]:     return forward_call(*args, **kwargs)
[default5]:[rank61]: frame #28: _PyFunction_Vectorcall + 0x6c (0x564060783a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fc6e4da6897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default7]:[rank55]: frame #63: PyObject_Call + 0xbc (0x55799182df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55dc64d912b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #55: PyObject_Call + 0x207 (0x559313053067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default2]:[rank50]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default5]:[rank61]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5640607748fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #1: <unknown function> + 0x5b3a23e (0x7fc71e8c323e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default7]:[rank63]: frame #43: _PyFunction_Vectorcall + 0x6c (0x5633c3c87a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #15: <unknown function> + 0x5124446 (0x7f5c5cc2e446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5593130392b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default1]:[rank49]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7effc93a7897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default7]:[rank63]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5633c3c788fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #30: <unknown function> + 0x150582 (0x56406078f582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default3]:[rank59]: frame #57: <unknown function> + 0x150582 (0x559313052582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]:     pipeline_state.run_communication()
[default7]:[rank63]: frame #45: <unknown function> + 0x150582 (0x5633c3c93582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f6b83a84978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank52]: frame #12: <unknown function> + 0x5adc309 (0x7f6bbc243309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5640607748fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank55]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default3]:[rank59]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5593130378fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default2]:[rank50]:     recv_activation_tensor = recv_activation()
[default2]:[rank58]: frame #57: <unknown function> + 0x150582 (0x55dc64daa582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #32: <unknown function> + 0x150582 (0x56406078f582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #1: <unknown function> + 0x5b3a23e (0x7f0002ec423e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank59]: frame #59: <unknown function> + 0x150582 (0x559313052582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f0002ebec87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #16: <unknown function> + 0x1acf4b8 (0x7f5c595d94b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]: frame #13: <unknown function> + 0x5ae6f10 (0x7f6bbc24df10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #46: PyObject_Call + 0xbc (0x5633c3c93f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f2a61e72897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default2]:[rank58]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55dc64d8f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f0002ebef82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default2]:[rank50]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default7]:[rank63]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5633c3c7a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5640607748fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f6bbc24dfa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: frame #59: <unknown function> + 0x150582 (0x55dc64daa582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f0002ebffd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0002e74371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: frame #60: PyObject_Call + 0xbc (0x55dc64daaf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #60: PyObject_Call + 0xbc (0x559313052f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #34: <unknown function> + 0x150582 (0x56406078f582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #17: <unknown function> + 0x5aee004 (0x7f5c5d5f8004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: frame #1: <unknown function> + 0x5b3a23e (0x7f2a9b98f23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #48: <unknown function> + 0x150582 (0x5633c3c93582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f2a9b989c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f2a9b989f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank58]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55dc64d912b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default5]:[rank61]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5640607748fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0002e74371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0002e74371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #49: PyObject_Call + 0xbc (0x5633c3c93f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fc71e8bdc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default5]:[rank61]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x56406077bf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0002e74371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #37: _PyObject_Call_Prepend + 0x69 (0x56406078dc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5593130392b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank58]: frame #62: <unknown function> + 0x150582 (0x55dc64daa582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #18: <unknown function> + 0x5af36b5 (0x7f5c5d5fd6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #19: <unknown function> + 0xd2631e (0x7f5c701e731e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default5]:[rank61]: frame #38: <unknown function> + 0x211239 (0x564060850239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fc71e8bdf82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fc71e8befd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5633c3c7a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #20: <unknown function> + 0x47def4 (0x7f5c6f93eef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default2]:[rank58]: frame #63: PyObject_Call + 0xbc (0x55dc64daaf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #21: <unknown function> + 0x1445a6 (0x559d5ecb05a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default2]:[rank58]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default5]:[rank61]: frame #39: _PyObject_MakeTpCall + 0x26b (0x56406077ca6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default7]:[rank63]: frame #51: _PyFunction_Vectorcall + 0x6c (0x5633c3c87a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7effca681189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank49]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7effca688610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:[rank59]: frame #62: <unknown function> + 0x150582 (0x559313052582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f2a9b98afd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5640607783e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2a9b93f371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]: frame #15: <unknown function> + 0x5124446 (0x7f6bbb88b446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x5633c3c80007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #16: <unknown function> + 0x1acf4b8 (0x7f6bb82364b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #53: _PyObject_Call_Prepend + 0x69 (0x5633c3c91c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #41: _PyFunction_Vectorcall + 0x6c (0x564060783a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: frame #63: PyObject_Call + 0xbc (0x559313052f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #22: _PyObject_MakeTpCall + 0x26b (0x559d5eca9a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank59]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default6]:[rank54]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc71e873371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc71e873371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #54: <unknown function> + 0x211239 (0x5633c3d54239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc71e873371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]:     dist.recv(
[default5]:[rank61]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x564060773c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default7]:[rank63]: frame #55: PyObject_Call + 0x207 (0x5633c3c94067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc71e873371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #43: _PyFunction_Vectorcall + 0x6c (0x564060783a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2a9b93f371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]:     return func(*args, **kwargs)
[default2]:[rank50]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default5]:[rank61]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5640607748fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #45: <unknown function> + 0x150582 (0x56406078f582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fc6e6080189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank63]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5633c3c7a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #46: PyObject_Call + 0xbc (0x56406078ff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #23: <unknown function> + 0x150866 (0x559d5ecbc866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x559d5eca5142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5640607762b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #57: <unknown function> + 0x150582 (0x5633c3c93582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5633c3c788fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]:     pg.recv([tensor], group_src_rank, tag).wait()
[default7]:[rank63]: frame #59: <unknown function> + 0x150582 (0x5633c3c93582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default4]:[rank52]: frame #17: <unknown function> + 0x5aee004 (0x7f6bbc255004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #60: PyObject_Call + 0xbc (0x5633c3c93f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #18: <unknown function> + 0x5af36b5 (0x7f6bbc25a6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5633c3c7a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank63]: frame #62: <unknown function> + 0x150582 (0x5633c3c93582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #19: <unknown function> + 0xd2631e (0x7f6bcee4431e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:[rank54]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fc6e6087610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank63]: frame #63: PyObject_Call + 0xbc (0x5633c3c93f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #25: _PyFunction_Vectorcall + 0x6c (0x559d5ecb0a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #48: <unknown function> + 0x150582 (0x56406078f582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7effca6a7978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank49]: frame #12: <unknown function> + 0x5adc309 (0x7f0002e66309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank63]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default1]:[rank49]: frame #13: <unknown function> + 0x5ae6f10 (0x7f0002e70f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2a9b93f371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank61]: frame #49: PyObject_Call + 0xbc (0x56406078ff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5640607762b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #51: _PyFunction_Vectorcall + 0x6c (0x564060783a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2a9b93f371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fc6e60a6978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank61]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x56406077c007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #53: _PyObject_Call_Prepend + 0x69 (0x56406078dc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #54: <unknown function> + 0x211239 (0x564060850239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #55: PyObject_Call + 0x207 (0x564060790067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5640607762b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #57: <unknown function> + 0x150582 (0x56406078f582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5640607748fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #59: <unknown function> + 0x150582 (0x56406078f582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #60: PyObject_Call + 0xbc (0x56406078ff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5640607762b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #62: <unknown function> + 0x150582 (0x56406078f582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: frame #63: PyObject_Call + 0xbc (0x56406078ff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank61]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default6]:[rank54]: frame #12: <unknown function> + 0x5adc309 (0x7fc71e865309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default5]:[rank53]: frame #26: PyObject_Call + 0xbc (0x559d5ecbcf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x559d5eca32b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #28: _PyFunction_Vectorcall + 0x6c (0x559d5ecb0a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fc651db5897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default4]:[rank52]: frame #20: <unknown function> + 0x47def4 (0x7f6bce59bef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:[rank52]: frame #21: <unknown function> + 0x1445a6 (0x5613e36b75a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #13: <unknown function> + 0x5ae6f10 (0x7fc71e86ff10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f2a6314c189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank48]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f2a63153610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank48]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f2a63172978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank48]: frame #12: <unknown function> + 0x5adc309 (0x7f2a9b931309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x559d5eca18fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #30: <unknown function> + 0x150582 (0x559d5ecbc582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x559d5eca18fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #22: _PyObject_MakeTpCall + 0x26b (0x5613e36b0a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fc71e86ffa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]: frame #15: <unknown function> + 0x5124446 (0x7fc71dead446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]: frame #16: <unknown function> + 0x1acf4b8 (0x7fc71a8584b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #32: <unknown function> + 0x150582 (0x559d5ecbc582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x559d5eca18fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #13: <unknown function> + 0x5ae6f10 (0x7f2a9b93bf10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]: frame #23: <unknown function> + 0x150866 (0x5613e36c3866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #34: <unknown function> + 0x150582 (0x559d5ecbc582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #17: <unknown function> + 0x5aee004 (0x7fc71e877004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]: frame #18: <unknown function> + 0x5af36b5 (0x7fc71e87c6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f0002e70fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x559d5eca18fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x559d5eca8f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x5613e36ac142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #25: _PyFunction_Vectorcall + 0x6c (0x5613e36b7a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f2a9b93bfa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]: frame #1: <unknown function> + 0x5b3a23e (0x7fc68b8d223e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fc68b8ccc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #37: _PyObject_Call_Prepend + 0x69 (0x559d5ecbac39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #38: <unknown function> + 0x211239 (0x559d5ed7d239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #39: _PyObject_MakeTpCall + 0x26b (0x559d5eca9a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #26: PyObject_Call + 0xbc (0x5613e36c3f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #15: <unknown function> + 0x5124446 (0x7f00024ae446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]: frame #16: <unknown function> + 0x1acf4b8 (0x7efffee594b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]: frame #17: <unknown function> + 0x5aee004 (0x7f0002e78004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x559d5eca53e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #41: _PyFunction_Vectorcall + 0x6c (0x559d5ecb0a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fc68b8ccf82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: frame #15: <unknown function> + 0x5124446 (0x7f2a9af79446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: frame #16: <unknown function> + 0x1acf4b8 (0x7f2a979244b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x559d5eca0c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #43: _PyFunction_Vectorcall + 0x6c (0x559d5ecb0a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fc68b8cdfd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank54]: frame #19: <unknown function> + 0xd2631e (0x7fc73146631e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:[rank54]: frame #20: <unknown function> + 0x47def4 (0x7fc730bbdef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:[rank52]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5613e36aa2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #28: _PyFunction_Vectorcall + 0x6c (0x5613e36b7a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #18: <unknown function> + 0x5af36b5 (0x7f0002e7d6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]: frame #19: <unknown function> + 0xd2631e (0x7f0015a6731e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:[rank52]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5613e36a88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #30: <unknown function> + 0x150582 (0x5613e36c3582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #17: <unknown function> + 0x5aee004 (0x7f2a9b943004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5613e36a88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x559d5eca18fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #20: <unknown function> + 0x47def4 (0x7f00151beef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default1]:[rank49]: frame #21: <unknown function> + 0x1445a6 (0x5629c11c85a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc68b882371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc68b882371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc68b882371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank49]: frame #22: _PyObject_MakeTpCall + 0x26b (0x5629c11c1a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #23: <unknown function> + 0x150866 (0x5629c11d4866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x5629c11bd142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #45: <unknown function> + 0x150582 (0x559d5ecbc582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #46: PyObject_Call + 0xbc (0x559d5ecbcf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #18: <unknown function> + 0x5af36b5 (0x7f2a9b9486b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: frame #19: <unknown function> + 0xd2631e (0x7f2aae53231e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default1]:[rank49]: frame #25: _PyFunction_Vectorcall + 0x6c (0x5629c11c8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc68b882371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank52]: frame #32: <unknown function> + 0x150582 (0x5613e36c3582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #20: <unknown function> + 0x47def4 (0x7f2aadc89ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:[rank54]: frame #21: <unknown function> + 0x1445a6 (0x55b6c34ff5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55b6c34f8a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x559d5eca32b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #48: <unknown function> + 0x150582 (0x559d5ecbc582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #49: PyObject_Call + 0xbc (0x559d5ecbcf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x559d5eca32b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #23: <unknown function> + 0x150866 (0x55b6c350b866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55b6c34f4142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #21: <unknown function> + 0x1445a6 (0x562a3b0165a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fc65308f189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank50]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fc653096610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank54]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55b6c34ffa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #51: _PyFunction_Vectorcall + 0x6c (0x559d5ecb0a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #22: _PyObject_MakeTpCall + 0x26b (0x562a3b00fa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fc6530b5978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank54]: frame #26: PyObject_Call + 0xbc (0x55b6c350bf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55b6c34f22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #26: PyObject_Call + 0xbc (0x5629c11d4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5613e36a88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #34: <unknown function> + 0x150582 (0x5613e36c3582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #23: <unknown function> + 0x150866 (0x562a3b022866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x562a3b00b142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x559d5eca9007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5629c11bb2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #28: _PyFunction_Vectorcall + 0x6c (0x5629c11c8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #12: <unknown function> + 0x5adc309 (0x7fc68b874309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: frame #25: _PyFunction_Vectorcall + 0x6c (0x562a3b016a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5613e36a88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x5613e36aff50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #13: <unknown function> + 0x5ae6f10 (0x7fc68b87ef10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #53: _PyObject_Call_Prepend + 0x69 (0x559d5ecbac39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #54: <unknown function> + 0x211239 (0x559d5ed7d239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #55: PyObject_Call + 0x207 (0x559d5ecbd067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x559d5eca32b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #57: <unknown function> + 0x150582 (0x559d5ecbc582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x559d5eca18fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #59: <unknown function> + 0x150582 (0x559d5ecbc582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #60: PyObject_Call + 0xbc (0x559d5ecbcf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x559d5eca32b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5629c11b98fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fc68b87efa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]: frame #15: <unknown function> + 0x5124446 (0x7fc68aebc446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: frame #62: <unknown function> + 0x150582 (0x559d5ecbc582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank53]: frame #63: PyObject_Call + 0xbc (0x559d5ecbcf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55b6c34ffa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #16: <unknown function> + 0x1acf4b8 (0x7fc6878674b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank50]: frame #17: <unknown function> + 0x5aee004 (0x7fc68b886004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank53]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default4]:[rank52]: frame #37: _PyObject_Call_Prepend + 0x69 (0x5613e36c1c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #18: <unknown function> + 0x5af36b5 (0x7fc68b88b6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank48]: frame #26: PyObject_Call + 0xbc (0x562a3b022f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #38: <unknown function> + 0x211239 (0x5613e3784239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55b6c34f08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #30: <unknown function> + 0x150582 (0x55b6c350b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x562a3b0092b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55b6c34f08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #30: <unknown function> + 0x150582 (0x5629c11d4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #39: _PyObject_MakeTpCall + 0x26b (0x5613e36b0a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5613e36ac3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #28: _PyFunction_Vectorcall + 0x6c (0x562a3b016a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #19: <unknown function> + 0xd2631e (0x7fc69e47531e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default2]:[rank50]: frame #20: <unknown function> + 0x47def4 (0x7fc69dbccef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default1]:[rank49]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5629c11b98fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #41: _PyFunction_Vectorcall + 0x6c (0x5613e36b7a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #21: <unknown function> + 0x1445a6 (0x55d936c265a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55d936c1fa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x562a3b0078fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x5613e36a7c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #43: _PyFunction_Vectorcall + 0x6c (0x5613e36b7a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #23: <unknown function> + 0x150866 (0x55d936c32866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #32: <unknown function> + 0x150582 (0x5629c11d4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5629c11b98fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #30: <unknown function> + 0x150582 (0x562a3b022582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55d936c1b142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55d936c26a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x562a3b0078fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5613e36a88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #32: <unknown function> + 0x150582 (0x55b6c350b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55b6c34f08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #34: <unknown function> + 0x150582 (0x55b6c350b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #45: <unknown function> + 0x150582 (0x5613e36c3582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #34: <unknown function> + 0x150582 (0x5629c11d4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #26: PyObject_Call + 0xbc (0x55d936c32f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #46: PyObject_Call + 0xbc (0x5613e36c3f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5613e36aa2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55b6c34f08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #48: <unknown function> + 0x150582 (0x5613e36c3582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5629c11b98fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x5629c11c0f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55b6c34f7f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #49: PyObject_Call + 0xbc (0x5613e36c3f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5613e36aa2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #51: _PyFunction_Vectorcall + 0x6c (0x5613e36b7a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55b6c3509c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #37: _PyObject_Call_Prepend + 0x69 (0x5629c11d2c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x5613e36b0007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #53: _PyObject_Call_Prepend + 0x69 (0x5613e36c1c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #38: <unknown function> + 0x211239 (0x5629c1295239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #38: <unknown function> + 0x211239 (0x55b6c35cc239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #32: <unknown function> + 0x150582 (0x562a3b022582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #39: _PyObject_MakeTpCall + 0x26b (0x5629c11c1a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5629c11bd3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x562a3b0078fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #54: <unknown function> + 0x211239 (0x5613e3784239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #41: _PyFunction_Vectorcall + 0x6c (0x5629c11c8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x5629c11b8c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #55: PyObject_Call + 0x207 (0x5613e36c4067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #34: <unknown function> + 0x150582 (0x562a3b022582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x562a3b0078fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #43: _PyFunction_Vectorcall + 0x6c (0x5629c11c8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5629c11b98fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55b6c34f8a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55b6c34f43e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5613e36aa2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55d936c192b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55d936c26a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x562a3b00ef50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #37: _PyObject_Call_Prepend + 0x69 (0x562a3b020c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #57: <unknown function> + 0x150582 (0x5613e36c3582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #45: <unknown function> + 0x150582 (0x5629c11d4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #46: PyObject_Call + 0xbc (0x5629c11d4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55d936c178fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #38: <unknown function> + 0x211239 (0x562a3b0e3239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #39: _PyObject_MakeTpCall + 0x26b (0x562a3b00fa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x562a3b00b3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55b6c34ffa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5629c11bb2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #48: <unknown function> + 0x150582 (0x5629c11d4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55b6c34efc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55b6c34ffa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5613e36a88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #30: <unknown function> + 0x150582 (0x55d936c32582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55d936c178fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #32: <unknown function> + 0x150582 (0x55d936c32582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #59: <unknown function> + 0x150582 (0x5613e36c3582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #60: PyObject_Call + 0xbc (0x5613e36c3f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55d936c178fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #49: PyObject_Call + 0xbc (0x5629c11d4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5629c11bb2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #34: <unknown function> + 0x150582 (0x55d936c32582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55b6c34f08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #45: <unknown function> + 0x150582 (0x55b6c350b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5613e36aa2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #51: _PyFunction_Vectorcall + 0x6c (0x5629c11c8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x5629c11c1007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55d936c178fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55d936c1ef50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #41: _PyFunction_Vectorcall + 0x6c (0x562a3b016a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #62: <unknown function> + 0x150582 (0x5613e36c3582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: frame #63: PyObject_Call + 0xbc (0x5613e36c3f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank52]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default6]:[rank54]: frame #46: PyObject_Call + 0xbc (0x55b6c350bf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #53: _PyObject_Call_Prepend + 0x69 (0x5629c11d2c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #54: <unknown function> + 0x211239 (0x5629c1295239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55b6c34f22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #48: <unknown function> + 0x150582 (0x55b6c350b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x562a3b006c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #43: _PyFunction_Vectorcall + 0x6c (0x562a3b016a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #55: PyObject_Call + 0x207 (0x5629c11d5067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #49: PyObject_Call + 0xbc (0x55b6c350bf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55b6c34f22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5629c11bb2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #57: <unknown function> + 0x150582 (0x5629c11d4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55d936c30c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x562a3b0078fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #45: <unknown function> + 0x150582 (0x562a3b022582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55b6c34ffa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55b6c34f8007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #46: PyObject_Call + 0xbc (0x562a3b022f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #38: <unknown function> + 0x211239 (0x55d936cf3239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55b6c3509c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x562a3b0092b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55d936c1fa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #48: <unknown function> + 0x150582 (0x562a3b022582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5629c11b98fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #59: <unknown function> + 0x150582 (0x5629c11d4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #49: PyObject_Call + 0xbc (0x562a3b022f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x562a3b0092b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #51: _PyFunction_Vectorcall + 0x6c (0x562a3b016a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x562a3b00f007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #54: <unknown function> + 0x211239 (0x55b6c35cc239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #60: PyObject_Call + 0xbc (0x5629c11d4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5629c11bb2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #55: PyObject_Call + 0x207 (0x55b6c350c067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55b6c34f22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55d936c1b3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55d936c26a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55d936c16c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #57: <unknown function> + 0x150582 (0x55b6c350b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55d936c26a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55b6c34f08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #59: <unknown function> + 0x150582 (0x55b6c350b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #60: PyObject_Call + 0xbc (0x55b6c350bf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55d936c178fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #62: <unknown function> + 0x150582 (0x5629c11d4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: frame #63: PyObject_Call + 0xbc (0x5629c11d4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #53: _PyObject_Call_Prepend + 0x69 (0x562a3b020c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #45: <unknown function> + 0x150582 (0x55d936c32582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #54: <unknown function> + 0x211239 (0x562a3b0e3239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #55: PyObject_Call + 0x207 (0x562a3b023067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #46: PyObject_Call + 0xbc (0x55d936c32f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55d936c192b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank49]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default2]:[rank50]: frame #48: <unknown function> + 0x150582 (0x55d936c32582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55b6c34f22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #62: <unknown function> + 0x150582 (0x55b6c350b582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #49: PyObject_Call + 0xbc (0x55d936c32f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55d936c192b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: frame #63: PyObject_Call + 0xbc (0x55b6c350bf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x562a3b0092b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #57: <unknown function> + 0x150582 (0x562a3b022582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank54]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default0]:[rank48]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x562a3b0078fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #59: <unknown function> + 0x150582 (0x562a3b022582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #60: PyObject_Call + 0xbc (0x562a3b022f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x562a3b0092b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #62: <unknown function> + 0x150582 (0x562a3b022582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: frame #63: PyObject_Call + 0xbc (0x562a3b022f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55d936c26a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55d936c1f007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55d936c30c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #54: <unknown function> + 0x211239 (0x55d936cf3239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #55: PyObject_Call + 0x207 (0x55d936c33067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55d936c192b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #57: <unknown function> + 0x150582 (0x55d936c32582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55d936c178fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #59: <unknown function> + 0x150582 (0x55d936c32582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #60: PyObject_Call + 0xbc (0x55d936c32f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank48]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default2]:[rank50]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55d936c192b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #62: <unknown function> + 0x150582 (0x55d936c32582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: frame #63: PyObject_Call + 0xbc (0x55d936c32f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank50]: . This may indicate a possible application crash on rank 0 or a network set up issue.
E0702 21:47:08.161000 139896234325824 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 1737002) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-07-02_21:47:08
  host      : ip-26-0-160-225.ec2.internal
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 1737003)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2024-07-02_21:47:08
  host      : ip-26-0-160-225.ec2.internal
  rank      : 2 (local_rank: 2)
  exitcode  : 1 (pid: 1737004)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
  time      : 2024-07-02_21:47:08
  host      : ip-26-0-160-225.ec2.internal
  rank      : 3 (local_rank: 3)
  exitcode  : 1 (pid: 1737005)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[4]:
  time      : 2024-07-02_21:47:08
  host      : ip-26-0-160-225.ec2.internal
  rank      : 4 (local_rank: 4)
  exitcode  : 1 (pid: 1737006)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[5]:
  time      : 2024-07-02_21:47:08
  host      : ip-26-0-160-225.ec2.internal
  rank      : 5 (local_rank: 5)
  exitcode  : 1 (pid: 1737007)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[6]:
  time      : 2024-07-02_21:47:08
  host      : ip-26-0-160-225.ec2.internal
  rank      : 6 (local_rank: 6)
  exitcode  : 1 (pid: 1737008)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[7]:
  time      : 2024-07-02_21:47:08
  host      : ip-26-0-160-225.ec2.internal
  rank      : 7 (local_rank: 7)
  exitcode  : 1 (pid: 1737009)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-02_21:47:08
  host      : ip-26-0-160-225.ec2.internal
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 1737002)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
srun: error: ip-26-0-160-225: task 0: Exited with exit code 1
W0702 21:47:12.033000 139984724735744 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-171-62.ec2.internal_3853684_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0702 21:47:12.278000 139673051064064 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-161-153.ec2.internal_1382157_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0702 21:47:12.547000 140391475324672 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-162-233.ec2.internal_1360550_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0702 21:47:12.582000 140463420364544 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-171-88.ec2.internal_843383_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0702 21:47:12.632000 139770468677376 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-161-103.ec2.internal_830490_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0702 21:47:12.709000 140041302849280 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-171-102.ec2.internal_3726035_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0702 21:47:12.875000 140100434835200 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-161-78.ec2.internal_1103218_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0702 21:47:13.014000 140046963582784 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3726114 closing signal SIGTERM
W0702 21:47:13.014000 140046963582784 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3726115 closing signal SIGTERM
W0702 21:47:13.014000 140046963582784 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3726116 closing signal SIGTERM
W0702 21:47:13.016000 140046963582784 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3726117 closing signal SIGTERM
W0702 21:47:13.016000 140046963582784 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3726118 closing signal SIGTERM
W0702 21:47:13.016000 140046963582784 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3726119 closing signal SIGTERM
W0702 21:47:13.018000 140046963582784 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3726120 closing signal SIGTERM
W0702 21:47:13.018000 140046963582784 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3726121 closing signal SIGTERM
W0702 21:47:13.029000 140397136058176 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1360628 closing signal SIGTERM
W0702 21:47:13.030000 140397136058176 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1360629 closing signal SIGTERM
W0702 21:47:13.030000 140397136058176 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1360630 closing signal SIGTERM
W0702 21:47:13.031000 140397136058176 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1360631 closing signal SIGTERM
W0702 21:47:13.031000 140397136058176 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1360632 closing signal SIGTERM
W0702 21:47:13.032000 140397136058176 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1360633 closing signal SIGTERM
W0702 21:47:13.032000 140397136058176 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1360634 closing signal SIGTERM
W0702 21:47:13.033000 140397136058176 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1360635 closing signal SIGTERM
W0702 21:47:13.046000 139990385469248 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3853764 closing signal SIGTERM
W0702 21:47:13.046000 139990385469248 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3853765 closing signal SIGTERM
W0702 21:47:13.047000 139990385469248 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3853766 closing signal SIGTERM
W0702 21:47:13.047000 139990385469248 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3853767 closing signal SIGTERM
W0702 21:47:13.048000 139990385469248 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3853768 closing signal SIGTERM
W0702 21:47:13.048000 139990385469248 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3853769 closing signal SIGTERM
W0702 21:47:13.049000 139990385469248 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3853770 closing signal SIGTERM
W0702 21:47:13.050000 139990385469248 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3853771 closing signal SIGTERM
E0702 21:47:13.163000 140106095568704 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 1103296) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10
E0702 21:47:13.169000 139776129410880 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 830568) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10
W0702 21:47:13.172000 140106095568704 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-161-78.ec2.internal_1103218_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
E0702 21:47:13.173000 139678711797568 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 1382236) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10
E0702 21:47:13.176000 140469081098048 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 843461) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10
W0702 21:47:13.175000 139776129410880 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-161-103.ec2.internal_830490_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0702 21:47:13.178000 139678711797568 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-161-153.ec2.internal_1382157_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0702 21:47:13.182000 140469081098048 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-171-88.ec2.internal_843383_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0702 21:47:13.200000 140106095568704 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-161-78.ec2.internal_1103218_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0702 21:47:13.205000 139678711797568 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-161-153.ec2.internal_1382157_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0702 21:47:13.203000 139776129410880 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-161-103.ec2.internal_830490_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0702 21:47:13.211000 140469081098048 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-171-88.ec2.internal_843383_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0702 21:47:13.228000 140106095568704 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-161-78.ec2.internal_1103218_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-161-78.ec2.internal
  rank      : 25 (local_rank: 1)
  exitcode  : 1 (pid: 1103297)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-161-78.ec2.internal
  rank      : 26 (local_rank: 2)
  exitcode  : 1 (pid: 1103298)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-161-78.ec2.internal
  rank      : 27 (local_rank: 3)
  exitcode  : 1 (pid: 1103299)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[4]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-161-78.ec2.internal
  rank      : 28 (local_rank: 4)
  exitcode  : 1 (pid: 1103300)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[5]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-161-78.ec2.internal
  rank      : 29 (local_rank: 5)
  exitcode  : 1 (pid: 1103301)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[6]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-161-78.ec2.internal
  rank      : 30 (local_rank: 6)
  exitcode  : 1 (pid: 1103302)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[7]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-161-78.ec2.internal
  rank      : 31 (local_rank: 7)
  exitcode  : 1 (pid: 1103303)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-161-78.ec2.internal
  rank      : 24 (local_rank: 0)
  exitcode  : 1 (pid: 1103296)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
W0702 21:47:13.233000 139678711797568 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-161-153.ec2.internal_1382157_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
W0702 21:47:13.235000 139776129410880 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-161-103.ec2.internal_830490_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-161-153.ec2.internal
  rank      : 17 (local_rank: 1)
  exitcode  : 1 (pid: 1382237)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-161-153.ec2.internal
  rank      : 18 (local_rank: 2)
  exitcode  : 1 (pid: 1382238)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-161-153.ec2.internal
  rank      : 19 (local_rank: 3)
  exitcode  : 1 (pid: 1382239)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[4]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-161-153.ec2.internal
  rank      : 20 (local_rank: 4)
  exitcode  : 1 (pid: 1382240)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[5]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-161-153.ec2.internal
  rank      : 21 (local_rank: 5)
  exitcode  : 1 (pid: 1382241)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[6]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-161-153.ec2.internal
  rank      : 22 (local_rank: 6)
  exitcode  : 1 (pid: 1382242)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[7]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-161-153.ec2.internal
  rank      : 23 (local_rank: 7)
  exitcode  : 1 (pid: 1382243)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-161-153.ec2.internal
  rank      : 16 (local_rank: 0)
  exitcode  : 1 (pid: 1382236)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-161-103.ec2.internal
  rank      : 9 (local_rank: 1)
  exitcode  : 1 (pid: 830569)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-161-103.ec2.internal
  rank      : 10 (local_rank: 2)
  exitcode  : 1 (pid: 830570)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-161-103.ec2.internal
  rank      : 11 (local_rank: 3)
  exitcode  : 1 (pid: 830571)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[4]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-161-103.ec2.internal
  rank      : 12 (local_rank: 4)
  exitcode  : 1 (pid: 830572)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[5]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-161-103.ec2.internal
  rank      : 13 (local_rank: 5)
  exitcode  : 1 (pid: 830573)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[6]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-161-103.ec2.internal
  rank      : 14 (local_rank: 6)
  exitcode  : 1 (pid: 830574)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[7]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-161-103.ec2.internal
  rank      : 15 (local_rank: 7)
  exitcode  : 1 (pid: 830575)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-161-103.ec2.internal
  rank      : 8 (local_rank: 0)
  exitcode  : 1 (pid: 830568)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
W0702 21:47:13.244000 140469081098048 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-171-88.ec2.internal_843383_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-171-88.ec2.internal
  rank      : 57 (local_rank: 1)
  exitcode  : 1 (pid: 843462)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-171-88.ec2.internal
  rank      : 58 (local_rank: 2)
  exitcode  : 1 (pid: 843463)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-171-88.ec2.internal
  rank      : 59 (local_rank: 3)
  exitcode  : 1 (pid: 843464)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[4]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-171-88.ec2.internal
  rank      : 60 (local_rank: 4)
  exitcode  : 1 (pid: 843465)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[5]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-171-88.ec2.internal
  rank      : 61 (local_rank: 5)
  exitcode  : 1 (pid: 843466)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[6]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-171-88.ec2.internal
  rank      : 62 (local_rank: 6)
  exitcode  : 1 (pid: 843467)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[7]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-171-88.ec2.internal
  rank      : 63 (local_rank: 7)
  exitcode  : 1 (pid: 843468)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-02_21:47:13
  host      : ip-26-0-171-88.ec2.internal
  rank      : 56 (local_rank: 0)
  exitcode  : 1 (pid: 843461)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
srun: error: ip-26-0-161-78: task 1: Exited with exit code 1
srun: error: ip-26-0-161-153: task 3: Exited with exit code 1
srun: error: ip-26-0-161-103: task 2: Exited with exit code 1
srun: error: ip-26-0-171-88: task 6: Exited with exit code 1
W0702 21:47:15.184000 140397136058176 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-162-233.ec2.internal_1360550_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0702 21:47:15.196000 140397136058176 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-162-233.ec2.internal_1360550_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 113, in _call_store
    return getattr(self._store, store_op)(*args, **kwargs)
torch.distributed.DistNetworkError: Broken pipe

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 254, in launch_agent
    result = agent.run()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 123, in wrapper
    result = f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 733, in run
    result = self._invoke_run(role)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 908, in _invoke_run
    num_nodes_waiting = rdzv_handler.num_nodes_waiting()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 1174, in num_nodes_waiting
    self._state_holder.sync()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 419, in sync
    get_response = self._backend.get_state()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 73, in get_state
    base64_state: bytes = self._call_store("get", self._key)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 115, in _call_store
    raise RendezvousConnectionError(
torch.distributed.elastic.rendezvous.api.RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details.
srun: error: ip-26-0-162-233: task 4: Exited with exit code 1
W0702 21:47:16.777000 140046963582784 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-171-102.ec2.internal_3726035_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0702 21:47:16.789000 140046963582784 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-171-102.ec2.internal_3726035_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 113, in _call_store
    return getattr(self._store, store_op)(*args, **kwargs)
torch.distributed.DistNetworkError: Broken pipe

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 254, in launch_agent
    result = agent.run()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 123, in wrapper
    result = f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 733, in run
    result = self._invoke_run(role)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 908, in _invoke_run
    num_nodes_waiting = rdzv_handler.num_nodes_waiting()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 1174, in num_nodes_waiting
    self._state_holder.sync()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 419, in sync
    get_response = self._backend.get_state()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 73, in get_state
    base64_state: bytes = self._call_store("get", self._key)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 115, in _call_store
    raise RendezvousConnectionError(
torch.distributed.elastic.rendezvous.api.RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details.
W0702 21:47:17.037000 139984724735744 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-171-62.ec2.internal_3853684_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
srun: error: ip-26-0-171-102: task 7: Exited with exit code 1
W0702 21:47:17.088000 139990385469248 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-171-62.ec2.internal_3853684_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0702 21:47:17.100000 139990385469248 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-171-62.ec2.internal_3853684_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 113, in _call_store
    return getattr(self._store, store_op)(*args, **kwargs)
torch.distributed.DistNetworkError: Broken pipe

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 254, in launch_agent
    result = agent.run()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 123, in wrapper
    result = f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 733, in run
    result = self._invoke_run(role)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 908, in _invoke_run
    num_nodes_waiting = rdzv_handler.num_nodes_waiting()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 1174, in num_nodes_waiting
    self._state_holder.sync()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 419, in sync
    get_response = self._backend.get_state()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 73, in get_state
    base64_state: bytes = self._call_store("get", self._key)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 115, in _call_store
    raise RendezvousConnectionError(
torch.distributed.elastic.rendezvous.api.RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details.
srun: error: ip-26-0-171-62: task 5: Exited with exit code 1
Consider using `hf_transfer` for faster uploads. This solution comes with some limitations. See https://huggingface.co/docs/huggingface_hub/hf_transfer for more details.