llama-1B/16_GPUS/dp-2_tp-4_pp-2_mbz-256/log.out

========================
START TIME: Tue Jul  2 16:30:22 UTC 2024
python3 version = Python 3.10.14
========================
The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well.
Token is valid (permission: write).
Your token has been saved to /admin/home/ferdinand_mom/.cache/huggingface/token
Login successful
Already on 'bench_cluster'
M	examples/config_tiny_llama.py
M	examples/config_tiny_llama.yaml
M	examples/train_tiny_llama.sh
M	src/nanotron/models/llama.py
M	src/nanotron/trainer.py
Your branch is up to date with 'origin/bench_cluster'.
Job status: RUNNING
W0702 16:30:24.760000 140513205032768 torch/distributed/run.py:757] 
W0702 16:30:24.760000 140513205032768 torch/distributed/run.py:757] *****************************************
W0702 16:30:24.760000 140513205032768 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0702 16:30:24.760000 140513205032768 torch/distributed/run.py:757] *****************************************
W0702 16:30:24.809000 140206204749632 torch/distributed/run.py:757] 
W0702 16:30:24.809000 140206204749632 torch/distributed/run.py:757] *****************************************
W0702 16:30:24.809000 140206204749632 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0702 16:30:24.809000 140206204749632 torch/distributed/run.py:757] *****************************************
[default0]:07/02/2024 16:30:42 [WARNING|DP=0|PP=0|TP=0|ip-26-0-171-62]: [Vocab Size Padding] Padded vocab (size: 50257) with 3 dummy tokens (new size: 50260)
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]: Config:
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]: Config(general=GeneralArgs(project='bench_cluster',
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                            run='%date_%jobid',
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                            seed=42,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                            step=None,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                            consumed_train_samples=None,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                            benchmark_csv_path=None,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                            ignore_sanity_checks=True),
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:        parallelism=ParallelismArgs(dp=2,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                    pp=2,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                    tp=4,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                    pp_engine=<nanotron.parallel.pipeline_parallel.engine.OneForwardOneBackwardPipelineEngine object at 0x7f62dae906a0>,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                    tp_mode=<TensorParallelLinearMode.REDUCE_SCATTER: 2>,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                    tp_linear_async_communication=False,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                    expert_parallel_size=1),
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:        model=ModelArgs(model_config=LlamaConfig(bos_token_id=1,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                 eos_token_id=2,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                 hidden_act='silu',
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                 hidden_size=2048,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                 initializer_range=0.02,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                 intermediate_size=4096,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                 is_llama_config=True,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                 max_position_embeddings=4096,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                 num_attention_heads=32,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                 num_hidden_layers=24,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                 num_key_value_heads=32,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                 pad_token_id=None,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                 pretraining_tp=1,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                 rms_norm_eps=1e-05,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                 rope_scaling=None,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                 rope_theta=10000.0,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                 tie_word_embeddings=True,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                 use_cache=True,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                 vocab_size=50260),
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                        init_method=RandomInit(std=0.025),
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                        dtype=torch.bfloat16,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                        make_vocab_size_divisible_by=1,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                        ddp_bucket_cap_mb=25),
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:        tokenizer=TokenizerArgs(tokenizer_name_or_path='openai-community/gpt2',
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                tokenizer_revision=None,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                tokenizer_max_length=None),
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:        checkpoints=CheckpointsArgs(checkpoints_path=Path('/dev/null'),
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                    checkpoint_interval=100000,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                    save_initial_state=False,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                    resume_checkpoint_path=None,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                    checkpoints_path_is_shared_file_system=False),
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:        logging=LoggingArgs(log_level='info',
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                            log_level_replica='info',
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                            iteration_step_info_interval=1),
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:        tokens=TokensArgs(sequence_length=4096,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                          train_steps=20,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                          micro_batch_size=256,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                          batch_accumulation_per_replica=2,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                          val_check_interval=-1,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                          limit_val_batches=0,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                          limit_test_batches=0),
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:        optimizer=OptimizerArgs(optimizer_factory=AdamWOptimizerArgs(adam_eps=1e-08,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                                     adam_beta1=0.9,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                                     adam_beta2=0.95,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                                     torch_adam_is_fused=True,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                                     name='adamW'),
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                zero_stage=1,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                weight_decay=0.01,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                clip_grad=1.0,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                accumulate_grad_in_fp32=True,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                learning_rate_scheduler=LRSchedulerArgs(learning_rate=0.0001,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                                        lr_warmup_steps=1,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                                        lr_warmup_style='linear',
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                                        lr_decay_style='linear',
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                                        lr_decay_steps=19,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                                        lr_decay_starting_step=None,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                                        min_decay_lr=1e-05)),
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:        data_stages=[DatasetStageArgs(name='Training Stage',
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                      start_training_step=1,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                      data=DataArgs(dataset=PretrainDatasetsArgs(hf_dataset_or_datasets='roneneldan/TinyStories',
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                                                 hf_dataset_splits='train',
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                                                 hf_dataset_config_name=None,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                                                 dataset_processing_num_proc_per_process=64,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                                                 dataset_overwrite_cache=False,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                                                 text_column_name='text'),
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                    seed=42,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:                                                    num_loading_workers=32))],
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:        profiler=ProfilerArgs(profiler_export_path=Path('/fsx/ferdinandmom/ferdinand-hf/bench_cluster/results/llama-1B/16_GPUS/dp-2_tp-4_pp-2_mbz-256')),
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:        lighteval=None)
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]: Model Config:
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]: LlamaConfig(bos_token_id=1,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:             eos_token_id=2,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:             hidden_act='silu',
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:             hidden_size=2048,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:             initializer_range=0.02,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:             intermediate_size=4096,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:             is_llama_config=True,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:             max_position_embeddings=4096,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:             num_attention_heads=32,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:             num_hidden_layers=24,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:             num_key_value_heads=32,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:             pad_token_id=None,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:             pretraining_tp=1,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:             rms_norm_eps=1e-05,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:             rope_scaling=None,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:             rope_theta=10000.0,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:             tie_word_embeddings=True,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:             use_cache=True,
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:             vocab_size=50260)
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]: Building model..
[default0]:07/02/2024 16:30:42 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]: Setting PP block ranks...
[default2]:07/02/2024 16:30:56 [INFO|DP=0|PP=1|TP=2|ip-26-0-171-88]: Local number of parameters: 131M (249.16MiB)
[default2]:07/02/2024 16:30:56 [INFO|DP=0|PP=1|TP=2|ip-26-0-171-88]: [After model building] Memory usage: 260.10MiB. Peak allocated: 262.13MiB Peak reserved: 264.00MiB
[default2]:07/02/2024 16:30:56 [INFO|DP=0|PP=1|TP=2|ip-26-0-171-88]: No checkpoint path provided.
[default1]:07/02/2024 16:30:56 [INFO|DP=0|PP=1|TP=1|ip-26-0-171-88]: Local number of parameters: 131M (249.16MiB)
[default1]:07/02/2024 16:30:56 [INFO|DP=0|PP=1|TP=1|ip-26-0-171-88]: [After model building] Memory usage: 260.10MiB. Peak allocated: 262.13MiB Peak reserved: 264.00MiB
[default1]:07/02/2024 16:30:56 [INFO|DP=0|PP=1|TP=1|ip-26-0-171-88]: No checkpoint path provided.
[default2]:07/02/2024 16:30:56 [INFO|DP=0|PP=0|TP=2|ip-26-0-171-62]: Local number of parameters: 173M (329.19MiB)
[default2]:07/02/2024 16:30:56 [INFO|DP=0|PP=0|TP=2|ip-26-0-171-62]: [After model building] Memory usage: 344.13MiB. Peak allocated: 346.16MiB Peak reserved: 348.00MiB
[default2]:07/02/2024 16:30:56 [INFO|DP=0|PP=0|TP=2|ip-26-0-171-62]: No checkpoint path provided.
[default3]:07/02/2024 16:30:56 [INFO|DP=0|PP=1|TP=3|ip-26-0-171-88]: Local number of parameters: 131M (249.16MiB)
[default3]:07/02/2024 16:30:56 [INFO|DP=0|PP=1|TP=3|ip-26-0-171-88]: [After model building] Memory usage: 260.10MiB. Peak allocated: 262.13MiB Peak reserved: 264.00MiB
[default3]:07/02/2024 16:30:56 [INFO|DP=0|PP=1|TP=3|ip-26-0-171-88]: No checkpoint path provided.
[default0]:07/02/2024 16:30:56 [INFO|DP=0|PP=1|TP=0|ip-26-0-171-88]: Local number of parameters: 131M (249.16MiB)
[default0]:07/02/2024 16:30:56 [INFO|DP=0|PP=1|TP=0|ip-26-0-171-88]: [After model building] Memory usage: 260.10MiB. Peak allocated: 262.13MiB Peak reserved: 264.00MiB
[default0]:07/02/2024 16:30:56 [INFO|DP=0|PP=1|TP=0|ip-26-0-171-88]: No checkpoint path provided.
[default3]:07/02/2024 16:30:56 [INFO|DP=0|PP=0|TP=3|ip-26-0-171-62]: Local number of parameters: 173M (329.19MiB)
[default3]:07/02/2024 16:30:56 [INFO|DP=0|PP=0|TP=3|ip-26-0-171-62]: [After model building] Memory usage: 344.13MiB. Peak allocated: 346.16MiB Peak reserved: 348.00MiB
[default3]:07/02/2024 16:30:56 [INFO|DP=0|PP=0|TP=3|ip-26-0-171-62]: No checkpoint path provided.
[default1]:07/02/2024 16:30:56 [INFO|DP=0|PP=0|TP=1|ip-26-0-171-62]: Local number of parameters: 173M (329.19MiB)
[default1]:07/02/2024 16:30:56 [INFO|DP=0|PP=0|TP=1|ip-26-0-171-62]: [After model building] Memory usage: 344.13MiB. Peak allocated: 346.16MiB Peak reserved: 348.00MiB
[default1]:07/02/2024 16:30:56 [INFO|DP=0|PP=0|TP=1|ip-26-0-171-62]: No checkpoint path provided.
[default0]:07/02/2024 16:30:56 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]: Total number of parameters: 1.21G (2313.42MiB)
[default0]:07/02/2024 16:30:56 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]: Local number of parameters: 173M (329.19MiB)
[default0]:07/02/2024 16:30:56 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]: [After model building] Memory usage: 344.13MiB. Peak allocated: 346.16MiB Peak reserved: 348.00MiB
[default0]:07/02/2024 16:30:56 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]: No checkpoint path provided.
[default0]:07/02/2024 16:30:56 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]: Parametrizing model parameters using StandardParametrizator
[default6]:07/02/2024 16:30:56 [INFO|DP=1|PP=1|TP=2|ip-26-0-171-88]: No checkpoint path provided.
[default5]:07/02/2024 16:30:56 [INFO|DP=1|PP=1|TP=1|ip-26-0-171-88]: No checkpoint path provided.
[default7]:07/02/2024 16:30:56 [INFO|DP=1|PP=1|TP=3|ip-26-0-171-88]: No checkpoint path provided.
[default4]:07/02/2024 16:30:56 [INFO|DP=1|PP=1|TP=0|ip-26-0-171-88]: No checkpoint path provided.
[default6]:07/02/2024 16:30:56 [INFO|DP=1|PP=0|TP=2|ip-26-0-171-62]: No checkpoint path provided.
[default4]:07/02/2024 16:30:56 [INFO|DP=1|PP=0|TP=0|ip-26-0-171-62]: No checkpoint path provided.
[default5]:07/02/2024 16:30:56 [INFO|DP=1|PP=0|TP=1|ip-26-0-171-62]: No checkpoint path provided.
[default7]:07/02/2024 16:30:56 [INFO|DP=1|PP=0|TP=3|ip-26-0-171-62]: No checkpoint path provided.
[default0]:07/02/2024 16:30:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]: [Optimizer Building] Using LearningRateForSP as learning rate
[default0]:07/02/2024 16:30:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]: [ZeRO sharding] Size of optimizer params per rank:
[default0]:07/02/2024 16:30:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]: [ZeRO sharding] DP Rank 0 has 86.3M out of 173M (50.00%) params' optimizer states
[default0]:07/02/2024 16:30:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]: [ZeRO sharding] DP Rank 1 has 86.3M out of 173M (50.00%) params' optimizer states
[default0]:07/02/2024 16:31:00 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]: [Training Plan] Stage Training Stage has 19 remaining training steps and has consumed 0 samples
[default0]:07/02/2024 16:31:00 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]: Using `datasets` library
[default0]:07/02/2024 16:31:00 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]: Loading tokenizer from openai-community/gpt2 and transformers/hf_hub versions ('4.41.2', '0.23.4')
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/02/2024 16:31:00 [WARNING|DP=0|PP=0|TP=0|ip-26-0-171-62]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/02/2024 16:31:01 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]: [Training Plan] There are 1 training stages 
[default0]:07/02/2024 16:31:01 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]: [Stage Training Stage] start from step 1 
[default0]:07/02/2024 16:31:01 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]: 
[default0]:07/02/2024 16:31:01 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]: [Start training] datetime: 2024-07-02 16:31:01.652103 | mbs: 256 | grad_accum: 2 | global_batch_size: 1024 | sequence_length: 4096 | train_steps: 20 | start_iteration_step: 0 | consumed_train_samples: 0
[default0]:07/02/2024 16:31:01 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]: Resuming training from stage Training Stage, it has trained for 0 samples and has 19 remaining train steps
[default0]:07/02/2024 16:31:01 [INFO|DP=0|PP=0|TP=0|ip-26-0-171-62]:  Memory usage: 1332.51MiB. Peak allocated 1332.51MiB. Peak reserved: 1338.00MiB
[default6]:07/02/2024 16:31:01 [WARNING|DP=1|PP=1|TP=2|ip-26-0-171-88]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/02/2024 16:31:01 [WARNING|DP=0|PP=1|TP=2|ip-26-0-171-88]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/02/2024 16:31:01 [WARNING|DP=1|PP=1|TP=3|ip-26-0-171-88]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/02/2024 16:31:01 [WARNING|DP=1|PP=1|TP=1|ip-26-0-171-88]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/02/2024 16:31:01 [WARNING|DP=0|PP=1|TP=1|ip-26-0-171-88]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/02/2024 16:31:01 [WARNING|DP=0|PP=1|TP=3|ip-26-0-171-88]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/02/2024 16:31:01 [WARNING|DP=0|PP=0|TP=2|ip-26-0-171-62]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/02/2024 16:31:01 [WARNING|DP=0|PP=1|TP=0|ip-26-0-171-88]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/02/2024 16:31:01 [WARNING|DP=1|PP=1|TP=0|ip-26-0-171-88]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/02/2024 16:31:01 [WARNING|DP=0|PP=0|TP=1|ip-26-0-171-62]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/02/2024 16:31:01 [WARNING|DP=1|PP=0|TP=2|ip-26-0-171-62]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:07/02/2024 16:31:01 [WARNING|DP=1|PP=0|TP=1|ip-26-0-171-62]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/02/2024 16:31:01 [WARNING|DP=1|PP=0|TP=3|ip-26-0-171-62]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/02/2024 16:31:01 [WARNING|DP=1|PP=0|TP=0|ip-26-0-171-62]: Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/02/2024 16:31:01 [WARNING|DP=0|PP=0|TP=3|ip-26-0-171-62]: Repo card metadata block was not found. Setting CardData to empty.
[default5]:[rank5]: OSError: [Errno 122] Disk quota exceeded
[default5]:
[default5]:[rank5]: During handling of the above exception, another exception occurred:
[default5]:
[default5]:[rank5]: Traceback (most recent call last):
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default5]:[rank5]:     trainer.train(dataloader)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default5]:[rank5]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default5]:[rank5]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default5]:[rank5]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default5]:[rank5]:     output = model(**micro_batch)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank5]:     return self._call_impl(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank5]:     return forward_call(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default5]:[rank5]:     sharded_logits = self.model(
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank5]:     return self._call_impl(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank5]:     return forward_call(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default5]:[rank5]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default5]:[rank5]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank5]:     return self._call_impl(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank5]:     return forward_call(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default5]:[rank5]:     output = self.pp_block(**new_kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank5]:     return self._call_impl(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank5]:     return forward_call(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 629, in forward
[default5]:[rank5]:     hidden_states = self.input_layernorm(hidden_states)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank5]:     return self._call_impl(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank5]:     return forward_call(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/nn/layer_norm.py", line 42, in forward
[default5]:[rank5]:     return layer_norm_fn(
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py", line 875, in layer_norm_fn
[default5]:[rank5]:     return LayerNormFn.apply(
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/function.py", line 598, in apply
[default5]:[rank5]:     return super().apply(*args, **kwargs)  # type: ignore[misc]
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py", line 748, in forward
[default5]:[rank5]:     y, y1, mean, rstd, residual_out, seeds, dropout_mask, dropout_mask1 = _layer_norm_fwd(
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py", line 335, in _layer_norm_fwd
[default5]:[rank5]:     _layer_norm_fwd_1pass_kernel[(M,)](
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/jit.py", line 167, in <lambda>
[default5]:[rank5]:     return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 143, in run
[default5]:[rank5]:     timings = {config: self._bench(*args, config=config, **kwargs) for config in pruned_configs}
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 143, in <dictcomp>
[default5]:[rank5]:     timings = {config: self._bench(*args, config=config, **kwargs) for config in pruned_configs}
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 122, in _bench
[default5]:[rank5]:     return do_bench(kernel_call, warmup=self.warmup, rep=self.rep, quantiles=(0.5, 0.2, 0.8))
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/testing.py", line 102, in do_bench
[default5]:[rank5]:     fn()
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 110, in kernel_call
[default5]:[rank5]:     self.fn.run(
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 305, in run
[default5]:[rank5]:     return self.fn.run(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 305, in run
[default5]:[rank5]:     return self.fn.run(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 305, in run
[default5]:[rank5]:     return self.fn.run(*args, **kwargs)
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/jit.py", line 416, in run
[default5]:[rank5]:     self.cache[device][key] = compile(
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/compiler/compiler.py", line 194, in compile
[default5]:[rank5]:     metadata_group[f"{src.name}.{ext}"] = fn_cache_manager.put(next_module, f"{src.name}.{ext}")
[default5]:[rank5]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/cache.py", line 123, in put
[default5]:[rank5]:     with open(temp_path, mode) as f:
[default5]:[rank5]: OSError: [Errno 122] Disk quota exceeded
[default5]:Exception in thread Thread-2 (_pin_memory_loop):
[default5]:Traceback (most recent call last):
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
[default5]:    self.run()
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/threading.py", line 953, in run
[default5]:    self._target(*self._args, **self._kwargs)
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 54, in _pin_memory_loop
[default5]:    do_one_step()
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 31, in do_one_step
[default5]:    r = in_queue.get(timeout=MP_STATUS_CHECK_INTERVAL)
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/queues.py", line 122, in get
[default5]:    return _ForkingPickler.loads(res)
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/multiprocessing/reductions.py", line 495, in rebuild_storage_fd
[default5]:    fd = df.detach()
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/resource_sharer.py", line 57, in detach
[default5]:    with _resource_sharer.get_connection(self._id) as conn:
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/resource_sharer.py", line 86, in get_connection
[default5]:    c = Client(address, authkey=process.current_process().authkey)
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 508, in Client
[default5]:    answer_challenge(c, authkey)
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 752, in answer_challenge
[default5]:    message = connection.recv_bytes(256)         # reject large message
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes
[default5]:    buf = self._recv_bytes(maxlength)
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
[default5]:    buf = self._recv(4)
[default5]:  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
[default5]:    chunk = read(handle, remaining)
[default5]:ConnectionResetError: [Errno 104] Connection reset by peer
[default2]:[rank2]: OSError: [Errno 122] Disk quota exceeded
[default2]:
[default2]:[rank2]: During handling of the above exception, another exception occurred:
[default2]:
[default2]:[rank2]: Traceback (most recent call last):
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default2]:[rank2]:     trainer.train(dataloader)
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default2]:[rank2]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default2]:[rank2]:     outputs = self.pipeline_engine.train_batch_iter(
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter
[default2]:[rank2]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default2]:[rank2]:     output = model(**micro_batch)
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank2]:     return self._call_impl(*args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank2]:     return forward_call(*args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default2]:[rank2]:     sharded_logits = self.model(
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank2]:     return self._call_impl(*args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank2]:     return forward_call(*args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default2]:[rank2]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default2]:[rank2]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank2]:     return self._call_impl(*args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank2]:     return forward_call(*args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward
[default2]:[rank2]:     output = self.pp_block(**new_kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank2]:     return self._call_impl(*args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank2]:     return forward_call(*args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward
[default2]:[rank2]:     output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask)
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank2]:     return self._call_impl(*args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank2]:     return forward_call(*args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 566, in forward
[default2]:[rank2]:     query_states, key_value_states = self.flash_rotary_embedding(query_states, kv=key_value_states)
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank2]:     return self._call_impl(*args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank2]:     return forward_call(*args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/layers/rotary.py", line 457, in forward
[default2]:[rank2]:     q = apply_rotary_emb_func(
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/layers/rotary.py", line 122, in apply_rotary_emb
[default2]:[rank2]:     return ApplyRotaryEmb.apply(
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/function.py", line 598, in apply
[default2]:[rank2]:     return super().apply(*args, **kwargs)  # type: ignore[misc]
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/layers/rotary.py", line 48, in forward
[default2]:[rank2]:     out = apply_rotary(
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/rotary.py", line 202, in apply_rotary
[default2]:[rank2]:     rotary_kernel[grid](
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/jit.py", line 167, in <lambda>
[default2]:[rank2]:     return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/jit.py", line 416, in run
[default2]:[rank2]:     self.cache[device][key] = compile(
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/compiler/compiler.py", line 194, in compile
[default2]:[rank2]:     metadata_group[f"{src.name}.{ext}"] = fn_cache_manager.put(next_module, f"{src.name}.{ext}")
[default2]:[rank2]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/cache.py", line 123, in put
[default2]:[rank2]:     with open(temp_path, mode) as f:
[default2]:[rank2]: OSError: [Errno 122] Disk quota exceeded
W0702 16:31:10.986000 140513205032768 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3391098 closing signal SIGTERM
W0702 16:31:10.991000 140513205032768 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3391099 closing signal SIGTERM
W0702 16:31:10.992000 140513205032768 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3391101 closing signal SIGTERM
W0702 16:31:10.998000 140513205032768 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3391102 closing signal SIGTERM
W0702 16:31:11.004000 140513205032768 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3391104 closing signal SIGTERM
W0702 16:31:11.034000 140513205032768 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3391105 closing signal SIGTERM
[default4]:[rank12]: Traceback (most recent call last):
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default4]:[rank12]:     trainer.train(dataloader)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default4]:[rank12]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default4]:[rank12]:     outputs = self.pipeline_engine.train_batch_iter(
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default4]:[rank12]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default4]:[rank12]:     output = model(**micro_batch)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank12]:     return self._call_impl(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank12]:     return forward_call(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default4]:[rank12]:     sharded_logits = self.model(
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank12]:     return self._call_impl(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank12]:     return forward_call(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default4]:[rank12]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default4]:[rank12]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default4]:[rank12]:     return self._call_impl(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default4]:[rank12]:     return forward_call(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default4]:[rank12]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default4]:[rank12]:     pipeline_state.run_communication()
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default4]:[rank12]:     recv_activation_tensor = recv_activation()
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default4]:[rank12]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default4]:[rank12]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default4]:[rank12]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default4]:[rank12]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default4]:[rank12]:     dist.recv(
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default4]:[rank12]:     return func(*args, **kwargs)
[default4]:[rank12]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default4]:[rank12]:     pg.recv([tensor], group_src_rank, tag).wait()
[default4]:[rank12]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default4]:[rank12]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default4]:[rank12]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f6869007897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default4]:[rank12]: frame #1: <unknown function> + 0x5b3a23e (0x7f68a2b2423e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank12]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f68a2b1ec87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank12]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f68a2b1ef82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank12]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f68a2b1ffd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank12]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f68a2ad4371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank12]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f68a2ad4371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank12]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f68a2ad4371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank12]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f68a2ad4371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank12]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f686a2e1189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank12]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f686a2e8610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank12]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f686a307978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default4]:[rank12]: frame #12: <unknown function> + 0x5adc309 (0x7f68a2ac6309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank12]: frame #13: <unknown function> + 0x5ae6f10 (0x7f68a2ad0f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank12]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f68a2ad0fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank12]: frame #15: <unknown function> + 0x5124446 (0x7f68a210e446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank12]: frame #16: <unknown function> + 0x1acf4b8 (0x7f689eab94b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank12]: frame #17: <unknown function> + 0x5aee004 (0x7f68a2ad8004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank12]: frame #18: <unknown function> + 0x5af36b5 (0x7f68a2add6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default4]:[rank12]: frame #19: <unknown function> + 0xd2631e (0x7f68b56c731e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:[rank12]: frame #20: <unknown function> + 0x47def4 (0x7f68b4e1eef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default4]:[rank12]: frame #21: <unknown function> + 0x1445a6 (0x564f64b315a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #22: _PyObject_MakeTpCall + 0x26b (0x564f64b2aa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #23: <unknown function> + 0x150866 (0x564f64b3d866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x564f64b26142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #25: _PyFunction_Vectorcall + 0x6c (0x564f64b31a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #26: PyObject_Call + 0xbc (0x564f64b3df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x564f64b242b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #28: _PyFunction_Vectorcall + 0x6c (0x564f64b31a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x564f64b228fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #30: <unknown function> + 0x150582 (0x564f64b3d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x564f64b228fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #32: <unknown function> + 0x150582 (0x564f64b3d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x564f64b228fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #34: <unknown function> + 0x150582 (0x564f64b3d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x564f64b228fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x564f64b29f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #37: _PyObject_Call_Prepend + 0x69 (0x564f64b3bc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #38: <unknown function> + 0x211239 (0x564f64bfe239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #39: _PyObject_MakeTpCall + 0x26b (0x564f64b2aa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x564f64b263e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #41: _PyFunction_Vectorcall + 0x6c (0x564f64b31a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x564f64b21c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #43: _PyFunction_Vectorcall + 0x6c (0x564f64b31a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x564f64b228fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #45: <unknown function> + 0x150582 (0x564f64b3d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #46: PyObject_Call + 0xbc (0x564f64b3df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x564f64b242b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #48: <unknown function> + 0x150582 (0x564f64b3d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #49: PyObject_Call + 0xbc (0x564f64b3df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x564f64b242b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #51: _PyFunction_Vectorcall + 0x6c (0x564f64b31a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x564f64b2a007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #53: _PyObject_Call_Prepend + 0x69 (0x564f64b3bc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #54: <unknown function> + 0x211239 (0x564f64bfe239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #55: PyObject_Call + 0x207 (0x564f64b3e067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x564f64b242b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #57: <unknown function> + 0x150582 (0x564f64b3d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x564f64b228fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #59: <unknown function> + 0x150582 (0x564f64b3d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #60: PyObject_Call + 0xbc (0x564f64b3df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x564f64b242b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #62: <unknown function> + 0x150582 (0x564f64b3d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: frame #63: PyObject_Call + 0xbc (0x564f64b3df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default4]:[rank12]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default5]:[rank13]: Traceback (most recent call last):
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default5]:[rank13]:     trainer.train(dataloader)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default5]:[rank13]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default5]:[rank13]:     outputs = self.pipeline_engine.train_batch_iter(
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default5]:[rank13]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default5]:[rank13]:     output = model(**micro_batch)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank13]:     return self._call_impl(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank13]:     return forward_call(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default5]:[rank13]:     sharded_logits = self.model(
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank13]:     return self._call_impl(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank13]:     return forward_call(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default5]:[rank13]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default5]:[rank13]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default5]:[rank13]:     return self._call_impl(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default5]:[rank13]:     return forward_call(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default5]:[rank13]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default5]:[rank13]:     pipeline_state.run_communication()
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default5]:[rank13]:     recv_activation_tensor = recv_activation()
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default5]:[rank13]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default5]:[rank13]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default5]:[rank13]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default5]:[rank13]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default5]:[rank13]:     dist.recv(
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default5]:[rank13]:     return func(*args, **kwargs)
[default5]:[rank13]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default5]:[rank13]:     pg.recv([tensor], group_src_rank, tag).wait()
[default5]:[rank13]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default5]:[rank13]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default5]:[rank13]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f5085367897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default5]:[rank13]: frame #1: <unknown function> + 0x5b3a23e (0x7f50bee8423e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank13]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f50bee7ec87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank13]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f50bee7ef82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank13]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f50bee7ffd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank13]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f50bee34371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank13]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f50bee34371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank13]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f50bee34371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank13]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f50bee34371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank13]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f5086641189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank13]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f5086648610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank13]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f5086667978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default5]:[rank13]: frame #12: <unknown function> + 0x5adc309 (0x7f50bee26309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank13]: frame #13: <unknown function> + 0x5ae6f10 (0x7f50bee30f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank13]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f50bee30fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank13]: frame #15: <unknown function> + 0x5124446 (0x7f50be46e446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank13]: frame #16: <unknown function> + 0x1acf4b8 (0x7f50bae194b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank13]: frame #17: <unknown function> + 0x5aee004 (0x7f50bee38004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank13]: frame #18: <unknown function> + 0x5af36b5 (0x7f50bee3d6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default5]:[rank13]: frame #19: <unknown function> + 0xd2631e (0x7f50d1a2731e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default5]:[rank13]: frame #20: <unknown function> + 0x47def4 (0x7f50d117eef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default5]:[rank13]: frame #21: <unknown function> + 0x1445a6 (0x5634866255a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #22: _PyObject_MakeTpCall + 0x26b (0x56348661ea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #23: <unknown function> + 0x150866 (0x563486631866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x56348661a142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #25: _PyFunction_Vectorcall + 0x6c (0x563486625a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #26: PyObject_Call + 0xbc (0x563486631f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5634866182b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #28: _PyFunction_Vectorcall + 0x6c (0x563486625a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5634866168fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #30: <unknown function> + 0x150582 (0x563486631582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5634866168fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #32: <unknown function> + 0x150582 (0x563486631582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5634866168fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #34: <unknown function> + 0x150582 (0x563486631582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5634866168fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x56348661df50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #37: _PyObject_Call_Prepend + 0x69 (0x56348662fc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #38: <unknown function> + 0x211239 (0x5634866f2239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #39: _PyObject_MakeTpCall + 0x26b (0x56348661ea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x56348661a3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #41: _PyFunction_Vectorcall + 0x6c (0x563486625a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x563486615c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #43: _PyFunction_Vectorcall + 0x6c (0x563486625a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5634866168fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #45: <unknown function> + 0x150582 (0x563486631582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #46: PyObject_Call + 0xbc (0x563486631f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5634866182b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #48: <unknown function> + 0x150582 (0x563486631582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #49: PyObject_Call + 0xbc (0x563486631f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5634866182b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #51: _PyFunction_Vectorcall + 0x6c (0x563486625a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x56348661e007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #53: _PyObject_Call_Prepend + 0x69 (0x56348662fc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #54: <unknown function> + 0x211239 (0x5634866f2239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #55: PyObject_Call + 0x207 (0x563486632067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5634866182b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #57: <unknown function> + 0x150582 (0x563486631582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5634866168fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #59: <unknown function> + 0x150582 (0x563486631582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #60: PyObject_Call + 0xbc (0x563486631f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5634866182b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #62: <unknown function> + 0x150582 (0x563486631582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: frame #63: PyObject_Call + 0xbc (0x563486631f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default5]:[rank13]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default0]:[rank8]: Traceback (most recent call last):
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default0]:[rank8]:     trainer.train(dataloader)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default0]:[rank8]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default0]:[rank8]:     outputs = self.pipeline_engine.train_batch_iter(
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default0]:[rank8]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default0]:[rank8]:     output = model(**micro_batch)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank8]:     return self._call_impl(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank8]:     return forward_call(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default0]:[rank8]:     sharded_logits = self.model(
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank8]:     return self._call_impl(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank8]:     return forward_call(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default0]:[rank8]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default0]:[rank8]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default0]:[rank8]:     return self._call_impl(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default0]:[rank8]:     return forward_call(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default0]:[rank8]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default0]:[rank8]:     pipeline_state.run_communication()
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default0]:[rank8]:     recv_activation_tensor = recv_activation()
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default0]:[rank8]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default0]:[rank8]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default0]:[rank8]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default0]:[rank8]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default0]:[rank8]:     dist.recv(
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default0]:[rank8]:     return func(*args, **kwargs)
[default0]:[rank8]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default0]:[rank8]:     pg.recv([tensor], group_src_rank, tag).wait()
[default0]:[rank8]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default0]:[rank8]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default0]:[rank8]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f8909d17897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default0]:[rank8]: frame #1: <unknown function> + 0x5b3a23e (0x7f894383423e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank8]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f894382ec87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank8]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f894382ef82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank8]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f894382ffd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank8]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f89437e4371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank8]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f89437e4371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank8]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f89437e4371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank8]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f89437e4371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank8]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f890aff1189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank8]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f890aff8610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank8]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f890b017978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default0]:[rank8]: frame #12: <unknown function> + 0x5adc309 (0x7f89437d6309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank8]: frame #13: <unknown function> + 0x5ae6f10 (0x7f89437e0f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank8]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f89437e0fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank8]: frame #15: <unknown function> + 0x5124446 (0x7f8942e1e446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank8]: frame #16: <unknown function> + 0x1acf4b8 (0x7f893f7c94b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank8]: frame #17: <unknown function> + 0x5aee004 (0x7f89437e8004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank8]: frame #18: <unknown function> + 0x5af36b5 (0x7f89437ed6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default0]:[rank8]: frame #19: <unknown function> + 0xd2631e (0x7f89563d731e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default0]:[rank8]: frame #20: <unknown function> + 0x47def4 (0x7f8955b2eef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default0]:[rank8]: frame #21: <unknown function> + 0x1445a6 (0x55b6fcee45a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55b6fcedda6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #23: <unknown function> + 0x150866 (0x55b6fcef0866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55b6fced9142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55b6fcee4a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #26: PyObject_Call + 0xbc (0x55b6fcef0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55b6fced72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55b6fcee4a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55b6fced58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #30: <unknown function> + 0x150582 (0x55b6fcef0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55b6fced58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #32: <unknown function> + 0x150582 (0x55b6fcef0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55b6fced58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #34: <unknown function> + 0x150582 (0x55b6fcef0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55b6fced58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55b6fcedcf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55b6fceeec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #38: <unknown function> + 0x211239 (0x55b6fcfb1239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55b6fcedda6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55b6fced93e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55b6fcee4a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55b6fced4c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55b6fcee4a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55b6fced58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #45: <unknown function> + 0x150582 (0x55b6fcef0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #46: PyObject_Call + 0xbc (0x55b6fcef0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55b6fced72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #48: <unknown function> + 0x150582 (0x55b6fcef0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #49: PyObject_Call + 0xbc (0x55b6fcef0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55b6fced72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55b6fcee4a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55b6fcedd007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55b6fceeec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #54: <unknown function> + 0x211239 (0x55b6fcfb1239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #55: PyObject_Call + 0x207 (0x55b6fcef1067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55b6fced72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #57: <unknown function> + 0x150582 (0x55b6fcef0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55b6fced58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #59: <unknown function> + 0x150582 (0x55b6fcef0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #60: PyObject_Call + 0xbc (0x55b6fcef0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55b6fced72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #62: <unknown function> + 0x150582 (0x55b6fcef0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: frame #63: PyObject_Call + 0xbc (0x55b6fcef0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default0]:[rank8]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default1]:[rank9]: Traceback (most recent call last):
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default1]:[rank9]:     trainer.train(dataloader)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default1]:[rank9]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default1]:[rank9]:     outputs = self.pipeline_engine.train_batch_iter(
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default1]:[rank9]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default1]:[rank9]:     output = model(**micro_batch)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank9]:     return self._call_impl(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank9]:     return forward_call(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default1]:[rank9]:     sharded_logits = self.model(
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank9]:     return self._call_impl(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank9]:     return forward_call(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default1]:[rank9]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default1]:[rank9]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default1]:[rank9]:     return self._call_impl(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default1]:[rank9]:     return forward_call(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default1]:[rank9]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default1]:[rank9]:     pipeline_state.run_communication()
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default1]:[rank9]:     recv_activation_tensor = recv_activation()
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default1]:[rank9]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default1]:[rank9]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default1]:[rank9]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default1]:[rank9]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default1]:[rank9]:     dist.recv(
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default1]:[rank9]:     return func(*args, **kwargs)
[default1]:[rank9]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default1]:[rank9]:     pg.recv([tensor], group_src_rank, tag).wait()
[default1]:[rank9]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default1]:[rank9]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default1]:[rank9]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f539b7cf897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default1]:[rank9]: frame #1: <unknown function> + 0x5b3a23e (0x7f53d52ec23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank9]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f53d52e6c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank9]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f53d52e6f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank9]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f53d52e7fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank9]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f53d529c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank9]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f53d529c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank9]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f53d529c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank9]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f53d529c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank9]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f539caa9189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank9]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f539cab0610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank9]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f539cacf978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default1]:[rank9]: frame #12: <unknown function> + 0x5adc309 (0x7f53d528e309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank9]: frame #13: <unknown function> + 0x5ae6f10 (0x7f53d5298f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank9]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f53d5298fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank9]: frame #15: <unknown function> + 0x5124446 (0x7f53d48d6446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank9]: frame #16: <unknown function> + 0x1acf4b8 (0x7f53d12814b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank9]: frame #17: <unknown function> + 0x5aee004 (0x7f53d52a0004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank9]: frame #18: <unknown function> + 0x5af36b5 (0x7f53d52a56b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default1]:[rank9]: frame #19: <unknown function> + 0xd2631e (0x7f53e7e8f31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default1]:[rank9]: frame #20: <unknown function> + 0x47def4 (0x7f53e75e6ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default1]:[rank9]: frame #21: <unknown function> + 0x1445a6 (0x5624506635a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #22: _PyObject_MakeTpCall + 0x26b (0x56245065ca6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #23: <unknown function> + 0x150866 (0x56245066f866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x562450658142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #25: _PyFunction_Vectorcall + 0x6c (0x562450663a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #26: PyObject_Call + 0xbc (0x56245066ff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5624506562b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #28: _PyFunction_Vectorcall + 0x6c (0x562450663a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5624506548fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #30: <unknown function> + 0x150582 (0x56245066f582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5624506548fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #32: <unknown function> + 0x150582 (0x56245066f582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5624506548fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #34: <unknown function> + 0x150582 (0x56245066f582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5624506548fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x56245065bf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #37: _PyObject_Call_Prepend + 0x69 (0x56245066dc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #38: <unknown function> + 0x211239 (0x562450730239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #39: _PyObject_MakeTpCall + 0x26b (0x56245065ca6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5624506583e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #41: _PyFunction_Vectorcall + 0x6c (0x562450663a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x562450653c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #43: _PyFunction_Vectorcall + 0x6c (0x562450663a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5624506548fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #45: <unknown function> + 0x150582 (0x56245066f582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #46: PyObject_Call + 0xbc (0x56245066ff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5624506562b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #48: <unknown function> + 0x150582 (0x56245066f582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #49: PyObject_Call + 0xbc (0x56245066ff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5624506562b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #51: _PyFunction_Vectorcall + 0x6c (0x562450663a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x56245065c007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #53: _PyObject_Call_Prepend + 0x69 (0x56245066dc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #54: <unknown function> + 0x211239 (0x562450730239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #55: PyObject_Call + 0x207 (0x562450670067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5624506562b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #57: <unknown function> + 0x150582 (0x56245066f582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5624506548fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #59: <unknown function> + 0x150582 (0x56245066f582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #60: PyObject_Call + 0xbc (0x56245066ff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5624506562b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #62: <unknown function> + 0x150582 (0x56245066f582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: frame #63: PyObject_Call + 0xbc (0x56245066ff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default1]:[rank9]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default3]:[rank11]: Traceback (most recent call last):
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default3]:[rank11]:     trainer.train(dataloader)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default3]:[rank11]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default3]:[rank11]:     outputs = self.pipeline_engine.train_batch_iter(
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default3]:[rank11]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default3]:[rank11]:     output = model(**micro_batch)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank11]:     return self._call_impl(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank11]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default3]:[rank11]:     sharded_logits = self.model(
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank11]:     return self._call_impl(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank11]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default3]:[rank11]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default3]:[rank11]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default3]:[rank11]:     return self._call_impl(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default3]:[rank11]:     return forward_call(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default3]:[rank11]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default3]:[rank11]:     pipeline_state.run_communication()
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default3]:[rank11]:     recv_activation_tensor = recv_activation()
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default3]:[rank11]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default3]:[rank11]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default3]:[rank11]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default3]:[rank11]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default3]:[rank11]:     dist.recv(
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default3]:[rank11]:     return func(*args, **kwargs)
[default3]:[rank11]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default3]:[rank11]:     pg.recv([tensor], group_src_rank, tag).wait()
[default3]:[rank11]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default3]:[rank11]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default3]:[rank11]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fdac16b5897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default3]:[rank11]: frame #1: <unknown function> + 0x5b3a23e (0x7fdafb1d223e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank11]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fdafb1ccc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank11]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fdafb1ccf82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank11]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fdafb1cdfd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank11]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fdafb182371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank11]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fdafb182371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank11]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fdafb182371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank11]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fdafb182371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank11]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fdac298f189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:[rank11]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fdac2996610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:[rank11]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fdac29b5978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default3]:[rank11]: frame #12: <unknown function> + 0x5adc309 (0x7fdafb174309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank11]: frame #13: <unknown function> + 0x5ae6f10 (0x7fdafb17ef10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank11]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fdafb17efa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank11]: frame #15: <unknown function> + 0x5124446 (0x7fdafa7bc446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank11]: frame #16: <unknown function> + 0x1acf4b8 (0x7fdaf71674b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank11]: frame #17: <unknown function> + 0x5aee004 (0x7fdafb186004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank11]: frame #18: <unknown function> + 0x5af36b5 (0x7fdafb18b6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default3]:[rank11]: frame #19: <unknown function> + 0xd2631e (0x7fdb0dd7531e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default3]:[rank11]: frame #20: <unknown function> + 0x47def4 (0x7fdb0d4ccef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default3]:[rank11]: frame #21: <unknown function> + 0x1445a6 (0x56264d3715a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #22: _PyObject_MakeTpCall + 0x26b (0x56264d36aa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #23: <unknown function> + 0x150866 (0x56264d37d866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x56264d366142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #25: _PyFunction_Vectorcall + 0x6c (0x56264d371a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #26: PyObject_Call + 0xbc (0x56264d37df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x56264d3642b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #28: _PyFunction_Vectorcall + 0x6c (0x56264d371a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x56264d3628fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #30: <unknown function> + 0x150582 (0x56264d37d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x56264d3628fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #32: <unknown function> + 0x150582 (0x56264d37d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x56264d3628fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #34: <unknown function> + 0x150582 (0x56264d37d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x56264d3628fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x56264d369f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #37: _PyObject_Call_Prepend + 0x69 (0x56264d37bc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #38: <unknown function> + 0x211239 (0x56264d43e239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #39: _PyObject_MakeTpCall + 0x26b (0x56264d36aa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x56264d3663e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #41: _PyFunction_Vectorcall + 0x6c (0x56264d371a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x56264d361c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #43: _PyFunction_Vectorcall + 0x6c (0x56264d371a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x56264d3628fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #45: <unknown function> + 0x150582 (0x56264d37d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #46: PyObject_Call + 0xbc (0x56264d37df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x56264d3642b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #48: <unknown function> + 0x150582 (0x56264d37d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #49: PyObject_Call + 0xbc (0x56264d37df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x56264d3642b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #51: _PyFunction_Vectorcall + 0x6c (0x56264d371a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x56264d36a007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #53: _PyObject_Call_Prepend + 0x69 (0x56264d37bc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #54: <unknown function> + 0x211239 (0x56264d43e239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #55: PyObject_Call + 0x207 (0x56264d37e067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x56264d3642b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #57: <unknown function> + 0x150582 (0x56264d37d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x56264d3628fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #59: <unknown function> + 0x150582 (0x56264d37d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #60: PyObject_Call + 0xbc (0x56264d37df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x56264d3642b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #62: <unknown function> + 0x150582 (0x56264d37d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: frame #63: PyObject_Call + 0xbc (0x56264d37df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default3]:[rank11]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default7]:[rank15]: Traceback (most recent call last):
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default7]:[rank15]:     trainer.train(dataloader)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default7]:[rank15]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default7]:[rank15]:     outputs = self.pipeline_engine.train_batch_iter(
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default7]:[rank15]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default7]:[rank15]:     output = model(**micro_batch)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank15]:     return self._call_impl(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank15]:     return forward_call(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default7]:[rank15]:     sharded_logits = self.model(
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank15]:     return self._call_impl(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank15]:     return forward_call(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default7]:[rank15]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default7]:[rank15]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default7]:[rank15]:     return self._call_impl(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default7]:[rank15]:     return forward_call(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default7]:[rank15]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default7]:[rank15]:     pipeline_state.run_communication()
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default7]:[rank15]:     recv_activation_tensor = recv_activation()
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default7]:[rank15]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default7]:[rank15]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default7]:[rank15]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default7]:[rank15]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default7]:[rank15]:     dist.recv(
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default7]:[rank15]:     return func(*args, **kwargs)
[default7]:[rank15]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default7]:[rank15]:     pg.recv([tensor], group_src_rank, tag).wait()
[default7]:[rank15]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default7]:[rank15]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default7]:[rank15]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f68339d5897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default7]:[rank15]: frame #1: <unknown function> + 0x5b3a23e (0x7f686d4f223e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank15]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f686d4ecc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank15]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f686d4ecf82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank15]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f686d4edfd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank15]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f686d4a2371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank15]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f686d4a2371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank15]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f686d4a2371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank15]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f686d4a2371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank15]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f6834caf189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank15]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f6834cb6610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank15]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f6834cd5978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default7]:[rank15]: frame #12: <unknown function> + 0x5adc309 (0x7f686d494309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank15]: frame #13: <unknown function> + 0x5ae6f10 (0x7f686d49ef10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank15]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f686d49efa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank15]: frame #15: <unknown function> + 0x5124446 (0x7f686cadc446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank15]: frame #16: <unknown function> + 0x1acf4b8 (0x7f68694874b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank15]: frame #17: <unknown function> + 0x5aee004 (0x7f686d4a6004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank15]: frame #18: <unknown function> + 0x5af36b5 (0x7f686d4ab6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default7]:[rank15]: frame #19: <unknown function> + 0xd2631e (0x7f688009531e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:[rank15]: frame #20: <unknown function> + 0x47def4 (0x7f687f7ecef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default7]:[rank15]: frame #21: <unknown function> + 0x1445a6 (0x564337fd55a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #22: _PyObject_MakeTpCall + 0x26b (0x564337fcea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #23: <unknown function> + 0x150866 (0x564337fe1866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x564337fca142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #25: _PyFunction_Vectorcall + 0x6c (0x564337fd5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #26: PyObject_Call + 0xbc (0x564337fe1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x564337fc82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #28: _PyFunction_Vectorcall + 0x6c (0x564337fd5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x564337fc68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #30: <unknown function> + 0x150582 (0x564337fe1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x564337fc68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #32: <unknown function> + 0x150582 (0x564337fe1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x564337fc68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #34: <unknown function> + 0x150582 (0x564337fe1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x564337fc68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x564337fcdf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #37: _PyObject_Call_Prepend + 0x69 (0x564337fdfc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #38: <unknown function> + 0x211239 (0x5643380a2239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #39: _PyObject_MakeTpCall + 0x26b (0x564337fcea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x564337fca3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #41: _PyFunction_Vectorcall + 0x6c (0x564337fd5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x564337fc5c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #43: _PyFunction_Vectorcall + 0x6c (0x564337fd5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x564337fc68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #45: <unknown function> + 0x150582 (0x564337fe1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #46: PyObject_Call + 0xbc (0x564337fe1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x564337fc82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #48: <unknown function> + 0x150582 (0x564337fe1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #49: PyObject_Call + 0xbc (0x564337fe1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x564337fc82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #51: _PyFunction_Vectorcall + 0x6c (0x564337fd5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x564337fce007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #53: _PyObject_Call_Prepend + 0x69 (0x564337fdfc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #54: <unknown function> + 0x211239 (0x5643380a2239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #55: PyObject_Call + 0x207 (0x564337fe2067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x564337fc82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #57: <unknown function> + 0x150582 (0x564337fe1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x564337fc68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #59: <unknown function> + 0x150582 (0x564337fe1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #60: PyObject_Call + 0xbc (0x564337fe1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x564337fc82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #62: <unknown function> + 0x150582 (0x564337fe1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: frame #63: PyObject_Call + 0xbc (0x564337fe1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default7]:[rank15]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default6]:[rank14]: Traceback (most recent call last):
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default6]:[rank14]:     trainer.train(dataloader)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default6]:[rank14]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default6]:[rank14]:     outputs = self.pipeline_engine.train_batch_iter(
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default6]:[rank14]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default6]:[rank14]:     output = model(**micro_batch)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank14]:     return self._call_impl(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank14]:     return forward_call(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default6]:[rank14]:     sharded_logits = self.model(
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank14]:     return self._call_impl(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank14]:     return forward_call(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default6]:[rank14]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default6]:[rank14]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default6]:[rank14]:     return self._call_impl(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default6]:[rank14]:     return forward_call(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default6]:[rank14]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default6]:[rank14]:     pipeline_state.run_communication()
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default6]:[rank14]:     recv_activation_tensor = recv_activation()
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default6]:[rank14]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default6]:[rank14]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default6]:[rank14]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default6]:[rank14]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default6]:[rank14]:     dist.recv(
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default6]:[rank14]:     return func(*args, **kwargs)
[default6]:[rank14]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default6]:[rank14]:     pg.recv([tensor], group_src_rank, tag).wait()
[default6]:[rank14]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default6]:[rank14]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default6]:[rank14]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fa606697897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default6]:[rank14]: frame #1: <unknown function> + 0x5b3a23e (0x7fa6401b423e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank14]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fa6401aec87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank14]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fa6401aef82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank14]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fa6401affd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank14]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa640164371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank14]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa640164371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank14]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa640164371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank14]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa640164371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank14]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fa607971189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank14]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fa607978610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank14]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fa607997978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default6]:[rank14]: frame #12: <unknown function> + 0x5adc309 (0x7fa640156309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank14]: frame #13: <unknown function> + 0x5ae6f10 (0x7fa640160f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank14]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fa640160fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank14]: frame #15: <unknown function> + 0x5124446 (0x7fa63f79e446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank14]: frame #16: <unknown function> + 0x1acf4b8 (0x7fa63c1494b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank14]: frame #17: <unknown function> + 0x5aee004 (0x7fa640168004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank14]: frame #18: <unknown function> + 0x5af36b5 (0x7fa64016d6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default6]:[rank14]: frame #19: <unknown function> + 0xd2631e (0x7fa652d5731e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:[rank14]: frame #20: <unknown function> + 0x47def4 (0x7fa6524aeef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default6]:[rank14]: frame #21: <unknown function> + 0x1445a6 (0x564dfc5145a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #22: _PyObject_MakeTpCall + 0x26b (0x564dfc50da6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #23: <unknown function> + 0x150866 (0x564dfc520866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x564dfc509142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #25: _PyFunction_Vectorcall + 0x6c (0x564dfc514a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #26: PyObject_Call + 0xbc (0x564dfc520f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x564dfc5072b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #28: _PyFunction_Vectorcall + 0x6c (0x564dfc514a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x564dfc5058fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #30: <unknown function> + 0x150582 (0x564dfc520582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x564dfc5058fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #32: <unknown function> + 0x150582 (0x564dfc520582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x564dfc5058fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #34: <unknown function> + 0x150582 (0x564dfc520582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x564dfc5058fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x564dfc50cf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #37: _PyObject_Call_Prepend + 0x69 (0x564dfc51ec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #38: <unknown function> + 0x211239 (0x564dfc5e1239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #39: _PyObject_MakeTpCall + 0x26b (0x564dfc50da6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x564dfc5093e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #41: _PyFunction_Vectorcall + 0x6c (0x564dfc514a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x564dfc504c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #43: _PyFunction_Vectorcall + 0x6c (0x564dfc514a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x564dfc5058fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #45: <unknown function> + 0x150582 (0x564dfc520582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #46: PyObject_Call + 0xbc (0x564dfc520f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x564dfc5072b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #48: <unknown function> + 0x150582 (0x564dfc520582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #49: PyObject_Call + 0xbc (0x564dfc520f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x564dfc5072b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #51: _PyFunction_Vectorcall + 0x6c (0x564dfc514a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x564dfc50d007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #53: _PyObject_Call_Prepend + 0x69 (0x564dfc51ec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #54: <unknown function> + 0x211239 (0x564dfc5e1239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #55: PyObject_Call + 0x207 (0x564dfc521067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x564dfc5072b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #57: <unknown function> + 0x150582 (0x564dfc520582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x564dfc5058fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #59: <unknown function> + 0x150582 (0x564dfc520582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #60: PyObject_Call + 0xbc (0x564dfc520f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x564dfc5072b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #62: <unknown function> + 0x150582 (0x564dfc520582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: frame #63: PyObject_Call + 0xbc (0x564dfc520f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default6]:[rank14]: . This may indicate a possible application crash on rank 0 or a network set up issue.
[default2]:[rank10]: Traceback (most recent call last):
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module>
[default2]:[rank10]:     trainer.train(dataloader)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train
[default2]:[rank10]:     outputs, loss_avg = self.training_step(dataloader=self.current_dataloader)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step
[default2]:[rank10]:     outputs = self.pipeline_engine.train_batch_iter(
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter
[default2]:[rank10]:     output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward
[default2]:[rank10]:     output = model(**micro_batch)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank10]:     return self._call_impl(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank10]:     return forward_call(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward
[default2]:[rank10]:     sharded_logits = self.model(
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank10]:     return self._call_impl(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank10]:     return forward_call(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward
[default2]:[rank10]:     return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0]
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states
[default2]:[rank10]:     hidden_encoder_states = encoder_block(**hidden_encoder_states)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[default2]:[rank10]:     return self._call_impl(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[default2]:[rank10]:     return forward_call(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward
[default2]:[rank10]:     new_kwargs[name] = recv_from_pipeline_state_buffer(
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer
[default2]:[rank10]:     pipeline_state.run_communication()
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication
[default2]:[rank10]:     recv_activation_tensor = recv_activation()
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__
[default2]:[rank10]:     return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0]
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors
[default2]:[rank10]:     buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors
[default2]:[rank10]:     meta = self._recv_meta(from_rank=from_rank, tag=tag)
[default2]:[rank10]:   File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta
[default2]:[rank10]:     dist.recv(
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
[default2]:[rank10]:     return func(*args, **kwargs)
[default2]:[rank10]:   File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv
[default2]:[rank10]:     pg.recv([tensor], group_src_rank, tag).wait()
[default2]:[rank10]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer
[default2]:[rank10]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first):
[default2]:[rank10]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f4a1f30f897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so)
[default2]:[rank10]: frame #1: <unknown function> + 0x5b3a23e (0x7f4a58e2c23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank10]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f4a58e26c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank10]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f4a58e26f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank10]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f4a58e27fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank10]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4a58ddc371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank10]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4a58ddc371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank10]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4a58ddc371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank10]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4a58ddc371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank10]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f4a205e9189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank10]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f4a205f0610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank10]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f4a2060f978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
[default2]:[rank10]: frame #12: <unknown function> + 0x5adc309 (0x7f4a58dce309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank10]: frame #13: <unknown function> + 0x5ae6f10 (0x7f4a58dd8f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank10]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f4a58dd8fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank10]: frame #15: <unknown function> + 0x5124446 (0x7f4a58416446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank10]: frame #16: <unknown function> + 0x1acf4b8 (0x7f4a54dc14b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank10]: frame #17: <unknown function> + 0x5aee004 (0x7f4a58de0004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank10]: frame #18: <unknown function> + 0x5af36b5 (0x7f4a58de56b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
[default2]:[rank10]: frame #19: <unknown function> + 0xd2631e (0x7f4a6b9cf31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default2]:[rank10]: frame #20: <unknown function> + 0x47def4 (0x7f4a6b126ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
[default2]:[rank10]: frame #21: <unknown function> + 0x1445a6 (0x55cce95bf5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55cce95b8a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #23: <unknown function> + 0x150866 (0x55cce95cb866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55cce95b4142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55cce95bfa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #26: PyObject_Call + 0xbc (0x55cce95cbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55cce95b22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55cce95bfa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55cce95b08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #30: <unknown function> + 0x150582 (0x55cce95cb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55cce95b08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #32: <unknown function> + 0x150582 (0x55cce95cb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55cce95b08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #34: <unknown function> + 0x150582 (0x55cce95cb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55cce95b08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55cce95b7f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55cce95c9c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #38: <unknown function> + 0x211239 (0x55cce968c239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55cce95b8a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55cce95b43e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55cce95bfa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55cce95afc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55cce95bfa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55cce95b08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #45: <unknown function> + 0x150582 (0x55cce95cb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #46: PyObject_Call + 0xbc (0x55cce95cbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55cce95b22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #48: <unknown function> + 0x150582 (0x55cce95cb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #49: PyObject_Call + 0xbc (0x55cce95cbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55cce95b22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55cce95bfa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55cce95b8007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55cce95c9c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #54: <unknown function> + 0x211239 (0x55cce968c239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #55: PyObject_Call + 0x207 (0x55cce95cc067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55cce95b22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #57: <unknown function> + 0x150582 (0x55cce95cb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55cce95b08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #59: <unknown function> + 0x150582 (0x55cce95cb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #60: PyObject_Call + 0xbc (0x55cce95cbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55cce95b22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #62: <unknown function> + 0x150582 (0x55cce95cb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: frame #63: PyObject_Call + 0xbc (0x55cce95cbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10)
[default2]:[rank10]: . This may indicate a possible application crash on rank 0 or a network set up issue.
E0702 16:31:14.157000 140513205032768 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 2 (pid: 3391100) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-07-02_16:31:10
  host      : ip-26-0-171-62.ec2.internal
  rank      : 5 (local_rank: 5)
  exitcode  : 1 (pid: 3391103)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-02_16:31:10
  host      : ip-26-0-171-62.ec2.internal
  rank      : 2 (local_rank: 2)
  exitcode  : 1 (pid: 3391100)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
srun: error: ip-26-0-171-62: task 0: Exited with exit code 1
W0702 16:31:14.988000 140200537929472 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-171-88.ec2.internal_384344_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0702 16:31:16.018000 140206204749632 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 384413 closing signal SIGTERM
W0702 16:31:16.018000 140206204749632 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 384414 closing signal SIGTERM
W0702 16:31:16.018000 140206204749632 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 384415 closing signal SIGTERM
W0702 16:31:16.019000 140206204749632 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 384416 closing signal SIGTERM
W0702 16:31:16.019000 140206204749632 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 384417 closing signal SIGTERM
W0702 16:31:16.020000 140206204749632 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 384418 closing signal SIGTERM
W0702 16:31:16.023000 140206204749632 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 384419 closing signal SIGTERM
W0702 16:31:16.026000 140206204749632 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 384420 closing signal SIGTERM
W0702 16:31:18.219000 140206204749632 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-171-88.ec2.internal_384344_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0702 16:31:18.226000 140206204749632 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-171-88.ec2.internal_384344_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 113, in _call_store
    return getattr(self._store, store_op)(*args, **kwargs)
torch.distributed.DistNetworkError: Broken pipe

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 254, in launch_agent
    result = agent.run()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 123, in wrapper
    result = f(*args, **kwargs)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 733, in run
    result = self._invoke_run(role)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 908, in _invoke_run
    num_nodes_waiting = rdzv_handler.num_nodes_waiting()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 1174, in num_nodes_waiting
    self._state_holder.sync()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 419, in sync
    get_response = self._backend.get_state()
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 73, in get_state
    base64_state: bytes = self._call_store("get", self._key)
  File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 115, in _call_store
    raise RendezvousConnectionError(
torch.distributed.elastic.rendezvous.api.RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details.
srun: error: ip-26-0-171-88: task 1: Exited with exit code 1
Consider using `hf_transfer` for faster uploads. This solution comes with some limitations. See https://huggingface.co/docs/huggingface_hub/hf_transfer for more details.