======================== START TIME: Sat Jul 6 09:20:24 UTC 2024 python3 version = Python 3.10.14 ======================== The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well. Token is valid (permission: write). Your token has been saved to /admin/home/ferdinand_mom/.cache/huggingface/token Login successful Already on 'bench_cluster' M examples/config_tiny_llama.py M examples/config_tiny_llama.yaml M examples/train_tiny_llama.sh Your branch is up to date with 'origin/bench_cluster'. Job status: RUNNING [2024-07-06 09:20:27,443] torch.distributed.run: [WARNING] [2024-07-06 09:20:27,443] torch.distributed.run: [WARNING] ***************************************** [2024-07-06 09:20:27,443] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. [2024-07-06 09:20:27,443] torch.distributed.run: [WARNING] ***************************************** [2024-07-06 09:20:27,441] torch.distributed.run: [WARNING] [2024-07-06 09:20:27,441] torch.distributed.run: [WARNING] ***************************************** [2024-07-06 09:20:27,441] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. [2024-07-06 09:20:27,441] torch.distributed.run: [WARNING] ***************************************** [2024-07-06 09:20:27,442] torch.distributed.run: [WARNING] [2024-07-06 09:20:27,442] torch.distributed.run: [WARNING] ***************************************** [2024-07-06 09:20:27,442] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. [2024-07-06 09:20:27,442] torch.distributed.run: [WARNING] ***************************************** [2024-07-06 09:20:27,444] torch.distributed.run: [WARNING] [2024-07-06 09:20:27,444] torch.distributed.run: [WARNING] ***************************************** [2024-07-06 09:20:27,444] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. [2024-07-06 09:20:27,444] torch.distributed.run: [WARNING] ***************************************** [2024-07-06 09:20:27,443] torch.distributed.run: [WARNING] [2024-07-06 09:20:27,443] torch.distributed.run: [WARNING] ***************************************** [2024-07-06 09:20:27,443] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. [2024-07-06 09:20:27,443] torch.distributed.run: [WARNING] ***************************************** [2024-07-06 09:20:27,445] torch.distributed.run: [WARNING] [2024-07-06 09:20:27,445] torch.distributed.run: [WARNING] ***************************************** [2024-07-06 09:20:27,445] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. [2024-07-06 09:20:27,445] torch.distributed.run: [WARNING] ***************************************** [2024-07-06 09:20:27,447] torch.distributed.run: [WARNING] [2024-07-06 09:20:27,447] torch.distributed.run: [WARNING] ***************************************** [2024-07-06 09:20:27,447] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. [2024-07-06 09:20:27,447] torch.distributed.run: [WARNING] ***************************************** [2024-07-06 09:20:27,491] torch.distributed.run: [WARNING] [2024-07-06 09:20:27,491] torch.distributed.run: [WARNING] ***************************************** [2024-07-06 09:20:27,491] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. [2024-07-06 09:20:27,491] torch.distributed.run: [WARNING] ***************************************** [default0]:07/06/2024 09:20:46 [WARNING|DP=0|PP=0|TP=0|ip-26-0-166-15]: [Vocab Size Padding] Padded vocab (size: 50257) with 1 dummy tokens (new size: 50258) [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: Config: [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: Config(general=GeneralArgs(project='bench_cluster', [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: run='%date_%jobid', [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: seed=42, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: step=None, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: consumed_train_samples=None, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: benchmark_csv_path=None, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: ignore_sanity_checks=True), [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: parallelism=ParallelismArgs(dp=1, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: pp=32, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: tp=2, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: pp_engine=, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: tp_mode=, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: tp_linear_async_communication=False, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: expert_parallel_size=1), [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: model=ModelArgs(model_config=LlamaConfig(bos_token_id=1, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: eos_token_id=2, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: hidden_act='silu', [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: hidden_size=2048, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: initializer_range=0.02, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: intermediate_size=4096, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: is_llama_config=True, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: max_position_embeddings=4096, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: num_attention_heads=32, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: num_hidden_layers=24, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: num_key_value_heads=32, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: pad_token_id=None, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: pretraining_tp=1, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: rms_norm_eps=1e-05, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: rope_scaling=None, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: rope_theta=10000.0, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: tie_word_embeddings=True, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: use_cache=True, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: vocab_size=50258), [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: init_method=RandomInit(std=0.025), [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: dtype=torch.bfloat16, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: make_vocab_size_divisible_by=1, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: ddp_bucket_cap_mb=25), [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: tokenizer=TokenizerArgs(tokenizer_name_or_path='openai-community/gpt2', [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: tokenizer_revision=None, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: tokenizer_max_length=None), [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: checkpoints=CheckpointsArgs(checkpoints_path=PosixPath('/dev/null'), [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: checkpoint_interval=100000, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: save_initial_state=False, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: resume_checkpoint_path=None, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: checkpoints_path_is_shared_file_system=False), [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: logging=LoggingArgs(log_level='info', [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: log_level_replica='info', [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: iteration_step_info_interval=1), [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: tokens=TokensArgs(sequence_length=4096, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: train_steps=20, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: micro_batch_size=32, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: batch_accumulation_per_replica=32, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: val_check_interval=-1, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: limit_val_batches=0, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: limit_test_batches=0), [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: optimizer=OptimizerArgs(optimizer_factory=AdamWOptimizerArgs(adam_eps=1e-08, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: adam_beta1=0.9, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: adam_beta2=0.95, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: torch_adam_is_fused=True, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: name='adamW'), [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: zero_stage=1, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: weight_decay=0.01, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: clip_grad=1.0, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: accumulate_grad_in_fp32=True, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: learning_rate_scheduler=LRSchedulerArgs(learning_rate=0.0001, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: lr_warmup_steps=1, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: lr_warmup_style='linear', [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: lr_decay_style='linear', [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: lr_decay_steps=19, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: lr_decay_starting_step=None, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: min_decay_lr=1e-05)), [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: data_stages=[DatasetStageArgs(name='Training Stage', [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: start_training_step=1, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: data=DataArgs(dataset=PretrainDatasetsArgs(hf_dataset_or_datasets='roneneldan/TinyStories', [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: hf_dataset_splits='train', [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: hf_dataset_config_name=None, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: dataset_processing_num_proc_per_process=64, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: dataset_overwrite_cache=False, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: text_column_name='text'), [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: seed=42, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: num_loading_workers=0))], [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: profiler=ProfilerArgs(profiler_export_path=PosixPath('/fsx/ferdinandmom/ferdinand-hf/bench_cluster/results/llama-1B/64_GPUS/dp-1_tp-2_pp-32_mbz-32')), [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: lighteval=None) [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: Model Config: [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: LlamaConfig(bos_token_id=1, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: eos_token_id=2, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: hidden_act='silu', [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: hidden_size=2048, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: initializer_range=0.02, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: intermediate_size=4096, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: is_llama_config=True, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: max_position_embeddings=4096, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: num_attention_heads=32, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: num_hidden_layers=24, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: num_key_value_heads=32, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: pad_token_id=None, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: pretraining_tp=1, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: rms_norm_eps=1e-05, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: rope_scaling=None, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: rope_theta=10000.0, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: tie_word_embeddings=True, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: use_cache=True, [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: vocab_size=50258) [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: Building model.. [default0]:07/06/2024 09:20:46 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: Setting PP block ranks... [default3]:07/06/2024 09:21:03 [INFO|DP=0|PP=21|TP=1|ip-26-0-175-19]: Local number of parameters: 21M (40.01MiB) [default3]:07/06/2024 09:21:03 [INFO|DP=0|PP=21|TP=1|ip-26-0-175-19]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default3]:07/06/2024 09:21:03 [INFO|DP=0|PP=21|TP=1|ip-26-0-175-19]: No checkpoint path provided. [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=8|TP=0|ip-26-0-175-132]: Local number of parameters: 21M (40.01MiB) [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=8|TP=0|ip-26-0-175-132]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=8|TP=0|ip-26-0-175-132]: No checkpoint path provided. [default6]:07/06/2024 09:21:03 [INFO|DP=0|PP=11|TP=0|ip-26-0-175-132]: Local number of parameters: 21M (40.01MiB) [default6]:07/06/2024 09:21:03 [INFO|DP=0|PP=11|TP=0|ip-26-0-175-132]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default6]:07/06/2024 09:21:03 [INFO|DP=0|PP=11|TP=0|ip-26-0-175-132]: No checkpoint path provided. [default2]:07/06/2024 09:21:03 [INFO|DP=0|PP=21|TP=0|ip-26-0-175-19]: Local number of parameters: 21M (40.01MiB) [default1]:07/06/2024 09:21:03 [INFO|DP=0|PP=8|TP=1|ip-26-0-175-132]: Local number of parameters: 21M (40.01MiB) [default1]:07/06/2024 09:21:03 [INFO|DP=0|PP=8|TP=1|ip-26-0-175-132]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default1]:07/06/2024 09:21:03 [INFO|DP=0|PP=8|TP=1|ip-26-0-175-132]: No checkpoint path provided. [default2]:07/06/2024 09:21:03 [INFO|DP=0|PP=21|TP=0|ip-26-0-175-19]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default2]:07/06/2024 09:21:03 [INFO|DP=0|PP=21|TP=0|ip-26-0-175-19]: No checkpoint path provided. [default6]:07/06/2024 09:21:03 [INFO|DP=0|PP=23|TP=0|ip-26-0-175-19]: Local number of parameters: 21M (40.01MiB) [default6]:07/06/2024 09:21:03 [INFO|DP=0|PP=23|TP=0|ip-26-0-175-19]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default4]:07/06/2024 09:21:03 [INFO|DP=0|PP=22|TP=0|ip-26-0-175-19]: Local number of parameters: 21M (40.01MiB) [default4]:07/06/2024 09:21:03 [INFO|DP=0|PP=22|TP=0|ip-26-0-175-19]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default6]:07/06/2024 09:21:03 [INFO|DP=0|PP=23|TP=0|ip-26-0-175-19]: No checkpoint path provided. [default4]:07/06/2024 09:21:03 [INFO|DP=0|PP=22|TP=0|ip-26-0-175-19]: No checkpoint path provided. [default1]:07/06/2024 09:21:03 [INFO|DP=0|PP=20|TP=1|ip-26-0-175-19]: Local number of parameters: 21M (40.01MiB) [default1]:07/06/2024 09:21:03 [INFO|DP=0|PP=20|TP=1|ip-26-0-175-19]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default1]:07/06/2024 09:21:03 [INFO|DP=0|PP=20|TP=1|ip-26-0-175-19]: No checkpoint path provided. [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=20|TP=0|ip-26-0-175-19]: Local number of parameters: 21M (40.01MiB) [default3]:07/06/2024 09:21:03 [INFO|DP=0|PP=9|TP=1|ip-26-0-175-132]: Local number of parameters: 21M (40.01MiB) [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=20|TP=0|ip-26-0-175-19]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=20|TP=0|ip-26-0-175-19]: No checkpoint path provided. [default3]:07/06/2024 09:21:03 [INFO|DP=0|PP=9|TP=1|ip-26-0-175-132]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default3]:07/06/2024 09:21:03 [INFO|DP=0|PP=9|TP=1|ip-26-0-175-132]: No checkpoint path provided. [default7]:07/06/2024 09:21:03 [INFO|DP=0|PP=23|TP=1|ip-26-0-175-19]: Local number of parameters: 21M (40.01MiB) [default7]:07/06/2024 09:21:03 [INFO|DP=0|PP=23|TP=1|ip-26-0-175-19]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default7]:07/06/2024 09:21:03 [INFO|DP=0|PP=23|TP=1|ip-26-0-175-19]: No checkpoint path provided. [default4]:07/06/2024 09:21:03 [INFO|DP=0|PP=10|TP=0|ip-26-0-175-132]: Local number of parameters: 21M (40.01MiB) [default4]:07/06/2024 09:21:03 [INFO|DP=0|PP=10|TP=0|ip-26-0-175-132]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default4]:07/06/2024 09:21:03 [INFO|DP=0|PP=10|TP=0|ip-26-0-175-132]: No checkpoint path provided. [default5]:07/06/2024 09:21:03 [INFO|DP=0|PP=22|TP=1|ip-26-0-175-19]: Local number of parameters: 21M (40.01MiB) [default5]:07/06/2024 09:21:03 [INFO|DP=0|PP=22|TP=1|ip-26-0-175-19]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default5]:07/06/2024 09:21:03 [INFO|DP=0|PP=22|TP=1|ip-26-0-175-19]: No checkpoint path provided. [default2]:07/06/2024 09:21:03 [INFO|DP=0|PP=9|TP=0|ip-26-0-175-132]: Local number of parameters: 21M (40.01MiB) [default2]:07/06/2024 09:21:03 [INFO|DP=0|PP=9|TP=0|ip-26-0-175-132]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default2]:07/06/2024 09:21:03 [INFO|DP=0|PP=9|TP=0|ip-26-0-175-132]: No checkpoint path provided. [default5]:07/06/2024 09:21:03 [INFO|DP=0|PP=10|TP=1|ip-26-0-175-132]: Local number of parameters: 21M (40.01MiB) [default5]:07/06/2024 09:21:03 [INFO|DP=0|PP=10|TP=1|ip-26-0-175-132]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default1]:07/06/2024 09:21:03 [INFO|DP=0|PP=4|TP=1|ip-26-0-174-240]: Local number of parameters: 21M (40.01MiB) [default1]:07/06/2024 09:21:03 [INFO|DP=0|PP=4|TP=1|ip-26-0-174-240]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default1]:07/06/2024 09:21:03 [INFO|DP=0|PP=4|TP=1|ip-26-0-174-240]: No checkpoint path provided. [default5]:07/06/2024 09:21:03 [INFO|DP=0|PP=10|TP=1|ip-26-0-175-132]: No checkpoint path provided. [default6]:07/06/2024 09:21:03 [INFO|DP=0|PP=27|TP=0|ip-26-0-175-241]: Local number of parameters: 0 (0.00MiB) [default6]:07/06/2024 09:21:03 [INFO|DP=0|PP=27|TP=0|ip-26-0-175-241]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB [default6]:07/06/2024 09:21:03 [INFO|DP=0|PP=27|TP=0|ip-26-0-175-241]: No checkpoint path provided. [default3]:07/06/2024 09:21:03 [INFO|DP=0|PP=25|TP=1|ip-26-0-175-241]: Local number of parameters: 51.5M (98.16MiB) [default3]:07/06/2024 09:21:03 [INFO|DP=0|PP=25|TP=1|ip-26-0-175-241]: [After model building] Memory usage: 98.17MiB. Peak allocated: 98.19MiB Peak reserved: 102.00MiB [default3]:07/06/2024 09:21:03 [INFO|DP=0|PP=25|TP=1|ip-26-0-175-241]: No checkpoint path provided. [default7]:07/06/2024 09:21:03 [INFO|DP=0|PP=27|TP=1|ip-26-0-175-241]: Local number of parameters: 0 (0.00MiB) [default7]:07/06/2024 09:21:03 [INFO|DP=0|PP=27|TP=1|ip-26-0-175-241]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB [default7]:07/06/2024 09:21:03 [INFO|DP=0|PP=27|TP=1|ip-26-0-175-241]: No checkpoint path provided. [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=12|TP=0|ip-26-0-175-165]: Local number of parameters: 21M (40.01MiB) [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=12|TP=0|ip-26-0-175-165]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=12|TP=0|ip-26-0-175-165]: No checkpoint path provided. [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=4|TP=0|ip-26-0-174-240]: Local number of parameters: 21M (40.01MiB) [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=4|TP=0|ip-26-0-174-240]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=4|TP=0|ip-26-0-174-240]: No checkpoint path provided. [default7]:07/06/2024 09:21:03 [INFO|DP=0|PP=11|TP=1|ip-26-0-175-132]: Local number of parameters: 21M (40.01MiB) [default7]:07/06/2024 09:21:03 [INFO|DP=0|PP=11|TP=1|ip-26-0-175-132]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default7]:07/06/2024 09:21:03 [INFO|DP=0|PP=11|TP=1|ip-26-0-175-132]: No checkpoint path provided. [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=28|TP=0|ip-26-0-175-34]: Local number of parameters: 0 (0.00MiB) [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=28|TP=0|ip-26-0-175-34]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=28|TP=0|ip-26-0-175-34]: No checkpoint path provided. [default4]:07/06/2024 09:21:03 [INFO|DP=0|PP=6|TP=0|ip-26-0-174-240]: Local number of parameters: 21M (40.01MiB) [default4]:07/06/2024 09:21:03 [INFO|DP=0|PP=6|TP=0|ip-26-0-174-240]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default4]:07/06/2024 09:21:03 [INFO|DP=0|PP=6|TP=0|ip-26-0-174-240]: No checkpoint path provided. [default3]:07/06/2024 09:21:03 [INFO|DP=0|PP=13|TP=1|ip-26-0-175-165]: Local number of parameters: 21M (40.01MiB) [default3]:07/06/2024 09:21:03 [INFO|DP=0|PP=13|TP=1|ip-26-0-175-165]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default3]:07/06/2024 09:21:03 [INFO|DP=0|PP=13|TP=1|ip-26-0-175-165]: No checkpoint path provided. [default2]:07/06/2024 09:21:03 [INFO|DP=0|PP=29|TP=0|ip-26-0-175-34]: Local number of parameters: 0 (0.00MiB) [default2]:07/06/2024 09:21:03 [INFO|DP=0|PP=29|TP=0|ip-26-0-175-34]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB [default2]:07/06/2024 09:21:03 [INFO|DP=0|PP=29|TP=0|ip-26-0-175-34]: No checkpoint path provided. [default3]:07/06/2024 09:21:03 [INFO|DP=0|PP=29|TP=1|ip-26-0-175-34]: Local number of parameters: 0 (0.00MiB) [default3]:07/06/2024 09:21:03 [INFO|DP=0|PP=29|TP=1|ip-26-0-175-34]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB [default3]:07/06/2024 09:21:03 [INFO|DP=0|PP=29|TP=1|ip-26-0-175-34]: No checkpoint path provided. [default5]:07/06/2024 09:21:03 [INFO|DP=0|PP=30|TP=1|ip-26-0-175-34]: Local number of parameters: 0 (0.00MiB) [default5]:07/06/2024 09:21:03 [INFO|DP=0|PP=30|TP=1|ip-26-0-175-34]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB [default5]:07/06/2024 09:21:03 [INFO|DP=0|PP=30|TP=1|ip-26-0-175-34]: No checkpoint path provided. [default4]:07/06/2024 09:21:03 [INFO|DP=0|PP=30|TP=0|ip-26-0-175-34]: Local number of parameters: 0 (0.00MiB) [default4]:07/06/2024 09:21:03 [INFO|DP=0|PP=30|TP=0|ip-26-0-175-34]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB [default4]:07/06/2024 09:21:03 [INFO|DP=0|PP=30|TP=0|ip-26-0-175-34]: No checkpoint path provided. [default6]:07/06/2024 09:21:03 [INFO|DP=0|PP=7|TP=0|ip-26-0-174-240]: Local number of parameters: 21M (40.01MiB) [default6]:07/06/2024 09:21:03 [INFO|DP=0|PP=7|TP=0|ip-26-0-174-240]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default6]:07/06/2024 09:21:03 [INFO|DP=0|PP=7|TP=0|ip-26-0-174-240]: No checkpoint path provided. [default1]:07/06/2024 09:21:03 [INFO|DP=0|PP=12|TP=1|ip-26-0-175-165]: Local number of parameters: 21M (40.01MiB) [default7]:07/06/2024 09:21:03 [INFO|DP=0|PP=7|TP=1|ip-26-0-174-240]: Local number of parameters: 21M (40.01MiB) [default1]:07/06/2024 09:21:03 [INFO|DP=0|PP=12|TP=1|ip-26-0-175-165]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default1]:07/06/2024 09:21:03 [INFO|DP=0|PP=12|TP=1|ip-26-0-175-165]: No checkpoint path provided. [default7]:07/06/2024 09:21:03 [INFO|DP=0|PP=7|TP=1|ip-26-0-174-240]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default7]:07/06/2024 09:21:03 [INFO|DP=0|PP=7|TP=1|ip-26-0-174-240]: No checkpoint path provided. [default2]:07/06/2024 09:21:03 [INFO|DP=0|PP=13|TP=0|ip-26-0-175-165]: Local number of parameters: 21M (40.01MiB) [default2]:07/06/2024 09:21:03 [INFO|DP=0|PP=13|TP=0|ip-26-0-175-165]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default2]:07/06/2024 09:21:03 [INFO|DP=0|PP=13|TP=0|ip-26-0-175-165]: No checkpoint path provided. [default4]:07/06/2024 09:21:03 [INFO|DP=0|PP=14|TP=0|ip-26-0-175-165]: Local number of parameters: 21M (40.01MiB) [default4]:07/06/2024 09:21:03 [INFO|DP=0|PP=14|TP=0|ip-26-0-175-165]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default5]:07/06/2024 09:21:03 [INFO|DP=0|PP=2|TP=1|ip-26-0-166-15]: Local number of parameters: 21M (40.01MiB) [default5]:07/06/2024 09:21:03 [INFO|DP=0|PP=2|TP=1|ip-26-0-166-15]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default5]:07/06/2024 09:21:03 [INFO|DP=0|PP=2|TP=1|ip-26-0-166-15]: No checkpoint path provided. [default4]:07/06/2024 09:21:03 [INFO|DP=0|PP=14|TP=0|ip-26-0-175-165]: No checkpoint path provided. [default2]:07/06/2024 09:21:03 [INFO|DP=0|PP=1|TP=0|ip-26-0-166-15]: Local number of parameters: 21M (40.01MiB) [default2]:07/06/2024 09:21:03 [INFO|DP=0|PP=1|TP=0|ip-26-0-166-15]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default2]:07/06/2024 09:21:03 [INFO|DP=0|PP=1|TP=0|ip-26-0-166-15]: No checkpoint path provided. [default4]:07/06/2024 09:21:03 [INFO|DP=0|PP=26|TP=0|ip-26-0-175-241]: Local number of parameters: 0 (0.00MiB) [default2]:07/06/2024 09:21:03 [INFO|DP=0|PP=5|TP=0|ip-26-0-174-240]: Local number of parameters: 21M (40.01MiB) [default2]:07/06/2024 09:21:03 [INFO|DP=0|PP=5|TP=0|ip-26-0-174-240]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default2]:07/06/2024 09:21:03 [INFO|DP=0|PP=5|TP=0|ip-26-0-174-240]: No checkpoint path provided. [default5]:07/06/2024 09:21:03 [INFO|DP=0|PP=26|TP=1|ip-26-0-175-241]: Local number of parameters: 0 (0.00MiB) [default5]:07/06/2024 09:21:03 [INFO|DP=0|PP=26|TP=1|ip-26-0-175-241]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB [default4]:07/06/2024 09:21:03 [INFO|DP=0|PP=26|TP=0|ip-26-0-175-241]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=24|TP=0|ip-26-0-175-241]: Local number of parameters: 2.05K (0.00MiB) [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=24|TP=0|ip-26-0-175-241]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB [default4]:07/06/2024 09:21:03 [INFO|DP=0|PP=26|TP=0|ip-26-0-175-241]: No checkpoint path provided. [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=24|TP=0|ip-26-0-175-241]: No checkpoint path provided. [default5]:07/06/2024 09:21:03 [INFO|DP=0|PP=26|TP=1|ip-26-0-175-241]: No checkpoint path provided. [default5]:07/06/2024 09:21:03 [INFO|DP=0|PP=14|TP=1|ip-26-0-175-165]: Local number of parameters: 21M (40.01MiB) [default5]:07/06/2024 09:21:03 [INFO|DP=0|PP=14|TP=1|ip-26-0-175-165]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default5]:07/06/2024 09:21:03 [INFO|DP=0|PP=14|TP=1|ip-26-0-175-165]: No checkpoint path provided. [default6]:07/06/2024 09:21:03 [INFO|DP=0|PP=31|TP=0|ip-26-0-175-34]: Local number of parameters: 0 (0.00MiB) [default6]:07/06/2024 09:21:03 [INFO|DP=0|PP=31|TP=0|ip-26-0-175-34]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB [default6]:07/06/2024 09:21:03 [INFO|DP=0|PP=31|TP=0|ip-26-0-175-34]: No checkpoint path provided. [default1]:07/06/2024 09:21:03 [INFO|DP=0|PP=28|TP=1|ip-26-0-175-34]: Local number of parameters: 0 (0.00MiB) [default1]:07/06/2024 09:21:03 [INFO|DP=0|PP=28|TP=1|ip-26-0-175-34]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB [default1]:07/06/2024 09:21:03 [INFO|DP=0|PP=28|TP=1|ip-26-0-175-34]: No checkpoint path provided. [default3]:07/06/2024 09:21:03 [INFO|DP=0|PP=5|TP=1|ip-26-0-174-240]: Local number of parameters: 21M (40.01MiB) [default3]:07/06/2024 09:21:03 [INFO|DP=0|PP=5|TP=1|ip-26-0-174-240]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default3]:07/06/2024 09:21:03 [INFO|DP=0|PP=5|TP=1|ip-26-0-174-240]: No checkpoint path provided. [default5]:07/06/2024 09:21:03 [INFO|DP=0|PP=6|TP=1|ip-26-0-174-240]: Local number of parameters: 21M (40.01MiB) [default5]:07/06/2024 09:21:03 [INFO|DP=0|PP=6|TP=1|ip-26-0-174-240]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default5]:07/06/2024 09:21:03 [INFO|DP=0|PP=6|TP=1|ip-26-0-174-240]: No checkpoint path provided. [default7]:07/06/2024 09:21:03 [INFO|DP=0|PP=15|TP=1|ip-26-0-175-165]: Local number of parameters: 21M (40.01MiB) [default7]:07/06/2024 09:21:03 [INFO|DP=0|PP=15|TP=1|ip-26-0-175-165]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default7]:07/06/2024 09:21:03 [INFO|DP=0|PP=15|TP=1|ip-26-0-175-165]: No checkpoint path provided. [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: Total number of parameters: 1.21G (2313.02MiB) [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: Local number of parameters: 72.4M (138.17MiB) [default3]:07/06/2024 09:21:03 [INFO|DP=0|PP=1|TP=1|ip-26-0-166-15]: Local number of parameters: 21M (40.01MiB) [default3]:07/06/2024 09:21:03 [INFO|DP=0|PP=1|TP=1|ip-26-0-166-15]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default3]:07/06/2024 09:21:03 [INFO|DP=0|PP=1|TP=1|ip-26-0-166-15]: No checkpoint path provided. [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: [After model building] Memory usage: 139.18MiB. Peak allocated: 141.21MiB Peak reserved: 156.00MiB [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: No checkpoint path provided. [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: Parametrizing model parameters using StandardParametrizator [default3]:07/06/2024 09:21:03 [INFO|DP=0|PP=17|TP=1|ip-26-0-175-170]: Local number of parameters: 21M (40.01MiB) [default3]:07/06/2024 09:21:03 [INFO|DP=0|PP=17|TP=1|ip-26-0-175-170]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default3]:07/06/2024 09:21:03 [INFO|DP=0|PP=17|TP=1|ip-26-0-175-170]: No checkpoint path provided. [default4]:07/06/2024 09:21:03 [INFO|DP=0|PP=18|TP=0|ip-26-0-175-170]: Local number of parameters: 21M (40.01MiB) [default4]:07/06/2024 09:21:03 [INFO|DP=0|PP=18|TP=0|ip-26-0-175-170]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default4]:07/06/2024 09:21:03 [INFO|DP=0|PP=18|TP=0|ip-26-0-175-170]: No checkpoint path provided. [default7]:07/06/2024 09:21:03 [INFO|DP=0|PP=31|TP=1|ip-26-0-175-34]: Local number of parameters: 0 (0.00MiB) [default7]:07/06/2024 09:21:03 [INFO|DP=0|PP=31|TP=1|ip-26-0-175-34]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB [default7]:07/06/2024 09:21:03 [INFO|DP=0|PP=31|TP=1|ip-26-0-175-34]: No checkpoint path provided. [default6]:07/06/2024 09:21:03 [INFO|DP=0|PP=15|TP=0|ip-26-0-175-165]: Local number of parameters: 21M (40.01MiB) [default6]:07/06/2024 09:21:03 [INFO|DP=0|PP=15|TP=0|ip-26-0-175-165]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default6]:07/06/2024 09:21:03 [INFO|DP=0|PP=15|TP=0|ip-26-0-175-165]: No checkpoint path provided. [default5]:07/06/2024 09:21:03 [INFO|DP=0|PP=18|TP=1|ip-26-0-175-170]: Local number of parameters: 21M (40.01MiB) [default5]:07/06/2024 09:21:03 [INFO|DP=0|PP=18|TP=1|ip-26-0-175-170]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default5]:07/06/2024 09:21:03 [INFO|DP=0|PP=18|TP=1|ip-26-0-175-170]: No checkpoint path provided. [default7]:07/06/2024 09:21:03 [INFO|DP=0|PP=19|TP=1|ip-26-0-175-170]: Local number of parameters: 21M (40.01MiB) [default7]:07/06/2024 09:21:03 [INFO|DP=0|PP=19|TP=1|ip-26-0-175-170]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default7]:07/06/2024 09:21:03 [INFO|DP=0|PP=19|TP=1|ip-26-0-175-170]: No checkpoint path provided. [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=16|TP=0|ip-26-0-175-170]: Local number of parameters: 21M (40.01MiB) [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=16|TP=0|ip-26-0-175-170]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default0]:07/06/2024 09:21:03 [INFO|DP=0|PP=16|TP=0|ip-26-0-175-170]: No checkpoint path provided. [default1]:07/06/2024 09:21:03 [INFO|DP=0|PP=16|TP=1|ip-26-0-175-170]: Local number of parameters: 21M (40.01MiB) [default1]:07/06/2024 09:21:03 [INFO|DP=0|PP=16|TP=1|ip-26-0-175-170]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default1]:07/06/2024 09:21:03 [INFO|DP=0|PP=16|TP=1|ip-26-0-175-170]: No checkpoint path provided. [default1]:07/06/2024 09:21:03 [INFO|DP=0|PP=0|TP=1|ip-26-0-166-15]: Local number of parameters: 72.4M (138.17MiB) [default1]:07/06/2024 09:21:03 [INFO|DP=0|PP=0|TP=1|ip-26-0-166-15]: [After model building] Memory usage: 139.18MiB. Peak allocated: 141.21MiB Peak reserved: 156.00MiB [default1]:07/06/2024 09:21:03 [INFO|DP=0|PP=0|TP=1|ip-26-0-166-15]: No checkpoint path provided. [default1]:07/06/2024 09:21:03 [INFO|DP=0|PP=24|TP=1|ip-26-0-175-241]: Local number of parameters: 2.05K (0.00MiB) [default1]:07/06/2024 09:21:03 [INFO|DP=0|PP=24|TP=1|ip-26-0-175-241]: [After model building] Memory usage: 0.01MiB. Peak allocated: 0.03MiB Peak reserved: 2.00MiB [default1]:07/06/2024 09:21:03 [INFO|DP=0|PP=24|TP=1|ip-26-0-175-241]: No checkpoint path provided. [default2]:07/06/2024 09:21:03 [INFO|DP=0|PP=25|TP=0|ip-26-0-175-241]: Local number of parameters: 51.5M (98.16MiB) [default2]:07/06/2024 09:21:03 [INFO|DP=0|PP=25|TP=0|ip-26-0-175-241]: [After model building] Memory usage: 98.17MiB. Peak allocated: 98.19MiB Peak reserved: 102.00MiB [default2]:07/06/2024 09:21:03 [INFO|DP=0|PP=25|TP=0|ip-26-0-175-241]: No checkpoint path provided. [default6]:07/06/2024 09:21:03 [INFO|DP=0|PP=19|TP=0|ip-26-0-175-170]: Local number of parameters: 21M (40.01MiB) [default6]:07/06/2024 09:21:03 [INFO|DP=0|PP=19|TP=0|ip-26-0-175-170]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default6]:07/06/2024 09:21:03 [INFO|DP=0|PP=19|TP=0|ip-26-0-175-170]: No checkpoint path provided. [default2]:07/06/2024 09:21:03 [INFO|DP=0|PP=17|TP=0|ip-26-0-175-170]: Local number of parameters: 21M (40.01MiB) [default2]:07/06/2024 09:21:03 [INFO|DP=0|PP=17|TP=0|ip-26-0-175-170]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default2]:07/06/2024 09:21:03 [INFO|DP=0|PP=17|TP=0|ip-26-0-175-170]: No checkpoint path provided. [default6]:07/06/2024 09:21:03 [INFO|DP=0|PP=3|TP=0|ip-26-0-166-15]: Local number of parameters: 21M (40.01MiB) [default6]:07/06/2024 09:21:03 [INFO|DP=0|PP=3|TP=0|ip-26-0-166-15]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default6]:07/06/2024 09:21:03 [INFO|DP=0|PP=3|TP=0|ip-26-0-166-15]: No checkpoint path provided. [default7]:07/06/2024 09:21:03 [INFO|DP=0|PP=3|TP=1|ip-26-0-166-15]: Local number of parameters: 21M (40.01MiB) [default7]:07/06/2024 09:21:03 [INFO|DP=0|PP=3|TP=1|ip-26-0-166-15]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default7]:07/06/2024 09:21:03 [INFO|DP=0|PP=3|TP=1|ip-26-0-166-15]: No checkpoint path provided. [default4]:07/06/2024 09:21:03 [INFO|DP=0|PP=2|TP=0|ip-26-0-166-15]: Local number of parameters: 21M (40.01MiB) [default4]:07/06/2024 09:21:03 [INFO|DP=0|PP=2|TP=0|ip-26-0-166-15]: [After model building] Memory usage: 41.02MiB. Peak allocated: 43.05MiB Peak reserved: 56.00MiB [default4]:07/06/2024 09:21:03 [INFO|DP=0|PP=2|TP=0|ip-26-0-166-15]: No checkpoint path provided. [default0]:07/06/2024 09:21:05 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: [Optimizer Building] Using LearningRateForSP as learning rate [default0]:07/06/2024 09:21:05 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: [ZeRO sharding] Size of optimizer params per rank: [default0]:07/06/2024 09:21:05 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: [ZeRO sharding] DP Rank 0 has 72.4M out of 72.4M (100.00%) params' optimizer states [default3]:Traceback (most recent call last): [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default3]: trainer = DistributedTrainer(config_file) [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 185, in __init__ [default3]: self.optimizer, self.grad_accumulator = init_optimizer_and_grad_accumulator( [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/helpers.py", line 401, in init_optimizer_and_grad_accumulator [default3]: param = model.get_parameter(optim_model_param_name) [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 714, in get_parameter [default3]: mod: torch.nn.Module = self.get_submodule(module_path) [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 681, in get_submodule [default3]: raise AttributeError(mod._get_name() + " has no " [default3]:AttributeError: PipelineBlock has no attribute `pp_block` [default2]:Traceback (most recent call last): [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default2]: trainer = DistributedTrainer(config_file) [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 185, in __init__ [default2]: self.optimizer, self.grad_accumulator = init_optimizer_and_grad_accumulator( [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/helpers.py", line 401, in init_optimizer_and_grad_accumulator [default2]: param = model.get_parameter(optim_model_param_name) [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 714, in get_parameter [default2]: mod: torch.nn.Module = self.get_submodule(module_path) [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 681, in get_submodule [default2]: raise AttributeError(mod._get_name() + " has no " [default2]:AttributeError: PipelineBlock has no attribute `pp_block` [default0]:07/06/2024 09:21:05 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: [Training Plan] Stage Training Stage has 19 remaining training steps and has consumed 0 samples [default0]:07/06/2024 09:21:05 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: Using `datasets` library [default0]:07/06/2024 09:21:05 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: Loading tokenizer from openai-community/gpt2 and transformers/hf_hub versions ('4.41.2', '0.23.4') [default0]:07/06/2024 09:21:06 [WARNING|DP=0|PP=0|TP=0|ip-26-0-166-15]: Repo card metadata block was not found. Setting CardData to empty. [default0]:Repo card metadata block was not found. Setting CardData to empty. [2024-07-06 09:21:09,741] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1328750 closing signal SIGTERM [2024-07-06 09:21:09,741] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1328751 closing signal SIGTERM [2024-07-06 09:21:09,742] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1328754 closing signal SIGTERM [2024-07-06 09:21:09,743] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1328755 closing signal SIGTERM [2024-07-06 09:21:09,744] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1328756 closing signal SIGTERM [2024-07-06 09:21:09,744] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1328757 closing signal SIGTERM [default0]:07/06/2024 09:21:11 [WARNING|DP=0|PP=8|TP=0|ip-26-0-175-132]: Repo card metadata block was not found. Setting CardData to empty. [default6]:07/06/2024 09:21:11 [WARNING|DP=0|PP=11|TP=0|ip-26-0-175-132]: Repo card metadata block was not found. Setting CardData to empty. [default4]:07/06/2024 09:21:11 [WARNING|DP=0|PP=10|TP=0|ip-26-0-175-132]: Repo card metadata block was not found. Setting CardData to empty. [default2]:07/06/2024 09:21:11 [WARNING|DP=0|PP=9|TP=0|ip-26-0-175-132]: Repo card metadata block was not found. Setting CardData to empty. [default4]:Repo card metadata block was not found. Setting CardData to empty. [default5]:07/06/2024 09:21:11 [WARNING|DP=0|PP=10|TP=1|ip-26-0-175-132]: Repo card metadata block was not found. Setting CardData to empty. [default6]:Repo card metadata block was not found. Setting CardData to empty. [default5]:Repo card metadata block was not found. Setting CardData to empty. [default2]:Repo card metadata block was not found. Setting CardData to empty. [default4]:07/06/2024 09:21:11 [WARNING|DP=0|PP=6|TP=0|ip-26-0-174-240]: Repo card metadata block was not found. Setting CardData to empty. [default1]:07/06/2024 09:21:11 [WARNING|DP=0|PP=4|TP=1|ip-26-0-174-240]: Repo card metadata block was not found. Setting CardData to empty. [default4]:Repo card metadata block was not found. Setting CardData to empty. [default4]:Repo card metadata block was not found. Setting CardData to empty. [default3]:Repo card metadata block was not found. Setting CardData to empty. [default3]:07/06/2024 09:21:11 [WARNING|DP=0|PP=13|TP=1|ip-26-0-175-165]: Repo card metadata block was not found. Setting CardData to empty. [default0]:Repo card metadata block was not found. Setting CardData to empty. [default5]:Repo card metadata block was not found. Setting CardData to empty. [default6]:Repo card metadata block was not found. Setting CardData to empty. [default6]:Repo card metadata block was not found. Setting CardData to empty. [default1]:Repo card metadata block was not found. Setting CardData to empty. [default1]:07/06/2024 09:21:11 [WARNING|DP=0|PP=12|TP=1|ip-26-0-175-165]: Repo card metadata block was not found. Setting CardData to empty. [default7]:Repo card metadata block was not found. Setting CardData to empty. [default6]:07/06/2024 09:21:11 [WARNING|DP=0|PP=7|TP=0|ip-26-0-174-240]: Repo card metadata block was not found. Setting CardData to empty. [default4]:07/06/2024 09:21:11 [WARNING|DP=0|PP=14|TP=0|ip-26-0-175-165]: Repo card metadata block was not found. Setting CardData to empty. [default7]:07/06/2024 09:21:11 [WARNING|DP=0|PP=7|TP=1|ip-26-0-174-240]: Repo card metadata block was not found. Setting CardData to empty. [default2]:Repo card metadata block was not found. Setting CardData to empty. [default2]:07/06/2024 09:21:11 [WARNING|DP=0|PP=5|TP=0|ip-26-0-174-240]: Repo card metadata block was not found. Setting CardData to empty. [default5]:07/06/2024 09:21:11 [WARNING|DP=0|PP=14|TP=1|ip-26-0-175-165]: Repo card metadata block was not found. Setting CardData to empty. [default1]:Repo card metadata block was not found. Setting CardData to empty. [default6]:07/06/2024 09:21:11 [WARNING|DP=0|PP=15|TP=0|ip-26-0-175-165]: Repo card metadata block was not found. Setting CardData to empty. [default1]:07/06/2024 09:21:11 [WARNING|DP=0|PP=8|TP=1|ip-26-0-175-132]: Repo card metadata block was not found. Setting CardData to empty. [default3]:Repo card metadata block was not found. Setting CardData to empty. [default3]:07/06/2024 09:21:11 [WARNING|DP=0|PP=9|TP=1|ip-26-0-175-132]: Repo card metadata block was not found. Setting CardData to empty. [default1]:Repo card metadata block was not found. Setting CardData to empty. [default0]:07/06/2024 09:21:11 [WARNING|DP=0|PP=4|TP=0|ip-26-0-174-240]: Repo card metadata block was not found. Setting CardData to empty. [default0]:Repo card metadata block was not found. Setting CardData to empty. [default3]:Repo card metadata block was not found. Setting CardData to empty. [default2]:07/06/2024 09:21:11 [WARNING|DP=0|PP=13|TP=0|ip-26-0-175-165]: Repo card metadata block was not found. Setting CardData to empty. [default2]:Repo card metadata block was not found. Setting CardData to empty. [default7]:Repo card metadata block was not found. Setting CardData to empty. [default5]:Repo card metadata block was not found. Setting CardData to empty. [default7]:07/06/2024 09:21:11 [WARNING|DP=0|PP=15|TP=1|ip-26-0-175-165]: Repo card metadata block was not found. Setting CardData to empty. [default5]:07/06/2024 09:21:11 [WARNING|DP=0|PP=6|TP=1|ip-26-0-174-240]: Repo card metadata block was not found. Setting CardData to empty. [default3]:07/06/2024 09:21:11 [WARNING|DP=0|PP=5|TP=1|ip-26-0-174-240]: Repo card metadata block was not found. Setting CardData to empty. [default7]:07/06/2024 09:21:11 [WARNING|DP=0|PP=11|TP=1|ip-26-0-175-132]: Repo card metadata block was not found. Setting CardData to empty. [default0]:07/06/2024 09:21:11 [WARNING|DP=0|PP=12|TP=0|ip-26-0-175-165]: Repo card metadata block was not found. Setting CardData to empty. [default0]:Repo card metadata block was not found. Setting CardData to empty. [default7]:Repo card metadata block was not found. Setting CardData to empty. [2024-07-06 09:21:11,662] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 2 (pid: 1328752) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 Traceback (most recent call last): File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in sys.exit(main()) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper return f(*args, **kwargs) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main run(args) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run elastic_launch( File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED ------------------------------------------------------------ Failures: [1]: time : 2024-07-06_09:21:09 host : ip-26-0-175-241.ec2.internal rank : 51 (local_rank: 3) exitcode : 1 (pid: 1328753) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-07-06_09:21:09 host : ip-26-0-175-241.ec2.internal rank : 50 (local_rank: 2) exitcode : 1 (pid: 1328752) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================ srun: error: ip-26-0-175-241: task 7: Exited with exit code 1 [default0]:[rank56]:[E ProcessGroupNCCL.cpp:537] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. [default0]:[rank56]:[E ProcessGroupNCCL.cpp:543] To avoid data inconsistency, we are taking the entire process down. [default0]:[rank56]:[E ProcessGroupNCCL.cpp:1182] [Rank 56] NCCL watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.19.3 [default0]:ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely. [default0]:Last error: [default0]:socketProgress: Connection closed by remote peer ip-26-0-175-241.ec2.internal<34092> [default0]:Exception raised from checkForNCCLErrorsInternal at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1436 (most recent call first): [default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7efd0e49ad87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default0]:frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::vector, std::allocator > > const&) + 0x2f3 (0x7efd0f641fa3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7b (0x7efd0f64227b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x17d (0x7efd0f645c1d in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7efd0f646839 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #5: + 0xd3e95 (0x7efd5934ae95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) [default0]:frame #6: + 0x8609 (0x7efd5e452609 in /lib/x86_64-linux-gnu/libpthread.so.0) [default0]:frame #7: clone + 0x43 (0x7efd5e21d353 in /lib/x86_64-linux-gnu/libc.so.6) [default0]: [default0]:terminate called after throwing an instance of 'c10::DistBackendError' [default0]: what(): [Rank 56] NCCL watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.19.3 [default0]:ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely. [default0]:Last error: [default0]:socketProgress: Connection closed by remote peer ip-26-0-175-241.ec2.internal<34092> [default0]:Exception raised from checkForNCCLErrorsInternal at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1436 (most recent call first): [default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7efd0e49ad87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default0]:frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::vector, std::allocator > > const&) + 0x2f3 (0x7efd0f641fa3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7b (0x7efd0f64227b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x17d (0x7efd0f645c1d in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7efd0f646839 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #5: + 0xd3e95 (0x7efd5934ae95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) [default0]:frame #6: + 0x8609 (0x7efd5e452609 in /lib/x86_64-linux-gnu/libpthread.so.0) [default0]:frame #7: clone + 0x43 (0x7efd5e21d353 in /lib/x86_64-linux-gnu/libc.so.6) [default0]: [default0]:Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1186 (most recent call first): [default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7efd0e49ad87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default0]:frame #1: + 0xdf6b11 (0x7efd0f39cb11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #2: + 0xd3e95 (0x7efd5934ae95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) [default0]:frame #3: + 0x8609 (0x7efd5e452609 in /lib/x86_64-linux-gnu/libpthread.so.0) [default0]:frame #4: clone + 0x43 (0x7efd5e21d353 in /lib/x86_64-linux-gnu/libc.so.6) [default0]: [default7]:Repo card metadata block was not found. Setting CardData to empty. [default6]:Repo card metadata block was not found. Setting CardData to empty. [default2]:07/06/2024 09:21:15 [WARNING|DP=0|PP=29|TP=0|ip-26-0-175-34]: Repo card metadata block was not found. Setting CardData to empty. [default1]:07/06/2024 09:21:15 [WARNING|DP=0|PP=28|TP=1|ip-26-0-175-34]: Repo card metadata block was not found. Setting CardData to empty. [default6]:07/06/2024 09:21:15 [WARNING|DP=0|PP=31|TP=0|ip-26-0-175-34]: Repo card metadata block was not found. Setting CardData to empty. [default2]:Repo card metadata block was not found. Setting CardData to empty. [default7]:07/06/2024 09:21:15 [WARNING|DP=0|PP=31|TP=1|ip-26-0-175-34]: Repo card metadata block was not found. Setting CardData to empty. [default1]:Repo card metadata block was not found. Setting CardData to empty. [default5]:Repo card metadata block was not found. Setting CardData to empty. [default4]:07/06/2024 09:21:15 [WARNING|DP=0|PP=30|TP=0|ip-26-0-175-34]: Repo card metadata block was not found. Setting CardData to empty. [default5]:07/06/2024 09:21:15 [WARNING|DP=0|PP=30|TP=1|ip-26-0-175-34]: Repo card metadata block was not found. Setting CardData to empty. [default3]:07/06/2024 09:21:15 [WARNING|DP=0|PP=29|TP=1|ip-26-0-175-34]: Repo card metadata block was not found. Setting CardData to empty. [default0]:[rank32]:[E ProcessGroupNCCL.cpp:537] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. [default0]:[rank32]:[E ProcessGroupNCCL.cpp:543] To avoid data inconsistency, we are taking the entire process down. [default0]:[rank32]:[E ProcessGroupNCCL.cpp:1182] [Rank 32] NCCL watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.19.3 [default0]:ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely. [default0]:Last error: [default0]:socketProgress: Connection closed by remote peer ip-26-0-175-241.ec2.internal<33314> [default0]:Exception raised from checkForNCCLErrorsInternal at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1436 (most recent call first): [default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f6e51012d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default0]:frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::vector, std::allocator > > const&) + 0x2f3 (0x7f6e521b9fa3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7b (0x7f6e521ba27b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x17d (0x7f6e521bdc1d in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7f6e521be839 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #5: + 0xd3e95 (0x7f6e9bec2e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) [default0]:frame #6: + 0x8609 (0x7f6ea0fca609 in /lib/x86_64-linux-gnu/libpthread.so.0) [default0]:frame #7: clone + 0x43 (0x7f6ea0d95353 in /lib/x86_64-linux-gnu/libc.so.6) [default0]: [default0]:terminate called after throwing an instance of 'c10::DistBackendError' [default0]: what(): [Rank 32] NCCL watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.19.3 [default0]:ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely. [default0]:Last error: [default0]:socketProgress: Connection closed by remote peer ip-26-0-175-241.ec2.internal<33314> [default0]:Exception raised from checkForNCCLErrorsInternal at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1436 (most recent call first): [default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f6e51012d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default0]:frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::vector, std::allocator > > const&) + 0x2f3 (0x7f6e521b9fa3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7b (0x7f6e521ba27b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x17d (0x7f6e521bdc1d in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7f6e521be839 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #5: + 0xd3e95 (0x7f6e9bec2e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) [default0]:frame #6: + 0x8609 (0x7f6ea0fca609 in /lib/x86_64-linux-gnu/libpthread.so.0) [default0]:frame #7: clone + 0x43 (0x7f6ea0d95353 in /lib/x86_64-linux-gnu/libc.so.6) [default0]: [default0]:Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1186 (most recent call first): [default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f6e51012d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default0]:frame #1: + 0xdf6b11 (0x7f6e51f14b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default4]:Repo card metadata block was not found. Setting CardData to empty. [default0]:frame #2: + 0xd3e95 (0x7f6e9bec2e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) [default0]:frame #3: + 0x8609 (0x7f6ea0fca609 in /lib/x86_64-linux-gnu/libpthread.so.0) [default0]:frame #4: clone + 0x43 (0x7f6ea0d95353 in /lib/x86_64-linux-gnu/libc.so.6) [default0]: [default3]:Repo card metadata block was not found. Setting CardData to empty. [default0]:[rank40]:[E ProcessGroupNCCL.cpp:537] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. [default0]:[rank40]:[E ProcessGroupNCCL.cpp:543] To avoid data inconsistency, we are taking the entire process down. [default0]:[rank40]:[E ProcessGroupNCCL.cpp:1182] [Rank 40] NCCL watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.19.3 [default0]:ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely. [default0]:Last error: [default0]:socketProgress: Connection closed by remote peer ip-26-0-175-241.ec2.internal<46152> [default0]:Exception raised from checkForNCCLErrorsInternal at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1436 (most recent call first): [default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f9c3c830d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default0]:frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::vector, std::allocator > > const&) + 0x2f3 (0x7f9c3d9d7fa3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7b (0x7f9c3d9d827b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x17d (0x7f9c3d9dbc1d in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7f9c3d9dc839 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #5: + 0xd3e95 (0x7f9c876e0e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) [default0]:frame #6: + 0x8609 (0x7f9c8c7e8609 in /lib/x86_64-linux-gnu/libpthread.so.0) [default0]:frame #7: clone + 0x43 (0x7f9c8c5b3353 in /lib/x86_64-linux-gnu/libc.so.6) [default0]: [default0]:terminate called after throwing an instance of 'c10::DistBackendError' [default0]: what(): [Rank 40] NCCL watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.19.3 [default0]:ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely. [default0]:Last error: [default0]:socketProgress: Connection closed by remote peer ip-26-0-175-241.ec2.internal<46152> [default0]:Exception raised from checkForNCCLErrorsInternal at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1436 (most recent call first): [default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f9c3c830d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default0]:frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::vector, std::allocator > > const&) + 0x2f3 (0x7f9c3d9d7fa3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7b (0x7f9c3d9d827b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x17d (0x7f9c3d9dbc1d in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7f9c3d9dc839 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #5: + 0xd3e95 (0x7f9c876e0e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) [default0]:frame #6: + 0x8609 (0x7f9c8c7e8609 in /lib/x86_64-linux-gnu/libpthread.so.0) [default0]:frame #7: clone + 0x43 (0x7f9c8c5b3353 in /lib/x86_64-linux-gnu/libc.so.6) [default0]: [default0]:Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1186 (most recent call first): [default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f9c3c830d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default0]:frame #1: + 0xdf6b11 (0x7f9c3d732b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #2: + 0xd3e95 (0x7f9c876e0e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) [default0]:frame #3: + 0x8609 (0x7f9c8c7e8609 in /lib/x86_64-linux-gnu/libpthread.so.0) [default0]:frame #4: clone + 0x43 (0x7f9c8c5b3353 in /lib/x86_64-linux-gnu/libc.so.6) [default0]: [default4]:07/06/2024 09:21:16 [WARNING|DP=0|PP=18|TP=0|ip-26-0-175-170]: Repo card metadata block was not found. Setting CardData to empty. [default5]:07/06/2024 09:21:16 [WARNING|DP=0|PP=18|TP=1|ip-26-0-175-170]: Repo card metadata block was not found. Setting CardData to empty. [default4]:Repo card metadata block was not found. Setting CardData to empty. [default1]:07/06/2024 09:21:16 [WARNING|DP=0|PP=16|TP=1|ip-26-0-175-170]: Repo card metadata block was not found. Setting CardData to empty. [default2]:07/06/2024 09:21:16 [WARNING|DP=0|PP=17|TP=0|ip-26-0-175-170]: Repo card metadata block was not found. Setting CardData to empty. [default1]:Repo card metadata block was not found. Setting CardData to empty. [default2]:Repo card metadata block was not found. Setting CardData to empty. [default5]:Repo card metadata block was not found. Setting CardData to empty. [default7]:Repo card metadata block was not found. Setting CardData to empty. [default7]:07/06/2024 09:21:16 [WARNING|DP=0|PP=19|TP=1|ip-26-0-175-170]: Repo card metadata block was not found. Setting CardData to empty. [default6]:07/06/2024 09:21:16 [WARNING|DP=0|PP=19|TP=0|ip-26-0-175-170]: Repo card metadata block was not found. Setting CardData to empty. [default6]:Repo card metadata block was not found. Setting CardData to empty. [default0]:07/06/2024 09:21:16 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: [Training Plan] There are 1 training stages [default0]:07/06/2024 09:21:16 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: [Stage Training Stage] start from step 1 [default0]:07/06/2024 09:21:16 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: [default0]:07/06/2024 09:21:16 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: [Start training] datetime: 2024-07-06 09:21:16.586235 | mbs: 32 | grad_accum: 32 | global_batch_size: 1024 | sequence_length: 4096 | train_steps: 20 | start_iteration_step: 0 | consumed_train_samples: 0 [default6]:07/06/2024 09:21:16 [WARNING|DP=0|PP=23|TP=0|ip-26-0-175-19]: Repo card metadata block was not found. Setting CardData to empty. [default3]:07/06/2024 09:21:16 [WARNING|DP=0|PP=21|TP=1|ip-26-0-175-19]: Repo card metadata block was not found. Setting CardData to empty. [default5]:Repo card metadata block was not found. Setting CardData to empty. [default4]:07/06/2024 09:21:16 [WARNING|DP=0|PP=22|TP=0|ip-26-0-175-19]: Repo card metadata block was not found. Setting CardData to empty. [default1]:07/06/2024 09:21:16 [WARNING|DP=0|PP=20|TP=1|ip-26-0-175-19]: Repo card metadata block was not found. Setting CardData to empty. [default7]:07/06/2024 09:21:16 [WARNING|DP=0|PP=23|TP=1|ip-26-0-175-19]: Repo card metadata block was not found. Setting CardData to empty. [default4]:Repo card metadata block was not found. Setting CardData to empty. [default5]:07/06/2024 09:21:16 [WARNING|DP=0|PP=22|TP=1|ip-26-0-175-19]: Repo card metadata block was not found. Setting CardData to empty. [default1]:Repo card metadata block was not found. Setting CardData to empty. [default3]:Repo card metadata block was not found. Setting CardData to empty. [default6]:Repo card metadata block was not found. Setting CardData to empty. [default7]:Repo card metadata block was not found. Setting CardData to empty. [default2]:07/06/2024 09:21:16 [WARNING|DP=0|PP=21|TP=0|ip-26-0-175-19]: Repo card metadata block was not found. Setting CardData to empty. [default2]:Repo card metadata block was not found. Setting CardData to empty. [default3]:07/06/2024 09:21:17 [WARNING|DP=0|PP=17|TP=1|ip-26-0-175-170]: Repo card metadata block was not found. Setting CardData to empty. [default3]:Repo card metadata block was not found. Setting CardData to empty. [default0]:07/06/2024 09:21:17 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: Resuming training from stage Training Stage, it has trained for 0 samples and has 19 remaining train steps [default0]:07/06/2024 09:21:17 [INFO|DP=0|PP=0|TP=0|ip-26-0-166-15]: Memory usage: 691.85MiB. Peak allocated 691.85MiB. Peak reserved: 712.00MiB [2024-07-06 09:21:19,743] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1227618 closing signal SIGTERM [2024-07-06 09:21:19,743] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1227619 closing signal SIGTERM [2024-07-06 09:21:19,744] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1227620 closing signal SIGTERM [2024-07-06 09:21:19,744] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1227621 closing signal SIGTERM [2024-07-06 09:21:19,745] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1227622 closing signal SIGTERM [2024-07-06 09:21:19,745] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1227623 closing signal SIGTERM [2024-07-06 09:21:19,745] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1227624 closing signal SIGTERM [2024-07-06 09:21:19,748] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 3640142 closing signal SIGTERM [2024-07-06 09:21:19,749] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 3640143 closing signal SIGTERM [2024-07-06 09:21:19,750] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 3640144 closing signal SIGTERM [2024-07-06 09:21:19,751] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 3640145 closing signal SIGTERM [2024-07-06 09:21:19,751] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 3640146 closing signal SIGTERM [2024-07-06 09:21:19,752] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 3640147 closing signal SIGTERM [2024-07-06 09:21:19,752] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 3640148 closing signal SIGTERM [2024-07-06 09:21:19,752] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1198360 closing signal SIGTERM [2024-07-06 09:21:19,752] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1198361 closing signal SIGTERM [2024-07-06 09:21:19,753] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1198362 closing signal SIGTERM [2024-07-06 09:21:19,753] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1198363 closing signal SIGTERM [2024-07-06 09:21:19,754] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1198364 closing signal SIGTERM [2024-07-06 09:21:19,754] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1198365 closing signal SIGTERM [2024-07-06 09:21:19,754] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1198366 closing signal SIGTERM [2024-07-06 09:21:21,771] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -6) local_rank: 0 (pid: 1198359) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 Traceback (most recent call last): File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in sys.exit(main()) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper return f(*args, **kwargs) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main run(args) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run elastic_launch( File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED ------------------------------------------------------------ Failures: ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-07-06_09:21:19 host : ip-26-0-175-34.ec2.internal rank : 56 (local_rank: 0) exitcode : -6 (pid: 1198359) error_file: traceback : Signal 6 (SIGABRT) received by PID 1198359 ============================================================ [2024-07-06 09:21:21,977] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -6) local_rank: 0 (pid: 3640141) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 Traceback (most recent call last): File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in sys.exit(main()) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper return f(*args, **kwargs) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main run(args) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run elastic_launch( File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED ------------------------------------------------------------ Failures: ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-07-06_09:21:19 host : ip-26-0-175-170.ec2.internal rank : 32 (local_rank: 0) exitcode : -6 (pid: 3640141) error_file: traceback : Signal 6 (SIGABRT) received by PID 3640141 ============================================================ [2024-07-06 09:21:22,068] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -6) local_rank: 0 (pid: 1227617) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 Traceback (most recent call last): File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in sys.exit(main()) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper srun: error: ip-26-0-175-34: task 3: Exited with exit code 1 return f(*args, **kwargs) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main run(args) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run elastic_launch( File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED ------------------------------------------------------------ Failures: ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-07-06_09:21:19 host : ip-26-0-175-19.ec2.internal rank : 40 (local_rank: 0) exitcode : -6 (pid: 1227617) error_file: traceback : Signal 6 (SIGABRT) received by PID 1227617 ============================================================ srun: error: ip-26-0-175-170: task 6: Exited with exit code 1 srun: error: ip-26-0-175-19: task 2: Exited with exit code 1 [default0]:[rank16]:[E ProcessGroupNCCL.cpp:537] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. [default0]:[rank16]:[E ProcessGroupNCCL.cpp:543] To avoid data inconsistency, we are taking the entire process down. [default0]:[rank16]:[E ProcessGroupNCCL.cpp:1182] [Rank 16] NCCL watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.19.3 [default0]:ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely. [default0]:Last error: [default0]:socketProgress: Connection closed by remote peer ip-26-0-175-170.ec2.internal<59874> [default0]:Exception raised from checkForNCCLErrorsInternal at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1436 (most recent call first): [default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f3e123acd87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default0]:frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::vector, std::allocator > > const&) + 0x2f3 (0x7f3e13553fa3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7b (0x7f3e1355427b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x17d (0x7f3e13557c1d in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7f3e13558839 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #5: + 0xd3e95 (0x7f3e5d25ce95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) [default0]:frame #6: + 0x8609 (0x7f3e62364609 in /lib/x86_64-linux-gnu/libpthread.so.0) [default0]:frame #7: clone + 0x43 (0x7f3e6212f353 in /lib/x86_64-linux-gnu/libc.so.6) [default0]: [default0]:terminate called after throwing an instance of 'c10::DistBackendError' [default0]: what(): [Rank 16] NCCL watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.19.3 [default0]:ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely. [default0]:Last error: [default0]:socketProgress: Connection closed by remote peer ip-26-0-175-170.ec2.internal<59874> [default0]:Exception raised from checkForNCCLErrorsInternal at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1436 (most recent call first): [default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f3e123acd87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default0]:frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::vector, std::allocator > > const&) + 0x2f3 (0x7f3e13553fa3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7b (0x7f3e1355427b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x17d (0x7f3e13557c1d in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7f3e13558839 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #5: + 0xd3e95 (0x7f3e5d25ce95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) [default0]:frame #6: + 0x8609 (0x7f3e62364609 in /lib/x86_64-linux-gnu/libpthread.so.0) [default0]:frame #7: clone + 0x43 (0x7f3e6212f353 in /lib/x86_64-linux-gnu/libc.so.6) [default0]: [default0]:Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1186 (most recent call first): [default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f3e123acd87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default0]:frame #1: + 0xdf6b11 (0x7f3e132aeb11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #2: + 0xd3e95 (0x7f3e5d25ce95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) [default0]:frame #3: + 0x8609 (0x7f3e62364609 in /lib/x86_64-linux-gnu/libpthread.so.0) [default0]:frame #4: clone + 0x43 (0x7f3e6212f353 in /lib/x86_64-linux-gnu/libc.so.6) [default0]: [default3]:07/06/2024 09:21:26 [WARNING|DP=0|PP=1|TP=1|ip-26-0-166-15]: Repo card metadata block was not found. Setting CardData to empty. [default5]:07/06/2024 09:21:26 [WARNING|DP=0|PP=2|TP=1|ip-26-0-166-15]: Repo card metadata block was not found. Setting CardData to empty. [default2]:07/06/2024 09:21:26 [WARNING|DP=0|PP=1|TP=0|ip-26-0-166-15]: Repo card metadata block was not found. Setting CardData to empty. [default1]:07/06/2024 09:21:26 [WARNING|DP=0|PP=0|TP=1|ip-26-0-166-15]: Repo card metadata block was not found. Setting CardData to empty. [default4]:07/06/2024 09:21:26 [WARNING|DP=0|PP=2|TP=0|ip-26-0-166-15]: Repo card metadata block was not found. Setting CardData to empty. [default1]:Repo card metadata block was not found. Setting CardData to empty. [default7]:07/06/2024 09:21:26 [WARNING|DP=0|PP=3|TP=1|ip-26-0-166-15]: Repo card metadata block was not found. Setting CardData to empty. [default3]:Repo card metadata block was not found. Setting CardData to empty. [default2]:Repo card metadata block was not found. Setting CardData to empty. [default5]:Repo card metadata block was not found. Setting CardData to empty. [default6]:Repo card metadata block was not found. Setting CardData to empty. [default4]:Repo card metadata block was not found. Setting CardData to empty. [default7]:Repo card metadata block was not found. Setting CardData to empty. [default6]:07/06/2024 09:21:26 [WARNING|DP=0|PP=3|TP=0|ip-26-0-166-15]: Repo card metadata block was not found. Setting CardData to empty. [2024-07-06 09:21:29,763] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 486140 closing signal SIGTERM [2024-07-06 09:21:29,763] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 486141 closing signal SIGTERM [2024-07-06 09:21:29,764] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 486142 closing signal SIGTERM [2024-07-06 09:21:29,765] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 486143 closing signal SIGTERM [2024-07-06 09:21:29,766] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 486144 closing signal SIGTERM [2024-07-06 09:21:29,766] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 486145 closing signal SIGTERM [2024-07-06 09:21:29,766] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 486146 closing signal SIGTERM [2024-07-06 09:21:31,882] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -6) local_rank: 0 (pid: 486139) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 Traceback (most recent call last): File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in sys.exit(main()) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper return f(*args, **kwargs) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main run(args) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run elastic_launch( File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED ------------------------------------------------------------ Failures: ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-07-06_09:21:29 host : ip-26-0-175-132.ec2.internal rank : 16 (local_rank: 0) exitcode : -6 (pid: 486139) error_file: traceback : Signal 6 (SIGABRT) received by PID 486139 ============================================================ srun: error: ip-26-0-175-132: task 4: Exited with exit code 1 [default0]:[rank0]:[E ProcessGroupNCCL.cpp:537] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. [default0]:[rank0]:[E ProcessGroupNCCL.cpp:543] To avoid data inconsistency, we are taking the entire process down. [default0]:[rank0]:[E ProcessGroupNCCL.cpp:1182] [Rank 0] NCCL watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.19.3 [default0]:ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely. [default0]:Last error: [default0]:socketProgress: Connection closed by remote peer ip-26-0-175-170.ec2.internal<52812> [default0]:Exception raised from checkForNCCLErrorsInternal at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1436 (most recent call first): [default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f3889985d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default0]:frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::vector, std::allocator > > const&) + 0x2f3 (0x7f388ab2cfa3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7b (0x7f388ab2d27b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x17d (0x7f388ab30c1d in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7f388ab31839 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #5: + 0xd3e95 (0x7f38d4835e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) [default0]:frame #6: + 0x8609 (0x7f38d993d609 in /lib/x86_64-linux-gnu/libpthread.so.0) [default0]:frame #7: clone + 0x43 (0x7f38d9708353 in /lib/x86_64-linux-gnu/libc.so.6) [default0]: [default0]:terminate called after throwing an instance of 'c10::DistBackendError' [default0]: what(): [Rank 0] NCCL watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.19.3 [default0]:ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely. [default0]:Last error: [default0]:socketProgress: Connection closed by remote peer ip-26-0-175-170.ec2.internal<52812> [default0]:Exception raised from checkForNCCLErrorsInternal at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1436 (most recent call first): [default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f3889985d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default0]:frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::vector, std::allocator > > const&) + 0x2f3 (0x7f388ab2cfa3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7b (0x7f388ab2d27b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x17d (0x7f388ab30c1d in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7f388ab31839 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #5: + 0xd3e95 (0x7f38d4835e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) [default0]:frame #6: + 0x8609 (0x7f38d993d609 in /lib/x86_64-linux-gnu/libpthread.so.0) [default0]:frame #7: clone + 0x43 (0x7f38d9708353 in /lib/x86_64-linux-gnu/libc.so.6) [default0]: [default0]:Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1186 (most recent call first): [default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f3889985d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default0]:frame #1: + 0xdf6b11 (0x7f388a887b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #2: + 0xd3e95 (0x7f38d4835e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) [default0]:frame #3: + 0x8609 (0x7f38d993d609 in /lib/x86_64-linux-gnu/libpthread.so.0) [default0]:frame #4: clone + 0x43 (0x7f38d9708353 in /lib/x86_64-linux-gnu/libc.so.6) [default0]: [default4]:Traceback (most recent call last): [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in [default4]: trainer.train(dataloader) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train [default4]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step [default4]: outputs = self.pipeline_engine.train_batch_iter( [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter [default4]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward [default4]: output = model(**micro_batch) [default4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default4]: return self._call_impl(*args, **kwargs) [default4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default4]: return forward_call(*args, **kwargs) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward [default4]: sharded_logits = self.model( [default4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default4]: return self._call_impl(*args, **kwargs) [default4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default4]: return forward_call(*args, **kwargs) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward [default4]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states [default4]: hidden_encoder_states = encoder_block(**hidden_encoder_states) [default4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default4]: return self._call_impl(*args, **kwargs) [default4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default4]: return forward_call(*args, **kwargs) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward [default4]: new_kwargs[name] = recv_from_pipeline_state_buffer( [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer [default4]: pipeline_state.run_communication() [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication [default4]: recv_activation_tensor = recv_activation() [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ [default4]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors [default4]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors [default4]: meta = self._recv_meta(from_rank=from_rank, tag=tag) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta [default4]: dist.recv( [default4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper [default4]: return func(*args, **kwargs) [default4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv [default4]: pg.recv([tensor], group_src_rank, tag).wait() [default4]:torch.distributed.DistBackendError: [14] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '13:14', but store->get('13:14') got error: Connection reset by peer [default4]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [default4]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f9555a71d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default4]:frame #1: + 0x589518e (0x7f958da2b18e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x360 (0x7f958da259a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f958da25ce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f958da26b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f958d9dbf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f958d9dbf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f958d9dbf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f958d9dbf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f9556c19c69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default4]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector > const&, c10d::OpType, int, bool) + 0x22b (0x7f9556c20c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default4]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector >&, int, int) + 0x550 (0x7f9556c43b60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default4]:frame #12: + 0x5838439 (0x7f958d9ce439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #13: + 0x5843330 (0x7f958d9d9330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #14: + 0x58433c5 (0x7f958d9d93c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #15: + 0x4e893cc (0x7f958d01f3cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #16: + 0x1a08a88 (0x7f9589b9ea88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #17: + 0x5849a84 (0x7f958d9dfa84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #18: + 0x584ed35 (0x7f958d9e4d35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #19: + 0xc97eee (0x7f95a0296eee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default4]:frame #20: + 0x413ea4 (0x7f959fa12ea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default4]:frame #21: + 0x1445a6 (0x561125b895a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #22: _PyObject_MakeTpCall + 0x26b (0x561125b82a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #23: + 0x150866 (0x561125b95866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x561125b7e142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #25: _PyFunction_Vectorcall + 0x6c (0x561125b89a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #26: PyObject_Call + 0xbc (0x561125b95f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x561125b7c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #28: _PyFunction_Vectorcall + 0x6c (0x561125b89a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x561125b7a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #30: + 0x150582 (0x561125b95582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x561125b7a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #32: + 0x150582 (0x561125b95582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x561125b7a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #34: + 0x150582 (0x561125b95582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x561125b7a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x561125b81f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #37: _PyObject_Call_Prepend + 0x69 (0x561125b93c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #38: + 0x211239 (0x561125c56239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #39: _PyObject_MakeTpCall + 0x26b (0x561125b82a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x561125b7e3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #41: _PyFunction_Vectorcall + 0x6c (0x561125b89a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x561125b79c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #43: _PyFunction_Vectorcall + 0x6c (0x561125b89a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x561125b7a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #45: + 0x150582 (0x561125b95582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #46: PyObject_Call + 0xbc (0x561125b95f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x561125b7c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #48: + 0x150582 (0x561125b95582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #49: PyObject_Call + 0xbc (0x561125b95f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x561125b7c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #51: _PyFunction_Vectorcall + 0x6c (0x561125b89a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x561125b82007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #53: _PyObject_Call_Prepend + 0x69 (0x561125b93c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #54: + 0x211239 (0x561125c56239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #55: PyObject_Call + 0x207 (0x561125b96067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x561125b7c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #57: + 0x150582 (0x561125b95582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x561125b7a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #59: + 0x150582 (0x561125b95582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #60: PyObject_Call + 0xbc (0x561125b95f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x561125b7c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #62: + 0x150582 (0x561125b95582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #63: PyObject_Call + 0xbc (0x561125b95f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:. This may indicate a possible application crash on rank 0 or a network set up issue. [default0]:Traceback (most recent call last): [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in [default0]: trainer.train(dataloader) [default2]:Traceback (most recent call last): [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in [default4]:Traceback (most recent call last): [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in [default2]:Traceback (most recent call last): [default0]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in [default2]: trainer.train(dataloader) [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step [default0]: outputs = self.pipeline_engine.train_batch_iter( [default2]: trainer.train(dataloader) [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train [default4]: trainer.train(dataloader) [default3]:Traceback (most recent call last): [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter [default3]: trainer.train(dataloader) [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train [default6]:Traceback (most recent call last): [default2]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) [default1]:Traceback (most recent call last): [default3]:Traceback (most recent call last): [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in [default0]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train [default2]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) [default3]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in [default4]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step [default3]: outputs = self.pipeline_engine.train_batch_iter( [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in [default2]: outputs = self.pipeline_engine.train_batch_iter( [default0]: output = model(**micro_batch) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter [default3]: trainer.train(dataloader) [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step [default0]: return self._call_impl(*args, **kwargs) [default4]: outputs = self.pipeline_engine.train_batch_iter( [default3]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward [default6]: trainer.train(dataloader) [default2]: outputs = self.pipeline_engine.train_batch_iter( [default3]: output = model(**micro_batch) [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train [default2]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default3]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) [default0]: return forward_call(*args, **kwargs) [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default6]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) [default1]: trainer.train(dataloader) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step [default0]: sharded_logits = self.model( [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default1]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) [default2]: output = model(**micro_batch) [default4]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) [default3]: return self._call_impl(*args, **kwargs) [default0]: return self._call_impl(*args, **kwargs) [default3]: outputs = self.pipeline_engine.train_batch_iter( [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step [default2]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward [default3]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward [default6]: outputs = self.pipeline_engine.train_batch_iter( [default1]: outputs = self.pipeline_engine.train_batch_iter( [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward [default2]: return self._call_impl(*args, **kwargs) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter [default1]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) [default4]: output = model(**micro_batch) [default3]: output = model(**micro_batch) [default6]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward [default2]: output = model(**micro_batch) [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default6]:Traceback (most recent call last): [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default0]: return forward_call(*args, **kwargs) [default3]: return forward_call(*args, **kwargs) [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward [default4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default6]: trainer.train(dataloader) [default2]: return forward_call(*args, **kwargs) [default5]:Traceback (most recent call last): [default4]: return self._call_impl(*args, **kwargs) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward [default3]: return self._call_impl(*args, **kwargs) [default4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default1]: output = model(**micro_batch) [default6]: output = model(**micro_batch) [default4]: return forward_call(*args, **kwargs) [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default1]: return self._call_impl(*args, **kwargs) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward [default2]: sharded_logits = self.model( [default3]: return forward_call(*args, **kwargs) [default6]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) [default6]: return self._call_impl(*args, **kwargs) [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default1]: return forward_call(*args, **kwargs) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step [default6]: outputs = self.pipeline_engine.train_batch_iter( [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter [default2]: return self._call_impl(*args, **kwargs) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default6]: return forward_call(*args, **kwargs) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward [default4]: sharded_logits = self.model( [default3]: sharded_logits = self.model( [default6]: sharded_logits = self.model( [default2]: return self._call_impl(*args, **kwargs) [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default2]: return forward_call(*args, **kwargs) [default6]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) [default3]: sharded_logits = self.model( [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default0]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] [default2]: return forward_call(*args, **kwargs) [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward [default2]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] [default4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default3]: return self._call_impl(*args, **kwargs) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states [default6]: return self._call_impl(*args, **kwargs) [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward [default3]: return self._call_impl(*args, **kwargs) [default2]: hidden_encoder_states = encoder_block(**hidden_encoder_states) [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default5]: trainer.train(dataloader) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train [default4]: return self._call_impl(*args, **kwargs) [default2]: sharded_logits = self.model( [default3]: return forward_call(*args, **kwargs) [default6]: return forward_call(*args, **kwargs) [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward [default3]: return forward_call(*args, **kwargs) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward [default1]: sharded_logits = self.model( [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward [default6]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default6]: output = model(**micro_batch) [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states [default6]: hidden_encoder_states = encoder_block(**hidden_encoder_states) [default6]: return self._call_impl(*args, **kwargs) [default0]: hidden_encoder_states = encoder_block(**hidden_encoder_states) [default5]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default6]: return self._call_impl(*args, **kwargs) [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default2]: return self._call_impl(*args, **kwargs) [default2]: return self._call_impl(*args, **kwargs) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step [default5]: outputs = self.pipeline_engine.train_batch_iter( [default3]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default2]: return forward_call(*args, **kwargs) [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward [default3]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] [default6]: return forward_call(*args, **kwargs) [default1]: return self._call_impl(*args, **kwargs) [default3]: hidden_encoder_states = encoder_block(**hidden_encoder_states) [default6]: return forward_call(*args, **kwargs) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward [default4]: return forward_call(*args, **kwargs) [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default2]: return forward_call(*args, **kwargs) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward [default2]: new_kwargs[name] = recv_from_pipeline_state_buffer( [default5]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default6]: sharded_logits = self.model( [default3]: hidden_encoder_states = encoder_block(**hidden_encoder_states) [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default5]: output = model(**micro_batch) [default3]: return self._call_impl(*args, **kwargs) [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default2]: pipeline_state.run_communication() [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication [default0]: return self._call_impl(*args, **kwargs) [default3]: return forward_call(*args, **kwargs) [default6]: return self._call_impl(*args, **kwargs) [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default6]: new_kwargs[name] = recv_from_pipeline_state_buffer( [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default3]: return self._call_impl(*args, **kwargs) [default1]: return forward_call(*args, **kwargs) [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer [default6]: pipeline_state.run_communication() [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward [default5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication [default2]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] [default6]: return forward_call(*args, **kwargs) [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default0]: return forward_call(*args, **kwargs) [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward [default3]: new_kwargs[name] = recv_from_pipeline_state_buffer( [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward [default2]: recv_activation_tensor = recv_activation() [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer [default5]: return self._call_impl(*args, **kwargs) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward [default1]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] [default6]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] [default0]: new_kwargs[name] = recv_from_pipeline_state_buffer( [default6]: recv_activation_tensor = recv_activation() [default3]: pipeline_state.run_communication() [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ [default4]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] [default5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default6]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer [default0]: pipeline_state.run_communication() [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states [default5]: return forward_call(*args, **kwargs) [default3]: return forward_call(*args, **kwargs) [default2]: hidden_encoder_states = encoder_block(**hidden_encoder_states) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors [default6]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward [default3]: new_kwargs[name] = recv_from_pipeline_state_buffer( [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default2]: return self._call_impl(*args, **kwargs) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors [default1]: hidden_encoder_states = encoder_block(**hidden_encoder_states) [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default4]: hidden_encoder_states = encoder_block(**hidden_encoder_states) [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication [default3]: recv_activation_tensor = recv_activation() [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward [default5]: sharded_logits = self.model( [default6]: meta = self._recv_meta(from_rank=from_rank, tag=tag) [default0]: recv_activation_tensor = recv_activation() [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ [default3]: pipeline_state.run_communication() [default5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default5]: return self._call_impl(*args, **kwargs) [default1]: return self._call_impl(*args, **kwargs) [default2]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ [default2]: return forward_call(*args, **kwargs) [default6]: hidden_encoder_states = encoder_block(**hidden_encoder_states) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication [default2]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) [default0]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default6]: dist.recv( [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward [default2]: meta = self._recv_meta(from_rank=from_rank, tag=tag) [default3]: recv_activation_tensor = recv_activation() [default3]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] [default5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper [default5]: return forward_call(*args, **kwargs) [default1]: return forward_call(*args, **kwargs) [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ [default4]: return self._call_impl(*args, **kwargs) [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward [default6]: return func(*args, **kwargs) [default3]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] [default4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default1]: new_kwargs[name] = recv_from_pipeline_state_buffer( [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta [default2]: new_kwargs[name] = recv_from_pipeline_state_buffer( [default6]: return self._call_impl(*args, **kwargs) [default4]: return forward_call(*args, **kwargs) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer [default2]: dist.recv( [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv [default5]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors [default3]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward [default2]: return func(*args, **kwargs) [default4]: new_kwargs[name] = recv_from_pipeline_state_buffer( [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default1]: pipeline_state.run_communication() [default5]: hidden_encoder_states = encoder_block(**hidden_encoder_states) [default6]: pg.recv([tensor], group_src_rank, tag).wait() [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors [default0]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) [default2]: pipeline_state.run_communication() [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication [default6]: return forward_call(*args, **kwargs) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward [default6]:torch.distributed.DistBackendError: [7] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '6:7', but store->get('6:7') got error: Connection reset by peer [default5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors [default5]: return self._call_impl(*args, **kwargs) [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication [default2]: pg.recv([tensor], group_src_rank, tag).wait() [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors [default3]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors [default1]: recv_activation_tensor = recv_activation() [default0]: meta = self._recv_meta(from_rank=from_rank, tag=tag) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer [default3]: meta = self._recv_meta(from_rank=from_rank, tag=tag) [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ [default1]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] [default6]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [default3]: dist.recv( [default4]: pipeline_state.run_communication() [default5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default6]: new_kwargs[name] = recv_from_pipeline_state_buffer( [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer [default2]: recv_activation_tensor = recv_activation() [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper [default5]: return forward_call(*args, **kwargs) [default6]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f2268e9ad87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward [default3]: return func(*args, **kwargs) [default6]: pipeline_state.run_communication() [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ [default2]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication [default3]: meta = self._recv_meta(from_rank=from_rank, tag=tag) [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv [default2]:torch.distributed.DistBackendError: [13] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '12:13', but store->get('12:13') got error: Connection reset by peer [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication [default2]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta [default3]: dist.recv( [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper [default3]: pg.recv([tensor], group_src_rank, tag).wait() [default6]:frame #1: + 0x589518e (0x7f22a0e5418e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fb306fd6d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors [default2]:frame #1: + 0x589518e (0x7fb33ef9018e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) [default2]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x360 (0x7fb33ef8a9a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]: recv_activation_tensor = recv_activation() [default6]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x360 (0x7f22a0e4e9a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f22a0e4ece2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f22a0e4fb11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f22a0e04f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:torch.distributed.DistBackendError: [5] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '4:5', but store->get('4:5') got error: Connection reset by peer [default3]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [default6]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f22a0e04f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) [default0]: dist.recv( [default3]: return func(*args, **kwargs) [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ [default3]: pg.recv([tensor], group_src_rank, tag).wait() [default6]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f22a0e04f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]: recv_activation_tensor = recv_activation() [default5]: new_kwargs[name] = recv_from_pipeline_state_buffer( [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors [default4]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] [default2]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fb33ef8ace2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f22a0e04f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]:Traceback (most recent call last): [default5]: pipeline_state.run_communication() [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication [default6]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f226a042c69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default2]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fb33ef8bb11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:torch.distributed.DistBackendError: [13] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '12:13', but store->get('12:13') got error: Connection reset by peer [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ [default5]: recv_activation_tensor = recv_activation() [default2]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb33ef40f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector > const&, c10d::OpType, int, bool) + 0x22b (0x7f226a049c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default3]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [default6]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector >&, int, int) + 0x550 (0x7f226a06cb60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors [default4]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) [default7]:Traceback (most recent call last): [default0]:Traceback (most recent call last): [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in [default2]: meta = self._recv_meta(from_rank=from_rank, tag=tag) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors [default4]: meta = self._recv_meta(from_rank=from_rank, tag=tag) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in [default3]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f15a2b71d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default6]:frame #12: + 0x5838439 (0x7f22a0df7439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta [default1]: meta = self._recv_meta(from_rank=from_rank, tag=tag) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ [default3]:frame #1: + 0x589518e (0x7f15dab2b18e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x360 (0x7f15dab259a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] [default3]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f15dab25ce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb33ef40f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in [default3]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f15dab26b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: trainer.train(dataloader) [default3]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fda819b6d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default3]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f15daadbf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #1: + 0x589518e (0x7fdab997018e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f15daadbf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors [default0]: trainer.train(dataloader) [default6]:frame #13: + 0x5843330 (0x7f22a0e02330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb33ef40f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb33ef40f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: trainer.train(dataloader) [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper [default3]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f15daadbf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta [default4]: dist.recv( [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train [default2]: dist.recv( [default3]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f15daadbf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f15a3d19c69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper [default7]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper [default3]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector > const&, c10d::OpType, int, bool) + 0x22b (0x7f15a3d20c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default6]:frame #14: + 0x58433c5 (0x7f22a0e023c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train [default7]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) [default6]:frame #15: + 0x4e893cc (0x7f22a04483cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors [default6]:frame #16: + 0x1a08a88 (0x7f229cfc7a88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fb30817ec69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default3]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector >&, int, int) + 0x550 (0x7f15a3d43b60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step [default0]: return func(*args, **kwargs) [default6]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors [default6]: meta = self._recv_meta(from_rank=from_rank, tag=tag) [default2]: return func(*args, **kwargs) [default4]: return func(*args, **kwargs) [default3]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x360 (0x7fdab996a9a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta [default4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv [default6]: dist.recv( [default7]: outputs = self.pipeline_engine.train_batch_iter( [default3]:frame #12: + 0x5838439 (0x7f15daace439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #13: + 0x5843330 (0x7f15daad9330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #17: + 0x5849a84 (0x7f22a0e08a84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step [default4]: pg.recv([tensor], group_src_rank, tag).wait() [default4]:torch.distributed.DistBackendError: [6] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '5:6', but store->get('5:6') got error: Connection reset by peer [default3]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fdab996ace2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv [default7]: outputs = self.pipeline_engine.train_batch_iter( [default1]: dist.recv( [default0]: pg.recv([tensor], group_src_rank, tag).wait() [default2]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector > const&, c10d::OpType, int, bool) + 0x22b (0x7fb308185c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default6]:frame #18: + 0x584ed35 (0x7f22a0e0dd35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper [default5]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) [default3]:frame #14: + 0x58433c5 (0x7f15daad93c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #15: + 0x4e893cc (0x7f15da11f3cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]:torch.distributed.DistBackendError: [12] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '11:12', but store->get('11:12') got error: Connection reset by peer [default3]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fdab996bb11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [default2]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector >&, int, int) + 0x550 (0x7fb3081a8b60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default2]:frame #12: + 0x5838439 (0x7fb33ef33439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #19: + 0xc97eee (0x7f22b36bfeee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default3]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fdab9920f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fdab9920f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper [default0]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) [default6]:frame #20: + 0x413ea4 (0x7f22b2e3bea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default6]:frame #21: + 0x1445a6 (0x55d5da3865a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f71274d1d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default1]: return func(*args, **kwargs) [default3]:frame #16: + 0x1a08a88 (0x7f15d6c9ea88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #22: _PyObject_MakeTpCall + 0x26b (0x55d5da37fa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #23: + 0x150866 (0x55d5da392866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step [default7]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv [default4]:frame #1: + 0x589518e (0x7f715f48b18e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #17: + 0x5849a84 (0x7f15daadfa84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [default2]:frame #13: + 0x5843330 (0x7fb33ef3e330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward [default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fa456a95d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default4]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x360 (0x7f715f4859a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55d5da37b142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #1: + 0x589518e (0x7fa48ea4f18e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward [default6]: return func(*args, **kwargs) [default0]: outputs = self.pipeline_engine.train_batch_iter( [default6]:frame #25: _PyFunction_Vectorcall + 0x6c (0x55d5da386a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f715f485ce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #18: + 0x584ed35 (0x7f15daae4d35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #14: + 0x58433c5 (0x7fb33ef3e3c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]: pg.recv([tensor], group_src_rank, tag).wait() [default2]:frame #15: + 0x4e893cc (0x7fb33e5843cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x360 (0x7fa48ea499a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:torch.distributed.DistBackendError: [5] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '4:5', but store->get('4:5') got error: Connection reset by peer [default6]:frame #26: PyObject_Call + 0xbc (0x55d5da392f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: output = model(**micro_batch) [default3]:frame #19: + 0xc97eee (0x7f15ed396eee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default6]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55d5da3792b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv [default3]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fdab9920f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f715f486b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]:Traceback (most recent call last): [default4]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f715f43bf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter [default6]:frame #28: _PyFunction_Vectorcall + 0x6c (0x55d5da386a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]: pg.recv([tensor], group_src_rank, tag).wait() [default6]: pg.recv([tensor], group_src_rank, tag).wait() [default3]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fdab9920f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]: meta = self._recv_meta(from_rank=from_rank, tag=tag) [default4]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f715f43bf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55d5da3778fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [default3]:frame #20: + 0x413ea4 (0x7f15ecb12ea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default7]: output = model(**micro_batch) [default1]:torch.distributed.DistBackendError: [4] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '3:4', but store->get('3:4') got error: Connection reset by peer [default1]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [default4]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f715f43bf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #30: + 0x150582 (0x55d5da392582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55d5da3778fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:Traceback (most recent call last): [default6]:frame #32: + 0x150582 (0x55d5da392582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]: trainer.train(dataloader) [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train [default0]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fa48ea49ce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f715f43bf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default7]: return self._call_impl(*args, **kwargs) [default6]:torch.distributed.DistBackendError: [15] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '14:15', but store->get('14:15') got error: Connection reset by peer [default3]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fda82b5ec69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta [default6]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55d5da3778fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #34: + 0x150582 (0x55d5da392582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55d5da3778fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fa48ea4ab11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa48e9fff81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [default5]: dist.recv( [default5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper [default3]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector > const&, c10d::OpType, int, bool) + 0x22b (0x7fda82b65c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default6]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fc5ae26fd87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default0]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa48e9fff81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step [default3]:frame #21: + 0x1445a6 (0x5564748c95a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55d5da37ef50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward [default0]: output = model(**micro_batch) [default1]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fc725c94d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default4]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f7128679c69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default4]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector > const&, c10d::OpType, int, bool) + 0x22b (0x7f7128680c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default3]:frame #22: _PyObject_MakeTpCall + 0x26b (0x5564748c2a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #23: + 0x150866 (0x5564748d5866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: return self._call_impl(*args, **kwargs) [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default5]: return func(*args, **kwargs) [default3]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector >&, int, int) + 0x550 (0x7fda82b88b60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]: return self._call_impl(*args, **kwargs) [default7]: return forward_call(*args, **kwargs) [default1]: outputs = self.pipeline_engine.train_batch_iter( [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in [default6]:frame #37: _PyObject_Call_Prepend + 0x69 (0x55d5da390c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #38: + 0x211239 (0x55d5da453239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: return forward_call(*args, **kwargs) [default2]:frame #16: + 0x1a08a88 (0x7fb33b103a88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward [default7]: sharded_logits = self.model( [default3]:frame #12: + 0x5838439 (0x7fdab9913439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #13: + 0x5843330 (0x7fdab991e330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #39: _PyObject_MakeTpCall + 0x26b (0x55d5da37fa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward [default5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv [default6]:frame #1: + 0x589518e (0x7fc5e622918e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x360 (0x7fc5e62239a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa48e9fff81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa48e9fff81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]: return forward_call(*args, **kwargs) [default2]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f76d16b4d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default3]:frame #14: + 0x58433c5 (0x7fdab991e3c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter [default1]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) [default1]:frame #1: + 0x589518e (0x7fc75dc4e18e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fa457c3dc69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default7]: sharded_logits = self.model( [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default1]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x360 (0x7fc75dc489a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fc75dc48ce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: return self._call_impl(*args, **kwargs) [default6]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55d5da37b3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward [default2]:frame #1: + 0x589518e (0x7f770966e18e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fc5e6223ce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fc5e6224b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x360 (0x7f77096689a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector > const&, c10d::OpType, int, bool) + 0x22b (0x7fa457c44c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default1]: output = model(**micro_batch) [default7]: return self._call_impl(*args, **kwargs) [default5]: pg.recv([tensor], group_src_rank, tag).wait() [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default1]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fc75dc49b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #15: + 0x4e893cc (0x7fdab8f643cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #16: + 0x1a08a88 (0x7fdab5ae3a88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]: sharded_logits = self.model( [default2]:frame #17: + 0x5849a84 (0x7fb33ef44a84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector >&, int, int) + 0x550 (0x7fa457c67b60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default4]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector >&, int, int) + 0x550 (0x7f71286a3b60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default4]:frame #12: + 0x5838439 (0x7f715f42e439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #17: + 0x5849a84 (0x7fdab9924a84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x5564748be142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #25: _PyFunction_Vectorcall + 0x6c (0x5564748c9a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:torch.distributed.DistBackendError: [14] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '13:14', but store->get('13:14') got error: Connection reset by peer [default5]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [default3]:frame #26: PyObject_Call + 0xbc (0x5564748d5f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f7709668ce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]: trainer.train(dataloader) [default0]:frame #12: + 0x5838439 (0x7fa48e9f2439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]:frame #13: + 0x5843330 (0x7fa48e9fd330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train [default0]:frame #14: + 0x58433c5 (0x7fa48e9fd3c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default6]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc5e61d9f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc5e61d9f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default0]: return self._call_impl(*args, **kwargs) [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default7]: return forward_call(*args, **kwargs) [default4]:frame #13: + 0x5843330 (0x7f715f439330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #14: + 0x58433c5 (0x7f715f4393c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]:frame #15: + 0x4e893cc (0x7fa48e0433cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #15: + 0x4e893cc (0x7f715ea7f3cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #16: + 0x1a08a88 (0x7f715b5fea88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]: return self._call_impl(*args, **kwargs) [default1]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc75dbfef81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f7709669b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #41: _PyFunction_Vectorcall + 0x6c (0x55d5da386a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward [default7]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] [default2]:frame #18: + 0x584ed35 (0x7fb33ef49d35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #18: + 0x584ed35 (0x7fdab9929d35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #19: + 0xc97eee (0x7fdacc1dbeee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default3]:frame #20: + 0x413ea4 (0x7fdacb957ea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default0]:frame #16: + 0x1a08a88 (0x7fa48abc2a88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]:frame #17: + 0x5849a84 (0x7fa48ea03a84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default1]: return forward_call(*args, **kwargs) [default4]:frame #17: + 0x5849a84 (0x7f715f43fa84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #18: + 0x584ed35 (0x7f715f444d35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc75dbfef81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f30cfb2cd87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default5]:frame #1: + 0x589518e (0x7f3107ae618e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]: return forward_call(*args, **kwargs) [default3]:frame #21: + 0x1445a6 (0x55b13f5775a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward [default1]: sharded_logits = self.model( [default2]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f770961ef81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f770961ef81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #19: + 0xc97eee (0x7f7171cf6eee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default4]:frame #20: + 0x413ea4 (0x7f7171472ea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default4]:frame #21: + 0x1445a6 (0x55cabb31b5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: return forward_call(*args, **kwargs) [default5]: outputs = self.pipeline_engine.train_batch_iter( [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter [default2]:frame #19: + 0xc97eee (0x7fb3517fbeee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default2]:frame #20: + 0x413ea4 (0x7fb350f77ea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states [default7]: hidden_encoder_states = encoder_block(**hidden_encoder_states) [default3]:frame #22: _PyObject_MakeTpCall + 0x26b (0x55b13f570a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #21: + 0x1445a6 (0x56533878b5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #22: _PyObject_MakeTpCall + 0x26b (0x565338784a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default7]: return self._call_impl(*args, **kwargs) [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default1]: return self._call_impl(*args, **kwargs) [default3]:frame #23: + 0x150866 (0x55b13f583866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default2]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f770961ef81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55d5da376c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc75dbfef81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5564748bc2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #28: _PyFunction_Vectorcall + 0x6c (0x5564748c9a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5564748ba8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward [default7]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] [default5]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) [default4]:frame #22: _PyObject_MakeTpCall + 0x26b (0x55cabb314a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f770961ef81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default2]:frame #23: + 0x150866 (0x565338797866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc5e61d9f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]:frame #18: + 0x584ed35 (0x7fa48ea08d35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]:frame #19: + 0xc97eee (0x7fa4a12baeee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states [default1]: return forward_call(*args, **kwargs) [default4]:frame #23: + 0x150866 (0x55cabb327866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc5e61d9f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #30: + 0x150582 (0x5564748d5582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc75dbfef81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: return forward_call(*args, **kwargs) [default0]:frame #20: + 0x413ea4 (0x7fa4a0a36ea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward [default0]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] [default7]: hidden_encoder_states = encoder_block(**hidden_encoder_states) [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default3]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55b13f56c142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward [default7]: new_kwargs[name] = recv_from_pipeline_state_buffer( [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states [default1]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] [default5]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x360 (0x7f3107ae09a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f3107ae0ce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f3107ae1b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer [default2]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f76d285cc69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default2]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector > const&, c10d::OpType, int, bool) + 0x22b (0x7f76d2863c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default1]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fc726e3cc69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default2]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x565338780142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #43: _PyFunction_Vectorcall + 0x6c (0x55d5da386a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #25: _PyFunction_Vectorcall + 0x6c (0x55b13f577a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55d5da3778fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]: hidden_encoder_states = encoder_block(**hidden_encoder_states) [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default7]: pipeline_state.run_communication() [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication [default3]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5564748ba8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #32: + 0x150582 (0x5564748d5582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3107a96f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3107a96f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector > const&, c10d::OpType, int, bool) + 0x22b (0x7fc726e43c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default3]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5564748ba8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #21: + 0x1445a6 (0x55729f0645a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #22: _PyObject_MakeTpCall + 0x26b (0x55729f05da6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #34: + 0x150582 (0x5564748d5582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #23: + 0x150866 (0x55729f070866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fc5af417c69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states [default1]: hidden_encoder_states = encoder_block(**hidden_encoder_states) [default6]:frame #45: + 0x150582 (0x55d5da392582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55729f059142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector >&, int, int) + 0x550 (0x7fc726e66b60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default7]: return self._call_impl(*args, **kwargs) [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default0]: return self._call_impl(*args, **kwargs) [default4]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55cabb310142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #46: PyObject_Call + 0xbc (0x55d5da392f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5564748ba8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector > const&, c10d::OpType, int, bool) + 0x22b (0x7fc5af41ec5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default3]:frame #26: PyObject_Call + 0xbc (0x55b13f583f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #25: _PyFunction_Vectorcall + 0x6c (0x56533878ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3107a96f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3107a96f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f30d0cd4c69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default1]: return self._call_impl(*args, **kwargs) [default5]: output = model(**micro_batch) [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default6]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector >&, int, int) + 0x550 (0x7fc5af441b60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default4]:frame #25: _PyFunction_Vectorcall + 0x6c (0x55cabb31ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #12: + 0x5838439 (0x7fc5e61cc439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #26: PyObject_Call + 0xbc (0x55cabb327f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #12: + 0x5838439 (0x7fc75dbf1439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #13: + 0x5843330 (0x7fc5e61d7330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: return forward_call(*args, **kwargs) [default0]:frame #25: _PyFunction_Vectorcall + 0x6c (0x55729f064a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #26: PyObject_Call + 0xbc (0x55729f070f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55d5da3792b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector >&, int, int) + 0x550 (0x7f76d2886b60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default4]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55cabb30e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward [default3]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x5564748c1f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: return self._call_impl(*args, **kwargs) [default5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default7]: new_kwargs[name] = recv_from_pipeline_state_buffer( [default2]:frame #12: + 0x5838439 (0x7f7709611439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #13: + 0x5843330 (0x7f770961c330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]: return forward_call(*args, **kwargs) [default2]:frame #26: PyObject_Call + 0xbc (0x565338797f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x56533877e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55b13f56a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #14: + 0x58433c5 (0x7fc5e61d73c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #15: + 0x4e893cc (0x7fc5e581d3cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]: return forward_call(*args, **kwargs) [default0]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55729f0572b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #28: _PyFunction_Vectorcall + 0x6c (0x56533878ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x56533877c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer [default7]: recv_activation_tensor = recv_activation() [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward [default5]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector > const&, c10d::OpType, int, bool) + 0x22b (0x7f30d0cdbc5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default4]:frame #28: _PyFunction_Vectorcall + 0x6c (0x55cabb31ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #28: _PyFunction_Vectorcall + 0x6c (0x55b13f577a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #13: + 0x5843330 (0x7fc75dbfc330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #16: + 0x1a08a88 (0x7fc5e239ca88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]:frame #14: + 0x58433c5 (0x7fc75dbfc3c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: pipeline_state.run_communication() [default6]:frame #17: + 0x5849a84 (0x7fc5e61dda84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #48: + 0x150582 (0x55d5da392582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #49: PyObject_Call + 0xbc (0x55d5da392f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward [default5]: sharded_logits = self.model( [default7]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] [default3]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55b13f5688fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector >&, int, int) + 0x550 (0x7f30d0cfeb60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]: return forward_call(*args, **kwargs) [default5]:frame #12: + 0x5838439 (0x7f3107a89439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #18: + 0x584ed35 (0x7fc5e61e2d35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward [default2]:frame #30: + 0x150582 (0x565338797582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #30: + 0x150582 (0x55b13f583582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55b13f5688fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #15: + 0x4e893cc (0x7fc75d2423cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #13: + 0x5843330 (0x7f3107a94330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]:frame #16: + 0x1a08a88 (0x7fc759dc1a88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x56533877c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #14: + 0x58433c5 (0x7f770961c3c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #19: + 0xc97eee (0x7fc5f8a94eee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default2]:frame #15: + 0x4e893cc (0x7f7708c623cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]:frame #28: _PyFunction_Vectorcall + 0x6c (0x55729f064a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55729f0558fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default2]:frame #32: + 0x150582 (0x565338797582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: return self._call_impl(*args, **kwargs) [default1]:frame #17: + 0x5849a84 (0x7fc75dc02a84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]: new_kwargs[name] = recv_from_pipeline_state_buffer( [default6]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55d5da3792b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #14: + 0x58433c5 (0x7f3107a943c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #20: + 0x413ea4 (0x7fc5f8210ea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default6]:frame #51: _PyFunction_Vectorcall + 0x6c (0x55d5da386a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #16: + 0x1a08a88 (0x7f77057e1a88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]:frame #30: + 0x150582 (0x55729f070582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55729f0558fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #32: + 0x150582 (0x55729f070582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default1]: new_kwargs[name] = recv_from_pipeline_state_buffer( [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer [default3]:frame #37: _PyObject_Call_Prepend + 0x69 (0x5564748d3c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x56533877c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #38: + 0x211239 (0x556474996239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #17: + 0x5849a84 (0x7f7709622a84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors [default7]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) [default5]: return forward_call(*args, **kwargs) [default6]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55d5da37f007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #39: _PyObject_MakeTpCall + 0x26b (0x5564748c2a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #18: + 0x584ed35 (0x7f7709627d35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer [default1]: pipeline_state.run_communication() [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward [default5]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] [default0]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55729f0558fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #34: + 0x150582 (0x55729f070582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5564748be3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: meta = self._recv_meta(from_rank=from_rank, tag=tag) [default3]:frame #41: _PyFunction_Vectorcall + 0x6c (0x5564748c9a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication [default6]:frame #53: _PyObject_Call_Prepend + 0x69 (0x55d5da390c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55cabb30c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #30: + 0x150582 (0x55cabb327582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #19: + 0xc97eee (0x7f771bed9eee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default6]:frame #21: + 0x1445a6 (0x55a097ea45a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: recv_activation_tensor = recv_activation() [default3]:frame #32: + 0x150582 (0x55b13f583582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55b13f5688fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #54: + 0x211239 (0x55d5da453239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #34: + 0x150582 (0x55b13f583582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]: recv_activation_tensor = recv_activation() [default6]:frame #55: PyObject_Call + 0x207 (0x55d5da393067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ [default2]:frame #20: + 0x413ea4 (0x7f771b655ea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default2]:frame #34: + 0x150582 (0x565338797582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x56533877c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states [default6]:frame #22: _PyObject_MakeTpCall + 0x26b (0x55a097e9da6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta [default7]: dist.recv( [default3]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55b13f5688fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55729f0558fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55729f05cf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x565338783f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #23: + 0x150866 (0x55a097eb0866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55a097e99142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #37: _PyObject_Call_Prepend + 0x69 (0x55729f06ec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors [default0]: pipeline_state.run_communication() [default6]:frame #25: _PyFunction_Vectorcall + 0x6c (0x55a097ea4a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55b13f56ff50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ [default7]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] [default4]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55cabb30c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #32: + 0x150582 (0x55cabb327582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: hidden_encoder_states = encoder_block(**hidden_encoder_states) [default5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default0]: recv_activation_tensor = recv_activation() [default5]:frame #15: + 0x4e893cc (0x7f31070da3cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #21: + 0x1445a6 (0x560d3105c5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #22: _PyObject_MakeTpCall + 0x26b (0x560d31055a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #23: + 0x150866 (0x560d31068866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) [default5]: return self._call_impl(*args, **kwargs) [default1]:frame #18: + 0x584ed35 (0x7fc75dc07d35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]:frame #38: + 0x211239 (0x55729f131239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #19: + 0xc97eee (0x7fc7704b9eee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper [default5]:frame #16: + 0x1a08a88 (0x7f3103c59a88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors [default2]:frame #37: _PyObject_Call_Prepend + 0x69 (0x565338795c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: return func(*args, **kwargs) [default5]:frame #17: + 0x5849a84 (0x7f3107a9aa84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ [default1]: meta = self._recv_meta(from_rank=from_rank, tag=tag) [default3]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x5564748b9c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta [default1]:frame #20: + 0x413ea4 (0x7fc76fc35ea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default7]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv [default7]: pg.recv([tensor], group_src_rank, tag).wait() [default5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default5]: return forward_call(*args, **kwargs) [default6]:frame #26: PyObject_Call + 0xbc (0x55a097eb0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55a097e972b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #43: _PyFunction_Vectorcall + 0x6c (0x5564748c9a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x560d31051142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors [default0]:frame #39: _PyObject_MakeTpCall + 0x26b (0x55729f05da6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55729f0593e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55d5da3792b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #57: + 0x150582 (0x55d5da392582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #37: _PyObject_Call_Prepend + 0x69 (0x55b13f581c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #21: + 0x1445a6 (0x55ef624265a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] [default1]: dist.recv( [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper [default4]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55cabb30c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #34: + 0x150582 (0x55cabb327582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #38: + 0x211239 (0x565338858239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #25: _PyFunction_Vectorcall + 0x6c (0x560d3105ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #38: + 0x211239 (0x55b13f644239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #22: _PyObject_MakeTpCall + 0x26b (0x55ef6241fa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors [default0]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors [default7]:torch.distributed.DistBackendError: [15] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '14:15', but store->get('14:15') got error: Connection reset by peer [default7]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [default1]: return func(*args, **kwargs) [default2]:frame #39: _PyObject_MakeTpCall + 0x26b (0x565338784a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5653387803e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #18: + 0x584ed35 (0x7f3107a9fd35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #19: + 0xc97eee (0x7f311a351eee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default7]: meta = self._recv_meta(from_rank=from_rank, tag=tag) [default5]:frame #20: + 0x413ea4 (0x7f3119acdea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default3]:frame #39: _PyObject_MakeTpCall + 0x26b (0x55b13f570a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55b13f56c3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5564748ba8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55d5da3778fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #41: _PyFunction_Vectorcall + 0x6c (0x55b13f577a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv [default2]:frame #26: PyObject_Call + 0xbc (0x560d31068f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x560d3104f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #41: _PyFunction_Vectorcall + 0x6c (0x56533878ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #28: _PyFunction_Vectorcall + 0x6c (0x55a097ea4a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta [default3]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55b13f567c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward [default5]: new_kwargs[name] = recv_from_pipeline_state_buffer( [default5]:frame #21: + 0x1445a6 (0x55b2e4d715a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55cabb30c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #45: + 0x150582 (0x5564748d5582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #41: _PyFunction_Vectorcall + 0x6c (0x55729f064a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55729f054c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #46: PyObject_Call + 0xbc (0x5564748d5f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #43: _PyFunction_Vectorcall + 0x6c (0x55729f064a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #43: _PyFunction_Vectorcall + 0x6c (0x55b13f577a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #22: _PyObject_MakeTpCall + 0x26b (0x55b2e4d6aa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #23: + 0x150866 (0x55ef62432866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55cabb313f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #59: + 0x150582 (0x55d5da392582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fa8f2aa9d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default2]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x56533877bc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #60: PyObject_Call + 0xbc (0x55d5da392f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]: meta = self._recv_meta(from_rank=from_rank, tag=tag) [default5]:frame #23: + 0x150866 (0x55b2e4d7d866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55d5da3792b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55729f0558fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #43: _PyFunction_Vectorcall + 0x6c (0x56533878ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #62: + 0x150582 (0x55d5da392582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]: pg.recv([tensor], group_src_rank, tag).wait() [default4]:frame #37: _PyObject_Call_Prepend + 0x69 (0x55cabb325c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:torch.distributed.DistBackendError: [12] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '11:12', but store->get('11:12') got error: Connection reset by peer [default7]:frame #1: + 0x589518e (0x7fa92aa6318e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #38: + 0x211239 (0x55cabb3e8239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #45: + 0x150582 (0x55729f070582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #46: PyObject_Call + 0xbc (0x55729f070f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55b2e4d66142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [default1]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fa6536a5d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default2]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x56533877c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta [default6]:frame #63: PyObject_Call + 0xbc (0x55d5da392f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5564748bc2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #28: _PyFunction_Vectorcall + 0x6c (0x560d3105ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #39: _PyObject_MakeTpCall + 0x26b (0x55cabb314a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55cabb3103e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x360 (0x7fa92aa5d9a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55a097e958fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #30: + 0x150582 (0x55a097eb0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer [default5]: pipeline_state.run_communication() [default0]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55729f0572b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x560d3104d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fa92aa5dce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fa92aa5eb11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #30: + 0x150582 (0x560d31068582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55ef6241b142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #41: _PyFunction_Vectorcall + 0x6c (0x55cabb31ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #25: _PyFunction_Vectorcall + 0x6c (0x55b2e4d71a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: dist.recv( [default2]:frame #45: + 0x150582 (0x565338797582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper [default2]:frame #46: PyObject_Call + 0xbc (0x565338797f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #48: + 0x150582 (0x55729f070582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #48: + 0x150582 (0x5564748d5582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #1: + 0x589518e (0x7fa68b65f18e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]:frame #25: _PyFunction_Vectorcall + 0x6c (0x55ef62426a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55b13f5688fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #26: PyObject_Call + 0xbc (0x55ef62432f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55a097e958fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #32: + 0x150582 (0x55a097eb0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: return func(*args, **kwargs) [default7]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa92aa13f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa92aa13f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]: dist.recv( [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication [default1]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x360 (0x7fa68b6599a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55a097e958fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #34: + 0x150582 (0x55a097eb0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x560d3104d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #49: PyObject_Call + 0xbc (0x5564748d5f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #26: PyObject_Call + 0xbc (0x55b2e4d7df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5564748bc2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fa68b659ce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x56533877e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #45: + 0x150582 (0x55b13f583582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:. This may indicate a possible application crash on rank 0 or a network set up issue. [default3]:frame #46: PyObject_Call + 0xbc (0x55b13f583f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #49: PyObject_Call + 0xbc (0x55729f070f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa92aa13f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55a097e958fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55a097e9cf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #48: + 0x150582 (0x565338797582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:Traceback (most recent call last): [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in [default3]: trainer.train(dataloader) [default5]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55b2e4d642b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper [default3]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) [default3]:frame #51: _PyFunction_Vectorcall + 0x6c (0x5564748c9a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x5564748c2007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: pg.recv([tensor], group_src_rank, tag).wait() [default7]:torch.distributed.DistBackendError: [7] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '6:7', but store->get('6:7') got error: Connection reset by peer [default3]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55b13f56a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #32: + 0x150582 (0x560d31068582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55729f0572b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: recv_activation_tensor = recv_activation() [default0]:frame #51: _PyFunction_Vectorcall + 0x6c (0x55729f064a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]: return func(*args, **kwargs) [default0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step [default3]: outputs = self.pipeline_engine.train_batch_iter( [default6]:frame #37: _PyObject_Call_Prepend + 0x69 (0x55a097eaec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #28: _PyFunction_Vectorcall + 0x6c (0x55b2e4d71a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #53: _PyObject_Call_Prepend + 0x69 (0x5564748d3c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #54: + 0x211239 (0x556474996239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:Traceback (most recent call last): [default7]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa92aa13f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55cabb30bc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55ef624192b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #28: _PyFunction_Vectorcall + 0x6c (0x55ef62426a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fa68b65ab11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter [default6]:frame #38: + 0x211239 (0x55a097f71239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]: pg.recv([tensor], group_src_rank, tag).wait() [default3]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) [default0]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55729f05d007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fa8f3c51c69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default3]:frame #55: PyObject_Call + 0x207 (0x5564748d6067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in [default7]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector > const&, c10d::OpType, int, bool) + 0x22b (0x7fa8f3c58c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default5]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55b2e4d628fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #49: PyObject_Call + 0xbc (0x565338797f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x56533877e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5564748bc2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:Traceback (most recent call last): [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in [default1]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa68b60ff81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #43: _PyFunction_Vectorcall + 0x6c (0x55cabb31ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f42d93b8d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default7]:frame #1: + 0x589518e (0x7f431137218e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #30: + 0x150582 (0x55b2e4d7d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:Traceback (most recent call last): [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward [default3]:frame #48: + 0x150582 (0x55b13f583582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector >&, int, int) + 0x550 (0x7fa8f3c7bb60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default3]: output = model(**micro_batch) [default6]:frame #39: _PyObject_MakeTpCall + 0x26b (0x55a097e9da6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55a097e993e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]: trainer.train(dataloader) [default2]:frame #51: _PyFunction_Vectorcall + 0x6c (0x56533878ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x565338784007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default5]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55b2e4d628fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #32: + 0x150582 (0x55b2e4d7d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55b2e4d628fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55ef624178fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #30: + 0x150582 (0x55ef62432582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in [default7]:frame #12: + 0x5838439 (0x7fa92aa06439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55ef624178fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ [default7]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x360 (0x7f431136c9a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]: trainer.train(dataloader) [default1]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa68b60ff81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa68b60ff81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #57: + 0x150582 (0x5564748d5582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #53: _PyObject_Call_Prepend + 0x69 (0x565338795c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #32: + 0x150582 (0x55ef62432582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #13: + 0x5843330 (0x7fa92aa11330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]:frame #14: + 0x58433c5 (0x7fa92aa113c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55ef624178fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] [default7]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f431136cce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f431136db11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #49: PyObject_Call + 0xbc (0x55b13f583f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:torch.distributed.DistBackendError: [4] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '3:4', but store->get('3:4') got error: Connection reset by peer [default0]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [default6]:frame #41: _PyFunction_Vectorcall + 0x6c (0x55a097ea4a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55a097e94c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #34: + 0x150582 (0x55b2e4d7d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #34: + 0x150582 (0x55ef62432582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #54: + 0x211239 (0x565338858239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #55: PyObject_Call + 0x207 (0x565338798067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors [default7]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4311322f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4311322f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x560d3104d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55cabb30c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55ef624178fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5564748ba8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors [default7]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4311322f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4311322f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]: trainer.train(dataloader) [default4]:Traceback (most recent call last): [default3]: return self._call_impl(*args, **kwargs) [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train [default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fe9a3df1d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train [default6]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) [default2]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in [default6]:frame #43: _PyFunction_Vectorcall + 0x6c (0x55a097ea4a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: meta = self._recv_meta(from_rank=from_rank, tag=tag) [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step [default2]: outputs = self.pipeline_engine.train_batch_iter( [default6]: outputs = self.pipeline_engine.train_batch_iter( [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train [default3]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55b13f56a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #53: _PyObject_Call_Prepend + 0x69 (0x55729f06ec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter [default7]:frame #15: + 0x4e893cc (0x7fa92a0573cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]: trainer.train(dataloader) [default6]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55a097e958fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #51: _PyFunction_Vectorcall + 0x6c (0x55b13f577a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter [default5]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55b2e4d628fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward [default5]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55b2e4d69f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #59: + 0x150582 (0x5564748d5582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #60: PyObject_Call + 0xbc (0x5564748d5f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train [default1]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa68b60ff81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5564748bc2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) [default3]: return forward_call(*args, **kwargs) [default0]:frame #54: + 0x211239 (0x55729f131239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #34: + 0x150582 (0x560d31068582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) [default7]:frame #16: + 0x1a08a88 (0x7fa926bd6a88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]:frame #17: + 0x5849a84 (0x7fa92aa17a84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x560d3104d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55ef6241ef50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step [default6]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward [default6]:frame #45: + 0x150582 (0x55a097eb0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta [default2]: output = model(**micro_batch) [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default6]:frame #46: PyObject_Call + 0xbc (0x55a097eb0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f42da560c69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default5]: outputs = self.pipeline_engine.train_batch_iter( [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter [default2]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x56533877e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #62: + 0x150582 (0x5564748d5582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #63: PyObject_Call + 0xbc (0x5564748d5f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]: output = model(**micro_batch) [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default1]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fa65484dc69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default3]:. This may indicate a possible application crash on rank 0 or a network set up issue. [default5]: dist.recv( [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward [default1]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector > const&, c10d::OpType, int, bool) + 0x22b (0x7fa654854c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default2]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x560d31054f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #37: _PyObject_Call_Prepend + 0x69 (0x560d31066c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]: sharded_logits = self.model( [default2]: return self._call_impl(*args, **kwargs) [default6]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55a097e972b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #37: _PyObject_Call_Prepend + 0x69 (0x55b2e4d7bc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #1: + 0x589518e (0x7fe9dbdab18e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default7]:frame #18: + 0x584ed35 (0x7fa92aa1cd35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #38: + 0x211239 (0x55b2e4e3e239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #37: _PyObject_Call_Prepend + 0x69 (0x55ef62430c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default6]:frame #48: + 0x150582 (0x55a097eb0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #38: + 0x211239 (0x55ef624f3239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]: return self._call_impl(*args, **kwargs) [default7]:frame #19: + 0xc97eee (0x7fa93d2ceeee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default7]:frame #20: + 0x413ea4 (0x7fa93ca4aea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default7]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector > const&, c10d::OpType, int, bool) + 0x22b (0x7f42da567c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default2]: return forward_call(*args, **kwargs) [default1]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector >&, int, int) + 0x550 (0x7fa654877b60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default1]:frame #12: + 0x5838439 (0x7fa68b602439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector >&, int, int) + 0x550 (0x7f42da58ab60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x360 (0x7fe9dbda59a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) [default1]:frame #13: + 0x5843330 (0x7fa68b60d330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #57: + 0x150582 (0x565338797582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fe9dbda5ce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step [default4]: outputs = self.pipeline_engine.train_batch_iter( [default2]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x56533877c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper [default6]: return forward_call(*args, **kwargs) [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward [default5]:frame #39: _PyObject_MakeTpCall + 0x26b (0x55b2e4d6aa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55b2e4d663e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #12: + 0x5838439 (0x7f4311315439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]: sharded_logits = self.model( [default6]:frame #49: PyObject_Call + 0xbc (0x55a097eb0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward [default3]: return self._call_impl(*args, **kwargs) [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default6]: sharded_logits = self.model( [default2]:frame #59: + 0x150582 (0x565338797582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]: return forward_call(*args, **kwargs) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward [default5]: output = model(**micro_batch) [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward [default0]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fe9dbda6b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] [default5]: return func(*args, **kwargs) [default5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv [default6]: return self._call_impl(*args, **kwargs) [default7]:frame #13: + 0x5843330 (0x7f4311320330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #60: PyObject_Call + 0xbc (0x565338797f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #38: + 0x211239 (0x560d31129239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #39: _PyObject_MakeTpCall + 0x26b (0x560d31055a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55b13f570007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #55: PyObject_Call + 0x207 (0x55729f071067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe9dbd5bf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe9dbd5bf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default7]:frame #21: + 0x1445a6 (0x55fbcf6d15a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #45: + 0x150582 (0x55cabb327582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]: return forward_call(*args, **kwargs) [default5]:frame #41: _PyFunction_Vectorcall + 0x6c (0x55b2e4d71a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #46: PyObject_Call + 0xbc (0x55cabb327f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward [default3]:frame #53: _PyObject_Call_Prepend + 0x69 (0x55b13f581c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #54: + 0x211239 (0x55b13f644239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: pg.recv([tensor], group_src_rank, tag).wait() [default6]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] [default2]: return self._call_impl(*args, **kwargs) [default0]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55729f0572b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #14: + 0x58433c5 (0x7f43113203c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]:frame #15: + 0x4e893cc (0x7f43109663cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states [default3]: hidden_encoder_states = encoder_block(**hidden_encoder_states) [default0]:frame #57: + 0x150582 (0x55729f070582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55cabb30e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x560d310513e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default3]: return self._call_impl(*args, **kwargs) [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default3]: return forward_call(*args, **kwargs) [default1]:frame #14: + 0x58433c5 (0x7fa68b60d3c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #41: _PyFunction_Vectorcall + 0x6c (0x560d3105ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward [default7]:frame #22: _PyObject_MakeTpCall + 0x26b (0x55fbcf6caa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #16: + 0x1a08a88 (0x7f430d4e5a88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]: new_kwargs[name] = recv_from_pipeline_state_buffer( [default5]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55b2e4d61c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #43: _PyFunction_Vectorcall + 0x6c (0x55b2e4d71a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #48: + 0x150582 (0x55cabb327582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #39: _PyObject_MakeTpCall + 0x26b (0x55ef6241fa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer [default3]:frame #55: PyObject_Call + 0x207 (0x55b13f584067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x560d3104cc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #17: + 0x5849a84 (0x7f4311326a84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]: pipeline_state.run_communication() [default5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default2]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x56533877e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #62: + 0x150582 (0x565338797582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #43: _PyFunction_Vectorcall + 0x6c (0x560d3105ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x560d3104d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states [default6]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55a097e972b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55b2e4d628fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #45: + 0x150582 (0x55b2e4d7d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #45: + 0x150582 (0x560d31068582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication [default1]:frame #15: + 0x4e893cc (0x7fa68ac533cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #63: PyObject_Call + 0xbc (0x565338797f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #46: PyObject_Call + 0xbc (0x560d31068f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]: recv_activation_tensor = recv_activation() [default2]:. This may indicate a possible application crash on rank 0 or a network set up issue. [default2]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x560d3104f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ [default0]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55729f0558fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #48: + 0x150582 (0x560d31068582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #49: PyObject_Call + 0xbc (0x560d31068f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default2]: return forward_call(*args, **kwargs) [default1]:frame #16: + 0x1a08a88 (0x7fa6877d2a88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]: hidden_encoder_states = encoder_block(**hidden_encoder_states) [default7]:frame #23: + 0x150866 (0x55fbcf6dd866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55fbcf6c6142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] [default3]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55b13f56a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #46: PyObject_Call + 0xbc (0x55b2e4d7df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55b2e4d642b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #17: + 0x5849a84 (0x7fa68b613a84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #57: + 0x150582 (0x55b13f583582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55b13f5688fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward [default7]:frame #25: _PyFunction_Vectorcall + 0x6c (0x55fbcf6d1a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:torch.distributed.DistBackendError: [6] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '5:6', but store->get('5:6') got error: Connection reset by peer [default2]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default0]:frame #59: + 0x150582 (0x55729f070582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [default0]:frame #60: PyObject_Call + 0xbc (0x55729f070f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe9dbd5bf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #59: + 0x150582 (0x55b13f583582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #26: PyObject_Call + 0xbc (0x55fbcf6ddf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x560d3104f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #51: _PyFunction_Vectorcall + 0x6c (0x560d3105ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55fbcf6c42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f113e17dd87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default6]:frame #51: _PyFunction_Vectorcall + 0x6c (0x55a097ea4a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #48: + 0x150582 (0x55b2e4d7d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #1: + 0x589518e (0x7f117613718e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #60: PyObject_Call + 0xbc (0x55b13f583f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #49: PyObject_Call + 0xbc (0x55cabb327f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55a097e9d007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #49: PyObject_Call + 0xbc (0x55b2e4d7df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #28: _PyFunction_Vectorcall + 0x6c (0x55fbcf6d1a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55cabb30e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55ef6241b3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55b13f56a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x360 (0x7f11761319a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #53: _PyObject_Call_Prepend + 0x69 (0x55a097eaec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x560d31055007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55729f0572b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #53: _PyObject_Call_Prepend + 0x69 (0x560d31066c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #41: _PyFunction_Vectorcall + 0x6c (0x55ef62426a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55ef62416c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #62: + 0x150582 (0x55729f070582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #43: _PyFunction_Vectorcall + 0x6c (0x55ef62426a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #18: + 0x584ed35 (0x7f431132bd35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe9dbd5bf81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fe9a4f99c69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors [default7]:Traceback (most recent call last): [default4]: output = model(**micro_batch) [default1]:frame #18: + 0x584ed35 (0x7fa68b618d35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55ef624178fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states [default6]: return self._call_impl(*args, **kwargs) [default5]: return self._call_impl(*args, **kwargs) [default5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default5]: return forward_call(*args, **kwargs) [default5]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f1176131ce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f1176132b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default4]: return self._call_impl(*args, **kwargs) [default4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default5]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f11760e7f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #51: _PyFunction_Vectorcall + 0x6c (0x55cabb31ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector > const&, c10d::OpType, int, bool) + 0x22b (0x7fe9a4fa0c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default0]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector >&, int, int) + 0x550 (0x7fe9a4fc3b60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default1]:frame #45: + 0x150582 (0x55ef62432582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #46: PyObject_Call + 0xbc (0x55ef62432f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default6]: return forward_call(*args, **kwargs) [default5]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f11760e7f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]: return forward_call(*args, **kwargs) [default7]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55fbcf6c28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55cabb314007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward [default5]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55b2e4d642b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #54: + 0x211239 (0x55a097f71239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #55: PyObject_Call + 0x207 (0x55a097eb1067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #53: _PyObject_Call_Prepend + 0x69 (0x55cabb325c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]: hidden_encoder_states = encoder_block(**hidden_encoder_states) [default7]:frame #30: + 0x150582 (0x55fbcf6dd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #19: + 0xc97eee (0x7f4323bddeee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default2]:frame #54: + 0x211239 (0x560d31129239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward [default1]:frame #19: + 0xc97eee (0x7fa69decaeee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default1]:frame #20: + 0x413ea4 (0x7fa69d646ea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default0]:frame #12: + 0x5838439 (0x7fe9dbd4e439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]:frame #13: + 0x5843330 (0x7fe9dbd59330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]: new_kwargs[name] = recv_from_pipeline_state_buffer( [default6]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55a097e972b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55fbcf6c28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #14: + 0x58433c5 (0x7fe9dbd593c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #54: + 0x211239 (0x55cabb3e8239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #55: PyObject_Call + 0x207 (0x55cabb328067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer [default0]:frame #63: PyObject_Call + 0xbc (0x55729f070f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]: pipeline_state.run_communication() [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors [default1]:frame #21: + 0x1445a6 (0x55da5401e5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #22: _PyObject_MakeTpCall + 0x26b (0x55da54017a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in [default7]:frame #32: + 0x150582 (0x55fbcf6dd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: trainer.train(dataloader) [default6]:frame #57: + 0x150582 (0x55a097eb0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #51: _PyFunction_Vectorcall + 0x6c (0x55b2e4d71a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: sharded_logits = self.model( [default5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default5]: return self._call_impl(*args, **kwargs) [default0]:. This may indicate a possible application crash on rank 0 or a network set up issue. [default5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 430, in train [default1]:frame #23: + 0x150866 (0x55da5402a866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default1]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55da54013142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]: return self._call_impl(*args, **kwargs) [default3]:frame #62: + 0x150582 (0x55b13f583582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]: return forward_call(*args, **kwargs) [default7]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55fbcf6c28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 459, in training_step [default6]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55a097e958fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #59: + 0x150582 (0x55a097eb0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default7]:frame #34: + 0x150582 (0x55fbcf6dd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]: return forward_call(*args, **kwargs) [default5]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55b2e4d6a007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #53: _PyObject_Call_Prepend + 0x69 (0x55b2e4d7bc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #55: PyObject_Call + 0x207 (0x560d31069067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f11760e7f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f11760e7f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55fbcf6c28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #25: _PyFunction_Vectorcall + 0x6c (0x55da5401ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55ef624192b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #48: + 0x150582 (0x55ef62432582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #26: PyObject_Call + 0xbc (0x55da5402af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #20: + 0x413ea4 (0x7f4323359ea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default0]:frame #15: + 0x4e893cc (0x7fe9db39f3cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #60: PyObject_Call + 0xbc (0x55a097eb0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55a097e972b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #16: + 0x1a08a88 (0x7fe9d7f1ea88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55cabb30e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #62: + 0x150582 (0x55a097eb0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55da540112b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #63: PyObject_Call + 0xbc (0x55b13f583f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #21: + 0x1445a6 (0x5640f9dbf5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f113f325c69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default5]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector > const&, c10d::OpType, int, bool) + 0x22b (0x7f113f32cc5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default7]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55fbcf6c9f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #28: _PyFunction_Vectorcall + 0x6c (0x55da5401ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #49: PyObject_Call + 0xbc (0x55ef62432f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55ef624192b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55da5400f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #63: PyObject_Call + 0xbc (0x55a097eb0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #57: + 0x150582 (0x55cabb327582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:. This may indicate a possible application crash on rank 0 or a network set up issue. [default7]:frame #22: _PyObject_MakeTpCall + 0x26b (0x5640f9db8a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #54: + 0x211239 (0x55b2e4e3e239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #55: PyObject_Call + 0x207 (0x55b2e4d7e067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #51: _PyFunction_Vectorcall + 0x6c (0x55ef62426a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #37: _PyObject_Call_Prepend + 0x69 (0x55fbcf6dbc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55ef6241f007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55cabb30c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:. This may indicate a possible application crash on rank 0 or a network set up issue. [default7]:frame #23: + 0x150866 (0x5640f9dcb866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x5640f9db4142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55b2e4d642b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #25: _PyFunction_Vectorcall + 0x6c (0x5640f9dbfa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #38: + 0x211239 (0x55fbcf79e239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #17: + 0x5849a84 (0x7fe9dbd5fa84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #57: + 0x150582 (0x55b2e4d7d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #26: PyObject_Call + 0xbc (0x5640f9dcbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #39: _PyObject_MakeTpCall + 0x26b (0x55fbcf6caa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector >&, int, int) + 0x550 (0x7f113f34fb60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default5]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55b2e4d628fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #59: + 0x150582 (0x55b2e4d7d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #12: + 0x5838439 (0x7f11760da439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]:frame #30: + 0x150582 (0x55da5402a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #53: _PyObject_Call_Prepend + 0x69 (0x55ef62430c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #59: + 0x150582 (0x55cabb327582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5640f9db22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #28: _PyFunction_Vectorcall + 0x6c (0x5640f9dbfa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #60: PyObject_Call + 0xbc (0x55b2e4d7df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55b2e4d642b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #60: PyObject_Call + 0xbc (0x55cabb327f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55da5400f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55fbcf6c63e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x560d3104f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5640f9db08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #41: _PyFunction_Vectorcall + 0x6c (0x55fbcf6d1a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #62: + 0x150582 (0x55b2e4d7d582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55fbcf6c1c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #63: PyObject_Call + 0xbc (0x55b2e4d7df1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #43: _PyFunction_Vectorcall + 0x6c (0x55fbcf6d1a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:. This may indicate a possible application crash on rank 0 or a network set up issue. [default7]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55fbcf6c28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #13: + 0x5843330 (0x7f11760e5330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]:frame #32: + 0x150582 (0x55da5402a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55da5400f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55cabb30e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #62: + 0x150582 (0x55cabb327582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #34: + 0x150582 (0x55da5402a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #45: + 0x150582 (0x55fbcf6dd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #30: + 0x150582 (0x5640f9dcb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55da5400f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #54: + 0x211239 (0x55ef624f3239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55da54016f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #37: _PyObject_Call_Prepend + 0x69 (0x55da54028c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #18: + 0x584ed35 (0x7fe9dbd64d35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default0]:frame #19: + 0xc97eee (0x7fe9ee616eee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default1]:frame #38: + 0x211239 (0x55da540eb239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #20: + 0x413ea4 (0x7fe9edd92ea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default7]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5640f9db08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #46: PyObject_Call + 0xbc (0x55fbcf6ddf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #14: + 0x58433c5 (0x7f11760e53c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]:frame #39: _PyObject_MakeTpCall + 0x26b (0x55da54017a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #21: + 0x1445a6 (0x559f27a1b5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55fbcf6c42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #32: + 0x150582 (0x5640f9dcb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55da540133e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #15: + 0x4e893cc (0x7f117572b3cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]:frame #55: PyObject_Call + 0x207 (0x55ef62433067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55ef624192b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5640f9db08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #34: + 0x150582 (0x5640f9dcb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #57: + 0x150582 (0x55ef62432582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #22: _PyObject_MakeTpCall + 0x26b (0x559f27a14a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #63: PyObject_Call + 0xbc (0x55cabb327f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #41: _PyFunction_Vectorcall + 0x6c (0x55da5401ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:. This may indicate a possible application crash on rank 0 or a network set up issue. [default1]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55ef624178fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55da5400ec5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #43: _PyFunction_Vectorcall + 0x6c (0x55da5401ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #23: + 0x150866 (0x559f27a27866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #48: + 0x150582 (0x55fbcf6dd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #16: + 0x1a08a88 (0x7f11722aaa88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55da5400f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5640f9db08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #49: PyObject_Call + 0xbc (0x55fbcf6ddf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55fbcf6c42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #59: + 0x150582 (0x55ef62432582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #51: _PyFunction_Vectorcall + 0x6c (0x55fbcf6d1a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #60: PyObject_Call + 0xbc (0x55ef62432f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55fbcf6ca007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #45: + 0x150582 (0x55da5402a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #17: + 0x5849a84 (0x7f11760eba84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default1]:frame #46: PyObject_Call + 0xbc (0x55da5402af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x5640f9db7f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55da540112b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #48: + 0x150582 (0x55da5402a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #53: _PyObject_Call_Prepend + 0x69 (0x55fbcf6dbc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #54: + 0x211239 (0x55fbcf79e239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #55: PyObject_Call + 0x207 (0x55fbcf6de067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55fbcf6c42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #57: + 0x150582 (0x55fbcf6dd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55fbcf6c28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #59: + 0x150582 (0x55fbcf6dd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #57: + 0x150582 (0x560d31068582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #60: PyObject_Call + 0xbc (0x55fbcf6ddf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55fbcf6c42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x560d3104d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #62: + 0x150582 (0x55fbcf6dd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x559f27a10142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #63: PyObject_Call + 0xbc (0x55fbcf6ddf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #49: PyObject_Call + 0xbc (0x55da5402af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #25: _PyFunction_Vectorcall + 0x6c (0x559f27a1ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55da540112b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #59: + 0x150582 (0x560d31068582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #60: PyObject_Call + 0xbc (0x560d31068f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #51: _PyFunction_Vectorcall + 0x6c (0x55da5401ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #18: + 0x584ed35 (0x7f11760f0d35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]:frame #37: _PyObject_Call_Prepend + 0x69 (0x5640f9dc9c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55da54017007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #38: + 0x211239 (0x5640f9e8c239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:. This may indicate a possible application crash on rank 0 or a network set up issue. [default2]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x560d3104f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #53: _PyObject_Call_Prepend + 0x69 (0x55da54028c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #54: + 0x211239 (0x55da540eb239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #55: PyObject_Call + 0x207 (0x55da5402b067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55da540112b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #62: + 0x150582 (0x560d31068582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #57: + 0x150582 (0x55da5402a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55da5400f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #59: + 0x150582 (0x55da5402a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #19: + 0xc97eee (0x7f11889a2eee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default0]:frame #26: PyObject_Call + 0xbc (0x559f27a27f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #39: _PyObject_MakeTpCall + 0x26b (0x5640f9db8a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #60: PyObject_Call + 0xbc (0x55da5402af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55da540112b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #62: + 0x150582 (0x55da5402a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #63: PyObject_Call + 0xbc (0x55da5402af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #63: PyObject_Call + 0xbc (0x560d31068f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:. This may indicate a possible application crash on rank 0 or a network set up issue. [default2]:. This may indicate a possible application crash on rank 0 or a network set up issue. [default1]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55ef624192b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #20: + 0x413ea4 (0x7f118811eea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default5]:frame #21: + 0x1445a6 (0x55f8733075a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #62: + 0x150582 (0x55ef62432582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:frame #63: PyObject_Call + 0xbc (0x55ef62432f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #22: _PyObject_MakeTpCall + 0x26b (0x55f873300a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #23: + 0x150866 (0x55f873313866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55f8732fc142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x559f27a0e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #28: _PyFunction_Vectorcall + 0x6c (0x559f27a1ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default1]:. This may indicate a possible application crash on rank 0 or a network set up issue. [default5]:frame #25: _PyFunction_Vectorcall + 0x6c (0x55f873307a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #26: PyObject_Call + 0xbc (0x55f873313f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55f8732fa2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x559f27a0c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5640f9db43e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #30: + 0x150582 (0x559f27a27582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #41: _PyFunction_Vectorcall + 0x6c (0x5640f9dbfa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x559f27a0c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #28: _PyFunction_Vectorcall + 0x6c (0x55f873307a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x5640f9dafc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55f8732f88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #43: _PyFunction_Vectorcall + 0x6c (0x5640f9dbfa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #30: + 0x150582 (0x55f873313582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5640f9db08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55f8732f88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #45: + 0x150582 (0x5640f9dcb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #32: + 0x150582 (0x55f873313582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #46: PyObject_Call + 0xbc (0x5640f9dcbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55f8732f88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5640f9db22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #34: + 0x150582 (0x55f873313582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #48: + 0x150582 (0x5640f9dcb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55f8732f88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #49: PyObject_Call + 0xbc (0x5640f9dcbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #32: + 0x150582 (0x559f27a27582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55f8732fff50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x559f27a0c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #37: _PyObject_Call_Prepend + 0x69 (0x55f873311c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #34: + 0x150582 (0x559f27a27582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #38: + 0x211239 (0x55f8733d4239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x559f27a0c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #39: _PyObject_MakeTpCall + 0x26b (0x55f873300a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x559f27a13f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55f8732fc3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #37: _PyObject_Call_Prepend + 0x69 (0x559f27a25c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #41: _PyFunction_Vectorcall + 0x6c (0x55f873307a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #38: + 0x211239 (0x559f27ae8239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55f8732f7c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #39: _PyObject_MakeTpCall + 0x26b (0x559f27a14a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #43: _PyFunction_Vectorcall + 0x6c (0x55f873307a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x559f27a103e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55f8732f88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5640f9db22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #51: _PyFunction_Vectorcall + 0x6c (0x5640f9dbfa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #45: + 0x150582 (0x55f873313582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #41: _PyFunction_Vectorcall + 0x6c (0x559f27a1ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #46: PyObject_Call + 0xbc (0x55f873313f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x559f27a0bc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55f8732fa2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x5640f9db8007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #48: + 0x150582 (0x55f873313582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #53: _PyObject_Call_Prepend + 0x69 (0x5640f9dc9c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #49: PyObject_Call + 0xbc (0x55f873313f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #54: + 0x211239 (0x5640f9e8c239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55f8732fa2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #55: PyObject_Call + 0x207 (0x5640f9dcc067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #51: _PyFunction_Vectorcall + 0x6c (0x55f873307a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5640f9db22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55f873300007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #57: + 0x150582 (0x5640f9dcb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #53: _PyObject_Call_Prepend + 0x69 (0x55f873311c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5640f9db08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #54: + 0x211239 (0x55f8733d4239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #59: + 0x150582 (0x5640f9dcb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #55: PyObject_Call + 0x207 (0x55f873314067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55f8732fa2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #43: _PyFunction_Vectorcall + 0x6c (0x559f27a1ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #60: PyObject_Call + 0xbc (0x5640f9dcbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x559f27a0c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5640f9db22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #45: + 0x150582 (0x559f27a27582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #62: + 0x150582 (0x5640f9dcb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #46: PyObject_Call + 0xbc (0x559f27a27f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #63: PyObject_Call + 0xbc (0x5640f9dcbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x559f27a0e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #57: + 0x150582 (0x55f873313582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #48: + 0x150582 (0x559f27a27582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55f8732f88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #49: PyObject_Call + 0xbc (0x559f27a27f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #59: + 0x150582 (0x55f873313582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x559f27a0e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #60: PyObject_Call + 0xbc (0x55f873313f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #51: _PyFunction_Vectorcall + 0x6c (0x559f27a1ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55f8732fa2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x559f27a14007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:. This may indicate a possible application crash on rank 0 or a network set up issue. [default5]:frame #62: + 0x150582 (0x55f873313582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #63: PyObject_Call + 0xbc (0x55f873313f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:. This may indicate a possible application crash on rank 0 or a network set up issue. [default0]:frame #53: _PyObject_Call_Prepend + 0x69 (0x559f27a25c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #54: + 0x211239 (0x559f27ae8239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #55: PyObject_Call + 0x207 (0x559f27a28067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x559f27a0e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #57: + 0x150582 (0x559f27a27582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x559f27a0c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #59: + 0x150582 (0x559f27a27582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #60: PyObject_Call + 0xbc (0x559f27a27f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x559f27a0e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #62: + 0x150582 (0x559f27a27582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:frame #63: PyObject_Call + 0xbc (0x559f27a27f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default0]:. This may indicate a possible application crash on rank 0 or a network set up issue. [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication [default7]: outputs = self.pipeline_engine.train_batch_iter( [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 187, in train_batch_iter [default3]: meta = self._recv_meta(from_rank=from_rank, tag=tag) [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward [default6]: recv_activation_tensor = recv_activation() [default7]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward [default7]: output = model(**micro_batch) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward [default2]: new_kwargs[name] = recv_from_pipeline_state_buffer( [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default7]: return self._call_impl(*args, **kwargs) [default4]: sharded_logits = self.model( [default4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default4]: return self._call_impl(*args, **kwargs) [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default7]: return forward_call(*args, **kwargs) [default5]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states [default3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 890, in forward [default4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default4]: return forward_call(*args, **kwargs) [default3]: dist.recv( [default2]: pipeline_state.run_communication() [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication [default5]: hidden_encoder_states = encoder_block(**hidden_encoder_states) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward [default4]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper [default2]: recv_activation_tensor = recv_activation() [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states [default4]: hidden_encoder_states = encoder_block(**hidden_encoder_states) [default3]: return func(*args, **kwargs) [default3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv [default6]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ [default2]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] [default7]: sharded_logits = self.model( [default5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors [default3]: pg.recv([tensor], group_src_rank, tag).wait() [default3]:torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default7]: return self._call_impl(*args, **kwargs) [default4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default5]: return self._call_impl(*args, **kwargs) [default3]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [default3]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f34192abd87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default7]: return forward_call(*args, **kwargs) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward [default2]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors [default5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default5]: return forward_call(*args, **kwargs) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward [default6]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) [default3]:frame #1: + 0x589518e (0x7f345126518e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]: return self._call_impl(*args, **kwargs) [default7]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] [default5]: new_kwargs[name] = recv_from_pipeline_state_buffer( [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors [default6]: meta = self._recv_meta(from_rank=from_rank, tag=tag) [default4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states [default7]: hidden_encoder_states = encoder_block(**hidden_encoder_states) [default2]: meta = self._recv_meta(from_rank=from_rank, tag=tag) [default6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta [default4]: return forward_call(*args, **kwargs) [default2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl [default7]: return self._call_impl(*args, **kwargs) [default3]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x360 (0x7f345125f9a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward [default2]: dist.recv( [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper [default2]: return func(*args, **kwargs) [default5]: pipeline_state.run_communication() [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication [default2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv [default6]: dist.recv( [default5]: recv_activation_tensor = recv_activation() [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ [default5]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper [default6]: return func(*args, **kwargs) [default6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors [default5]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) [default6]: pg.recv([tensor], group_src_rank, tag).wait() [default4]: new_kwargs[name] = recv_from_pipeline_state_buffer( [default3]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f345125fce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f3451260b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:torch.distributed.DistBackendError: [3] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '2:3', but store->get('2:3') got error: Connection reset by peer [default6]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer [default4]: pipeline_state.run_communication() [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors [default5]: meta = self._recv_meta(from_rank=from_rank, tag=tag) [default2]: pg.recv([tensor], group_src_rank, tag).wait() [default2]:torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl [default6]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fbc94eb0d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default6]:frame #1: + 0x589518e (0x7fbccce6a18e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: return forward_call(*args, **kwargs) [default2]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [default2]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f22bbe7bd87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta [default5]: dist.recv( [default3]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3451215f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3451215f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper [default5]: return func(*args, **kwargs) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward [default7]: new_kwargs[name] = recv_from_pipeline_state_buffer( [default3]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3451215f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3451215f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv [default5]: pg.recv([tensor], group_src_rank, tag).wait() [default6]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x360 (0x7fbccce649a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fbccce64ce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer [default7]: pipeline_state.run_communication() [default2]:frame #1: + 0x589518e (0x7f22f3e3518e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication [default3]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f341a453c69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default4]: recv_activation_tensor = recv_activation() [default2]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x360 (0x7f22f3e2f9a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector > const&, c10d::OpType, int, bool) + 0x22b (0x7f341a45ac5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default5]:torch.distributed.DistBackendError: [2] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '1:2', but store->get('1:2') got error: Connection reset by peer [default5]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [default6]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fbccce65b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fbccce1af81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ [default4]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] [default3]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector >&, int, int) + 0x550 (0x7f341a47db60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default3]:frame #12: + 0x5838439 (0x7f3451208439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: recv_activation_tensor = recv_activation() [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ [default6]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fbccce1af81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fbccce1af81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f22f3e2fce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #13: + 0x5843330 (0x7f3451213330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fbccce1af81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors [default4]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) [default7]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors [default4]: meta = self._recv_meta(from_rank=from_rank, tag=tag) [default2]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f22f3e30b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f22f3de5f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors [default6]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fbc96058c69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default3]:frame #14: + 0x58433c5 (0x7f34512133c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #15: + 0x4e893cc (0x7f34508593cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta [default4]: dist.recv( [default4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper [default3]:frame #16: + 0x1a08a88 (0x7f344d3d8a88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector > const&, c10d::OpType, int, bool) + 0x22b (0x7fbc9605fc5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default5]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fee969d4d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default5]:frame #1: + 0x589518e (0x7feece98e18e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f22f3de5f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f22f3de5f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #17: + 0x5849a84 (0x7f3451219a84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]: return func(*args, **kwargs) [default4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv [default6]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector >&, int, int) + 0x550 (0x7fbc96082b60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default6]:frame #12: + 0x5838439 (0x7fbccce0d439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f22f3de5f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: meta = self._recv_meta(from_rank=from_rank, tag=tag) [default7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta [default7]: dist.recv( [default3]:frame #18: + 0x584ed35 (0x7f345121ed35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #13: + 0x5843330 (0x7fbccce18330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #14: + 0x58433c5 (0x7fbccce183c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]: pg.recv([tensor], group_src_rank, tag).wait() [default3]:frame #19: + 0xc97eee (0x7f3463ad0eee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default3]:frame #20: + 0x413ea4 (0x7f346324cea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default6]:frame #15: + 0x4e893cc (0x7fbccc45e3cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 72, in wrapper [default7]: return func(*args, **kwargs) [default4]:torch.distributed.DistBackendError: [2] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '1:2', but store->get('1:2') got error: Connection reset by peer [default4]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [default6]:frame #16: + 0x1a08a88 (0x7fbcc8fdda88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #17: + 0x5849a84 (0x7fbccce1ea84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x360 (0x7feece9889a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #21: + 0x1445a6 (0x56278fab05a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #22: _PyObject_MakeTpCall + 0x26b (0x56278faa9a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f22bd023c69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default5]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7feece988ce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7feece989b11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7feece93ef81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7feece93ef81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1706, in recv [default2]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector > const&, c10d::OpType, int, bool) + 0x22b (0x7f22bd02ac5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default2]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector >&, int, int) + 0x550 (0x7f22bd04db60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default4]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f10c79b5d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default4]:frame #1: + 0x589518e (0x7f10ff96f18e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]: pg.recv([tensor], group_src_rank, tag).wait() [default2]:frame #12: + 0x5838439 (0x7f22f3dd8439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #13: + 0x5843330 (0x7f22f3de3330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7feece93ef81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7feece93ef81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #18: + 0x584ed35 (0x7fbccce23d35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #19: + 0xc97eee (0x7fbcdf6d5eee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default6]:frame #20: + 0x413ea4 (0x7fbcdee51ea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default2]:frame #14: + 0x58433c5 (0x7f22f3de33c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x360 (0x7f10ff9699a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #21: + 0x1445a6 (0x55df175d35a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #22: _PyObject_MakeTpCall + 0x26b (0x55df175cca6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #23: + 0x150866 (0x56278fabc866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f10ff969ce2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f10ff96ab11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f10ff91ff81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x56278faa5142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #25: _PyFunction_Vectorcall + 0x6c (0x56278fab0a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:torch.distributed.DistBackendError: [3] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '2:3', but store->get('2:3') got error: Connection reset by peer [default7]:Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [default6]:frame #23: + 0x150866 (0x55df175df866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55df175c8142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #25: _PyFunction_Vectorcall + 0x6c (0x55df175d3a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #26: PyObject_Call + 0xbc (0x56278fabcf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x56278faa32b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f0e03406d87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) [default2]:frame #15: + 0x4e893cc (0x7f22f34293cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #16: + 0x1a08a88 (0x7f22effa8a88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f10ff91ff81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f10ff91ff81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #28: _PyFunction_Vectorcall + 0x6c (0x56278fab0a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x56278faa18fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #26: PyObject_Call + 0xbc (0x55df175dff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #1: + 0x589518e (0x7f0e3b3c018e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #30: + 0x150582 (0x56278fabc582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x56278faa18fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #32: + 0x150582 (0x56278fabc582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x360 (0x7f0e3b3ba9a0 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]:frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f0e3b3bace2 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fee97b7cc69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default3]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x56278faa18fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #34: + 0x150582 (0x56278fabc582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55df175c62b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #17: + 0x5849a84 (0x7f22f3de9a84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f10ff91ff81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #28: _PyFunction_Vectorcall + 0x6c (0x55df175d3a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55df175c48fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f0e3b3bbb11 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector > const&, c10d::OpType, int, bool) + 0x22b (0x7fee97b83c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default5]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector >&, int, int) + 0x550 (0x7fee97ba6b60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default3]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x56278faa18fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x56278faa8f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #18: + 0x584ed35 (0x7f22f3deed35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #12: + 0x5838439 (0x7feece931439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #13: + 0x5843330 (0x7feece93c330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f10c8b5dc69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default3]:frame #37: _PyObject_Call_Prepend + 0x69 (0x56278fabac39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #38: + 0x211239 (0x56278fb7d239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #39: _PyObject_MakeTpCall + 0x26b (0x56278faa9a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector > const&, c10d::OpType, int, bool) + 0x22b (0x7f10c8b64c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default4]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector >&, int, int) + 0x550 (0x7f10c8b87b60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default6]:frame #30: + 0x150582 (0x55df175df582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55df175c48fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #19: + 0xc97eee (0x7f23066a0eee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default5]:frame #14: + 0x58433c5 (0x7feece93c3c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]:frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0e3b370f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #32: + 0x150582 (0x55df175df582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55df175c48fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #15: + 0x4e893cc (0x7feecdf823cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #16: + 0x1a08a88 (0x7feecab01a88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x56278faa53e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #41: _PyFunction_Vectorcall + 0x6c (0x56278fab0a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0e3b370f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]:frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0e3b370f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]:frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0e3b370f81 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #17: + 0x5849a84 (0x7feece942a84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #34: + 0x150582 (0x55df175df582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #20: + 0x413ea4 (0x7f2305e1cea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default7]:frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f0e045aec69 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default5]:frame #18: + 0x584ed35 (0x7feece947d35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55df175c48fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55df175cbf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x56278faa0c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #21: + 0x1445a6 (0x5623e78ef5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #22: _PyObject_MakeTpCall + 0x26b (0x5623e78e8a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #23: + 0x150866 (0x5623e78fb866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #19: + 0xc97eee (0x7feee11f9eee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default5]:frame #20: + 0x413ea4 (0x7feee0975ea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default7]:frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector > const&, c10d::OpType, int, bool) + 0x22b (0x7f0e045b5c5b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default2]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x5623e78e4142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #25: _PyFunction_Vectorcall + 0x6c (0x5623e78efa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #43: _PyFunction_Vectorcall + 0x6c (0x56278fab0a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #11: c10d::ProcessGroupNCCL::recv(std::vector >&, int, int) + 0x550 (0x7f0e045d8b60 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [default7]:frame #12: + 0x5838439 (0x7f0e3b363439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #12: + 0x5838439 (0x7f10ff912439 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #13: + 0x5843330 (0x7f10ff91d330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #21: + 0x1445a6 (0x562aaeab55a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #13: + 0x5843330 (0x7f0e3b36e330 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]:frame #14: + 0x58433c5 (0x7f0e3b36e3c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #26: PyObject_Call + 0xbc (0x5623e78fbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5623e78e22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #28: _PyFunction_Vectorcall + 0x6c (0x5623e78efa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #22: _PyObject_MakeTpCall + 0x26b (0x562aaeaaea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #23: + 0x150866 (0x562aaeac1866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x56278faa18fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #45: + 0x150582 (0x56278fabc582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5623e78e08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x562aaeaaa142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #25: _PyFunction_Vectorcall + 0x6c (0x562aaeab5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #15: + 0x4e893cc (0x7f0e3a9b43cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #30: + 0x150582 (0x5623e78fb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #46: PyObject_Call + 0xbc (0x56278fabcf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x56278faa32b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #26: PyObject_Call + 0xbc (0x562aaeac1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x562aaeaa82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #37: _PyObject_Call_Prepend + 0x69 (0x55df175ddc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #38: + 0x211239 (0x55df176a0239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #39: _PyObject_MakeTpCall + 0x26b (0x55df175cca6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55df175c83e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5623e78e08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #48: + 0x150582 (0x56278fabc582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #28: _PyFunction_Vectorcall + 0x6c (0x562aaeab5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x562aaeaa68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #14: + 0x58433c5 (0x7f10ff91d3c5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #15: + 0x4e893cc (0x7f10fef633cc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default2]:frame #32: + 0x150582 (0x5623e78fb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5623e78e08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #49: PyObject_Call + 0xbc (0x56278fabcf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #41: _PyFunction_Vectorcall + 0x6c (0x55df175d3a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #16: + 0x1a08a88 (0x7f10fbae2a88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #17: + 0x5849a84 (0x7f10ff923a84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default3]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x56278faa32b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #51: _PyFunction_Vectorcall + 0x6c (0x56278fab0a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #34: + 0x150582 (0x5623e78fb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5623e78e08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x56278faa9007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #16: + 0x1a08a88 (0x7f0e37533a88 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default5]:frame #30: + 0x150582 (0x562aaeac1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #53: _PyObject_Call_Prepend + 0x69 (0x56278fabac39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #54: + 0x211239 (0x56278fb7d239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55df175c3c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x5623e78e7f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #17: + 0x5849a84 (0x7f0e3b374a84 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default6]:frame #43: _PyFunction_Vectorcall + 0x6c (0x55df175d3a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #18: + 0x584ed35 (0x7f10ff928d35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default4]:frame #19: + 0xc97eee (0x7f11121daeee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default3]:frame #55: PyObject_Call + 0x207 (0x56278fabd067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x56278faa32b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #37: _PyObject_Call_Prepend + 0x69 (0x5623e78f9c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #20: + 0x413ea4 (0x7f1111956ea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default5]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x562aaeaa68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #32: + 0x150582 (0x562aaeac1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55df175c48fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #45: + 0x150582 (0x55df175df582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #38: + 0x211239 (0x5623e79bc239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #39: _PyObject_MakeTpCall + 0x26b (0x5623e78e8a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #57: + 0x150582 (0x56278fabc582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x56278faa18fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #18: + 0x584ed35 (0x7f0e3b379d35 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [default7]:frame #19: + 0xc97eee (0x7f0e4dc2beee in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default2]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5623e78e43e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #41: _PyFunction_Vectorcall + 0x6c (0x5623e78efa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #46: PyObject_Call + 0xbc (0x55df175dff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x562aaeaa68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #34: + 0x150582 (0x562aaeac1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #20: + 0x413ea4 (0x7f0e4d3a7ea4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [default2]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x5623e78dfc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #21: + 0x1445a6 (0x5573f53365a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #22: _PyObject_MakeTpCall + 0x26b (0x5573f532fa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #21: + 0x1445a6 (0x55808a6dc5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #22: _PyObject_MakeTpCall + 0x26b (0x55808a6d5a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x562aaeaa68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x562aaeaadf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55df175c62b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #48: + 0x150582 (0x55df175df582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #43: _PyFunction_Vectorcall + 0x6c (0x5623e78efa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5623e78e08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #45: + 0x150582 (0x5623e78fb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #59: + 0x150582 (0x56278fabc582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #23: + 0x150866 (0x5573f5342866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x5573f532b142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #46: PyObject_Call + 0xbc (0x5623e78fbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5623e78e22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #48: + 0x150582 (0x5623e78fb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #37: _PyObject_Call_Prepend + 0x69 (0x562aaeabfc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #38: + 0x211239 (0x562aaeb82239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #39: _PyObject_MakeTpCall + 0x26b (0x562aaeaaea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #49: PyObject_Call + 0xbc (0x55df175dff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #23: + 0x150866 (0x55808a6e8866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55808a6d1142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #49: PyObject_Call + 0xbc (0x5623e78fbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5623e78e22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #60: PyObject_Call + 0xbc (0x56278fabcf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #25: _PyFunction_Vectorcall + 0x6c (0x55808a6dca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #51: _PyFunction_Vectorcall + 0x6c (0x5623e78efa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55df175c62b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #51: _PyFunction_Vectorcall + 0x6c (0x55df175d3a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #26: PyObject_Call + 0xbc (0x55808a6e8f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55808a6cf2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x562aaeaaa3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x5623e78e8007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #53: _PyObject_Call_Prepend + 0x69 (0x5623e78f9c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #25: _PyFunction_Vectorcall + 0x6c (0x5573f5336a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55df175cc007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #53: _PyObject_Call_Prepend + 0x69 (0x55df175ddc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #41: _PyFunction_Vectorcall + 0x6c (0x562aaeab5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x562aaeaa5c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #28: _PyFunction_Vectorcall + 0x6c (0x55808a6dca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #54: + 0x211239 (0x55df176a0239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #43: _PyFunction_Vectorcall + 0x6c (0x562aaeab5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x562aaeaa68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #45: + 0x150582 (0x562aaeac1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #55: PyObject_Call + 0x207 (0x55df175e0067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55df175c62b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x56278faa32b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #62: + 0x150582 (0x56278fabc582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #57: + 0x150582 (0x55df175df582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55df175c48fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55808a6cd8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #30: + 0x150582 (0x55808a6e8582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #54: + 0x211239 (0x5623e79bc239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #55: PyObject_Call + 0x207 (0x5623e78fc067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5623e78e22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #57: + 0x150582 (0x5623e78fb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #26: PyObject_Call + 0xbc (0x5573f5342f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5573f53292b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #46: PyObject_Call + 0xbc (0x562aaeac1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #59: + 0x150582 (0x55df175df582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #60: PyObject_Call + 0xbc (0x55df175dff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55df175c62b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:frame #63: PyObject_Call + 0xbc (0x56278fabcf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55808a6cd8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #32: + 0x150582 (0x55808a6e8582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #62: + 0x150582 (0x55df175df582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5623e78e08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #59: + 0x150582 (0x5623e78fb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55808a6cd8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:frame #63: PyObject_Call + 0xbc (0x55df175dff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default3]:. This may indicate a possible application crash on rank 0 or a network set up issue. [default2]:frame #60: PyObject_Call + 0xbc (0x5623e78fbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5623e78e22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default6]:. This may indicate a possible application crash on rank 0 or a network set up issue. [default4]:frame #28: _PyFunction_Vectorcall + 0x6c (0x5573f5336a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5573f53278fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #30: + 0x150582 (0x5573f5342582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #34: + 0x150582 (0x55808a6e8582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5573f53278fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #32: + 0x150582 (0x5573f5342582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5573f53278fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55808a6cd8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x562aaeaa82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #34: + 0x150582 (0x5573f5342582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55808a6d4f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #37: _PyObject_Call_Prepend + 0x69 (0x55808a6e6c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #62: + 0x150582 (0x5623e78fb582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:frame #63: PyObject_Call + 0xbc (0x5623e78fbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #48: + 0x150582 (0x562aaeac1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #49: PyObject_Call + 0xbc (0x562aaeac1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x562aaeaa82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5573f53278fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #38: + 0x211239 (0x55808a7a9239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #39: _PyObject_MakeTpCall + 0x26b (0x55808a6d5a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55808a6d13e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default2]:. This may indicate a possible application crash on rank 0 or a network set up issue. [default7]:frame #41: _PyFunction_Vectorcall + 0x6c (0x55808a6dca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #51: _PyFunction_Vectorcall + 0x6c (0x562aaeab5a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x562aaeaae007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x5573f532ef50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #37: _PyObject_Call_Prepend + 0x69 (0x5573f5340c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #53: _PyObject_Call_Prepend + 0x69 (0x562aaeabfc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #54: + 0x211239 (0x562aaeb82239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55808a6ccc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #38: + 0x211239 (0x5573f5403239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #39: _PyObject_MakeTpCall + 0x26b (0x5573f532fa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #43: _PyFunction_Vectorcall + 0x6c (0x55808a6dca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55808a6cd8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #55: PyObject_Call + 0x207 (0x562aaeac2067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x562aaeaa82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #45: + 0x150582 (0x55808a6e8582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #46: PyObject_Call + 0xbc (0x55808a6e8f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #57: + 0x150582 (0x562aaeac1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x562aaeaa68fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #59: + 0x150582 (0x562aaeac1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5573f532b3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #41: _PyFunction_Vectorcall + 0x6c (0x5573f5336a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55808a6cf2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #48: + 0x150582 (0x55808a6e8582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #42: _PyEval_EvalFrameDefault + 0x72c (0x5573f5326c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #43: _PyFunction_Vectorcall + 0x6c (0x5573f5336a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #49: PyObject_Call + 0xbc (0x55808a6e8f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #60: PyObject_Call + 0xbc (0x562aaeac1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5573f53278fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #45: + 0x150582 (0x5573f5342582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #46: PyObject_Call + 0xbc (0x5573f5342f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55808a6cf2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #51: _PyFunction_Vectorcall + 0x6c (0x55808a6dca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x562aaeaa82b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #62: + 0x150582 (0x562aaeac1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:frame #63: PyObject_Call + 0xbc (0x562aaeac1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55808a6d5007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default5]:. This may indicate a possible application crash on rank 0 or a network set up issue. [default7]:frame #53: _PyObject_Call_Prepend + 0x69 (0x55808a6e6c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #54: + 0x211239 (0x55808a7a9239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #55: PyObject_Call + 0x207 (0x55808a6e9067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55808a6cf2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #57: + 0x150582 (0x55808a6e8582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55808a6cd8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5573f53292b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #48: + 0x150582 (0x5573f5342582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #49: PyObject_Call + 0xbc (0x5573f5342f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5573f53292b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #51: _PyFunction_Vectorcall + 0x6c (0x5573f5336a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #52: _PyObject_FastCallDictTstate + 0x187 (0x5573f532f007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #53: _PyObject_Call_Prepend + 0x69 (0x5573f5340c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #54: + 0x211239 (0x5573f5403239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #55: PyObject_Call + 0x207 (0x5573f5343067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #59: + 0x150582 (0x55808a6e8582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #60: PyObject_Call + 0xbc (0x55808a6e8f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5573f53292b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #57: + 0x150582 (0x5573f5342582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5573f53278fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #59: + 0x150582 (0x5573f5342582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #60: PyObject_Call + 0xbc (0x5573f5342f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5573f53292b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #62: + 0x150582 (0x5573f5342582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default4]:frame #63: PyObject_Call + 0xbc (0x5573f5342f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55808a6cf2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #62: + 0x150582 (0x55808a6e8582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:frame #63: PyObject_Call + 0xbc (0x55808a6e8f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) [default7]:. This may indicate a possible application crash on rank 0 or a network set up issue. [default4]:. This may indicate a possible application crash on rank 0 or a network set up issue. [2024-07-06 09:21:39,768] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 2132133 closing signal SIGTERM [2024-07-06 09:21:39,768] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 2132134 closing signal SIGTERM [2024-07-06 09:21:39,769] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 2132135 closing signal SIGTERM [2024-07-06 09:21:39,769] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 2132136 closing signal SIGTERM [2024-07-06 09:21:39,769] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 2132138 closing signal SIGTERM [2024-07-06 09:21:39,770] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1194839 closing signal SIGTERM [2024-07-06 09:21:39,770] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1194840 closing signal SIGTERM [2024-07-06 09:21:39,771] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1194841 closing signal SIGTERM [2024-07-06 09:21:39,771] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1194842 closing signal SIGTERM [2024-07-06 09:21:39,777] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 70213 closing signal SIGTERM [2024-07-06 09:21:39,778] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 70214 closing signal SIGTERM [2024-07-06 09:21:39,779] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 70215 closing signal SIGTERM [2024-07-06 09:21:41,102] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -6) local_rank: 0 (pid: 70212) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 Traceback (most recent call last): File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in sys.exit(main()) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper return f(*args, **kwargs) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main run(args) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run elastic_launch( File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED ------------------------------------------------------------ Failures: [1]: time : 2024-07-06_09:21:39 host : ip-26-0-166-15.ec2.internal rank : 4 (local_rank: 4) exitcode : 1 (pid: 70216) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [2]: time : 2024-07-06_09:21:39 host : ip-26-0-166-15.ec2.internal rank : 5 (local_rank: 5) exitcode : 1 (pid: 70217) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [3]: time : 2024-07-06_09:21:39 host : ip-26-0-166-15.ec2.internal rank : 6 (local_rank: 6) exitcode : 1 (pid: 70218) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [4]: time : 2024-07-06_09:21:39 host : ip-26-0-166-15.ec2.internal rank : 7 (local_rank: 7) exitcode : 1 (pid: 70219) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-07-06_09:21:39 host : ip-26-0-166-15.ec2.internal rank : 0 (local_rank: 0) exitcode : -6 (pid: 70212) error_file: traceback : Signal 6 (SIGABRT) received by PID 70212 ============================================================ [2024-07-06 09:21:41,306] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 4 (pid: 1194843) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 [2024-07-06 09:21:41,350] torch.distributed.elastic.rendezvous.dynamic_rendezvous: [WARNING] The node 'ip-26-0-174-240.ec2.internal_1194769_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError. Traceback (most recent call last): File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in sys.exit(main()) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper return f(*args, **kwargs) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main run(args) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run elastic_launch( File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED ------------------------------------------------------------ Failures: [1]: time : 2024-07-06_09:21:39 host : ip-26-0-174-240.ec2.internal rank : 13 (local_rank: 5) exitcode : 1 (pid: 1194844) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [2]: time : 2024-07-06_09:21:39 host : ip-26-0-174-240.ec2.internal rank : 14 (local_rank: 6) exitcode : 1 (pid: 1194845) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [3]: time : 2024-07-06_09:21:39 host : ip-26-0-174-240.ec2.internal rank : 15 (local_rank: 7) exitcode : 1 (pid: 1194846) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-07-06_09:21:39 host : ip-26-0-174-240.ec2.internal rank : 12 (local_rank: 4) exitcode : 1 (pid: 1194843) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================ [2024-07-06 09:21:41,496] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 4 (pid: 2132137) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 [2024-07-06 09:21:41,541] torch.distributed.elastic.rendezvous.dynamic_rendezvous: [WARNING] The node 'ip-26-0-175-165.ec2.internal_2132062_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError. Traceback (most recent call last): File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in sys.exit(main()) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper return f(*args, **kwargs) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 812, in main run(args) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 803, in run elastic_launch( File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 135, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED ------------------------------------------------------------ Failures: [1]: time : 2024-07-06_09:21:39 host : ip-26-0-175-165.ec2.internal rank : 30 (local_rank: 6) exitcode : 1 (pid: 2132139) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [2]: time : 2024-07-06_09:21:39 host : ip-26-0-175-165.ec2.internal rank : 31 (local_rank: 7) exitcode : 1 (pid: 2132140) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-07-06_09:21:39 host : ip-26-0-175-165.ec2.internal rank : 28 (local_rank: 4) exitcode : 1 (pid: 2132137) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================ srun: error: ip-26-0-166-15: task 0: Exited with exit code 1 srun: error: ip-26-0-174-240: task 1: Exited with exit code 1 srun: error: ip-26-0-175-165: task 5: Exited with exit code 1 Consider using `hf_transfer` for faster uploads. This solution comes with some limitations. See https://huggingface.co/docs/huggingface_hub/hf_transfer for more details.