| ======================== | |
| START TIME: Tue Jul 2 19:34:11 UTC 2024 | |
| python3 version = Python 3.10.14 | |
| ======================== | |
| The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well. | |
| Token is valid (permission: write). | |
| Your token has been saved to /admin/home/ferdinand_mom/.cache/huggingface/token | |
| Login successful | |
| Already on 'bench_cluster' | |
| M examples/config_tiny_llama.py | |
| M examples/config_tiny_llama.yaml | |
| M examples/train_tiny_llama.sh | |
| M src/nanotron/models/llama.py | |
| M src/nanotron/trainer.py | |
| Your branch is up to date with 'origin/bench_cluster'. | |
| Job status: RUNNING | |
| W0702 19:34:14.431000 140094188271424 torch/distributed/run.py:757] | |
| W0702 19:34:14.431000 140094188271424 torch/distributed/run.py:757] ***************************************** | |
| W0702 19:34:14.431000 140094188271424 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. | |
| W0702 19:34:14.431000 140094188271424 torch/distributed/run.py:757] ***************************************** | |
| W0702 19:34:14.487000 140686904493888 torch/distributed/run.py:757] | |
| W0702 19:34:14.487000 140686904493888 torch/distributed/run.py:757] ***************************************** | |
| W0702 19:34:14.487000 140686904493888 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. | |
| W0702 19:34:14.487000 140686904493888 torch/distributed/run.py:757] ***************************************** | |
| [default0]:07/02/2024 19:34:33 [WARNING|DP=0|PP=0|TP=0|ip-26-0-169-139]: [Vocab Size Padding] Padded vocab (size: 50257) with 1 dummy tokens (new size: 50258) | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: Config: | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: Config(general=GeneralArgs(project='bench_cluster', | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: run='%date_%jobid', | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: seed=42, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: step=None, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: consumed_train_samples=None, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: benchmark_csv_path=None, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: ignore_sanity_checks=True), | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: parallelism=ParallelismArgs(dp=1, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: pp=8, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: tp=2, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: pp_engine=<nanotron.parallel.pipeline_parallel.engine.OneForwardOneBackwardPipelineEngine object at 0x7f93142c4910>, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: tp_mode=<TensorParallelLinearMode.REDUCE_SCATTER: 2>, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: tp_linear_async_communication=False, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: expert_parallel_size=1), | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: model=ModelArgs(model_config=LlamaConfig(bos_token_id=1, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: eos_token_id=2, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: hidden_act='silu', | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: hidden_size=2048, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: initializer_range=0.02, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: intermediate_size=4096, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: is_llama_config=True, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: max_position_embeddings=4096, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: num_attention_heads=32, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: num_hidden_layers=24, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: num_key_value_heads=32, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: pad_token_id=None, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: pretraining_tp=1, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: rms_norm_eps=1e-05, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: rope_scaling=None, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: rope_theta=10000.0, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: tie_word_embeddings=True, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: use_cache=True, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: vocab_size=50258), | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: init_method=RandomInit(std=0.025), | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: dtype=torch.bfloat16, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: make_vocab_size_divisible_by=1, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: ddp_bucket_cap_mb=25), | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: tokenizer=TokenizerArgs(tokenizer_name_or_path='openai-community/gpt2', | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: tokenizer_revision=None, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: tokenizer_max_length=None), | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: checkpoints=CheckpointsArgs(checkpoints_path=Path('/dev/null'), | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: checkpoint_interval=100000, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: save_initial_state=False, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: resume_checkpoint_path=None, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: checkpoints_path_is_shared_file_system=False), | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: logging=LoggingArgs(log_level='info', | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: log_level_replica='info', | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: iteration_step_info_interval=1), | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: tokens=TokensArgs(sequence_length=4096, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: train_steps=20, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: micro_batch_size=64, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: batch_accumulation_per_replica=16, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: val_check_interval=-1, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: limit_val_batches=0, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: limit_test_batches=0), | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: optimizer=OptimizerArgs(optimizer_factory=AdamWOptimizerArgs(adam_eps=1e-08, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: adam_beta1=0.9, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: adam_beta2=0.95, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: torch_adam_is_fused=True, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: name='adamW'), | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: zero_stage=1, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: weight_decay=0.01, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: clip_grad=1.0, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: accumulate_grad_in_fp32=True, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: learning_rate_scheduler=LRSchedulerArgs(learning_rate=0.0001, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: lr_warmup_steps=1, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: lr_warmup_style='linear', | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: lr_decay_style='linear', | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: lr_decay_steps=19, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: lr_decay_starting_step=None, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: min_decay_lr=1e-05)), | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: data_stages=[DatasetStageArgs(name='Training Stage', | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: start_training_step=1, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: data=DataArgs(dataset=PretrainDatasetsArgs(hf_dataset_or_datasets='roneneldan/TinyStories', | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: hf_dataset_splits='train', | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: hf_dataset_config_name=None, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: dataset_processing_num_proc_per_process=64, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: dataset_overwrite_cache=False, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: text_column_name='text'), | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: seed=42, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: num_loading_workers=32))], | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: profiler=ProfilerArgs(profiler_export_path=Path('/fsx/ferdinandmom/ferdinand-hf/bench_cluster/results/llama-1B/16_GPUS/dp-1_tp-2_pp-8_mbz-64')), | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: lighteval=None) | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: Model Config: | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: LlamaConfig(bos_token_id=1, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: eos_token_id=2, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: hidden_act='silu', | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: hidden_size=2048, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: initializer_range=0.02, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: intermediate_size=4096, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: is_llama_config=True, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: max_position_embeddings=4096, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: num_attention_heads=32, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: num_hidden_layers=24, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: num_key_value_heads=32, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: pad_token_id=None, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: pretraining_tp=1, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: rms_norm_eps=1e-05, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: rope_scaling=None, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: rope_theta=10000.0, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: tie_word_embeddings=True, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: use_cache=True, | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: vocab_size=50258) | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: Building model.. | |
| [default0]:07/02/2024 19:34:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: Setting PP block ranks... | |
| [default3]:07/02/2024 19:34:48 [INFO|DP=0|PP=1|TP=1|ip-26-0-169-139]: Local number of parameters: 62.9M (120.02MiB) | |
| [default3]:07/02/2024 19:34:48 [INFO|DP=0|PP=1|TP=1|ip-26-0-169-139]: [After model building] Memory usage: 123.03MiB. Peak allocated: 125.06MiB Peak reserved: 138.00MiB | |
| [default3]:07/02/2024 19:34:48 [INFO|DP=0|PP=1|TP=1|ip-26-0-169-139]: No checkpoint path provided. | |
| [default4]:07/02/2024 19:34:48 [INFO|DP=0|PP=2|TP=0|ip-26-0-169-139]: Local number of parameters: 62.9M (120.02MiB) | |
| [default4]:07/02/2024 19:34:48 [INFO|DP=0|PP=2|TP=0|ip-26-0-169-139]: [After model building] Memory usage: 123.03MiB. Peak allocated: 125.06MiB Peak reserved: 138.00MiB | |
| [default2]:07/02/2024 19:34:48 [INFO|DP=0|PP=1|TP=0|ip-26-0-169-139]: Local number of parameters: 62.9M (120.02MiB) | |
| [default2]:07/02/2024 19:34:48 [INFO|DP=0|PP=1|TP=0|ip-26-0-169-139]: [After model building] Memory usage: 123.03MiB. Peak allocated: 125.06MiB Peak reserved: 138.00MiB | |
| [default2]:07/02/2024 19:34:48 [INFO|DP=0|PP=1|TP=0|ip-26-0-169-139]: No checkpoint path provided. | |
| [default4]:07/02/2024 19:34:48 [INFO|DP=0|PP=2|TP=0|ip-26-0-169-139]: No checkpoint path provided. | |
| [default1]:07/02/2024 19:34:48 [INFO|DP=0|PP=0|TP=1|ip-26-0-169-139]: Local number of parameters: 135M (258.19MiB) | |
| [default1]:07/02/2024 19:34:48 [INFO|DP=0|PP=0|TP=1|ip-26-0-169-139]: [After model building] Memory usage: 262.20MiB. Peak allocated: 264.23MiB Peak reserved: 280.00MiB | |
| [default1]:07/02/2024 19:34:48 [INFO|DP=0|PP=0|TP=1|ip-26-0-169-139]: No checkpoint path provided. | |
| [default3]:07/02/2024 19:34:48 [INFO|DP=0|PP=5|TP=1|ip-26-0-172-73]: Local number of parameters: 62.9M (120.02MiB) | |
| [default3]:07/02/2024 19:34:48 [INFO|DP=0|PP=5|TP=1|ip-26-0-172-73]: [After model building] Memory usage: 123.03MiB. Peak allocated: 125.06MiB Peak reserved: 138.00MiB | |
| [default0]:07/02/2024 19:34:48 [INFO|DP=0|PP=4|TP=0|ip-26-0-172-73]: Local number of parameters: 62.9M (120.02MiB) | |
| [default0]:07/02/2024 19:34:48 [INFO|DP=0|PP=4|TP=0|ip-26-0-172-73]: [After model building] Memory usage: 123.03MiB. Peak allocated: 125.06MiB Peak reserved: 138.00MiB | |
| [default0]:07/02/2024 19:34:48 [INFO|DP=0|PP=4|TP=0|ip-26-0-172-73]: No checkpoint path provided. | |
| [default3]:07/02/2024 19:34:48 [INFO|DP=0|PP=5|TP=1|ip-26-0-172-73]: No checkpoint path provided. | |
| [default2]:07/02/2024 19:34:48 [INFO|DP=0|PP=5|TP=0|ip-26-0-172-73]: Local number of parameters: 62.9M (120.02MiB) | |
| [default2]:07/02/2024 19:34:48 [INFO|DP=0|PP=5|TP=0|ip-26-0-172-73]: [After model building] Memory usage: 123.03MiB. Peak allocated: 125.06MiB Peak reserved: 138.00MiB | |
| [default2]:07/02/2024 19:34:48 [INFO|DP=0|PP=5|TP=0|ip-26-0-172-73]: No checkpoint path provided. | |
| [default1]:07/02/2024 19:34:48 [INFO|DP=0|PP=4|TP=1|ip-26-0-172-73]: Local number of parameters: 62.9M (120.02MiB) | |
| [default1]:07/02/2024 19:34:48 [INFO|DP=0|PP=4|TP=1|ip-26-0-172-73]: [After model building] Memory usage: 123.03MiB. Peak allocated: 125.06MiB Peak reserved: 138.00MiB | |
| [default1]:07/02/2024 19:34:48 [INFO|DP=0|PP=4|TP=1|ip-26-0-172-73]: No checkpoint path provided. | |
| [default6]:07/02/2024 19:34:48 [INFO|DP=0|PP=3|TP=0|ip-26-0-169-139]: Local number of parameters: 83.9M (160.03MiB) | |
| [default6]:07/02/2024 19:34:48 [INFO|DP=0|PP=3|TP=0|ip-26-0-169-139]: [After model building] Memory usage: 164.04MiB. Peak allocated: 166.07MiB Peak reserved: 180.00MiB | |
| [default0]:07/02/2024 19:34:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: Total number of parameters: 1.21G (2313.02MiB) | |
| [default0]:07/02/2024 19:34:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: Local number of parameters: 135M (258.19MiB) | |
| [default0]:07/02/2024 19:34:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: [After model building] Memory usage: 262.20MiB. Peak allocated: 264.23MiB Peak reserved: 280.00MiB | |
| [default7]:07/02/2024 19:34:48 [INFO|DP=0|PP=3|TP=1|ip-26-0-169-139]: Local number of parameters: 83.9M (160.03MiB) | |
| [default6]:07/02/2024 19:34:48 [INFO|DP=0|PP=3|TP=0|ip-26-0-169-139]: No checkpoint path provided. | |
| [default7]:07/02/2024 19:34:48 [INFO|DP=0|PP=3|TP=1|ip-26-0-169-139]: [After model building] Memory usage: 164.04MiB. Peak allocated: 166.07MiB Peak reserved: 180.00MiB | |
| [default7]:07/02/2024 19:34:48 [INFO|DP=0|PP=3|TP=1|ip-26-0-169-139]: No checkpoint path provided. | |
| [default5]:07/02/2024 19:34:48 [INFO|DP=0|PP=2|TP=1|ip-26-0-169-139]: Local number of parameters: 62.9M (120.02MiB) | |
| [default0]:07/02/2024 19:34:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: No checkpoint path provided. | |
| [default0]:07/02/2024 19:34:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: Parametrizing model parameters using StandardParametrizator | |
| [default5]:07/02/2024 19:34:48 [INFO|DP=0|PP=2|TP=1|ip-26-0-169-139]: [After model building] Memory usage: 123.03MiB. Peak allocated: 125.06MiB Peak reserved: 138.00MiB | |
| [default5]:07/02/2024 19:34:48 [INFO|DP=0|PP=2|TP=1|ip-26-0-169-139]: No checkpoint path provided. | |
| [default6]:07/02/2024 19:34:48 [INFO|DP=0|PP=7|TP=0|ip-26-0-172-73]: Local number of parameters: 51.5M (98.16MiB) | |
| [default6]:07/02/2024 19:34:48 [INFO|DP=0|PP=7|TP=0|ip-26-0-172-73]: [After model building] Memory usage: 98.17MiB. Peak allocated: 98.18MiB Peak reserved: 102.00MiB | |
| [default6]:07/02/2024 19:34:48 [INFO|DP=0|PP=7|TP=0|ip-26-0-172-73]: No checkpoint path provided. | |
| [default4]:07/02/2024 19:34:48 [INFO|DP=0|PP=6|TP=0|ip-26-0-172-73]: Local number of parameters: 83.9M (160.03MiB) | |
| [default4]:07/02/2024 19:34:48 [INFO|DP=0|PP=6|TP=0|ip-26-0-172-73]: [After model building] Memory usage: 164.04MiB. Peak allocated: 166.07MiB Peak reserved: 180.00MiB | |
| [default4]:07/02/2024 19:34:48 [INFO|DP=0|PP=6|TP=0|ip-26-0-172-73]: No checkpoint path provided. | |
| [default5]:07/02/2024 19:34:48 [INFO|DP=0|PP=6|TP=1|ip-26-0-172-73]: Local number of parameters: 83.9M (160.03MiB) | |
| [default5]:07/02/2024 19:34:48 [INFO|DP=0|PP=6|TP=1|ip-26-0-172-73]: [After model building] Memory usage: 164.04MiB. Peak allocated: 166.07MiB Peak reserved: 180.00MiB | |
| [default5]:07/02/2024 19:34:48 [INFO|DP=0|PP=6|TP=1|ip-26-0-172-73]: No checkpoint path provided. | |
| [default7]:07/02/2024 19:34:48 [INFO|DP=0|PP=7|TP=1|ip-26-0-172-73]: Local number of parameters: 51.5M (98.16MiB) | |
| [default7]:07/02/2024 19:34:48 [INFO|DP=0|PP=7|TP=1|ip-26-0-172-73]: [After model building] Memory usage: 98.17MiB. Peak allocated: 98.18MiB Peak reserved: 102.00MiB | |
| [default7]:07/02/2024 19:34:48 [INFO|DP=0|PP=7|TP=1|ip-26-0-172-73]: No checkpoint path provided. | |
| [default0]:07/02/2024 19:34:50 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: [Optimizer Building] Using LearningRateForSP as learning rate | |
| [default0]:07/02/2024 19:34:50 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: [ZeRO sharding] Size of optimizer params per rank: | |
| [default0]:07/02/2024 19:34:50 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: [ZeRO sharding] DP Rank 0 has 135M out of 135M (100.00%) params' optimizer states | |
| [default0]:07/02/2024 19:34:51 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: [Training Plan] Stage Training Stage has 19 remaining training steps and has consumed 0 samples | |
| [default0]:07/02/2024 19:34:51 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: Using `datasets` library | |
| [default0]:07/02/2024 19:34:51 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: Loading tokenizer from openai-community/gpt2 and transformers/hf_hub versions ('4.41.2', '0.23.4') | |
| [default0]:07/02/2024 19:34:51 [WARNING|DP=0|PP=0|TP=0|ip-26-0-169-139]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default0]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default0]:07/02/2024 19:34:52 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: [Training Plan] There are 1 training stages | |
| [default0]:07/02/2024 19:34:52 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: [Stage Training Stage] start from step 1 | |
| [default0]:07/02/2024 19:34:52 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: | |
| [default0]:07/02/2024 19:34:52 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: [Start training] datetime: 2024-07-02 19:34:52.520338 | mbs: 64 | grad_accum: 16 | global_batch_size: 1024 | sequence_length: 4096 | train_steps: 20 | start_iteration_step: 0 | consumed_train_samples: 0 | |
| [default0]:07/02/2024 19:34:52 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: Resuming training from stage Training Stage, it has trained for 0 samples and has 19 remaining train steps | |
| [default0]:07/02/2024 19:34:52 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-139]: Memory usage: 1294.97MiB. Peak allocated 1294.97MiB. Peak reserved: 1316.00MiB | |
| [default1]:07/02/2024 19:34:52 [WARNING|DP=0|PP=0|TP=1|ip-26-0-169-139]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default4]:07/02/2024 19:34:52 [WARNING|DP=0|PP=2|TP=0|ip-26-0-169-139]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default3]:07/02/2024 19:34:52 [WARNING|DP=0|PP=1|TP=1|ip-26-0-169-139]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default0]:07/02/2024 19:34:52 [WARNING|DP=0|PP=4|TP=0|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default3]:07/02/2024 19:34:52 [WARNING|DP=0|PP=5|TP=1|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default1]:07/02/2024 19:34:52 [WARNING|DP=0|PP=4|TP=1|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default2]:07/02/2024 19:34:52 [WARNING|DP=0|PP=5|TP=0|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default6]:07/02/2024 19:34:52 [WARNING|DP=0|PP=3|TP=0|ip-26-0-169-139]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default5]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default5]:07/02/2024 19:34:52 [WARNING|DP=0|PP=2|TP=1|ip-26-0-169-139]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default4]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:07/02/2024 19:34:52 [WARNING|DP=0|PP=3|TP=1|ip-26-0-169-139]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default6]:07/02/2024 19:34:52 [WARNING|DP=0|PP=7|TP=0|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default3]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default6]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default4]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default0]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default6]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default4]:07/02/2024 19:34:52 [WARNING|DP=0|PP=6|TP=0|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default5]:07/02/2024 19:34:52 [WARNING|DP=0|PP=6|TP=1|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:07/02/2024 19:34:52 [WARNING|DP=0|PP=7|TP=1|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default1]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default3]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default5]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default2]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default1]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default2]:07/02/2024 19:34:53 [WARNING|DP=0|PP=1|TP=0|ip-26-0-169-139]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default2]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default0]:[rank0]: Traceback (most recent call last): | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default0]:[rank0]: trainer.train(dataloader) | |
| [default1]:[rank1]: Traceback (most recent call last): | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default1]:[rank1]: trainer.train(dataloader) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default1]:[rank1]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default1]:[rank1]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default0]:[rank0]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default1]:[rank1]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default0]:[rank0]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default1]:[rank1]: output = model(**micro_batch) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default1]:[rank1]: sharded_logits = self.model( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank0]: output = model(**micro_batch) | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank0]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank0]: return forward_call(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default0]:[rank0]: sharded_logits = self.model( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default1]:[rank1]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default0]:[rank0]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank0]: return forward_call(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default0]:[rank0]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default0]:[rank0]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank0]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank0]: return forward_call(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward | |
| [default0]:[rank0]: output = self.pp_block(**new_kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default0]:[rank0]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank0]: return forward_call(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward | |
| [default0]:[rank0]: output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank0]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank0]: return forward_call(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 563, in forward | |
| [default0]:[rank0]: key_value_states = torch.cat([key_states.unsqueeze(0), value_states.unsqueeze(0)], dim=0) | |
| [default1]:[rank1]: output = self.pp_block(**new_kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward | |
| [default1]:[rank1]: output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1024.00 MiB. GPU | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 563, in forward | |
| [default1]:[rank1]: key_value_states = torch.cat([key_states.unsqueeze(0), value_states.unsqueeze(0)], dim=0) | |
| [default1]:[rank1]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1024.00 MiB. GPU has a total capacity of 79.33 GiB of which 439.94 MiB is free. Including non-PyTorch memory, this process has 78.88 GiB memory in use. Of the allocated memory 69.40 GiB is allocated by PyTorch, and 1.42 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) | |
| [default4]:[rank4]: Traceback (most recent call last): | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default4]:[rank4]: trainer.train(dataloader) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default4]:[rank4]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default7]:[rank15]: Traceback (most recent call last): | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default7]:[rank15]: trainer.train(dataloader) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default7]:[rank15]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default7]:[rank15]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default7]:[rank15]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanot[default4]:[rank4]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default5]:[rank5]: Traceback (most recent call last): | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default5]:[rank5]: trainer.train(dataloader) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default5]:[rank5]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default5]:[rank5]: outputs = self.pipeline_engine.train_batch_iter( | |
| ron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default7]:[rank15]: output = model(**micro_batch) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default5]:[rank5]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank15]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank15]: return forward_call(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default7]:[rank15]: sharded_logits = self.model( | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank15]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.p[default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| y", line 1541, in _call_impl | |
| [default7]:[rank15]: return forward_call(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default7]:[rank15]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 782, in forward_with_hidden_states | |
| [default7]:[rank15]: hidden_states = self.final_layer_norm(input=hidden_encoder_states["hidden_states"])["hidden_states"] | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank15]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank4]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default7]:[rank15]: return forward_call(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default5]:[rank5]: output = model(**micro_batch) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default7]:[rank15]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default5]:[rank5]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank5]: return forward_call(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default7]:[rank15]: pipeline_state.run_communication() | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default4]:[rank4]: output = model(**micro_batch) | |
| [default7]:[rank15]: recv_activation_tensor = recv_activation() | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default7]:[rank15]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default7]:[rank15]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default5]:[rank5]: sharded_logits = self.model( | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default7]:[rank15]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default7]:[rank15]: dist.recv( | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default7]:[rank15]: return func(*args, **kwargs) | |
| [default5]:[rank5]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default7]:[rank15]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default5]:[rank5]: return forward_call(*args, **kwargs) | |
| [default7]:[rank15]: torch.distributed.DistBackendError: [7] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '6:7', but store->get('6:7') got error: Connection reset by peer | |
| [default4]:[rank4]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank15]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default7]:[rank15]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f931d390897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default5]:[rank5]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank15]: frame #1: <unknown function> + 0x5b3a23e (0x7f9356ead23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default7]:[rank15]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f9356ea7c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f9356ea7f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: return forward_call(*args, **kwargs) | |
| [default5]:[rank5]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default7]:[rank15]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f9356ea8fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9356e5d371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9356e5d371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9356e5d371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9356e5d371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libt[default4]:[rank4]: sharded_logits = self.model( | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| orch_cpu.so) | |
| [default7]:[rank15]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f931e66a189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:[rank15]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f931e671610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:[rank15]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f931e690978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank4]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank15]: frame #12: <unknown function> + 0x5adc309 (0x7f9356e4f309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #13: <unknown function> + 0x5ae6f10 (0x7f9356e59f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank4]: return forward_call(*args, **kwargs) | |
| [default7]:[rank15]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f9356e59fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #15: <unknown function> + 0x5124446 (0x7f9356497446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #16: <unknown function> + 0x1acf4b8 (0x7f9352e424b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank15]: frame #17: <unknown function> + 0x5aee004 (0x7f9356e61004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #18: <unknown function> + 0x5af36b5 (0x7f9356e666b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #19: <unknown function> + 0xd2631e (0x7f9369a5031e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default4]:[rank4]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default4]:[rank4]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default7]:[rank15]: frame #20: <unknown function> + 0x47def4 (0x7f93691a7ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default5]:[rank5]: return forward_call(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank15]: frame #21: <unknown function> + 0x1445a6 (0x557e639fd5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #22: _PyObject_MakeTpCall + 0x26b (0x557e639f6a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #23: <unknown function> + 0x150866 (0x557e63a09866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank15]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x557e639f2142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #25: _PyFunction_Vectorcall + 0x6c (0x557e639fda2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #26: PyObject_Call + 0xbc (0x557e63a09f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x557e639f02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank15]: frame #28: _PyFunction_Vectorcall + 0x6c (0x557e639fda2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x557e639ee8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #30: <unknown function> + 0x150582 (0x557e63a09582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x557e639ee8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #32: <unknown function> + 0x150582 (0x557e63a09582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default7]:[rank15]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x557e639ee8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #34: <unknown function> + 0x150582 (0x557e63a09582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x557e639ee8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default7]:[rank15]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x557e639f5f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: return forward_call(*args, **kwargs) | |
| [default7]:[rank15]: frame #37: _PyObject_Call_Prepend + 0x69 (0x557e63a07c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #38: <unknown function> + 0x211239 (0x557e63aca239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default7]:[rank15]: frame #39: _PyObject_MakeTpCall + 0x26b (0x557e639f6a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x557e639f23e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: pipeline_state.run_communication() | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default7]:[rank15]: frame #41: _PyFunction_Vectorcall + 0x6c (0x557e639fda2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: Traceback (most recent call last): | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default7]:[rank15]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x557e639edc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #43: _PyFunction_Vectorcall + 0x6c (0x557e639fda2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: trainer.train(dataloader) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default4]:[rank12]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default4]:[rank12]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default4]:[rank12]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default7]:[rank15]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x557e639ee8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #45: <unknown function> + 0x150582 (0x557e63a09582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #46: PyObject_Call + 0xbc (0x557e63a09f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x557e639f02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default7]:[rank15]: frame #48: <unknown function> + 0x150582 (0x557e63a09582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: output = model(**micro_batch) | |
| [default7]:[rank15]: frame #49: PyObject_Call + 0xbc (0x557e63a09f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x557e639f02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #51: _PyFunction_Vectorcall + 0x6c (0x557e639fda2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x557e639f6007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #53: _PyObject_Call_Prepend + 0x69 (0x557e63a07c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #54: <unknown function> + 0x211239 (0x557e63aca239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #55: _PyObject_MakeTpCall + 0x26b (0x557e639f6a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #56: _PyEval_EvalFrameDefault + 0x5723 (0x557e639f2c53 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #57: <unknown function> + 0x150582 (0x557e63a09582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x557e639ee8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #59: <unknown function> + 0x150582 (0x557e63a09582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #60: PyObject_Call + 0xbc (0x557e63a09f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank15]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x557e639f02b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #62: <unknown function> + 0x150582 (0x557e63a09582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank15]: frame #63: PyObject_Call + 0xbc (0x557e63a09f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default4]:[rank12]: return forward_call(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default4]:[rank12]: sharded_logits = self.model( | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank12]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank12]: return forward_call(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default4]:[rank12]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default4]:[rank12]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank12]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank12]: return forward_call(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default4]:[rank12]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default4]:[rank12]: pipeline_state.run_communication() | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default4]:[rank12]: recv_activation_tensor = recv_activation() | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default4]:[rank12]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default4]:[rank12]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default4]:[rank12]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default4]:[rank12]: dist.recv( | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default4]:[rank12]: return func(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default4]:[rank12]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default4]:[rank12]: torch.distributed.DistBackendError: [6] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '5:6', but store->get('5:6') got error: Connection reset by peer | |
| [default4]:[rank12]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default4]:[rank12]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f4ebbb90897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default4]:[rank12]: frame #1: <unknown function> + 0x5b3a23e (0x7f4ef56ad23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f4ef56a7c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f4ef56a7f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f4ef56a8fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4ef565d371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4ef565d371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4ef565d371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4ef565d371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f4ebce6a189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:[rank12]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f4ebce71610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:[rank12]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f4ebce90978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:[rank12]: frame #12: <unknown function> + 0x5adc309 (0x7f4ef564f309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #13: <unknown function> + 0x5ae6f10 (0x7f4ef5659f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f4ef5659fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #15: <unknown function> + 0x5124446 (0x7f4ef4c97446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #16: <unknown function> + 0x1acf4b8 (0x7f4ef16424b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #17: <unknown function> + 0x5aee004 (0x7f4ef5661004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #18: <unknown function> + 0x5af36b5 (0x7f4ef56666b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #19: <unknown function> + 0xd2631e (0x7f4f0825031e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default4]:[rank12]: frame #20: <unknown function> + 0x47def4 (0x7f4f079a7ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default4]:[rank12]: frame #21: <unknown function> + 0x1445a6 (0x55e6795845a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55e67957da6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #23: <unknown function> + 0x150866 (0x55e679590866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55e679579142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55e679584a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #26: PyObject_Call + 0xbc (0x55e679590f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55e6795772b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55e679584a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55e6795758fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #30: <unknown function> + 0x150582 (0x55e679590582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55e6795758fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #32: <unknown function> + 0x150582 (0x55e679590582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55e6795758fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #34: <unknown function> + 0x150582 (0x55e679590582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55e6795758fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55e67957cf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55e67958ec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #38: <unknown function> + 0x211239 (0x55e679651239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55e67957da6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55e6795793e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55e679584a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55e679574c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55e679584a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55e6795758fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #45: <unknown function> + 0x150582 (0x55e679590582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #46: PyObject_Call + 0xbc (0x55e679590f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55e6795772b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #48: <unknown function> + 0x150582 (0x55e679590582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #49: PyObject_Call + 0xbc (0x55e679590f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55e6795772b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55e679584a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55e67957d007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55e67958ec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #54: <unknown function> + 0x211239 (0x55e679651239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #55: PyObject_Call + 0x207 (0x55e679591067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55e6795772b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #57: <unknown function> + 0x150582 (0x55e679590582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55e6795758fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #59: <unknown function> + 0x150582 (0x55e679590582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #60: PyObject_Call + 0xbc (0x55e679590f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55e6795772b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #62: <unknown function> + 0x150582 (0x55e679590582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #63: PyObject_Call + 0xbc (0x55e679590f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default0]:[rank8]: Traceback (most recent call last): | |
| [default3]:[rank11]: Traceback (most recent call last): | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default6]:[rank14]: Traceback (most recent call last): | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default3]:[rank11]: trainer.train(dataloader) | |
| [default0]:[rank8]: trainer.train(dataloader) | |
| [default6]:[rank14]: trainer.train(dataloader) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default3]:[rank11]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default0]:[rank8]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default6]:[rank14]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default0]:[rank8]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default3]:[rank11]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default6]:[rank14]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default0]:[rank8]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default6]:[rank14]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default3]:[rank11]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default3]:[rank11]: output = model(**micro_batch) | |
| [default0]:[rank8]: output = model(**micro_batch) | |
| [default6]:[rank14]: output = model(**micro_batch) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank11]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank8]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank11]: return forward_call(*args, **kwargs) | |
| [default6]:[rank14]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default6]:[rank14]: return forward_call(*args, **kwargs) | |
| [default0]:[rank8]: return forward_call(*args, **kwargs) | |
| [default3]:[rank11]: sharded_logits = self.model( | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default6]:[rank14]: sharded_logits = self.model( | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank11]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank14]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank8]: sharded_logits = self.model( | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank8]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank8]: return forward_call(*args, **kwargs) | |
| [default6]:[rank14]: return forward_call(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default3]:[rank11]: return forward_call(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default0]:[rank8]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default6]:[rank14]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default0]:[rank8]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 782, in forward_with_hidden_states | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default6]:[rank14]: hidden_states = self.final_layer_norm(input=hidden_encoder_states["hidden_states"])["hidden_states"] | |
| [default5]:[rank5]: recv_activation_tensor = recv_activation() | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default0]:[rank8]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank5]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default4]:[rank4]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default5]:[rank5]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default6]:[rank6]: Traceback (most recent call last): | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default4]:[rank4]: pipeline_state.run_communication() | |
| [default6]:[rank6]: trainer.train(dataloader) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default5]:[rank5]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default4]:[rank4]: recv_activation_tensor = recv_activation() | |
| [default6]:[rank6]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank4]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default3]:[rank11]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank6]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default6]:[rank14]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default5]:[rank5]: dist.recv( | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default0]:[rank8]: return forward_call(*args, **kwargs) | |
| [default6]:[rank6]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank5]: return func(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default3]:[rank11]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default6]:[rank6]: output = model(**micro_batch) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank8]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default5]:[rank5]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank14]: return forward_call(*args, **kwargs) | |
| [default4]:[rank4]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default3]:[rank11]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default6]:[rank14]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default6]:[rank6]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default5]:[rank5]: torch.distributed.DistBackendError: [2] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '1:2', but store->get('1:2') got error: Connection reset by peer | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank5]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default4]:[rank4]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default6]:[rank6]: return forward_call(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default6]:[rank6]: sharded_logits = self.model( | |
| [default0]:[rank8]: pipeline_state.run_communication() | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default4]:[rank4]: dist.recv( | |
| [default3]:[rank11]: return forward_call(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default5]:[rank5]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f5f06bde897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default0]:[rank8]: recv_activation_tensor = recv_activation() | |
| [default5]:[rank5]: frame #1: <unknown function> + 0x5b3a23e (0x7f5f406fb23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: pipeline_state.run_communication() | |
| [default6]:[rank6]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank5]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f5f406f5c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default3]:[rank11]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default6]:[rank6]: return forward_call(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default5]:[rank5]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f5f406f5f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f5f406f6fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: recv_activation_tensor = recv_activation() | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default6]:[rank6]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default3]:[rank11]: pipeline_state.run_communication() | |
| [default4]:[rank4]: return func(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default6]:[rank14]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default0]:[rank8]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default5]:[rank5]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f5f406ab371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default5]:[rank5]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f5f406ab371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f5f406ab371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default6]:[rank6]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank14]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default5]:[rank5]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f5f406ab371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default5]:[rank5]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f5f07eb8189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default6]:[rank6]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default5]:[rank5]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f5f07ebf610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:[rank5]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f5f07ede978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:[rank5]: frame #12: <unknown function> + 0x5adc309 (0x7f5f4069d309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default5]:[rank5]: frame #13: <unknown function> + 0x5ae6f10 (0x7f5f406a7f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: recv_activation_tensor = recv_activation() | |
| [default4]:[rank4]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default4]:[rank4]: torch.distributed.DistBackendError: [2] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '1:2', but store->get('1:2') got error: Connection reset by peer | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default6]:[rank6]: return forward_call(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default0]:[rank8]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default5]:[rank5]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f5f406a7fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default6]:[rank6]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default4]:[rank4]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default4]:[rank4]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fd38ef7b897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default6]:[rank14]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default6]:[rank6]: pipeline_state.run_communication() | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default4]:[rank4]: frame #1: <unknown function> + 0x5b3a23e (0x7fd3c8a9823e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: dist.recv( | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default4]:[rank4]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fd3c8a92c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default0]:[rank8]: return func(*args, **kwargs) | |
| [default5]:[rank5]: frame #15: <unknown function> + 0x5124446 (0x7f5f3fce5446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #16: <unknown function> + 0x1acf4b8 (0x7f5f3c6904b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default5]:[rank5]: frame #17: <unknown function> + 0x5aee004 (0x7f5f406af004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default6]:[rank14]: dist.recv( | |
| [default4]:[rank4]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fd3c8a92f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default6]:[rank6]: recv_activation_tensor = recv_activation() | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default5]:[rank5]: frame #18: <unknown function> + 0x5af36b5 (0x7f5f406b46b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default5]:[rank5]: frame #19: <unknown function> + 0xd2631e (0x7f5f5329e31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default3]:[rank11]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default0]:[rank8]: torch.distributed.DistBackendError: [4] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '3:4', but store->get('3:4') got error: Connection reset by peer | |
| [default5]:[rank5]: frame #20: <unknown function> + 0x47def4 (0x7f5f529f5ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default5]:[rank5]: frame #21: <unknown function> + 0x1445a6 (0x55af73e465a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: return func(*args, **kwargs) | |
| [default5]:[rank5]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55af73e3fa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #23: <unknown function> + 0x150866 (0x55af73e52866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default6]:[rank6]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default5]:[rank5]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55af73e3b142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default0]:[rank8]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f79166b4897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default6]:[rank14]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default5]:[rank5]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55af73e46a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fd3c8a93fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default3]:[rank11]: dist.recv( | |
| [default4]:[rank4]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd3c8a48371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd3c8a48371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd3c8a48371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #1: <unknown function> + 0x5b3a23e (0x7f79501d123e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default5]:[rank5]: frame #26: PyObject_Call + 0xbc (0x55af73e52f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: torch.distributed.DistBackendError: [7] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '6:7', but store->get('6:7') got error: Connection reset by peer | |
| [default5]:[rank5]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55af73e392b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: return func(*args, **kwargs) | |
| [default0]:[rank8]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f79501cbc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default3]:[rank11]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default4]:[rank4]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd3c8a48371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default4]:[rank4]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fd390255189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default0]:[rank8]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f79501cbf82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55af73e46a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: torch.distributed.DistBackendError: [5] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '4:5', but store->get('4:5') got error: Connection reset by peer | |
| [default6]:[rank14]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7ff12ef0c897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default4]:[rank4]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fd39025c610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:[rank11]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default4]:[rank4]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fd39027b978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:[rank4]: frame #12: <unknown function> + 0x5adc309 (0x7fd3c8a3a309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #1: <unknown function> + 0x5b3a23e (0x7ff168a2923e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55af73e378fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f79501ccfd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #13: <unknown function> + 0x5ae6f10 (0x7fd3c8a44f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: dist.recv( | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default3]:[rank11]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f36d1cb4897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default5]:[rank5]: frame #30: <unknown function> + 0x150582 (0x55af73e52582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7950181371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7ff168a23c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55af73e378fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #1: <unknown function> + 0x5b3a23e (0x7f370b7d123e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: return func(*args, **kwargs) | |
| [default6]:[rank14]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7ff168a23f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default3]:[rank11]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f370b7cbc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default0]:[rank8]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7950181371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: torch.distributed.DistBackendError: [3] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '2:3', but store->get('2:3') got error: Connection reset by peer | |
| [default6]:[rank14]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7ff168a24fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default0]:[rank8]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7950181371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f370b7cbf82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fbf20fea897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default6]:[rank14]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff1689d9371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: frame #1: <unknown function> + 0x5b3a23e (0x7fbf5ab0723e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7950181371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #32: <unknown function> + 0x150582 (0x55af73e52582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f370b7ccfd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55af73e378fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #34: <unknown function> + 0x150582 (0x55af73e52582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff1689d9371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55af73e378fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f791798e189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:[rank5]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55af73e3ef50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff1689d9371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f370b781371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fbf5ab01c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fbf5ab01f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff1689d9371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fd3c8a44fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #15: <unknown function> + 0x5124446 (0x7fd3c8082446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f370b781371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fbf5ab02fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7ff1301e6189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:[rank5]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55af73e50c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f370b781371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #16: <unknown function> + 0x1acf4b8 (0x7fd3c4a2d4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f7917995610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:[rank14]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7ff1301ed610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:[rank5]: frame #38: <unknown function> + 0x211239 (0x55af73f13239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f370b781371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7ff13020c978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:[rank14]: frame #12: <unknown function> + 0x5adc309 (0x7ff1689cb309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #17: <unknown function> + 0x5aee004 (0x7fd3c8a4c004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fbf5aab7371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fbf5aab7371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f79179b4978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:[rank14]: frame #13: <unknown function> + 0x5ae6f10 (0x7ff1689d5f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55af73e3fa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #12: <unknown function> + 0x5adc309 (0x7f7950173309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f36d2f8e189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:[rank14]: frame #14: <unknown function> + 0x5ae6fa5 (0x7ff1689d5fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55af73e3b3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f36d2f95610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:[rank6]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fbf5aab7371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #13: <unknown function> + 0x5ae6f10 (0x7f795017df10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #18: <unknown function> + 0x5af36b5 (0x7fd3c8a516b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #19: <unknown function> + 0xd2631e (0x7fd3db63b31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default4]:[rank4]: frame #20: <unknown function> + 0x47def4 (0x7fd3dad92ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default6]:[rank14]: frame #15: <unknown function> + 0x5124446 (0x7ff168013446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #21: <unknown function> + 0x1445a6 (0x55d93d1d45a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #16: <unknown function> + 0x1acf4b8 (0x7ff1649be4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f36d2fb4978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:[rank6]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fbf5aab7371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #12: <unknown function> + 0x5adc309 (0x7f370b773309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fbf222c4189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:[rank11]: frame #13: <unknown function> + 0x5ae6f10 (0x7f370b77df10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55d93d1cda6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55af73e46a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f795017dfa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #15: <unknown function> + 0x5124446 (0x7f794f7bb446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #23: <unknown function> + 0x150866 (0x55d93d1e0866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55af73e36c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #17: <unknown function> + 0x5aee004 (0x7ff1689dd004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #18: <unknown function> + 0x5af36b5 (0x7ff1689e26b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55d93d1c9142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #19: <unknown function> + 0xd2631e (0x7ff17b5cc31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default6]:[rank6]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fbf222cb610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:[rank14]: frame #20: <unknown function> + 0x47def4 (0x7ff17ad23ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default4]:[rank4]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55d93d1d4a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #21: <unknown function> + 0x1445a6 (0x55a2b6fa85a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55a2b6fa1a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #26: PyObject_Call + 0xbc (0x55d93d1e0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55af73e46a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55af73e378fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #45: <unknown function> + 0x150582 (0x55af73e52582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #23: <unknown function> + 0x150866 (0x55a2b6fb4866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fbf222ea978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:[rank6]: frame #12: <unknown function> + 0x5adc309 (0x7fbf5aaa9309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55a2b6f9d142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f370b77dfa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #16: <unknown function> + 0x1acf4b8 (0x7f794c1664b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55d93d1c72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #46: PyObject_Call + 0xbc (0x55af73e52f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55af73e392b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #17: <unknown function> + 0x5aee004 (0x7f7950185004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: frame #13: <unknown function> + 0x5ae6f10 (0x7fbf5aab3f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55a2b6fa8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #48: <unknown function> + 0x150582 (0x55af73e52582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #18: <unknown function> + 0x5af36b5 (0x7f795018a6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #49: PyObject_Call + 0xbc (0x55af73e52f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55af73e392b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #15: <unknown function> + 0x5124446 (0x7f370adbb446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fbf5aab3fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: frame #15: <unknown function> + 0x5124446 (0x7fbf5a0f1446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #19: <unknown function> + 0xd2631e (0x7f7962d7431e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default6]:[rank14]: frame #26: PyObject_Call + 0xbc (0x55a2b6fb4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #16: <unknown function> + 0x1acf4b8 (0x7fbf56a9c4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55af73e46a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #16: <unknown function> + 0x1acf4b8 (0x7f37077664b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55d93d1d4a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #17: <unknown function> + 0x5aee004 (0x7f370b785004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55a2b6f9b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55a2b6fa8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55af73e3f007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #18: <unknown function> + 0x5af36b5 (0x7f370b78a6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55af73e50c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55a2b6f998fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #17: <unknown function> + 0x5aee004 (0x7fbf5aabb004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #19: <unknown function> + 0xd2631e (0x7f371e37431e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default3]:[rank11]: frame #20: <unknown function> + 0x47def4 (0x7f371dacbef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default4]:[rank4]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55d93d1c58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #30: <unknown function> + 0x150582 (0x55d93d1e0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55d93d1c58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #30: <unknown function> + 0x150582 (0x55a2b6fb4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #18: <unknown function> + 0x5af36b5 (0x7fbf5aac06b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #20: <unknown function> + 0x47def4 (0x7f79624cbef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default6]:[rank6]: frame #19: <unknown function> + 0xd2631e (0x7fbf6d6aa31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default3]:[rank11]: frame #21: <unknown function> + 0x1445a6 (0x562acfd685a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #54: <unknown function> + 0x211239 (0x55af73f13239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #21: <unknown function> + 0x1445a6 (0x560e313ab5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55a2b6f998fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #20: <unknown function> + 0x47def4 (0x7fbf6ce01ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default6]:[rank6]: frame #21: <unknown function> + 0x1445a6 (0x5602622f15a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #22: _PyObject_MakeTpCall + 0x26b (0x562acfd61a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #55: PyObject_Call + 0x207 (0x55af73e53067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #22: _PyObject_MakeTpCall + 0x26b (0x560e313a4a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55af73e392b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #32: <unknown function> + 0x150582 (0x55a2b6fb4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #57: <unknown function> + 0x150582 (0x55af73e52582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #22: _PyObject_MakeTpCall + 0x26b (0x5602622eaa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #23: <unknown function> + 0x150866 (0x562acfd74866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #32: <unknown function> + 0x150582 (0x55d93d1e0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55d93d1c58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x562acfd5d142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55a2b6f998fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #25: _PyFunction_Vectorcall + 0x6c (0x562acfd68a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #23: <unknown function> + 0x150866 (0x5602622fd866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #34: <unknown function> + 0x150582 (0x55a2b6fb4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #34: <unknown function> + 0x150582 (0x55d93d1e0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55a2b6f998fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #23: <unknown function> + 0x150866 (0x560e313b7866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #26: PyObject_Call + 0xbc (0x562acfd74f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55d93d1c58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x560e313a0142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55af73e378fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x5602622e6142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55a2b6fa0f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55d93d1ccf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #25: _PyFunction_Vectorcall + 0x6c (0x560e313aba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #25: _PyFunction_Vectorcall + 0x6c (0x5602622f1a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55a2b6fb2c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #59: <unknown function> + 0x150582 (0x55af73e52582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #60: PyObject_Call + 0xbc (0x55af73e52f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #26: PyObject_Call + 0xbc (0x560e313b7f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #26: PyObject_Call + 0xbc (0x5602622fdf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #38: <unknown function> + 0x211239 (0x55a2b7075239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55d93d1dec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x562acfd5b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x560e3139e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #38: <unknown function> + 0x211239 (0x55d93d2a1239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55d93d1cda6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #28: _PyFunction_Vectorcall + 0x6c (0x562acfd68a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55a2b6fa1a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55a2b6f9d3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55d93d1c93e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55a2b6fa8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55d93d1d4a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55af73e392b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5602622e42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #28: _PyFunction_Vectorcall + 0x6c (0x5602622f1a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x562acfd598fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5602622e28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55a2b6f98c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #28: _PyFunction_Vectorcall + 0x6c (0x560e313aba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55d93d1c4c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55d93d1d4a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55d93d1c58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55a2b6fa8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55a2b6f998fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #30: <unknown function> + 0x150582 (0x5602622fd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #30: <unknown function> + 0x150582 (0x562acfd74582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #45: <unknown function> + 0x150582 (0x55d93d1e0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x560e3139c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5602622e28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #30: <unknown function> + 0x150582 (0x560e313b7582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #62: <unknown function> + 0x150582 (0x55af73e52582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x562acfd598fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #63: PyObject_Call + 0xbc (0x55af73e52f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #46: PyObject_Call + 0xbc (0x55d93d1e0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #45: <unknown function> + 0x150582 (0x55a2b6fb4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x560e3139c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #32: <unknown function> + 0x150582 (0x562acfd74582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default4]:[rank4]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55d93d1c72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #32: <unknown function> + 0x150582 (0x560e313b7582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x562acfd598fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #48: <unknown function> + 0x150582 (0x55d93d1e0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #49: PyObject_Call + 0xbc (0x55d93d1e0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #34: <unknown function> + 0x150582 (0x562acfd74582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55d93d1c72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x560e3139c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #34: <unknown function> + 0x150582 (0x560e313b7582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55d93d1d4a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55d93d1cd007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: Traceback (most recent call last): | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default6]:[rank14]: frame #46: PyObject_Call + 0xbc (0x55a2b6fb4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x562acfd598fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #32: <unknown function> + 0x150582 (0x5602622fd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x560e3139c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55a2b6f9b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x562acfd60f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x560e313a3f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5602622e28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55d93d1dec39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #37: _PyObject_Call_Prepend + 0x69 (0x562acfd72c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #54: <unknown function> + 0x211239 (0x55d93d2a1239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #48: <unknown function> + 0x150582 (0x55a2b6fb4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #49: PyObject_Call + 0xbc (0x55a2b6fb4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #34: <unknown function> + 0x150582 (0x5602622fd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #38: <unknown function> + 0x211239 (0x562acfe35239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5602622e28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x5602622e9f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: trainer.train(dataloader) | |
| [default6]:[rank14]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55a2b6f9b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #37: _PyObject_Call_Prepend + 0x69 (0x560e313b5c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #55: PyObject_Call + 0x207 (0x55d93d1e1067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55d93d1c72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default6]:[rank6]: frame #37: _PyObject_Call_Prepend + 0x69 (0x5602622fbc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #39: _PyObject_MakeTpCall + 0x26b (0x562acfd61a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #38: <unknown function> + 0x211239 (0x560e31478239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #57: <unknown function> + 0x150582 (0x55d93d1e0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55a2b6fa8a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #38: <unknown function> + 0x211239 (0x5602623be239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default4]:[rank4]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55d93d1c58fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #59: <unknown function> + 0x150582 (0x55d93d1e0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x562acfd5d3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55a2b6fa1007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55a2b6fb2c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #60: PyObject_Call + 0xbc (0x55d93d1e0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #39: _PyObject_MakeTpCall + 0x26b (0x5602622eaa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #54: <unknown function> + 0x211239 (0x55a2b7075239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55d93d1c72b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #39: _PyObject_MakeTpCall + 0x26b (0x560e313a4a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #62: <unknown function> + 0x150582 (0x55d93d1e0582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #63: PyObject_Call + 0xbc (0x55d93d1e0f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #41: _PyFunction_Vectorcall + 0x6c (0x562acfd68a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5602622e63e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #41: _PyFunction_Vectorcall + 0x6c (0x5602622f1a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default6]:[rank14]: frame #55: _PyObject_MakeTpCall + 0x26b (0x55a2b6fa1a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #56: _PyEval_EvalFrameDefault + 0x5723 (0x55a2b6f9dc53 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default0]:[rank8]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x560e313a03e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x5602622e1c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #43: _PyFunction_Vectorcall + 0x6c (0x5602622f1a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default6]:[rank14]: frame #57: <unknown function> + 0x150582 (0x55a2b6fb4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55a2b6f998fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5602622e28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #45: <unknown function> + 0x150582 (0x5602622fd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x562acfd58c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #46: PyObject_Call + 0xbc (0x5602622fdf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5602622e42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #48: <unknown function> + 0x150582 (0x5602622fd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #49: PyObject_Call + 0xbc (0x5602622fdf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #59: <unknown function> + 0x150582 (0x55a2b6fb4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5602622e42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #51: _PyFunction_Vectorcall + 0x6c (0x5602622f1a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x5602622ea007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #60: PyObject_Call + 0xbc (0x55a2b6fb4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #43: _PyFunction_Vectorcall + 0x6c (0x562acfd68a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x562acfd598fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #53: _PyObject_Call_Prepend + 0x69 (0x5602622fbc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #54: <unknown function> + 0x211239 (0x5602623be239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #55: PyObject_Call + 0x207 (0x5602622fe067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5602622e42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default6]:[rank6]: frame #57: <unknown function> + 0x150582 (0x5602622fd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5602622e28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #59: <unknown function> + 0x150582 (0x5602622fd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #60: PyObject_Call + 0xbc (0x5602622fdf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default6]:[rank6]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5602622e42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #62: <unknown function> + 0x150582 (0x5602622fd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #63: PyObject_Call + 0xbc (0x5602622fdf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #45: <unknown function> + 0x150582 (0x562acfd74582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #46: PyObject_Call + 0xbc (0x562acfd74f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default6]:[rank14]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55a2b6f9b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #62: <unknown function> + 0x150582 (0x55a2b6fb4582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: Traceback (most recent call last): | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default7]:[rank7]: trainer.train(dataloader) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default7]:[rank7]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default7]:[rank7]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default7]:[rank7]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/na[default3]:[rank11]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x562acfd5b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #48: <unknown function> + 0x150582 (0x562acfd74582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #49: PyObject_Call + 0xbc (0x562acfd74f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| notron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default7]:[rank7]: output = model(**micro_batch) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default6]:[rank14]: frame #63: PyObject_Call + 0xbc (0x55a2b6fb4f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank7]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank7]: return forward_call(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default7]:[rank7]: sharded_logits = self.model( | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank7]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line [default6]:[rank14]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default0]:[rank8]: frame #41: _PyFunction_Vectorcall + 0x6c (0x560e313aba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x560e3139bc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #43: _PyFunction_Vectorcall + 0x6c (0x560e313aba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| 1541, in _call_impl | |
| [default7]:[rank7]: return forward_call(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default7]:[rank7]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default7]:[rank7]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank7]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank11]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x562acfd5b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #51: _PyFunction_Vectorcall + 0x6c (0x562acfd68a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: return forward_call(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default7]:[rank7]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default7]:[rank7]: pipeline_state.run_communication() | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default7]:[rank7]: recv_activation_tensor = recv_activation() | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default7]:[rank7]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default5]:[rank13]: output = model(**micro_batch) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default7]:[rank7]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default7]:[rank7]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank11]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x562acfd61007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: dist.recv( | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default7]:[rank7]: return func(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default5]:[rank13]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank7]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default7]:[rank7]: torch.distributed.DistBackendError: [3] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '2:3', but store->get('2:3') got error: Connection reset by peer | |
| [default0]:[rank8]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x560e3139c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default7]:[rank7]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f914b756897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default0]:[rank8]: frame #45: <unknown function> + 0x150582 (0x560e313b7582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #1: <unknown function> + 0x5b3a23e (0x7f918527323e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank7]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f918526dc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f918526df82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f918526efd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9185223371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: return forward_call(*args, **kwargs) | |
| [default7]:[rank7]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9185223371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9185223371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9185223371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f914ca30189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:[rank11]: frame #53: _PyObject_Call_Prepend + 0x69 (0x562acfd72c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #54: <unknown function> + 0x211239 (0x562acfe35239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f914ca37610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:[rank7]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f914ca56978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:[rank7]: frame #12: <unknown function> + 0x5adc309 (0x7f9185215309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #13: <unknown function> + 0x5ae6f10 (0x7f918521ff10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f918521ffa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #15: <unknown function> + 0x5124446 (0x7f918485d446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #16: <unknown function> + 0x1acf4b8 (0x7f91812084b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #17: <unknown function> + 0x5aee004 (0x7f9185227004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #18: <unknown function> + 0x5af36b5 (0x7f918522c6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #19: <unknown function> + 0xd2631e (0x7f9197e1631e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default7]:[rank7]: frame #20: <unknown function> + 0x47def4 (0x7f919756def4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default7]:[rank7]: frame #21: <unknown function> + 0x1445a6 (0x5610a55cf5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #22: _PyObject_MakeTpCall + 0x26b (0x5610a55c8a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #23: <unknown function> + 0x150866 (0x5610a55db866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x5610a55c4142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #25: _PyFunction_Vectorcall + 0x6c (0x5610a55cfa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #26: PyObject_Call + 0xbc (0x5610a55dbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5610a55c22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #28: _PyFunction_Vectorcall + 0x6c (0x5610a55cfa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5610a55c08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #30: <unknown function> + 0x150582 (0x5610a55db582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default7]:[rank7]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5610a55c08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #32: <unknown function> + 0x150582 (0x5610a55db582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5610a55c08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #34: <unknown function> + 0x150582 (0x5610a55db582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: sharded_logits = self.model( | |
| [default7]:[rank7]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5610a55c08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x5610a55c7f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #37: _PyObject_Call_Prepend + 0x69 (0x5610a55d9c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #38: <unknown function> + 0x211239 (0x5610a569c239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #39: _PyObject_MakeTpCall + 0x26b (0x5610a55c8a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #55: PyObject_Call + 0x207 (0x562acfd75067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x562acfd5b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5610a55c43e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #41: _PyFunction_Vectorcall + 0x6c (0x5610a55cfa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x5610a55bfc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #43: _PyFunction_Vectorcall + 0x6c (0x5610a55cfa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5610a55c08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank7]: frame #45: <unknown function> + 0x150582 (0x5610a55db582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #46: PyObject_Call + 0xbc (0x5610a55dbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5610a55c22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #48: <unknown function> + 0x150582 (0x5610a55db582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #46: PyObject_Call + 0xbc (0x560e313b7f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x560e3139e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #49: PyObject_Call + 0xbc (0x5610a55dbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5610a55c22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #51: _PyFunction_Vectorcall + 0x6c (0x5610a55cfa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x5610a55c8007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #57: <unknown function> + 0x150582 (0x562acfd74582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #53: _PyObject_Call_Prepend + 0x69 (0x5610a55d9c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #54: <unknown function> + 0x211239 (0x5610a569c239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #55: PyObject_Call + 0x207 (0x5610a55dc067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5610a55c22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #57: <unknown function> + 0x150582 (0x5610a55db582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5610a55c08fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #59: <unknown function> + 0x150582 (0x5610a55db582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/[default5]:[rank13]: return self._call_impl(*args, **kwargs) | |
| python3.10) | |
| [default7]:[rank7]: frame #60: PyObject_Call + 0xbc (0x5610a55dbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5610a55c22b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #62: <unknown function> + 0x150582 (0x5610a55db582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #48: <unknown function> + 0x150582 (0x560e313b7582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #49: PyObject_Call + 0xbc (0x560e313b7f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #63: PyObject_Call + 0xbc (0x5610a55dbf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank13]: return forward_call(*args, **kwargs) | |
| [default0]:[rank8]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x560e3139e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #51: _PyFunction_Vectorcall + 0x6c (0x560e313aba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x560e313a4007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default5]:[rank13]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default3]:[rank11]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x562acfd598fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #59: <unknown function> + 0x150582 (0x562acfd74582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #60: PyObject_Call + 0xbc (0x562acfd74f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x562acfd5b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #53: _PyObject_Call_Prepend + 0x69 (0x560e313b5c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #54: <unknown function> + 0x211239 (0x560e31478239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #55: PyObject_Call + 0x207 (0x560e313b8067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default5]:[rank13]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default0]:[rank8]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x560e3139e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #57: <unknown function> + 0x150582 (0x560e313b7582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x560e3139c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #62: <unknown function> + 0x150582 (0x562acfd74582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #59: <unknown function> + 0x150582 (0x560e313b7582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank8]: frame #60: PyObject_Call + 0xbc (0x560e313b7f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x560e3139e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #63: PyObject_Call + 0xbc (0x562acfd74f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default5]:[rank13]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank8]: frame #62: <unknown function> + 0x150582 (0x560e313b7582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #63: PyObject_Call + 0xbc (0x560e313b7f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default5]:[rank13]: return forward_call(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default5]:[rank13]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default5]:[rank13]: pipeline_state.run_communication() | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default5]:[rank13]: recv_activation_tensor = recv_activation() | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default5]:[rank13]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default5]:[rank13]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default5]:[rank13]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default5]:[rank13]: dist.recv( | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default5]:[rank13]: return func(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default5]:[rank13]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default5]:[rank13]: torch.distributed.DistBackendError: [6] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '5:6', but store->get('5:6') got error: Connection reset by peer | |
| [default5]:[rank13]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default5]:[rank13]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f4b2ccba897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default5]:[rank13]: frame #1: <unknown function> + 0x5b3a23e (0x7f4b667d723e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f4b667d1c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f4b667d1f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f4b667d2fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4b66787371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4b66787371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4b66787371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4b66787371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f4b2df94189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:[rank13]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f4b2df9b610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:[rank13]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f4b2dfba978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:[rank13]: frame #12: <unknown function> + 0x5adc309 (0x7f4b66779309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #13: <unknown function> + 0x5ae6f10 (0x7f4b66783f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f4b66783fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #15: <unknown function> + 0x5124446 (0x7f4b65dc1446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #16: <unknown function> + 0x1acf4b8 (0x7f4b6276c4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #17: <unknown function> + 0x5aee004 (0x7f4b6678b004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #18: <unknown function> + 0x5af36b5 (0x7f4b667906b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #19: <unknown function> + 0xd2631e (0x7f4b7937a31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default5]:[rank13]: frame #20: <unknown function> + 0x47def4 (0x7f4b78ad1ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default5]:[rank13]: frame #21: <unknown function> + 0x1445a6 (0x55d86542e5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55d865427a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #23: <unknown function> + 0x150866 (0x55d86543a866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55d865423142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55d86542ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #26: PyObject_Call + 0xbc (0x55d86543af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55d8654212b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55d86542ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55d86541f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #30: <unknown function> + 0x150582 (0x55d86543a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55d86541f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #32: <unknown function> + 0x150582 (0x55d86543a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55d86541f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #34: <unknown function> + 0x150582 (0x55d86543a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55d86541f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55d865426f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55d865438c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #38: <unknown function> + 0x211239 (0x55d8654fb239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55d865427a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55d8654233e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55d86542ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55d86541ec5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55d86542ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55d86541f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #45: <unknown function> + 0x150582 (0x55d86543a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #46: PyObject_Call + 0xbc (0x55d86543af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55d8654212b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #48: <unknown function> + 0x150582 (0x55d86543a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #49: PyObject_Call + 0xbc (0x55d86543af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55d8654212b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55d86542ea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55d865427007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55d865438c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #54: <unknown function> + 0x211239 (0x55d8654fb239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #55: PyObject_Call + 0x207 (0x55d86543b067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55d8654212b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #57: <unknown function> + 0x150582 (0x55d86543a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55d86541f8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #59: <unknown function> + 0x150582 (0x55d86543a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #60: PyObject_Call + 0xbc (0x55d86543af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55d8654212b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #62: <unknown function> + 0x150582 (0x55d86543a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #63: PyObject_Call + 0xbc (0x55d86543af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default2]:[rank10]: Traceback (most recent call last): | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default2]:[rank10]: trainer.train(dataloader) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default2]:[rank10]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default2]:[rank10]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default2]:[rank10]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default2]:[rank10]: output = model(**micro_batch) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank10]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank10]: return forward_call(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default2]:[rank10]: sharded_logits = self.model( | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank10]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank10]: return forward_call(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default2]:[rank10]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default2]:[rank10]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank10]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank10]: return forward_call(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default2]:[rank10]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default2]:[rank10]: pipeline_state.run_communication() | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default2]:[rank10]: recv_activation_tensor = recv_activation() | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default2]:[rank10]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default2]:[rank10]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default2]:[rank10]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default2]:[rank10]: dist.recv( | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default2]:[rank10]: return func(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default2]:[rank10]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default2]:[rank10]: torch.distributed.DistBackendError: [5] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '4:5', but store->get('4:5') got error: Connection reset by peer | |
| [default2]:[rank10]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default2]:[rank10]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f36796f0897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default2]:[rank10]: frame #1: <unknown function> + 0x5b3a23e (0x7f36b320d23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f36b3207c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f36b3207f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f36b3208fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f36b31bd371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f36b31bd371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f36b31bd371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f36b31bd371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f367a9ca189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:[rank10]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f367a9d1610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:[rank10]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f367a9f0978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:[rank10]: frame #12: <unknown function> + 0x5adc309 (0x7f36b31af309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #13: <unknown function> + 0x5ae6f10 (0x7f36b31b9f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f36b31b9fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #15: <unknown function> + 0x5124446 (0x7f36b27f7446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #16: <unknown function> + 0x1acf4b8 (0x7f36af1a24b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #17: <unknown function> + 0x5aee004 (0x7f36b31c1004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #18: <unknown function> + 0x5af36b5 (0x7f36b31c66b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #19: <unknown function> + 0xd2631e (0x7f36c5db031e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default2]:[rank10]: frame #20: <unknown function> + 0x47def4 (0x7f36c5507ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default2]:[rank10]: frame #21: <unknown function> + 0x1445a6 (0x55f7811fe5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55f7811f7a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #23: <unknown function> + 0x150866 (0x55f78120a866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55f7811f3142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55f7811fea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #26: PyObject_Call + 0xbc (0x55f78120af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55f7811f12b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55f7811fea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55f7811ef8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #30: <unknown function> + 0x150582 (0x55f78120a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55f7811ef8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #32: <unknown function> + 0x150582 (0x55f78120a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55f7811ef8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #34: <unknown function> + 0x150582 (0x55f78120a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55f7811ef8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55f7811f6f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55f781208c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #38: <unknown function> + 0x211239 (0x55f7812cb239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55f7811f7a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55f7811f33e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55f7811fea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55f7811eec5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55f7811fea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55f7811ef8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #45: <unknown function> + 0x150582 (0x55f78120a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #46: PyObject_Call + 0xbc (0x55f78120af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55f7811f12b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #48: <unknown function> + 0x150582 (0x55f78120a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #49: PyObject_Call + 0xbc (0x55f78120af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55f7811f12b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55f7811fea2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55f7811f7007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55f781208c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #54: <unknown function> + 0x211239 (0x55f7812cb239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #55: PyObject_Call + 0x207 (0x55f78120b067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55f7811f12b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #57: <unknown function> + 0x150582 (0x55f78120a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55f7811ef8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #59: <unknown function> + 0x150582 (0x55f78120a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #60: PyObject_Call + 0xbc (0x55f78120af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55f7811f12b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #62: <unknown function> + 0x150582 (0x55f78120a582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #63: PyObject_Call + 0xbc (0x55f78120af1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default1]:[rank9]: Traceback (most recent call last): | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default1]:[rank9]: trainer.train(dataloader) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default1]:[rank9]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default1]:[rank9]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default1]:[rank9]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default1]:[rank9]: output = model(**micro_batch) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank9]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank9]: return forward_call(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default1]:[rank9]: sharded_logits = self.model( | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank9]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank9]: return forward_call(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default1]:[rank9]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default1]:[rank9]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank9]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank9]: return forward_call(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default1]:[rank9]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default1]:[rank9]: pipeline_state.run_communication() | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default1]:[rank9]: recv_activation_tensor = recv_activation() | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default1]:[rank9]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default1]:[rank9]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default1]:[rank9]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default1]:[rank9]: dist.recv( | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default1]:[rank9]: return func(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default1]:[rank9]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default1]:[rank9]: torch.distributed.DistBackendError: [4] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '3:4', but store->get('3:4') got error: Connection reset by peer | |
| [default1]:[rank9]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default1]:[rank9]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f26e7f37897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default1]:[rank9]: frame #1: <unknown function> + 0x5b3a23e (0x7f2721a5423e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f2721a4ec87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f2721a4ef82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f2721a4ffd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2721a04371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2721a04371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2721a04371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2721a04371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f26e9211189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default1]:[rank9]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f26e9218610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default1]:[rank9]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f26e9237978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default1]:[rank9]: frame #12: <unknown function> + 0x5adc309 (0x7f27219f6309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #13: <unknown function> + 0x5ae6f10 (0x7f2721a00f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f2721a00fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #15: <unknown function> + 0x5124446 (0x7f272103e446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #16: <unknown function> + 0x1acf4b8 (0x7f271d9e94b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #17: <unknown function> + 0x5aee004 (0x7f2721a08004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #18: <unknown function> + 0x5af36b5 (0x7f2721a0d6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #19: <unknown function> + 0xd2631e (0x7f27345f731e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default1]:[rank9]: frame #20: <unknown function> + 0x47def4 (0x7f2733d4eef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default1]:[rank9]: frame #21: <unknown function> + 0x1445a6 (0x5586dcd695a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #22: _PyObject_MakeTpCall + 0x26b (0x5586dcd62a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #23: <unknown function> + 0x150866 (0x5586dcd75866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x5586dcd5e142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #25: _PyFunction_Vectorcall + 0x6c (0x5586dcd69a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #26: PyObject_Call + 0xbc (0x5586dcd75f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5586dcd5c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #28: _PyFunction_Vectorcall + 0x6c (0x5586dcd69a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5586dcd5a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #30: <unknown function> + 0x150582 (0x5586dcd75582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5586dcd5a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #32: <unknown function> + 0x150582 (0x5586dcd75582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5586dcd5a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #34: <unknown function> + 0x150582 (0x5586dcd75582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5586dcd5a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x5586dcd61f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #37: _PyObject_Call_Prepend + 0x69 (0x5586dcd73c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #38: <unknown function> + 0x211239 (0x5586dce36239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #39: _PyObject_MakeTpCall + 0x26b (0x5586dcd62a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5586dcd5e3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #41: _PyFunction_Vectorcall + 0x6c (0x5586dcd69a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x5586dcd59c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #43: _PyFunction_Vectorcall + 0x6c (0x5586dcd69a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5586dcd5a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #45: <unknown function> + 0x150582 (0x5586dcd75582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #46: PyObject_Call + 0xbc (0x5586dcd75f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5586dcd5c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #48: <unknown function> + 0x150582 (0x5586dcd75582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #49: PyObject_Call + 0xbc (0x5586dcd75f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5586dcd5c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #51: _PyFunction_Vectorcall + 0x6c (0x5586dcd69a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x5586dcd62007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #53: _PyObject_Call_Prepend + 0x69 (0x5586dcd73c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #54: <unknown function> + 0x211239 (0x5586dce36239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #55: PyObject_Call + 0x207 (0x5586dcd76067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5586dcd5c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #57: <unknown function> + 0x150582 (0x5586dcd75582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5586dcd5a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #59: <unknown function> + 0x150582 (0x5586dcd75582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #60: PyObject_Call + 0xbc (0x5586dcd75f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5586dcd5c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #62: <unknown function> + 0x150582 (0x5586dcd75582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #63: PyObject_Call + 0xbc (0x5586dcd75f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| W0702 19:35:06.747000 140686904493888 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2516372 closing signal SIGTERM | |
| W0702 19:35:06.747000 140686904493888 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2516373 closing signal SIGTERM | |
| W0702 19:35:06.748000 140686904493888 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2516374 closing signal SIGTERM | |
| W0702 19:35:06.749000 140686904493888 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2516375 closing signal SIGTERM | |
| W0702 19:35:06.749000 140686904493888 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2516376 closing signal SIGTERM | |
| W0702 19:35:06.749000 140686904493888 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2516377 closing signal SIGTERM | |
| E0702 19:35:08.373000 140686904493888 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 2516370) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 | |
| Traceback (most recent call last): | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module> | |
| sys.exit(main()) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper | |
| return f(*args, **kwargs) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main | |
| run(args) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run | |
| elastic_launch( | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__ | |
| return launch_agent(self._config, self._entrypoint, list(args)) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent | |
| raise ChildFailedError( | |
| torch.distributed.elastic.multiprocessing.errors.ChildFailedError: | |
| ============================================================ | |
| /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED | |
| ------------------------------------------------------------ | |
| Failures: | |
| [1]: | |
| time : 2024-07-02_19:35:06 | |
| host : ip-26-0-169-139.ec2.internal | |
| rank : 1 (local_rank: 1) | |
| exitcode : 1 (pid: 2516371) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| ------------------------------------------------------------ | |
| Root Cause (first observed failure): | |
| [0]: | |
| time : 2024-07-02_19:35:06 | |
| host : ip-26-0-169-139.ec2.internal | |
| rank : 0 (local_rank: 0) | |
| exitcode : 1 (pid: 2516370) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| ============================================================ | |
| srun: error: ip-26-0-169-139: task 0: Exited with exit code 1 | |
| W0702 19:35:10.747000 140088521451264 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-172-73.ec2.internal_728571_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError. | |
| E0702 19:35:11.876000 140094188271424 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 728641) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 | |
| W0702 19:35:11.882000 140094188271424 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-172-73.ec2.internal_728571_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError. | |
| W0702 19:35:11.911000 140094188271424 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-172-73.ec2.internal_728571_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError. | |
| W0702 19:35:11.939000 140094188271424 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-172-73.ec2.internal_728571_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError. | |
| Traceback (most recent call last): | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module> | |
| sys.exit(main()) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper | |
| return f(*args, **kwargs) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main | |
| run(args) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run | |
| elastic_launch( | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__ | |
| return launch_agent(self._config, self._entrypoint, list(args)) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent | |
| raise ChildFailedError( | |
| torch.distributed.elastic.multiprocessing.errors.ChildFailedError: | |
| ============================================================ | |
| /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED | |
| ------------------------------------------------------------ | |
| Failures: | |
| [1]: | |
| time : 2024-07-02_19:35:11 | |
| host : ip-26-0-172-73.ec2.internal | |
| rank : 9 (local_rank: 1) | |
| exitcode : 1 (pid: 728642) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| [2]: | |
| time : 2024-07-02_19:35:11 | |
| host : ip-26-0-172-73.ec2.internal | |
| rank : 10 (local_rank: 2) | |
| exitcode : 1 (pid: 728643) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| [3]: | |
| time : 2024-07-02_19:35:11 | |
| host : ip-26-0-172-73.ec2.internal | |
| rank : 11 (local_rank: 3) | |
| exitcode : 1 (pid: 728644) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| [4]: | |
| time : 2024-07-02_19:35:11 | |
| host : ip-26-0-172-73.ec2.internal | |
| rank : 12 (local_rank: 4) | |
| exitcode : 1 (pid: 728645) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| [5]: | |
| time : 2024-07-02_19:35:11 | |
| host : ip-26-0-172-73.ec2.internal | |
| rank : 13 (local_rank: 5) | |
| exitcode : 1 (pid: 728646) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| [6]: | |
| time : 2024-07-02_19:35:11 | |
| host : ip-26-0-172-73.ec2.internal | |
| rank : 14 (local_rank: 6) | |
| exitcode : 1 (pid: 728647) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| [7]: | |
| time : 2024-07-02_19:35:11 | |
| host : ip-26-0-172-73.ec2.internal | |
| rank : 15 (local_rank: 7) | |
| exitcode : 1 (pid: 728648) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| ------------------------------------------------------------ | |
| Root Cause (first observed failure): | |
| [0]: | |
| time : 2024-07-02_19:35:11 | |
| host : ip-26-0-172-73.ec2.internal | |
| rank : 8 (local_rank: 0) | |
| exitcode : 1 (pid: 728641) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| ============================================================ | |
| srun: error: ip-26-0-172-73: task 1: Exited with exit code 1 | |
| Consider using `hf_transfer` for faster uploads. This solution comes with some limitations. See https://huggingface.co/docs/huggingface_hub/hf_transfer for more details. | |