| ======================== | |
| START TIME: Tue Jul 2 16:19:41 UTC 2024 | |
| python3 version = Python 3.10.14 | |
| ======================== | |
| The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well. | |
| Token is valid (permission: write). | |
| Your token has been saved to /admin/home/ferdinand_mom/.cache/huggingface/token | |
| Login successful | |
| Already on 'bench_cluster' | |
| M examples/config_tiny_llama.py | |
| M examples/config_tiny_llama.yaml | |
| M examples/train_tiny_llama.sh | |
| M src/nanotron/models/llama.py | |
| M src/nanotron/trainer.py | |
| Your branch is up to date with 'origin/bench_cluster'. | |
| Job status: RUNNING | |
| W0702 16:19:43.710000 140449488344896 torch/distributed/run.py:757] | |
| W0702 16:19:43.710000 140449488344896 torch/distributed/run.py:757] ***************************************** | |
| W0702 16:19:43.710000 140449488344896 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. | |
| W0702 16:19:43.710000 140449488344896 torch/distributed/run.py:757] ***************************************** | |
| W0702 16:19:43.804000 140680251041600 torch/distributed/run.py:757] | |
| W0702 16:19:43.804000 140680251041600 torch/distributed/run.py:757] ***************************************** | |
| W0702 16:19:43.804000 140680251041600 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. | |
| W0702 16:19:43.804000 140680251041600 torch/distributed/run.py:757] ***************************************** | |
| [default0]:07/02/2024 16:20:02 [WARNING|DP=0|PP=0|TP=0|ip-26-0-167-177]: [Vocab Size Padding] Padded vocab (size: 50257) with 7 dummy tokens (new size: 50264) | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: Config: | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: Config(general=GeneralArgs(project='bench_cluster', | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: run='%date_%jobid', | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: seed=42, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: step=None, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: consumed_train_samples=None, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: benchmark_csv_path=None, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: ignore_sanity_checks=True), | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: parallelism=ParallelismArgs(dp=1, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: pp=2, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: tp=8, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: pp_engine=<nanotron.parallel.pipeline_parallel.engine.OneForwardOneBackwardPipelineEngine object at 0x7f655575c910>, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: tp_mode=<TensorParallelLinearMode.REDUCE_SCATTER: 2>, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: tp_linear_async_communication=False, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: expert_parallel_size=1), | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: model=ModelArgs(model_config=LlamaConfig(bos_token_id=1, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: eos_token_id=2, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: hidden_act='silu', | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: hidden_size=2048, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: initializer_range=0.02, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: intermediate_size=4096, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: is_llama_config=True, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: max_position_embeddings=4096, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: num_attention_heads=32, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: num_hidden_layers=24, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: num_key_value_heads=32, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: pad_token_id=None, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: pretraining_tp=1, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: rms_norm_eps=1e-05, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: rope_scaling=None, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: rope_theta=10000.0, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: tie_word_embeddings=True, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: use_cache=True, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: vocab_size=50264), | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: init_method=RandomInit(std=0.025), | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: dtype=torch.bfloat16, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: make_vocab_size_divisible_by=1, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: ddp_bucket_cap_mb=25), | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: tokenizer=TokenizerArgs(tokenizer_name_or_path='openai-community/gpt2', | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: tokenizer_revision=None, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: tokenizer_max_length=None), | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: checkpoints=CheckpointsArgs(checkpoints_path=Path('/dev/null'), | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: checkpoint_interval=100000, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: save_initial_state=False, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: resume_checkpoint_path=None, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: checkpoints_path_is_shared_file_system=False), | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: logging=LoggingArgs(log_level='info', | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: log_level_replica='info', | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: iteration_step_info_interval=1), | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: tokens=TokensArgs(sequence_length=4096, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: train_steps=20, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: micro_batch_size=128, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: batch_accumulation_per_replica=8, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: val_check_interval=-1, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: limit_val_batches=0, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: limit_test_batches=0), | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: optimizer=OptimizerArgs(optimizer_factory=AdamWOptimizerArgs(adam_eps=1e-08, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: adam_beta1=0.9, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: adam_beta2=0.95, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: torch_adam_is_fused=True, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: name='adamW'), | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: zero_stage=1, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: weight_decay=0.01, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: clip_grad=1.0, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: accumulate_grad_in_fp32=True, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: learning_rate_scheduler=LRSchedulerArgs(learning_rate=0.0001, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: lr_warmup_steps=1, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: lr_warmup_style='linear', | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: lr_decay_style='linear', | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: lr_decay_steps=19, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: lr_decay_starting_step=None, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: min_decay_lr=1e-05)), | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: data_stages=[DatasetStageArgs(name='Training Stage', | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: start_training_step=1, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: data=DataArgs(dataset=PretrainDatasetsArgs(hf_dataset_or_datasets='roneneldan/TinyStories', | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: hf_dataset_splits='train', | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: hf_dataset_config_name=None, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: dataset_processing_num_proc_per_process=64, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: dataset_overwrite_cache=False, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: text_column_name='text'), | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: seed=42, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: num_loading_workers=32))], | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: profiler=ProfilerArgs(profiler_export_path=Path('/fsx/ferdinandmom/ferdinand-hf/bench_cluster/results/llama-1B/16_GPUS/dp-1_tp-8_pp-2_mbz-128')), | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: lighteval=None) | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: Model Config: | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: LlamaConfig(bos_token_id=1, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: eos_token_id=2, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: hidden_act='silu', | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: hidden_size=2048, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: initializer_range=0.02, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: intermediate_size=4096, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: is_llama_config=True, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: max_position_embeddings=4096, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: num_attention_heads=32, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: num_hidden_layers=24, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: num_key_value_heads=32, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: pad_token_id=None, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: pretraining_tp=1, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: rms_norm_eps=1e-05, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: rope_scaling=None, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: rope_theta=10000.0, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: tie_word_embeddings=True, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: use_cache=True, | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: vocab_size=50264) | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: Building model.. | |
| [default0]:07/02/2024 16:20:02 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: Setting PP block ranks... | |
| [default0]:07/02/2024 16:20:20 [INFO|DP=0|PP=1|TP=0|ip-26-0-170-31]: Local number of parameters: 65.3M (124.62MiB) | |
| [default0]:07/02/2024 16:20:20 [INFO|DP=0|PP=1|TP=0|ip-26-0-170-31]: [After model building] Memory usage: 135.64MiB. Peak allocated: 137.67MiB Peak reserved: 150.00MiB | |
| [default0]:07/02/2024 16:20:20 [INFO|DP=0|PP=1|TP=0|ip-26-0-170-31]: No checkpoint path provided. | |
| [default4]:07/02/2024 16:20:20 [INFO|DP=0|PP=1|TP=4|ip-26-0-170-31]: Local number of parameters: 65.3M (124.62MiB) | |
| [default4]:07/02/2024 16:20:20 [INFO|DP=0|PP=1|TP=4|ip-26-0-170-31]: [After model building] Memory usage: 135.64MiB. Peak allocated: 137.67MiB Peak reserved: 150.00MiB | |
| [default4]:07/02/2024 16:20:20 [INFO|DP=0|PP=1|TP=4|ip-26-0-170-31]: No checkpoint path provided. | |
| [default3]:07/02/2024 16:20:20 [INFO|DP=0|PP=1|TP=3|ip-26-0-170-31]: Local number of parameters: 65.3M (124.62MiB) | |
| [default3]:07/02/2024 16:20:20 [INFO|DP=0|PP=1|TP=3|ip-26-0-170-31]: [After model building] Memory usage: 135.64MiB. Peak allocated: 137.67MiB Peak reserved: 150.00MiB | |
| [default3]:07/02/2024 16:20:20 [INFO|DP=0|PP=1|TP=3|ip-26-0-170-31]: No checkpoint path provided. | |
| [default1]:07/02/2024 16:20:20 [INFO|DP=0|PP=1|TP=1|ip-26-0-170-31]: Local number of parameters: 65.3M (124.62MiB) | |
| [default1]:07/02/2024 16:20:20 [INFO|DP=0|PP=1|TP=1|ip-26-0-170-31]: [After model building] Memory usage: 135.64MiB. Peak allocated: 137.67MiB Peak reserved: 150.00MiB | |
| [default6]:07/02/2024 16:20:20 [INFO|DP=0|PP=1|TP=6|ip-26-0-170-31]: Local number of parameters: 65.3M (124.62MiB) | |
| [default6]:07/02/2024 16:20:20 [INFO|DP=0|PP=1|TP=6|ip-26-0-170-31]: [After model building] Memory usage: 135.64MiB. Peak allocated: 137.67MiB Peak reserved: 150.00MiB | |
| [default5]:07/02/2024 16:20:20 [INFO|DP=0|PP=1|TP=5|ip-26-0-170-31]: Local number of parameters: 65.3M (124.62MiB) | |
| [default5]:07/02/2024 16:20:20 [INFO|DP=0|PP=1|TP=5|ip-26-0-170-31]: [After model building] Memory usage: 135.64MiB. Peak allocated: 137.67MiB Peak reserved: 150.00MiB | |
| [default5]:07/02/2024 16:20:20 [INFO|DP=0|PP=1|TP=5|ip-26-0-170-31]: No checkpoint path provided. | |
| [default6]:07/02/2024 16:20:20 [INFO|DP=0|PP=1|TP=6|ip-26-0-170-31]: No checkpoint path provided. | |
| [default1]:07/02/2024 16:20:20 [INFO|DP=0|PP=1|TP=1|ip-26-0-170-31]: No checkpoint path provided. | |
| [default2]:07/02/2024 16:20:20 [INFO|DP=0|PP=1|TP=2|ip-26-0-170-31]: Local number of parameters: 65.3M (124.62MiB) | |
| [default0]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: Total number of parameters: 1.21G (2314.22MiB) | |
| [default0]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: Local number of parameters: 86.3M (164.65MiB) | |
| [default0]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: [After model building] Memory usage: 179.67MiB. Peak allocated: 181.70MiB Peak reserved: 198.00MiB | |
| [default0]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: No checkpoint path provided. | |
| [default0]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: Parametrizing model parameters using StandardParametrizator | |
| [default2]:07/02/2024 16:20:20 [INFO|DP=0|PP=1|TP=2|ip-26-0-170-31]: [After model building] Memory usage: 135.64MiB. Peak allocated: 137.67MiB Peak reserved: 150.00MiB | |
| [default2]:07/02/2024 16:20:20 [INFO|DP=0|PP=1|TP=2|ip-26-0-170-31]: No checkpoint path provided. | |
| [default1]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=1|ip-26-0-167-177]: Local number of parameters: 86.3M (164.65MiB) | |
| [default1]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=1|ip-26-0-167-177]: [After model building] Memory usage: 179.67MiB. Peak allocated: 181.70MiB Peak reserved: 198.00MiB | |
| [default1]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=1|ip-26-0-167-177]: No checkpoint path provided. | |
| [default2]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=2|ip-26-0-167-177]: Local number of parameters: 86.3M (164.65MiB) | |
| [default2]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=2|ip-26-0-167-177]: [After model building] Memory usage: 179.67MiB. Peak allocated: 181.70MiB Peak reserved: 198.00MiB | |
| [default2]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=2|ip-26-0-167-177]: No checkpoint path provided. | |
| [default3]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=3|ip-26-0-167-177]: Local number of parameters: 86.3M (164.65MiB) | |
| [default7]:07/02/2024 16:20:20 [INFO|DP=0|PP=1|TP=7|ip-26-0-170-31]: Local number of parameters: 65.3M (124.62MiB) | |
| [default7]:07/02/2024 16:20:20 [INFO|DP=0|PP=1|TP=7|ip-26-0-170-31]: [After model building] Memory usage: 135.64MiB. Peak allocated: 137.67MiB Peak reserved: 150.00MiB | |
| [default3]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=3|ip-26-0-167-177]: [After model building] Memory usage: 179.67MiB. Peak allocated: 181.70MiB Peak reserved: 198.00MiB | |
| [default4]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=4|ip-26-0-167-177]: Local number of parameters: 86.3M (164.65MiB) | |
| [default4]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=4|ip-26-0-167-177]: [After model building] Memory usage: 179.67MiB. Peak allocated: 181.70MiB Peak reserved: 198.00MiB | |
| [default5]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=5|ip-26-0-167-177]: Local number of parameters: 86.3M (164.65MiB) | |
| [default7]:07/02/2024 16:20:20 [INFO|DP=0|PP=1|TP=7|ip-26-0-170-31]: No checkpoint path provided. | |
| [default5]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=5|ip-26-0-167-177]: [After model building] Memory usage: 179.67MiB. Peak allocated: 181.70MiB Peak reserved: 198.00MiB | |
| [default5]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=5|ip-26-0-167-177]: No checkpoint path provided. | |
| [default6]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=6|ip-26-0-167-177]: Local number of parameters: 86.3M (164.65MiB) | |
| [default6]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=6|ip-26-0-167-177]: [After model building] Memory usage: 179.67MiB. Peak allocated: 181.70MiB Peak reserved: 198.00MiB | |
| [default3]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=3|ip-26-0-167-177]: No checkpoint path provided. | |
| [default4]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=4|ip-26-0-167-177]: No checkpoint path provided. | |
| [default7]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=7|ip-26-0-167-177]: Local number of parameters: 86.3M (164.65MiB) | |
| [default7]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=7|ip-26-0-167-177]: [After model building] Memory usage: 179.67MiB. Peak allocated: 181.70MiB Peak reserved: 198.00MiB | |
| [default6]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=6|ip-26-0-167-177]: No checkpoint path provided. | |
| [default7]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=7|ip-26-0-167-177]: No checkpoint path provided. | |
| [default0]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: [Optimizer Building] Using LearningRateForSP as learning rate | |
| [default0]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: [ZeRO sharding] Size of optimizer params per rank: | |
| [default0]:07/02/2024 16:20:20 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: [ZeRO sharding] DP Rank 0 has 86.3M out of 86.3M (100.00%) params' optimizer states | |
| [default0]:07/02/2024 16:20:22 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: [Training Plan] Stage Training Stage has 19 remaining training steps and has consumed 0 samples | |
| [default0]:07/02/2024 16:20:22 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: Using `datasets` library | |
| [default0]:07/02/2024 16:20:22 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: Loading tokenizer from openai-community/gpt2 and transformers/hf_hub versions ('4.41.2', '0.23.4') | |
| [default0]:07/02/2024 16:20:22 [WARNING|DP=0|PP=0|TP=0|ip-26-0-167-177]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default0]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default0]:07/02/2024 16:20:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: [Training Plan] There are 1 training stages | |
| [default0]:07/02/2024 16:20:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: [Stage Training Stage] start from step 1 | |
| [default0]:07/02/2024 16:20:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: | |
| [default0]:07/02/2024 16:20:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: [Start training] datetime: 2024-07-02 16:20:23.097559 | mbs: 128 | grad_accum: 8 | global_batch_size: 1024 | sequence_length: 4096 | train_steps: 20 | start_iteration_step: 0 | consumed_train_samples: 0 | |
| [default0]:07/02/2024 16:20:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: Resuming training from stage Training Stage, it has trained for 0 samples and has 19 remaining train steps | |
| [default0]:07/02/2024 16:20:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-167-177]: Memory usage: 839.67MiB. Peak allocated 839.67MiB. Peak reserved: 858.00MiB | |
| [default0]:07/02/2024 16:20:23 [WARNING|DP=0|PP=1|TP=0|ip-26-0-170-31]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default5]:07/02/2024 16:20:23 [WARNING|DP=0|PP=1|TP=5|ip-26-0-170-31]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default6]:07/02/2024 16:20:23 [WARNING|DP=0|PP=1|TP=6|ip-26-0-170-31]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default2]:07/02/2024 16:20:23 [WARNING|DP=0|PP=1|TP=2|ip-26-0-170-31]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default2]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:07/02/2024 16:20:23 [WARNING|DP=0|PP=1|TP=7|ip-26-0-170-31]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default0]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default1]:07/02/2024 16:20:23 [WARNING|DP=0|PP=0|TP=1|ip-26-0-167-177]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default2]:07/02/2024 16:20:23 [WARNING|DP=0|PP=0|TP=2|ip-26-0-167-177]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default6]:07/02/2024 16:20:23 [WARNING|DP=0|PP=0|TP=6|ip-26-0-167-177]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default3]:07/02/2024 16:20:23 [WARNING|DP=0|PP=0|TP=3|ip-26-0-167-177]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default5]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default5]:07/02/2024 16:20:23 [WARNING|DP=0|PP=0|TP=5|ip-26-0-167-177]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:07/02/2024 16:20:23 [WARNING|DP=0|PP=0|TP=7|ip-26-0-167-177]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default6]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default1]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default6]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default2]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default3]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default5]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default3]:07/02/2024 16:20:23 [WARNING|DP=0|PP=1|TP=3|ip-26-0-170-31]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default3]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default1]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default1]:07/02/2024 16:20:23 [WARNING|DP=0|PP=1|TP=1|ip-26-0-170-31]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default4]:07/02/2024 16:20:23 [WARNING|DP=0|PP=0|TP=4|ip-26-0-167-177]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default4]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default4]:07/02/2024 16:20:24 [WARNING|DP=0|PP=1|TP=4|ip-26-0-170-31]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default4]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default1]:[rank1]: OSError: [Errno 122] Disk quota exceeded | |
| [default1]: | |
| [default1]:[rank1]: During handling of the above exception, another exception occurred: | |
| [default1]: | |
| [default1]:[rank1]: Traceback (most recent call last): | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default1]:[rank1]: trainer.train(dataloader) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default1]:[rank1]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default1]:[rank1]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default1]:[rank1]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default1]:[rank1]: output = model(**micro_batch) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default1]:[rank1]: sharded_logits = self.model( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default1]:[rank1]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default1]:[rank1]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward | |
| [default1]:[rank1]: output = self.pp_block(**new_kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 629, in forward | |
| [default1]:[rank1]: hidden_states = self.input_layernorm(hidden_states) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/nn/layer_norm.py", line 42, in forward | |
| [default1]:[rank1]: return layer_norm_fn( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py", line 875, in layer_norm_fn | |
| [default1]:[rank1]: return LayerNormFn.apply( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/function.py", line 598, in apply | |
| [default1]:[rank1]: return super().apply(*args, **kwargs) # type: ignore[misc] | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py", line 748, in forward | |
| [default1]:[rank1]: y, y1, mean, rstd, residual_out, seeds, dropout_mask, dropout_mask1 = _layer_norm_fwd( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py", line 335, in _layer_norm_fwd | |
| [default1]:[rank1]: _layer_norm_fwd_1pass_kernel[(M,)]( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/jit.py", line 167, in <lambda> | |
| [default1]:[rank1]: return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 143, in run | |
| [default1]:[rank1]: timings = {config: self._bench(*args, config=config, **kwargs) for config in pruned_configs} | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 143, in <dictcomp> | |
| [default1]:[rank1]: timings = {config: self._bench(*args, config=config, **kwargs) for config in pruned_configs} | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 122, in _bench | |
| [default1]:[rank1]: return do_bench(kernel_call, warmup=self.warmup, rep=self.rep, quantiles=(0.5, 0.2, 0.8)) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/testing.py", line 102, in do_bench | |
| [default1]:[rank1]: fn() | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 110, in kernel_call | |
| [default1]:[rank1]: self.fn.run( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 305, in run | |
| [default1]:[rank1]: return self.fn.run(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 305, in run | |
| [default1]:[rank1]: return self.fn.run(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 305, in run | |
| [default1]:[rank1]: return self.fn.run(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/jit.py", line 416, in run | |
| [default1]:[rank1]: self.cache[device][key] = compile( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/compiler/compiler.py", line 194, in compile | |
| [default1]:[rank1]: metadata_group[f"{src.name}.{ext}"] = fn_cache_manager.put(next_module, f"{src.name}.{ext}") | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/cache.py", line 123, in put | |
| [default1]:[rank1]: with open(temp_path, mode) as f: | |
| [default1]:[rank1]: OSError: [Errno 122] Disk quota exceeded | |
| [default4]:[rank4]: OSError: [Errno 122] Disk quota exceeded | |
| [default4]: | |
| [default4]:[rank4]: During handling of the above exception, another exception occurred: | |
| [default4]: | |
| [default4]:[rank4]: Traceback (most recent call last): | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default4]:[rank4]: trainer.train(dataloader) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default4]:[rank4]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default4]:[rank4]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default4]:[rank4]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default4]:[rank4]: output = model(**micro_batch) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank4]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank4]: return forward_call(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default4]:[rank4]: sharded_logits = self.model( | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank4]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank4]: return forward_call(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default4]:[rank4]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default4]:[rank4]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank4]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank4]: return forward_call(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward | |
| [default4]:[rank4]: output = self.pp_block(**new_kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank4]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank4]: return forward_call(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 629, in forward | |
| [default4]:[rank4]: hidden_states = self.input_layernorm(hidden_states) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank4]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank4]: return forward_call(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/nn/layer_norm.py", line 42, in forward | |
| [default4]:[rank4]: return layer_norm_fn( | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py", line 875, in layer_norm_fn | |
| [default4]:[rank4]: return LayerNormFn.apply( | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/function.py", line 598, in apply | |
| [default4]:[rank4]: return super().apply(*args, **kwargs) # type: ignore[misc] | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py", line 748, in forward | |
| [default4]:[rank4]: y, y1, mean, rstd, residual_out, seeds, dropout_mask, dropout_mask1 = _layer_norm_fwd( | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py", line 335, in _layer_norm_fwd | |
| [default4]:[rank4]: _layer_norm_fwd_1pass_kernel[(M,)]( | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/jit.py", line 167, in <lambda> | |
| [default4]:[rank4]: return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 143, in run | |
| [default4]:[rank4]: timings = {config: self._bench(*args, config=config, **kwargs) for config in pruned_configs} | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 143, in <dictcomp> | |
| [default4]:[rank4]: timings = {config: self._bench(*args, config=config, **kwargs) for config in pruned_configs} | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 122, in _bench | |
| [default4]:[rank4]: return do_bench(kernel_call, warmup=self.warmup, rep=self.rep, quantiles=(0.5, 0.2, 0.8)) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/testing.py", line 102, in do_bench | |
| [default4]:[rank4]: fn() | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 110, in kernel_call | |
| [default4]:[rank4]: self.fn.run( | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 305, in run | |
| [default4]:[rank4]: return self.fn.run(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 305, in run | |
| [default4]:[rank4]: return self.fn.run(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 305, in run | |
| [default4]:[rank4]: return self.fn.run(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/jit.py", line 416, in run | |
| [default4]:[rank4]: self.cache[device][key] = compile( | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/compiler/compiler.py", line 194, in compile | |
| [default4]:[rank4]: metadata_group[f"{src.name}.{ext}"] = fn_cache_manager.put(next_module, f"{src.name}.{ext}") | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/cache.py", line 123, in put | |
| [default4]:[rank4]: with open(temp_path, mode) as f: | |
| [default4]:[rank4]: OSError: [Errno 122] Disk quota exceeded | |
| W0702 16:20:35.075000 140449488344896 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 528027 closing signal SIGTERM | |
| W0702 16:20:35.080000 140449488344896 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 528029 closing signal SIGTERM | |
| W0702 16:20:35.083000 140449488344896 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 528030 closing signal SIGTERM | |
| W0702 16:20:35.116000 140449488344896 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 528031 closing signal SIGTERM | |
| W0702 16:20:35.117000 140449488344896 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 528032 closing signal SIGTERM | |
| W0702 16:20:35.122000 140449488344896 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 528033 closing signal SIGTERM | |
| W0702 16:20:35.125000 140449488344896 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 528034 closing signal SIGTERM | |
| [default7]:[rank15]: Traceback (most recent call last): | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default7]:[rank15]: trainer.train(dataloader) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default7]:[rank15]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default7]:[rank15]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default7]:[rank15]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default7]:[rank15]: output = model(**micro_batch) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank15]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank15]: return forward_call(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default7]:[rank15]: sharded_logits = self.model( | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank15]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank15]: return forward_call(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default7]:[rank15]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default7]:[rank15]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank15]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank15]: return forward_call(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default7]:[rank15]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default7]:[rank15]: pipeline_state.run_communication() | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default7]:[rank15]: recv_activation_tensor = recv_activation() | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default7]:[rank15]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default7]:[rank15]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default7]:[rank15]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default7]:[rank15]: dist.recv( | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default7]:[rank15]: return func(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default7]:[rank15]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default7]:[rank15]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer | |
| [default7]:[rank15]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default7]:[rank15]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fc4556a6897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default7]:[rank15]: frame #1: <unknown function> + 0x5b3a23e (0x7fc48f1c323e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fc48f1bdc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fc48f1bdf82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fc48f1befd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc48f173371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc48f173371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc48f173371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc48f173371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fc456980189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:[rank15]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fc456987610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:[rank15]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fc4569a6978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:[rank15]: frame #12: <unknown function> + 0x5adc309 (0x7fc48f165309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #13: <unknown function> + 0x5ae6f10 (0x7fc48f16ff10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fc48f16ffa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #15: <unknown function> + 0x5124446 (0x7fc48e7ad446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #16: <unknown function> + 0x1acf4b8 (0x7fc48b1584b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #17: <unknown function> + 0x5aee004 (0x7fc48f177004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #18: <unknown function> + 0x5af36b5 (0x7fc48f17c6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #19: <unknown function> + 0xd2631e (0x7fc4a1d6631e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default7]:[rank15]: frame #20: <unknown function> + 0x47def4 (0x7fc4a14bdef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default7]:[rank15]: frame #21: <unknown function> + 0x1445a6 (0x56064daf65a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #22: _PyObject_MakeTpCall + 0x26b (0x56064daefa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #23: <unknown function> + 0x150866 (0x56064db02866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x56064daeb142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #25: _PyFunction_Vectorcall + 0x6c (0x56064daf6a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #26: PyObject_Call + 0xbc (0x56064db02f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x56064dae92b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #28: _PyFunction_Vectorcall + 0x6c (0x56064daf6a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x56064dae78fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #30: <unknown function> + 0x150582 (0x56064db02582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x56064dae78fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #32: <unknown function> + 0x150582 (0x56064db02582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x56064dae78fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #34: <unknown function> + 0x150582 (0x56064db02582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x56064dae78fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x56064daeef50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #37: _PyObject_Call_Prepend + 0x69 (0x56064db00c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #38: <unknown function> + 0x211239 (0x56064dbc3239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #39: _PyObject_MakeTpCall + 0x26b (0x56064daefa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x56064daeb3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #41: _PyFunction_Vectorcall + 0x6c (0x56064daf6a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x56064dae6c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #43: _PyFunction_Vectorcall + 0x6c (0x56064daf6a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x56064dae78fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #45: <unknown function> + 0x150582 (0x56064db02582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #46: PyObject_Call + 0xbc (0x56064db02f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x56064dae92b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #48: <unknown function> + 0x150582 (0x56064db02582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #49: PyObject_Call + 0xbc (0x56064db02f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x56064dae92b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #51: _PyFunction_Vectorcall + 0x6c (0x56064daf6a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x56064daef007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #53: _PyObject_Call_Prepend + 0x69 (0x56064db00c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #54: <unknown function> + 0x211239 (0x56064dbc3239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #55: PyObject_Call + 0x207 (0x56064db03067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x56064dae92b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #57: <unknown function> + 0x150582 (0x56064db02582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x56064dae78fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #59: <unknown function> + 0x150582 (0x56064db02582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #60: PyObject_Call + 0xbc (0x56064db02f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x56064dae92b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #62: <unknown function> + 0x150582 (0x56064db02582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #63: PyObject_Call + 0xbc (0x56064db02f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default1]:[rank9]: Traceback (most recent call last): | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default1]:[rank9]: trainer.train(dataloader) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default1]:[rank9]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default1]:[rank9]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default1]:[rank9]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default1]:[rank9]: output = model(**micro_batch) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank9]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank9]: return forward_call(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default1]:[rank9]: sharded_logits = self.model( | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank9]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank9]: return forward_call(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default1]:[rank9]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default1]:[rank9]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank9]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank9]: return forward_call(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default1]:[rank9]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default1]:[rank9]: pipeline_state.run_communication() | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default1]:[rank9]: recv_activation_tensor = recv_activation() | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default1]:[rank9]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default1]:[rank9]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default1]:[rank9]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default1]:[rank9]: dist.recv( | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default1]:[rank9]: return func(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default1]:[rank9]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default1]:[rank9]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer | |
| [default1]:[rank9]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default1]:[rank9]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f954d38e897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default1]:[rank9]: frame #1: <unknown function> + 0x5b3a23e (0x7f9586eab23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f9586ea5c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f9586ea5f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f9586ea6fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9586e5b371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9586e5b371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9586e5b371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9586e5b371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f954e668189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default1]:[rank9]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f954e66f610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default1]:[rank9]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f954e68e978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default1]:[rank9]: frame #12: <unknown function> + 0x5adc309 (0x7f9586e4d309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #13: <unknown function> + 0x5ae6f10 (0x7f9586e57f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f9586e57fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #15: <unknown function> + 0x5124446 (0x7f9586495446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #16: <unknown function> + 0x1acf4b8 (0x7f9582e404b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #17: <unknown function> + 0x5aee004 (0x7f9586e5f004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #18: <unknown function> + 0x5af36b5 (0x7f9586e646b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #19: <unknown function> + 0xd2631e (0x7f9599a4e31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default1]:[rank9]: frame #20: <unknown function> + 0x47def4 (0x7f95991a5ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default1]:[rank9]: frame #21: <unknown function> + 0x1445a6 (0x55b134f755a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55b134f6ea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #23: <unknown function> + 0x150866 (0x55b134f81866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55b134f6a142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55b134f75a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #26: PyObject_Call + 0xbc (0x55b134f81f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55b134f682b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55b134f75a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55b134f668fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #30: <unknown function> + 0x150582 (0x55b134f81582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55b134f668fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #32: <unknown function> + 0x150582 (0x55b134f81582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55b134f668fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #34: <unknown function> + 0x150582 (0x55b134f81582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55b134f668fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55b134f6df50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55b134f7fc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #38: <unknown function> + 0x211239 (0x55b135042239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55b134f6ea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55b134f6a3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55b134f75a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55b134f65c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55b134f75a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55b134f668fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #45: <unknown function> + 0x150582 (0x55b134f81582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #46: PyObject_Call + 0xbc (0x55b134f81f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55b134f682b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #48: <unknown function> + 0x150582 (0x55b134f81582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #49: PyObject_Call + 0xbc (0x55b134f81f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55b134f682b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55b134f75a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55b134f6e007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55b134f7fc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #54: <unknown function> + 0x211239 (0x55b135042239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #55: PyObject_Call + 0x207 (0x55b134f82067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55b134f682b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #57: <unknown function> + 0x150582 (0x55b134f81582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55b134f668fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #59: <unknown function> + 0x150582 (0x55b134f81582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #60: PyObject_Call + 0xbc (0x55b134f81f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55b134f682b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #62: <unknown function> + 0x150582 (0x55b134f81582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #63: PyObject_Call + 0xbc (0x55b134f81f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default2]:[rank10]: Traceback (most recent call last): | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default2]:[rank10]: trainer.train(dataloader) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default2]:[rank10]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default2]:[rank10]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default2]:[rank10]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default2]:[rank10]: output = model(**micro_batch) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank10]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank10]: return forward_call(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default2]:[rank10]: sharded_logits = self.model( | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank10]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank11]: Traceback (most recent call last): | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default3]:[rank11]: trainer.train(dataloader) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default3]:[rank11]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default3]:[rank11]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default3]:[rank11]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default3]:[rank11]: output = model(**micro_batch) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank11]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank11]: return forward_call(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default3]:[rank11]: sharded_logits = self.model( | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank11]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank11]: return forward_call(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank10]: return forward_call(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default2]:[rank10]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default2]:[rank10]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank14]: Traceback (most recent call last): | |
| [default2]:[rank10]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank10]: return forward_call(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default2]:[rank10]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default2]:[rank10]: pipeline_state.run_communication() | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default2]:[rank10]: recv_activation_tensor = recv_activation() | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default5]:[rank13]: Traceback (most recent call last): | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default3]:[rank11]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default2]:[rank10]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default5]:[rank13]: trainer.train(dataloader) | |
| [default3]:[rank11]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default5]:[rank13]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default5]:[rank13]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default3]:[rank11]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank13]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default5]:[rank13]: output = model(**micro_batch) | |
| [default3]:[rank11]: return forward_call(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default0]:[rank8]: Traceback (most recent call last): | |
| [default5]:[rank13]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank14]: trainer.train(dataloader) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank13]: return forward_call(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default5]:[rank13]: sharded_logits = self.model( | |
| [default0]:[rank8]: trainer.train(dataloader) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank13]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank13]: return forward_call(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default6]:[rank14]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default6]:[rank14]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default6]:[rank14]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default6]:[rank14]: output = model(**micro_batch) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank14]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank14]: return forward_call(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default6]:[rank14]: sharded_logits = self.model( | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank14]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank14]: return forward_call(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default6]:[rank14]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default6]:[rank14]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank14]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank14]: return forward_call(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default6]:[rank14]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default6]:[rank14]: pipeline_state.run_communication() | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default6]:[rank14]: recv_activation_tensor = recv_activation() | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default6]:[rank14]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default6]:[rank14]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default6]:[rank14]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default6]:[rank14]: dist.recv( | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default6]:[rank14]: return func(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default6]:[rank14]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default6]:[rank14]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer | |
| [default6]:[rank14]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default6]:[rank14]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7faced000897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default6]:[rank14]: frame #1: <unknown function> + 0x5b3a23e (0x7fad26b1d23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fad26b17c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fad26b17f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fad26b18fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fad26acd371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fad26acd371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fad26acd371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fad26acd371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7facee2da189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:[rank14]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7facee2e1610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:[rank14]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7facee300978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:[rank14]: frame #12: <unknown function> + 0x5adc309 (0x7fad26abf309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #13: <unknown function> + 0x5ae6f10 (0x7fad26ac9f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fad26ac9fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #15: <unknown function> + 0x5124446 (0x7fad26107446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #16: <unknown function> + 0x1acf4b8 (0x7fad22ab24b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #17: <unknown function> + 0x5aee004 (0x7fad26ad1004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #18: <unknown function> + 0x5af36b5 (0x7fad26ad66b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #19: <unknown function> + 0xd2631e (0x7fad396c031e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default6]:[rank14]: frame #20: <unknown function> + 0x47def4 (0x7fad38e17ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default6]:[rank14]: frame #21: <unknown function> + 0x1445a6 (0x55936551a5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #22: _PyObject_MakeTpCall + 0x26b (0x559365513a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #23: <unknown function> + 0x150866 (0x559365526866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55936550f142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55936551aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #26: PyObject_Call + 0xbc (0x559365526f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55936550d2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55936551aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: Traceback (most recent call last): | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default6]:[rank14]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55936550b8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #30: <unknown function> + 0x150582 (0x559365526582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55936550b8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #32: <unknown function> + 0x150582 (0x559365526582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55936550b8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #34: <unknown function> + 0x150582 (0x559365526582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55936550b8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: trainer.train(dataloader) | |
| [default6]:[rank14]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x559365512f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #37: _PyObject_Call_Prepend + 0x69 (0x559365524c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #38: <unknown function> + 0x211239 (0x5593655e7239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #39: _PyObject_MakeTpCall + 0x26b (0x559365513a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55936550f3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55936551aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55936550ac5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default4]:[rank12]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default0]:[rank8]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default0]:[rank8]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default0]:[rank8]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default0]:[rank8]: output = model(**micro_batch) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank8]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank8]: return forward_call(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default0]:[rank8]: sharded_logits = self.model( | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank8]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank8]: return forward_call(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default0]:[rank8]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default0]:[rank8]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank8]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank8]: return forward_call(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default0]:[rank8]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default0]:[rank8]: pipeline_state.run_communication() | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default0]:[rank8]: recv_activation_tensor = recv_activation() | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default0]:[rank8]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default0]:[rank8]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default0]:[rank8]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default0]:[rank8]: dist.recv( | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default0]:[rank8]: return func(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default0]:[rank8]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default0]:[rank8]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer | |
| [default0]:[rank8]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default0]:[rank8]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f3e03256897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default0]:[rank8]: frame #1: <unknown function> + 0x5b3a23e (0x7f3e3cd7323e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f3e3cd6dc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f3e3cd6df82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f3e3cd6efd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3e3cd23371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3e3cd23371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3e3cd23371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3e3cd23371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f3e04530189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default0]:[rank8]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f3e04537610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default0]:[rank8]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f3e04556978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default0]:[rank8]: frame #12: <unknown function> + 0x5adc309 (0x7f3e3cd15309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #13: <unknown function> + 0x5ae6f10 (0x7f3e3cd1ff10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f3e3cd1ffa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #15: <unknown function> + 0x5124446 (0x7f3e3c35d446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #16: <unknown function> + 0x1acf4b8 (0x7f3e38d084b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #17: <unknown function> + 0x5aee004 (0x7f3e3cd27004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #18: <unknown function> + 0x5af36b5 (0x7f3e3cd2c6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #19: <unknown function> + 0xd2631e (0x7f3e4f91631e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default0]:[rank8]: frame #20: <unknown function> + 0x47def4 (0x7f3e4f06def4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default0]:[rank8]: frame #21: <unknown function> + 0x1445a6 (0x561aa4e595a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #22: _PyObject_MakeTpCall + 0x26b (0x561aa4e52a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #23: <unknown function> + 0x150866 (0x561aa4e65866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x561aa4e4e142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #25: _PyFunction_Vectorcall + 0x6c (0x561aa4e59a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #26: PyObject_Call + 0xbc (0x561aa4e65f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x561aa4e4c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #28: _PyFunction_Vectorcall + 0x6c (0x561aa4e59a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x561aa4e4a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #30: <unknown function> + 0x150582 (0x561aa4e65582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x561aa4e4a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #32: <unknown function> + 0x150582 (0x561aa4e65582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x561aa4e4a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #34: <unknown function> + 0x150582 (0x561aa4e65582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x561aa4e4a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x561aa4e51f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #37: _PyObject_Call_Prepend + 0x69 (0x561aa4e63c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #38: <unknown function> + 0x211239 (0x561aa4f26239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #39: _PyObject_MakeTpCall + 0x26b (0x561aa4e52a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x561aa4e4e3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #41: _PyFunction_Vectorcall + 0x6c (0x561aa4e59a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x561aa4e49c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #43: _PyFunction_Vectorcall + 0x6c (0x561aa4e59a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x561aa4e4a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #45: <unknown function> + 0x150582 (0x561aa4e65582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #46: PyObject_Call + 0xbc (0x561aa4e65f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x561aa4e4c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #48: <unknown function> + 0x150582 (0x561aa4e65582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #49: PyObject_Call + 0xbc (0x561aa4e65f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x561aa4e4c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #51: _PyFunction_Vectorcall + 0x6c (0x561aa4e59a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x561aa4e52007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #53: _PyObject_Call_Prepend + 0x69 (0x561aa4e63c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #54: <unknown function> + 0x211239 (0x561aa4f26239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #55: PyObject_Call + 0x207 (0x561aa4e66067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x561aa4e4c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #57: <unknown function> + 0x150582 (0x561aa4e65582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x561aa4e4a8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #59: <unknown function> + 0x150582 (0x561aa4e65582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #60: PyObject_Call + 0xbc (0x561aa4e65f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x561aa4e4c2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #62: <unknown function> + 0x150582 (0x561aa4e65582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #63: PyObject_Call + 0xbc (0x561aa4e65f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default2]:[rank10]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default2]:[rank10]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default2]:[rank10]: dist.recv( | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default2]:[rank10]: return func(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default2]:[rank10]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default2]:[rank10]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer | |
| [default2]:[rank10]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default2]:[rank10]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f28a3a5b897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default2]:[rank10]: frame #1: <unknown function> + 0x5b3a23e (0x7f28dd57823e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f28dd572c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f28dd572f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f28dd573fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f28dd528371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f28dd528371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f28dd528371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f28dd528371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f28a4d35189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:[rank10]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f28a4d3c610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:[rank10]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f28a4d5b978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:[rank10]: frame #12: <unknown function> + 0x5adc309 (0x7f28dd51a309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #13: <unknown function> + 0x5ae6f10 (0x7f28dd524f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f28dd524fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #15: <unknown function> + 0x5124446 (0x7f28dcb62446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #16: <unknown function> + 0x1acf4b8 (0x7f28d950d4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #17: <unknown function> + 0x5aee004 (0x7f28dd52c004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #18: <unknown function> + 0x5af36b5 (0x7f28dd5316b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #19: <unknown function> + 0xd2631e (0x7f28f011b31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default2]:[rank10]: frame #20: <unknown function> + 0x47def4 (0x7f28ef872ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default2]:[rank10]: frame #21: <unknown function> + 0x1445a6 (0x556689a7b5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #22: _PyObject_MakeTpCall + 0x26b (0x556689a74a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #23: <unknown function> + 0x150866 (0x556689a87866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x556689a70142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #25: _PyFunction_Vectorcall + 0x6c (0x556689a7ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #26: PyObject_Call + 0xbc (0x556689a87f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x556689a6e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #28: _PyFunction_Vectorcall + 0x6c (0x556689a7ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x556689a6c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #30: <unknown function> + 0x150582 (0x556689a87582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x556689a6c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #32: <unknown function> + 0x150582 (0x556689a87582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x556689a6c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #34: <unknown function> + 0x150582 (0x556689a87582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x556689a6c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x556689a73f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #37: _PyObject_Call_Prepend + 0x69 (0x556689a85c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #38: <unknown function> + 0x211239 (0x556689b48239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #39: _PyObject_MakeTpCall + 0x26b (0x556689a74a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x556689a703e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #41: _PyFunction_Vectorcall + 0x6c (0x556689a7ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x556689a6bc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #43: _PyFunction_Vectorcall + 0x6c (0x556689a7ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x556689a6c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #45: <unknown function> + 0x150582 (0x556689a87582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #46: PyObject_Call + 0xbc (0x556689a87f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x556689a6e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #48: <unknown function> + 0x150582 (0x556689a87582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #49: PyObject_Call + 0xbc (0x556689a87f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x556689a6e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #51: _PyFunction_Vectorcall + 0x6c (0x556689a7ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x556689a74007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #53: _PyObject_Call_Prepend + 0x69 (0x556689a85c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #54: <unknown function> + 0x211239 (0x556689b48239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #55: PyObject_Call + 0x207 (0x556689a88067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x556689a6e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #57: <unknown function> + 0x150582 (0x556689a87582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x556689a6c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #59: <unknown function> + 0x150582 (0x556689a87582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #60: PyObject_Call + 0xbc (0x556689a87f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x556689a6e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #62: <unknown function> + 0x150582 (0x556689a87582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #63: PyObject_Call + 0xbc (0x556689a87f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default3]:[rank11]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default3]:[rank11]: pipeline_state.run_communication() | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default3]:[rank11]: recv_activation_tensor = recv_activation() | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default3]:[rank11]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default3]:[rank11]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default3]:[rank11]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default3]:[rank11]: dist.recv( | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default3]:[rank11]: return func(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default3]:[rank11]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default3]:[rank11]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer | |
| [default3]:[rank11]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default3]:[rank11]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f2c890ac897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default3]:[rank11]: frame #1: <unknown function> + 0x5b3a23e (0x7f2cc2bc923e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f2cc2bc3c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f2cc2bc3f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f2cc2bc4fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2cc2b79371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2cc2b79371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2cc2b79371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2cc2b79371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f2c8a386189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:[rank11]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f2c8a38d610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:[rank11]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f2c8a3ac978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:[rank11]: frame #12: <unknown function> + 0x5adc309 (0x7f2cc2b6b309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #13: <unknown function> + 0x5ae6f10 (0x7f2cc2b75f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f2cc2b75fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #15: <unknown function> + 0x5124446 (0x7f2cc21b3446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #16: <unknown function> + 0x1acf4b8 (0x7f2cbeb5e4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #17: <unknown function> + 0x5aee004 (0x7f2cc2b7d004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #18: <unknown function> + 0x5af36b5 (0x7f2cc2b826b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #19: <unknown function> + 0xd2631e (0x7f2cd576c31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default3]:[rank11]: frame #20: <unknown function> + 0x47def4 (0x7f2cd4ec3ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default3]:[rank11]: frame #21: <unknown function> + 0x1445a6 (0x560f494d15a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #22: _PyObject_MakeTpCall + 0x26b (0x560f494caa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #23: <unknown function> + 0x150866 (0x560f494dd866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x560f494c6142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #25: _PyFunction_Vectorcall + 0x6c (0x560f494d1a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #26: PyObject_Call + 0xbc (0x560f494ddf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x560f494c42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #28: _PyFunction_Vectorcall + 0x6c (0x560f494d1a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x560f494c28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #30: <unknown function> + 0x150582 (0x560f494dd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x560f494c28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #32: <unknown function> + 0x150582 (0x560f494dd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x560f494c28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #34: <unknown function> + 0x150582 (0x560f494dd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x560f494c28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x560f494c9f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #37: _PyObject_Call_Prepend + 0x69 (0x560f494dbc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #38: <unknown function> + 0x211239 (0x560f4959e239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #39: _PyObject_MakeTpCall + 0x26b (0x560f494caa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x560f494c63e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #41: _PyFunction_Vectorcall + 0x6c (0x560f494d1a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x560f494c1c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #43: _PyFunction_Vectorcall + 0x6c (0x560f494d1a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x560f494c28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #45: <unknown function> + 0x150582 (0x560f494dd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #46: PyObject_Call + 0xbc (0x560f494ddf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x560f494c42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #48: <unknown function> + 0x150582 (0x560f494dd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #49: PyObject_Call + 0xbc (0x560f494ddf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x560f494c42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #51: _PyFunction_Vectorcall + 0x6c (0x560f494d1a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x560f494ca007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #53: _PyObject_Call_Prepend + 0x69 (0x560f494dbc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #54: <unknown function> + 0x211239 (0x560f4959e239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #55: PyObject_Call + 0x207 (0x560f494de067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x560f494c42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #57: <unknown function> + 0x150582 (0x560f494dd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x560f494c28fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #59: <unknown function> + 0x150582 (0x560f494dd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #60: PyObject_Call + 0xbc (0x560f494ddf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x560f494c42b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #62: <unknown function> + 0x150582 (0x560f494dd582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #63: PyObject_Call + 0xbc (0x560f494ddf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default5]:[rank13]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default5]:[rank13]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank13]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank13]: return forward_call(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default5]:[rank13]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default5]:[rank13]: pipeline_state.run_communication() | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default5]:[rank13]: recv_activation_tensor = recv_activation() | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default5]:[rank13]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default5]:[rank13]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default5]:[rank13]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default5]:[rank13]: dist.recv( | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default5]:[rank13]: return func(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default5]:[rank13]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default5]:[rank13]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer | |
| [default5]:[rank13]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default5]:[rank13]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f14a9a23897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default5]:[rank13]: frame #1: <unknown function> + 0x5b3a23e (0x7f14e354023e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f14e353ac87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f14e353af82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f14e353bfd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f14e34f0371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f14e34f0371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f14e34f0371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f14e34f0371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f14aacfd189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:[rank13]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f14aad04610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:[rank13]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f14aad23978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:[rank13]: frame #12: <unknown function> + 0x5adc309 (0x7f14e34e2309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #13: <unknown function> + 0x5ae6f10 (0x7f14e34ecf10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f14e34ecfa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #15: <unknown function> + 0x5124446 (0x7f14e2b2a446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #16: <unknown function> + 0x1acf4b8 (0x7f14df4d54b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #17: <unknown function> + 0x5aee004 (0x7f14e34f4004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #18: <unknown function> + 0x5af36b5 (0x7f14e34f96b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #19: <unknown function> + 0xd2631e (0x7f14f60e331e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default5]:[rank13]: frame #20: <unknown function> + 0x47def4 (0x7f14f583aef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default5]:[rank13]: frame #21: <unknown function> + 0x1445a6 (0x559e466235a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #22: _PyObject_MakeTpCall + 0x26b (0x559e4661ca6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #23: <unknown function> + 0x150866 (0x559e4662f866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x559e46618142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #25: _PyFunction_Vectorcall + 0x6c (0x559e46623a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #26: PyObject_Call + 0xbc (0x559e4662ff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x559e466162b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #28: _PyFunction_Vectorcall + 0x6c (0x559e46623a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x559e466148fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #30: <unknown function> + 0x150582 (0x559e4662f582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x559e466148fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #32: <unknown function> + 0x150582 (0x559e4662f582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x559e466148fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #34: <unknown function> + 0x150582 (0x559e4662f582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x559e466148fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x559e4661bf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #37: _PyObject_Call_Prepend + 0x69 (0x559e4662dc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #38: <unknown function> + 0x211239 (0x559e466f0239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #39: _PyObject_MakeTpCall + 0x26b (0x559e4661ca6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x559e466183e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #41: _PyFunction_Vectorcall + 0x6c (0x559e46623a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x559e46613c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #43: _PyFunction_Vectorcall + 0x6c (0x559e46623a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x559e466148fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #45: <unknown function> + 0x150582 (0x559e4662f582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #46: PyObject_Call + 0xbc (0x559e4662ff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x559e466162b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #48: <unknown function> + 0x150582 (0x559e4662f582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #49: PyObject_Call + 0xbc (0x559e4662ff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x559e466162b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #51: _PyFunction_Vectorcall + 0x6c (0x559e46623a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x559e4661c007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #53: _PyObject_Call_Prepend + 0x69 (0x559e4662dc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #54: <unknown function> + 0x211239 (0x559e466f0239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #55: PyObject_Call + 0x207 (0x559e46630067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x559e466162b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #57: <unknown function> + 0x150582 (0x559e4662f582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x559e466148fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #59: <unknown function> + 0x150582 (0x559e4662f582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #60: PyObject_Call + 0xbc (0x559e4662ff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x559e466162b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #62: <unknown function> + 0x150582 (0x559e4662f582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #63: PyObject_Call + 0xbc (0x559e4662ff1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default6]:[rank14]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55936551aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55936550b8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #45: <unknown function> + 0x150582 (0x559365526582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #46: PyObject_Call + 0xbc (0x559365526f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55936550d2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #48: <unknown function> + 0x150582 (0x559365526582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #49: PyObject_Call + 0xbc (0x559365526f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55936550d2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55936551aa2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x559365513007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #53: _PyObject_Call_Prepend + 0x69 (0x559365524c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #54: <unknown function> + 0x211239 (0x5593655e7239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #55: PyObject_Call + 0x207 (0x559365527067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55936550d2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #57: <unknown function> + 0x150582 (0x559365526582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55936550b8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #59: <unknown function> + 0x150582 (0x559365526582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #60: PyObject_Call + 0xbc (0x559365526f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55936550d2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #62: <unknown function> + 0x150582 (0x559365526582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #63: PyObject_Call + 0xbc (0x559365526f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default4]:[rank12]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default4]:[rank12]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default4]:[rank12]: output = model(**micro_batch) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank12]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank12]: return forward_call(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default4]:[rank12]: sharded_logits = self.model( | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank12]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank12]: return forward_call(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default4]:[rank12]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default4]:[rank12]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank12]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank12]: return forward_call(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default4]:[rank12]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default4]:[rank12]: pipeline_state.run_communication() | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default4]:[rank12]: recv_activation_tensor = recv_activation() | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default4]:[rank12]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default4]:[rank12]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default4]:[rank12]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default4]:[rank12]: dist.recv( | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default4]:[rank12]: return func(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default4]:[rank12]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default4]:[rank12]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer | |
| [default4]:[rank12]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default4]:[rank12]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f83dbcca897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default4]:[rank12]: frame #1: <unknown function> + 0x5b3a23e (0x7f84157e723e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f84157e1c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f84157e1f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f84157e2fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f8415797371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f8415797371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f8415797371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f8415797371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f83dcfa4189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:[rank12]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f83dcfab610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:[rank12]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f83dcfca978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:[rank12]: frame #12: <unknown function> + 0x5adc309 (0x7f8415789309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #13: <unknown function> + 0x5ae6f10 (0x7f8415793f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f8415793fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #15: <unknown function> + 0x5124446 (0x7f8414dd1446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #16: <unknown function> + 0x1acf4b8 (0x7f841177c4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #17: <unknown function> + 0x5aee004 (0x7f841579b004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #18: <unknown function> + 0x5af36b5 (0x7f84157a06b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank12]: frame #19: <unknown function> + 0xd2631e (0x7f842838a31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default4]:[rank12]: frame #20: <unknown function> + 0x47def4 (0x7f8427ae1ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default4]:[rank12]: frame #21: <unknown function> + 0x1445a6 (0x55db769255a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55db7691ea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #23: <unknown function> + 0x150866 (0x55db76931866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55db7691a142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55db76925a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #26: PyObject_Call + 0xbc (0x55db76931f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55db769182b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55db76925a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55db769168fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #30: <unknown function> + 0x150582 (0x55db76931582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55db769168fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #32: <unknown function> + 0x150582 (0x55db76931582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55db769168fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #34: <unknown function> + 0x150582 (0x55db76931582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55db769168fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55db7691df50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55db7692fc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #38: <unknown function> + 0x211239 (0x55db769f2239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55db7691ea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55db7691a3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55db76925a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55db76915c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55db76925a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55db769168fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #45: <unknown function> + 0x150582 (0x55db76931582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #46: PyObject_Call + 0xbc (0x55db76931f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55db769182b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #48: <unknown function> + 0x150582 (0x55db76931582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #49: PyObject_Call + 0xbc (0x55db76931f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55db769182b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55db76925a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55db7691e007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55db7692fc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #54: <unknown function> + 0x211239 (0x55db769f2239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #55: PyObject_Call + 0x207 (0x55db76932067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55db769182b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #57: <unknown function> + 0x150582 (0x55db76931582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55db769168fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #59: <unknown function> + 0x150582 (0x55db76931582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #60: PyObject_Call + 0xbc (0x55db76931f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55db769182b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #62: <unknown function> + 0x150582 (0x55db76931582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: frame #63: PyObject_Call + 0xbc (0x55db76931f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank12]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| E0702 16:20:37.522000 140449488344896 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 1 (pid: 528028) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 | |
| Traceback (most recent call last): | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module> | |
| sys.exit(main()) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper | |
| return f(*args, **kwargs) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main | |
| run(args) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run | |
| elastic_launch( | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__ | |
| return launch_agent(self._config, self._entrypoint, list(args)) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent | |
| raise ChildFailedError( | |
| torch.distributed.elastic.multiprocessing.errors.ChildFailedError: | |
| ============================================================ | |
| /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED | |
| ------------------------------------------------------------ | |
| Failures: | |
| <NO_OTHER_FAILURES> | |
| ------------------------------------------------------------ | |
| Root Cause (first observed failure): | |
| [0]: | |
| time : 2024-07-02_16:20:35 | |
| host : ip-26-0-167-177.ec2.internal | |
| rank : 1 (local_rank: 1) | |
| exitcode : 1 (pid: 528028) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| ============================================================ | |
| srun: error: ip-26-0-167-177: task 0: Exited with exit code 1 | |
| W0702 16:20:39.127000 140674584221440 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1252] The node 'ip-26-0-170-31.ec2.internal_2780766_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError. | |
| W0702 16:20:40.070000 140680251041600 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2780837 closing signal SIGTERM | |
| W0702 16:20:40.070000 140680251041600 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2780838 closing signal SIGTERM | |
| W0702 16:20:40.072000 140680251041600 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2780839 closing signal SIGTERM | |
| W0702 16:20:40.072000 140680251041600 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2780840 closing signal SIGTERM | |
| W0702 16:20:40.073000 140680251041600 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2780841 closing signal SIGTERM | |
| W0702 16:20:40.078000 140680251041600 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2780842 closing signal SIGTERM | |
| W0702 16:20:40.080000 140680251041600 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2780843 closing signal SIGTERM | |
| W0702 16:20:40.081000 140680251041600 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2780844 closing signal SIGTERM | |
| W0702 16:20:42.416000 140680251041600 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-170-31.ec2.internal_2780766_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError. | |
| W0702 16:20:42.428000 140680251041600 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-170-31.ec2.internal_2780766_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError. | |
| Traceback (most recent call last): | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 113, in _call_store | |
| return getattr(self._store, store_op)(*args, **kwargs) | |
| torch.distributed.DistNetworkError: Broken pipe | |
| The above exception was the direct cause of the following exception: | |
| Traceback (most recent call last): | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module> | |
| sys.exit(main()) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper | |
| return f(*args, **kwargs) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main | |
| run(args) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run | |
| elastic_launch( | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__ | |
| return launch_agent(self._config, self._entrypoint, list(args)) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 254, in launch_agent | |
| result = agent.run() | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 123, in wrapper | |
| result = f(*args, **kwargs) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 733, in run | |
| result = self._invoke_run(role) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 908, in _invoke_run | |
| num_nodes_waiting = rdzv_handler.num_nodes_waiting() | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 1174, in num_nodes_waiting | |
| self._state_holder.sync() | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 419, in sync | |
| get_response = self._backend.get_state() | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 73, in get_state | |
| base64_state: bytes = self._call_store("get", self._key) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 115, in _call_store | |
| raise RendezvousConnectionError( | |
| torch.distributed.elastic.rendezvous.api.RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details. | |
| srun: error: ip-26-0-170-31: task 1: Exited with exit code 1 | |
| Consider using `hf_transfer` for faster uploads. This solution comes with some limitations. See https://huggingface.co/docs/huggingface_hub/hf_transfer for more details. | |