| ======================== | |
| START TIME: Thu Jul 4 00:05:41 UTC 2024 | |
| python3 version = Python 3.10.14 | |
| ======================== | |
| The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well. | |
| Token is valid (permission: write). | |
| Your token has been saved to /admin/home/ferdinand_mom/.cache/huggingface/token | |
| Login successful | |
| Already on 'bench_cluster' | |
| M examples/config_tiny_llama.py | |
| M examples/config_tiny_llama.yaml | |
| M examples/train_tiny_llama.sh | |
| M src/nanotron/models/llama.py | |
| M src/nanotron/trainer.py | |
| Your branch is up to date with 'origin/bench_cluster'. | |
| Job status: RUNNING | |
| W0704 00:05:44.592000 140695680661312 torch/distributed/run.py:757] | |
| W0704 00:05:44.592000 140695680661312 torch/distributed/run.py:757] ***************************************** | |
| W0704 00:05:44.592000 140695680661312 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. | |
| W0704 00:05:44.592000 140695680661312 torch/distributed/run.py:757] ***************************************** | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: Config: | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: Config(general=GeneralArgs(project='bench_cluster', | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: run='%date_%jobid', | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: seed=42, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: step=None, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: consumed_train_samples=None, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: benchmark_csv_path=None, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: ignore_sanity_checks=True), | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: parallelism=ParallelismArgs(dp=1, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: pp=8, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: tp=1, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: pp_engine=<nanotron.parallel.pipeline_parallel.engine.OneForwardOneBackwardPipelineEngine object at 0x7f091eefc8b0>, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: tp_mode=<TensorParallelLinearMode.REDUCE_SCATTER: 2>, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: tp_linear_async_communication=False, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: expert_parallel_size=1), | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: model=ModelArgs(model_config=LlamaConfig(bos_token_id=1, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: eos_token_id=2, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: hidden_act='silu', | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: hidden_size=2048, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: initializer_range=0.02, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: intermediate_size=4096, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: is_llama_config=True, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: max_position_embeddings=4096, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: num_attention_heads=32, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: num_hidden_layers=24, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: num_key_value_heads=32, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: pad_token_id=None, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: pretraining_tp=1, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: rms_norm_eps=1e-05, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: rope_scaling=None, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: rope_theta=10000.0, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: tie_word_embeddings=True, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: use_cache=True, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: vocab_size=50257), | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: init_method=RandomInit(std=0.025), | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: dtype=torch.bfloat16, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: make_vocab_size_divisible_by=1, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: ddp_bucket_cap_mb=25), | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: tokenizer=TokenizerArgs(tokenizer_name_or_path='openai-community/gpt2', | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: tokenizer_revision=None, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: tokenizer_max_length=None), | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: checkpoints=CheckpointsArgs(checkpoints_path=Path('/dev/null'), | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: checkpoint_interval=100000, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: save_initial_state=False, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: resume_checkpoint_path=None, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: checkpoints_path_is_shared_file_system=False), | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: logging=LoggingArgs(log_level='info', | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: log_level_replica='info', | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: iteration_step_info_interval=1), | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: tokens=TokensArgs(sequence_length=4096, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: train_steps=20, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: micro_batch_size=128, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: batch_accumulation_per_replica=8, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: val_check_interval=-1, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: limit_val_batches=0, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: limit_test_batches=0), | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: optimizer=OptimizerArgs(optimizer_factory=AdamWOptimizerArgs(adam_eps=1e-08, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: adam_beta1=0.9, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: adam_beta2=0.95, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: torch_adam_is_fused=True, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: name='adamW'), | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: zero_stage=1, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: weight_decay=0.01, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: clip_grad=1.0, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: accumulate_grad_in_fp32=True, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: learning_rate_scheduler=LRSchedulerArgs(learning_rate=0.0001, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: lr_warmup_steps=1, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: lr_warmup_style='linear', | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: lr_decay_style='linear', | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: lr_decay_steps=19, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: lr_decay_starting_step=None, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: min_decay_lr=1e-05)), | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: data_stages=[DatasetStageArgs(name='Training Stage', | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: start_training_step=1, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: data=DataArgs(dataset=PretrainDatasetsArgs(hf_dataset_or_datasets='roneneldan/TinyStories', | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: hf_dataset_splits='train', | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: hf_dataset_config_name=None, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: dataset_processing_num_proc_per_process=64, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: dataset_overwrite_cache=False, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: text_column_name='text'), | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: seed=42, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: num_loading_workers=0))], | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: profiler=ProfilerArgs(profiler_export_path=Path('/fsx/ferdinandmom/ferdinand-hf/bench_cluster/results/llama-1B/8_GPUS/dp-1_tp-1_pp-8_mbz-128')), | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: lighteval=None) | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: Model Config: | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: LlamaConfig(bos_token_id=1, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: eos_token_id=2, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: hidden_act='silu', | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: hidden_size=2048, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: initializer_range=0.02, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: intermediate_size=4096, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: is_llama_config=True, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: max_position_embeddings=4096, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: num_attention_heads=32, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: num_hidden_layers=24, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: num_key_value_heads=32, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: pad_token_id=None, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: pretraining_tp=1, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: rms_norm_eps=1e-05, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: rope_scaling=None, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: rope_theta=10000.0, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: tie_word_embeddings=True, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: use_cache=True, | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: vocab_size=50257) | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: Building model.. | |
| [default0]:07/04/2024 00:06:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: Setting PP block ranks... | |
| [default2]:07/04/2024 00:06:19 [INFO|DP=0|PP=2|TP=0|ip-26-0-169-247]: Local number of parameters: 126M (240.02MiB) | |
| [default2]:07/04/2024 00:06:19 [INFO|DP=0|PP=2|TP=0|ip-26-0-169-247]: [After model building] Memory usage: 243.03MiB. Peak allocated: 245.06MiB Peak reserved: 262.00MiB | |
| [default2]:07/04/2024 00:06:19 [INFO|DP=0|PP=2|TP=0|ip-26-0-169-247]: No checkpoint path provided. | |
| [default5]:07/04/2024 00:06:19 [INFO|DP=0|PP=5|TP=0|ip-26-0-169-247]: Local number of parameters: 126M (240.02MiB) | |
| [default5]:07/04/2024 00:06:19 [INFO|DP=0|PP=5|TP=0|ip-26-0-169-247]: [After model building] Memory usage: 243.03MiB. Peak allocated: 245.06MiB Peak reserved: 262.00MiB | |
| [default5]:07/04/2024 00:06:19 [INFO|DP=0|PP=5|TP=0|ip-26-0-169-247]: No checkpoint path provided. | |
| [default0]:07/04/2024 00:06:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: Total number of parameters: 1.21G (2312.82MiB) | |
| [default0]:07/04/2024 00:06:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: Local number of parameters: 271M (516.35MiB) | |
| [default0]:07/04/2024 00:06:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: [After model building] Memory usage: 520.36MiB. Peak allocated: 522.39MiB Peak reserved: 534.00MiB | |
| [default0]:07/04/2024 00:06:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: No checkpoint path provided. | |
| [default0]:07/04/2024 00:06:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: Parametrizing model parameters using StandardParametrizator | |
| [default3]:07/04/2024 00:06:19 [INFO|DP=0|PP=3|TP=0|ip-26-0-169-247]: Local number of parameters: 168M (320.03MiB) | |
| [default3]:07/04/2024 00:06:19 [INFO|DP=0|PP=3|TP=0|ip-26-0-169-247]: [After model building] Memory usage: 324.04MiB. Peak allocated: 326.07MiB Peak reserved: 336.00MiB | |
| [default3]:07/04/2024 00:06:19 [INFO|DP=0|PP=3|TP=0|ip-26-0-169-247]: No checkpoint path provided. | |
| [default1]:07/04/2024 00:06:19 [INFO|DP=0|PP=1|TP=0|ip-26-0-169-247]: Local number of parameters: 126M (240.02MiB) | |
| [default1]:07/04/2024 00:06:19 [INFO|DP=0|PP=1|TP=0|ip-26-0-169-247]: [After model building] Memory usage: 243.03MiB. Peak allocated: 245.06MiB Peak reserved: 262.00MiB | |
| [default1]:07/04/2024 00:06:19 [INFO|DP=0|PP=1|TP=0|ip-26-0-169-247]: No checkpoint path provided. | |
| [default4]:07/04/2024 00:06:19 [INFO|DP=0|PP=4|TP=0|ip-26-0-169-247]: Local number of parameters: 126M (240.02MiB) | |
| [default4]:07/04/2024 00:06:19 [INFO|DP=0|PP=4|TP=0|ip-26-0-169-247]: [After model building] Memory usage: 243.03MiB. Peak allocated: 245.06MiB Peak reserved: 262.00MiB | |
| [default4]:07/04/2024 00:06:19 [INFO|DP=0|PP=4|TP=0|ip-26-0-169-247]: No checkpoint path provided. | |
| [default6]:07/04/2024 00:06:19 [INFO|DP=0|PP=6|TP=0|ip-26-0-169-247]: Local number of parameters: 168M (320.03MiB) | |
| [default6]:07/04/2024 00:06:19 [INFO|DP=0|PP=6|TP=0|ip-26-0-169-247]: [After model building] Memory usage: 324.04MiB. Peak allocated: 326.07MiB Peak reserved: 336.00MiB | |
| [default6]:07/04/2024 00:06:19 [INFO|DP=0|PP=6|TP=0|ip-26-0-169-247]: No checkpoint path provided. | |
| [default7]:07/04/2024 00:06:19 [INFO|DP=0|PP=7|TP=0|ip-26-0-169-247]: Local number of parameters: 103M (196.32MiB) | |
| [default7]:07/04/2024 00:06:19 [INFO|DP=0|PP=7|TP=0|ip-26-0-169-247]: [After model building] Memory usage: 196.33MiB. Peak allocated: 196.33MiB Peak reserved: 200.00MiB | |
| [default7]:07/04/2024 00:06:19 [INFO|DP=0|PP=7|TP=0|ip-26-0-169-247]: No checkpoint path provided. | |
| [default0]:07/04/2024 00:06:20 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: [Optimizer Building] Using LearningRateForSP as learning rate | |
| [default0]:07/04/2024 00:06:20 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: [ZeRO sharding] Size of optimizer params per rank: | |
| [default0]:07/04/2024 00:06:20 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: [ZeRO sharding] DP Rank 0 has 271M out of 271M (100.00%) params' optimizer states | |
| [default0]:07/04/2024 00:06:21 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: [Training Plan] Stage Training Stage has 19 remaining training steps and has consumed 0 samples | |
| [default0]:07/04/2024 00:06:21 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: Using `datasets` library | |
| [default0]:07/04/2024 00:06:21 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: Loading tokenizer from openai-community/gpt2 and transformers/hf_hub versions ('4.41.2', '0.23.4') | |
| [default0]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default0]:07/04/2024 00:06:21 [WARNING|DP=0|PP=0|TP=0|ip-26-0-169-247]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default0]:07/04/2024 00:06:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: [Training Plan] There are 1 training stages | |
| [default0]:07/04/2024 00:06:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: [Stage Training Stage] start from step 1 | |
| [default0]:07/04/2024 00:06:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: | |
| [default0]:07/04/2024 00:06:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: [Start training] datetime: 2024-07-04 00:06:25.551663 | mbs: 128 | grad_accum: 8 | global_batch_size: 1024 | sequence_length: 4096 | train_steps: 20 | start_iteration_step: 0 | consumed_train_samples: 0 | |
| [default0]:07/04/2024 00:06:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: Resuming training from stage Training Stage, it has trained for 0 samples and has 19 remaining train steps | |
| [default0]:07/04/2024 00:06:25 [INFO|DP=0|PP=0|TP=0|ip-26-0-169-247]: Memory usage: 2585.75MiB. Peak allocated 2585.75MiB. Peak reserved: 2602.00MiB | |
| [default5]:07/04/2024 00:06:25 [WARNING|DP=0|PP=5|TP=0|ip-26-0-169-247]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:07/04/2024 00:06:25 [WARNING|DP=0|PP=7|TP=0|ip-26-0-169-247]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default5]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default4]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default4]:07/04/2024 00:06:25 [WARNING|DP=0|PP=4|TP=0|ip-26-0-169-247]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default6]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default6]:07/04/2024 00:06:25 [WARNING|DP=0|PP=6|TP=0|ip-26-0-169-247]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default3]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default3]:07/04/2024 00:06:25 [WARNING|DP=0|PP=3|TP=0|ip-26-0-169-247]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default2]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default2]:07/04/2024 00:06:25 [WARNING|DP=0|PP=2|TP=0|ip-26-0-169-247]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default1]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default1]:07/04/2024 00:06:26 [WARNING|DP=0|PP=1|TP=0|ip-26-0-169-247]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default0]:[rank0]: Traceback (most recent call last): | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default0]:[rank0]: trainer.train(dataloader) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default0]:[rank0]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default0]:[rank0]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default0]:[rank0]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default0]:[rank0]: output = model(**micro_batch) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank0]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank0]: return forward_call(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default0]:[rank0]: sharded_logits = self.model( | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank0]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank0]: return forward_call(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default0]:[rank0]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default0]:[rank0]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank0]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank0]: return forward_call(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward | |
| [default0]:[rank0]: output = self.pp_block(**new_kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank0]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank0]: return forward_call(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 637, in forward | |
| [default0]:[rank0]: hidden_states = self.mlp(hidden_states=hidden_states)["hidden_states"] | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank0]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank0]: return forward_call(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 171, in forward | |
| [default0]:[rank0]: merged_states = self.gate_up_proj(hidden_states) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank0]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank0]: return forward_call(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/nn.py", line 87, in forward | |
| [default0]:[rank0]: return column_linear( | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/tensor_parallel/functional.py", line 359, in column_linear | |
| [default0]:[rank0]: return F.linear(input, weight, bias) | |
| [default0]:[rank0]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 8.00 GiB. GPU | |
| [default3]:[rank3]: Traceback (most recent call last): | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default3]:[rank3]: trainer.train(dataloader) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default3]:[rank3]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default3]:[rank3]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default3]:[rank3]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default3]:[rank3]: output = model(**micro_batch) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank3]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank3]: return forward_call(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default3]:[rank3]: sharded_logits = self.model( | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank3]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank3]: return forward_call(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default3]:[rank3]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default3]:[rank3]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank3]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank3]: return forward_call(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default3]:[rank3]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default3]:[rank3]: pipeline_state.run_communication() | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default3]:[rank3]: recv_activation_tensor = recv_activation() | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default3]:[rank3]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default3]:[rank3]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default3]:[rank3]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default3]:[rank3]: dist.recv( | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default3]:[rank3]: return func(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default3]:[rank3]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default3]:[rank3]: torch.distributed.DistBackendError: [3] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '2:3', but store->get('2:3') got error: Connection reset by peer | |
| [default3]:[rank3]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default3]:[rank3]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7ff5c9aef897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default3]:[rank3]: frame #1: <unknown function> + 0x5b3a23e (0x7ff60360c23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank3]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7ff603606c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank3]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7ff603606f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank3]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7ff603607fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank3]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff6035bc371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank3]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff6035bc371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank3]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff6035bc371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank3]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff6035bc371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank3]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7ff5cadc9189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:[rank3]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7ff5cadd0610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:[rank3]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7ff5cadef978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:[rank3]: frame #12: <unknown function> + 0x5adc309 (0x7ff6035ae309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank3]: frame #13: <unknown function> + 0x5ae6f10 (0x7ff6035b8f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank3]: frame #14: <unknown function> + 0x5ae6fa5 (0x7ff6035b8fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank3]: frame #15: <unknown function> + 0x5124446 (0x7ff602bf6446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank3]: frame #16: <unknown function> + 0x1acf4b8 (0x7ff5ff5a14b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank3]: frame #17: <unknown function> + 0x5aee004 (0x7ff6035c0004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank3]: frame #18: <unknown function> + 0x5af36b5 (0x7ff6035c56b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank3]: frame #19: <unknown function> + 0xd2631e (0x7ff6161af31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default3]:[rank3]: frame #20: <unknown function> + 0x47def4 (0x7ff615906ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default3]:[rank3]: frame #21: <unknown function> + 0x1445a6 (0x556c7e8485a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #22: _PyObject_MakeTpCall + 0x26b (0x556c7e841a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #23: <unknown function> + 0x150866 (0x556c7e854866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x556c7e83d142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #25: _PyFunction_Vectorcall + 0x6c (0x556c7e848a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #26: PyObject_Call + 0xbc (0x556c7e854f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x556c7e83b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #28: _PyFunction_Vectorcall + 0x6c (0x556c7e848a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x556c7e8398fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #30: <unknown function> + 0x150582 (0x556c7e854582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x556c7e8398fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #32: <unknown function> + 0x150582 (0x556c7e854582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x556c7e8398fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #34: <unknown function> + 0x150582 (0x556c7e854582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x556c7e8398fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x556c7e840f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #37: _PyObject_Call_Prepend + 0x69 (0x556c7e852c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #38: <unknown function> + 0x211239 (0x556c7e915239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #39: _PyObject_MakeTpCall + 0x26b (0x556c7e841a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x556c7e83d3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #41: _PyFunction_Vectorcall + 0x6c (0x556c7e848a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x556c7e838c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #43: _PyFunction_Vectorcall + 0x6c (0x556c7e848a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x556c7e8398fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #45: <unknown function> + 0x150582 (0x556c7e854582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #46: PyObject_Call + 0xbc (0x556c7e854f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x556c7e83b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #48: <unknown function> + 0x150582 (0x556c7e854582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #49: PyObject_Call + 0xbc (0x556c7e854f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x556c7e83b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #51: _PyFunction_Vectorcall + 0x6c (0x556c7e848a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x556c7e841007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #53: _PyObject_Call_Prepend + 0x69 (0x556c7e852c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #54: <unknown function> + 0x211239 (0x556c7e915239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #55: PyObject_Call + 0x207 (0x556c7e855067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x556c7e83b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #57: <unknown function> + 0x150582 (0x556c7e854582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x556c7e8398fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #59: <unknown function> + 0x150582 (0x556c7e854582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #60: PyObject_Call + 0xbc (0x556c7e854f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x556c7e83b2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #62: <unknown function> + 0x150582 (0x556c7e854582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: frame #63: PyObject_Call + 0xbc (0x556c7e854f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank3]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default1]:[rank1]: Traceback (most recent call last): | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default1]:[rank1]: trainer.train(dataloader) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default1]:[rank1]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default1]:[rank1]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default1]:[rank1]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default1]:[rank1]: output = model(**micro_batch) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default1]:[rank1]: sharded_logits = self.model( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default1]:[rank1]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default1]:[rank1]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default1]:[rank1]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default1]:[rank1]: pipeline_state.run_communication() | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default1]:[rank1]: recv_activation_tensor = recv_activation() | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default1]:[rank1]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default1]:[rank1]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default1]:[rank1]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default1]:[rank1]: dist.recv( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default1]:[rank1]: return func(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default1]:[rank1]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default1]:[rank1]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer | |
| [default1]:[rank1]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default1]:[rank1]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7ff919a7f897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default1]:[rank1]: frame #1: <unknown function> + 0x5b3a23e (0x7ff95359c23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank1]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7ff953596c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank1]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7ff953596f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank1]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7ff953597fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank1]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff95354c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank1]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff95354c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank1]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff95354c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank1]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff95354c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank1]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7ff91ad59189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default1]:[rank1]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7ff91ad60610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default1]:[rank1]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7ff91ad7f978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default1]:[rank1]: frame #12: <unknown function> + 0x5adc309 (0x7ff95353e309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank1]: frame #13: <unknown function> + 0x5ae6f10 (0x7ff953548f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank1]: frame #14: <unknown function> + 0x5ae6fa5 (0x7ff953548fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank1]: frame #15: <unknown function> + 0x5124446 (0x7ff952b86446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank1]: frame #16: <unknown function> + 0x1acf4b8 (0x7ff94f5314b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank1]: frame #17: <unknown function> + 0x5aee004 (0x7ff953550004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank1]: frame #18: <unknown function> + 0x5af36b5 (0x7ff9535556b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank1]: frame #19: <unknown function> + 0xd2631e (0x7ff96613f31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default1]:[rank1]: frame #20: <unknown function> + 0x47def4 (0x7ff965896ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default1]:[rank1]: frame #21: <unknown function> + 0x1445a6 (0x55e73b4b65a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55e73b4afa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #23: <unknown function> + 0x150866 (0x55e73b4c2866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55e73b4ab142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55e73b4b6a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #26: PyObject_Call + 0xbc (0x55e73b4c2f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55e73b4a92b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55e73b4b6a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55e73b4a78fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #30: <unknown function> + 0x150582 (0x55e73b4c2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55e73b4a78fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #32: <unknown function> + 0x150582 (0x55e73b4c2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55e73b4a78fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #34: <unknown function> + 0x150582 (0x55e73b4c2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55e73b4a78fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55e73b4aef50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55e73b4c0c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #38: <unknown function> + 0x211239 (0x55e73b583239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55e73b4afa6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55e73b4ab3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55e73b4b6a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55e73b4a6c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55e73b4b6a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55e73b4a78fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #45: <unknown function> + 0x150582 (0x55e73b4c2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #46: PyObject_Call + 0xbc (0x55e73b4c2f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55e73b4a92b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #48: <unknown function> + 0x150582 (0x55e73b4c2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #49: PyObject_Call + 0xbc (0x55e73b4c2f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55e73b4a92b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55e73b4b6a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55e73b4af007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55e73b4c0c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #54: <unknown function> + 0x211239 (0x55e73b583239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #55: PyObject_Call + 0x207 (0x55e73b4c3067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55e73b4a92b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #57: <unknown function> + 0x150582 (0x55e73b4c2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55e73b4a78fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #59: <unknown function> + 0x150582 (0x55e73b4c2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #60: PyObject_Call + 0xbc (0x55e73b4c2f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55e73b4a92b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #62: <unknown function> + 0x150582 (0x55e73b4c2582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: frame #63: PyObject_Call + 0xbc (0x55e73b4c2f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank1]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default4]:[rank4]: Traceback (most recent call last): | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default4]:[rank4]: trainer.train(dataloader) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default4]:[rank4]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default4]:[rank4]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default4]:[rank4]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default4]:[rank4]: output = model(**micro_batch) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank4]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank4]: return forward_call(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default4]:[rank4]: sharded_logits = self.model( | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank4]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank4]: return forward_call(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default4]:[rank4]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default4]:[rank4]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank4]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank4]: return forward_call(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default4]:[rank4]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default4]:[rank4]: pipeline_state.run_communication() | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default4]:[rank4]: recv_activation_tensor = recv_activation() | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default4]:[rank4]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default4]:[rank4]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default4]:[rank4]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default4]:[rank4]: dist.recv( | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default4]:[rank4]: return func(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default4]:[rank4]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default4]:[rank4]: torch.distributed.DistBackendError: [4] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '3:4', but store->get('3:4') got error: Connection reset by peer | |
| [default4]:[rank4]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default4]:[rank4]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f0e2414f897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default4]:[rank4]: frame #1: <unknown function> + 0x5b3a23e (0x7f0e5dc6c23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f0e5dc66c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f0e5dc66f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f0e5dc67fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0e5dc1c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0e5dc1c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0e5dc1c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0e5dc1c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f0e25429189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:[rank4]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f0e25430610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:[rank4]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f0e2544f978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:[rank4]: frame #12: <unknown function> + 0x5adc309 (0x7f0e5dc0e309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #13: <unknown function> + 0x5ae6f10 (0x7f0e5dc18f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f0e5dc18fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #15: <unknown function> + 0x5124446 (0x7f0e5d256446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #16: <unknown function> + 0x1acf4b8 (0x7f0e59c014b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #17: <unknown function> + 0x5aee004 (0x7f0e5dc20004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #18: <unknown function> + 0x5af36b5 (0x7f0e5dc256b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default4]:[rank4]: frame #19: <unknown function> + 0xd2631e (0x7f0e7080f31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default4]:[rank4]: frame #20: <unknown function> + 0x47def4 (0x7f0e6ff66ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default4]:[rank4]: frame #21: <unknown function> + 0x1445a6 (0x565539b3c5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #22: _PyObject_MakeTpCall + 0x26b (0x565539b35a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #23: <unknown function> + 0x150866 (0x565539b48866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x565539b31142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #25: _PyFunction_Vectorcall + 0x6c (0x565539b3ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #26: PyObject_Call + 0xbc (0x565539b48f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x565539b2f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #28: _PyFunction_Vectorcall + 0x6c (0x565539b3ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x565539b2d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #30: <unknown function> + 0x150582 (0x565539b48582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x565539b2d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #32: <unknown function> + 0x150582 (0x565539b48582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x565539b2d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #34: <unknown function> + 0x150582 (0x565539b48582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x565539b2d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x565539b34f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #37: _PyObject_Call_Prepend + 0x69 (0x565539b46c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #38: <unknown function> + 0x211239 (0x565539c09239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #39: _PyObject_MakeTpCall + 0x26b (0x565539b35a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x565539b313e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #41: _PyFunction_Vectorcall + 0x6c (0x565539b3ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x565539b2cc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #43: _PyFunction_Vectorcall + 0x6c (0x565539b3ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x565539b2d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #45: <unknown function> + 0x150582 (0x565539b48582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #46: PyObject_Call + 0xbc (0x565539b48f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x565539b2f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #48: <unknown function> + 0x150582 (0x565539b48582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #49: PyObject_Call + 0xbc (0x565539b48f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x565539b2f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #51: _PyFunction_Vectorcall + 0x6c (0x565539b3ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x565539b35007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #53: _PyObject_Call_Prepend + 0x69 (0x565539b46c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #54: <unknown function> + 0x211239 (0x565539c09239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #55: PyObject_Call + 0x207 (0x565539b49067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x565539b2f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #57: <unknown function> + 0x150582 (0x565539b48582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x565539b2d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #59: <unknown function> + 0x150582 (0x565539b48582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #60: PyObject_Call + 0xbc (0x565539b48f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x565539b2f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #62: <unknown function> + 0x150582 (0x565539b48582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: frame #63: PyObject_Call + 0xbc (0x565539b48f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default4]:[rank4]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default6]:[rank6]: Traceback (most recent call last): | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default6]:[rank6]: trainer.train(dataloader) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default6]:[rank6]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default6]:[rank6]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default6]:[rank6]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default6]:[rank6]: output = model(**micro_batch) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank6]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank6]: return forward_call(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default6]:[rank6]: sharded_logits = self.model( | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank6]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank6]: return forward_call(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default6]:[rank6]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default6]:[rank6]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank6]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank6]: return forward_call(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default6]:[rank6]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default6]:[rank6]: pipeline_state.run_communication() | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default6]:[rank6]: recv_activation_tensor = recv_activation() | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default6]:[rank6]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default6]:[rank6]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default6]:[rank6]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default6]:[rank6]: dist.recv( | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default6]:[rank6]: return func(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default6]:[rank6]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default6]:[rank6]: torch.distributed.DistBackendError: [6] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '5:6', but store->get('5:6') got error: Connection reset by peer | |
| [default6]:[rank6]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default6]:[rank6]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f6d105af897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default5]:[rank5]: Traceback (most recent call last): | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default6]:[rank6]: frame #1: <unknown function> + 0x5b3a23e (0x7f6d4a0cc23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f6d4a0c6c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f6d4a0c6f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f6d4a0c7fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6d4a07c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6d4a07c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: trainer.train(dataloader) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default6]:[rank6]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6d4a07c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6d4a07c371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f6d11889189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:[rank6]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f6d11890610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:[rank6]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f6d118af978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:[rank6]: frame #12: <unknown function> + 0x5adc309 (0x7f6d4a06e309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: frame #13: <unknown function> + 0x5ae6f10 (0x7f6d4a078f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default6]:[rank6]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f6d4a078fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default6]:[rank6]: frame #15: <unknown function> + 0x5124446 (0x7f6d496b6446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: frame #16: <unknown function> + 0x1acf4b8 (0x7f6d460614b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: frame #17: <unknown function> + 0x5aee004 (0x7f6d4a080004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: frame #18: <unknown function> + 0x5af36b5 (0x7f6d4a0856b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank6]: frame #19: <unknown function> + 0xd2631e (0x7f6d5cc6f31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default6]:[rank6]: frame #20: <unknown function> + 0x47def4 (0x7f6d5c3c6ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default6]:[rank6]: frame #21: <unknown function> + 0x1445a6 (0x56221378d5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #22: _PyObject_MakeTpCall + 0x26b (0x562213786a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #23: <unknown function> + 0x150866 (0x562213799866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x562213782142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #25: _PyFunction_Vectorcall + 0x6c (0x56221378da2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #26: PyObject_Call + 0xbc (0x562213799f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5622137802b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default6]:[rank6]: frame #28: _PyFunction_Vectorcall + 0x6c (0x56221378da2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x56221377e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default6]:[rank6]: frame #30: <unknown function> + 0x150582 (0x562213799582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x56221377e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #32: <unknown function> + 0x150582 (0x562213799582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default6]:[rank6]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x56221377e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #34: <unknown function> + 0x150582 (0x562213799582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x56221377e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x562213785f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #37: _PyObject_Call_Prepend + 0x69 (0x562213797c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #38: <unknown function> + 0x211239 (0x56221385a239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #39: _PyObject_MakeTpCall + 0x26b (0x562213786a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5622137823e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #41: _PyFunction_Vectorcall + 0x6c (0x56221378da2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x56221377dc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #43: _PyFunction_Vectorcall + 0x6c (0x56221378da2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: output = model(**micro_batch) | |
| [default6]:[rank6]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x56221377e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank6]: frame #45: <unknown function> + 0x150582 (0x562213799582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #46: PyObject_Call + 0xbc (0x562213799f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5622137802b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #48: <unknown function> + 0x150582 (0x562213799582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #49: PyObject_Call + 0xbc (0x562213799f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5622137802b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #51: _PyFunction_Vectorcall + 0x6c (0x56221378da2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x562213786007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #53: _PyObject_Call_Prepend + 0x69 (0x562213797c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #54: <unknown function> + 0x211239 (0x56221385a239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #55: PyObject_Call + 0x207 (0x56221379a067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5622137802b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #57: <unknown function> + 0x150582 (0x562213799582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x56221377e8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #59: <unknown function> + 0x150582 (0x562213799582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #60: PyObject_Call + 0xbc (0x562213799f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank6]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5622137802b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank5]: return forward_call(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default5]:[rank5]: sharded_logits = self.model( | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank5]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank5]: return forward_call(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default5]:[rank5]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default6]:[rank6]: frame #62: <unknown function> + 0x150582 (0x562213799582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank5]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank5]: return forward_call(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default5]:[rank5]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default6]:[rank6]: frame #63: PyObject_Call + 0xbc (0x562213799f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default6]:[rank6]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default5]:[rank5]: pipeline_state.run_communication() | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default5]:[rank5]: recv_activation_tensor = recv_activation() | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default5]:[rank5]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default5]:[rank5]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default5]:[rank5]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default5]:[rank5]: dist.recv( | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default5]:[rank5]: return func(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default5]:[rank5]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default5]:[rank5]: torch.distributed.DistBackendError: [5] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '4:5', but store->get('4:5') got error: Connection reset by peer | |
| [default5]:[rank5]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default5]:[rank5]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f01960f1897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default5]:[rank5]: frame #1: <unknown function> + 0x5b3a23e (0x7f01cfc0e23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f01cfc08c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f01cfc08f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f01cfc09fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f01cfbbe371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f01cfbbe371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f01cfbbe371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f01cfbbe371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f01973cb189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:[rank5]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f01973d2610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:[rank5]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f01973f1978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:[rank5]: frame #12: <unknown function> + 0x5adc309 (0x7f01cfbb0309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #13: <unknown function> + 0x5ae6f10 (0x7f01cfbbaf10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f01cfbbafa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #15: <unknown function> + 0x5124446 (0x7f01cf1f8446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #16: <unknown function> + 0x1acf4b8 (0x7f01cbba34b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #17: <unknown function> + 0x5aee004 (0x7f01cfbc2004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #18: <unknown function> + 0x5af36b5 (0x7f01cfbc76b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank5]: frame #19: <unknown function> + 0xd2631e (0x7f01e27b131e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default5]:[rank5]: frame #20: <unknown function> + 0x47def4 (0x7f01e1f08ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default5]:[rank5]: frame #21: <unknown function> + 0x1445a6 (0x5649581355a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #22: _PyObject_MakeTpCall + 0x26b (0x56495812ea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #23: <unknown function> + 0x150866 (0x564958141866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x56495812a142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #25: _PyFunction_Vectorcall + 0x6c (0x564958135a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #26: PyObject_Call + 0xbc (0x564958141f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5649581282b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #28: _PyFunction_Vectorcall + 0x6c (0x564958135a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5649581268fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #30: <unknown function> + 0x150582 (0x564958141582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5649581268fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #32: <unknown function> + 0x150582 (0x564958141582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5649581268fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #34: <unknown function> + 0x150582 (0x564958141582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5649581268fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x56495812df50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #37: _PyObject_Call_Prepend + 0x69 (0x56495813fc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #38: <unknown function> + 0x211239 (0x564958202239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #39: _PyObject_MakeTpCall + 0x26b (0x56495812ea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x56495812a3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #41: _PyFunction_Vectorcall + 0x6c (0x564958135a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x564958125c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #43: _PyFunction_Vectorcall + 0x6c (0x564958135a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5649581268fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #45: <unknown function> + 0x150582 (0x564958141582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #46: PyObject_Call + 0xbc (0x564958141f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5649581282b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #48: <unknown function> + 0x150582 (0x564958141582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #49: PyObject_Call + 0xbc (0x564958141f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5649581282b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #51: _PyFunction_Vectorcall + 0x6c (0x564958135a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x56495812e007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #53: _PyObject_Call_Prepend + 0x69 (0x56495813fc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #54: <unknown function> + 0x211239 (0x564958202239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #55: PyObject_Call + 0x207 (0x564958142067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5649581282b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #57: <unknown function> + 0x150582 (0x564958141582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5649581268fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #59: <unknown function> + 0x150582 (0x564958141582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #60: PyObject_Call + 0xbc (0x564958141f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5649581282b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #62: <unknown function> + 0x150582 (0x564958141582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: frame #63: PyObject_Call + 0xbc (0x564958141f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank5]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default7]:[rank7]: Traceback (most recent call last): | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default7]:[rank7]: trainer.train(dataloader) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default2]:[rank2]: Traceback (most recent call last): | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default2]:[rank2]: trainer.train(dataloader) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default2]:[rank2]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default2]:[rank2]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default2]:[rank2]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default2]:[rank2]: output = model(**micro_batch) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank7]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default7]:[rank7]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default2]:[rank2]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank2]: return forward_call(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default2]:[rank2]: sharded_logits = self.model( | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank2]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank2]: return forward_call(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default2]:[rank2]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default2]:[rank2]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default7]:[rank7]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default7]:[rank7]: output = model(**micro_batch) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank7]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank2]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank2]: return forward_call(*args, **kwargs) | |
| [default7]:[rank7]: return forward_call(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default7]:[rank7]: sharded_logits = self.model( | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank7]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank2]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank7]: return forward_call(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default7]:[rank7]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 782, in forward_with_hidden_states | |
| [default7]:[rank7]: hidden_states = self.final_layer_norm(input=hidden_encoder_states["hidden_states"])["hidden_states"] | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank7]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default2]:[rank2]: pipeline_state.run_communication() | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default2]:[rank2]: recv_activation_tensor = recv_activation() | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default2]:[rank2]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default2]:[rank2]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default7]:[rank7]: return forward_call(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default2]:[rank2]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default7]:[rank7]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default2]:[rank2]: dist.recv( | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default2]:[rank2]: return func(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default2]:[rank2]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default7]:[rank7]: pipeline_state.run_communication() | |
| [default2]:[rank2]: torch.distributed.DistBackendError: [2] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '1:2', but store->get('1:2') got error: Connection reset by peer | |
| [default2]:[rank2]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default7]:[rank7]: recv_activation_tensor = recv_activation() | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default7]:[rank7]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default7]:[rank7]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default7]:[rank7]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default2]:[rank2]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f8c2f221897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default7]:[rank7]: dist.recv( | |
| [default2]:[rank2]: frame #1: <unknown function> + 0x5b3a23e (0x7f8c68d3e23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default7]:[rank7]: return func(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default7]:[rank7]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default7]:[rank7]: torch.distributed.DistBackendError: [7] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '6:7', but store->get('6:7') got error: Connection reset by peer | |
| [default7]:[rank7]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default2]:[rank2]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f8c68d38c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f718e523897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default2]:[rank2]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f8c68d38f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #1: <unknown function> + 0x5b3a23e (0x7f71c804023e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank2]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f8c68d39fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f71c803ac87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank2]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f8c68cee371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f71c803af82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f71c803bfd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f71c7ff0371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank2]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f8c68cee371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank2]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f8c68cee371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f71c7ff0371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank2]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f8c68cee371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank2]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f8c304fb189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:[rank2]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f8c30502610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:[rank2]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f8c30521978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:[rank2]: frame #12: <unknown function> + 0x5adc309 (0x7f8c68ce0309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank2]: frame #13: <unknown function> + 0x5ae6f10 (0x7f8c68ceaf10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f71c7ff0371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f71c7ff0371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f718f7fd189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:[rank7]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f718f804610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:[rank7]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f718f823978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:[rank7]: frame #12: <unknown function> + 0x5adc309 (0x7f71c7fe2309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank2]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f8c68ceafa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #13: <unknown function> + 0x5ae6f10 (0x7f71c7fecf10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank2]: frame #15: <unknown function> + 0x5124446 (0x7f8c68328446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank2]: frame #16: <unknown function> + 0x1acf4b8 (0x7f8c64cd34b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f71c7fecfa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank2]: frame #17: <unknown function> + 0x5aee004 (0x7f8c68cf2004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #15: <unknown function> + 0x5124446 (0x7f71c762a446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #16: <unknown function> + 0x1acf4b8 (0x7f71c3fd54b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #17: <unknown function> + 0x5aee004 (0x7f71c7ff4004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #18: <unknown function> + 0x5af36b5 (0x7f71c7ff96b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #19: <unknown function> + 0xd2631e (0x7f71dabe331e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default2]:[rank2]: frame #18: <unknown function> + 0x5af36b5 (0x7f8c68cf76b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank7]: frame #20: <unknown function> + 0x47def4 (0x7f71da33aef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default7]:[rank7]: frame #21: <unknown function> + 0x1445a6 (0x55ed3732b5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #19: <unknown function> + 0xd2631e (0x7f8c7b8e131e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default7]:[rank7]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55ed37324a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #20: <unknown function> + 0x47def4 (0x7f8c7b038ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default7]:[rank7]: frame #23: <unknown function> + 0x150866 (0x55ed37337866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #21: <unknown function> + 0x1445a6 (0x5574cb2475a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55ed37320142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #22: _PyObject_MakeTpCall + 0x26b (0x5574cb240a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #23: <unknown function> + 0x150866 (0x5574cb253866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x5574cb23c142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55ed3732ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #26: PyObject_Call + 0xbc (0x55ed37337f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #25: _PyFunction_Vectorcall + 0x6c (0x5574cb247a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55ed3731e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55ed3732ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55ed3731c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #30: <unknown function> + 0x150582 (0x55ed37337582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #26: PyObject_Call + 0xbc (0x5574cb253f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5574cb23a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #28: _PyFunction_Vectorcall + 0x6c (0x5574cb247a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5574cb2388fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #30: <unknown function> + 0x150582 (0x5574cb253582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5574cb2388fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55ed3731c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #32: <unknown function> + 0x150582 (0x5574cb253582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5574cb2388fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #34: <unknown function> + 0x150582 (0x5574cb253582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5574cb2388fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x5574cb23ff50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #37: _PyObject_Call_Prepend + 0x69 (0x5574cb251c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #38: <unknown function> + 0x211239 (0x5574cb314239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #39: _PyObject_MakeTpCall + 0x26b (0x5574cb240a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5574cb23c3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #41: _PyFunction_Vectorcall + 0x6c (0x5574cb247a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #32: <unknown function> + 0x150582 (0x55ed37337582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55ed3731c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #34: <unknown function> + 0x150582 (0x55ed37337582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x5574cb237c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55ed3731c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55ed37323f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #43: _PyFunction_Vectorcall + 0x6c (0x5574cb247a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55ed37335c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5574cb2388fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #45: <unknown function> + 0x150582 (0x5574cb253582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #46: PyObject_Call + 0xbc (0x5574cb253f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #38: <unknown function> + 0x211239 (0x55ed373f8239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55ed37324a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5574cb23a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55ed373203e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55ed3732ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55ed3731bc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55ed3732ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55ed3731c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #45: <unknown function> + 0x150582 (0x55ed37337582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #46: PyObject_Call + 0xbc (0x55ed37337f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55ed3731e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #48: <unknown function> + 0x150582 (0x5574cb253582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #49: PyObject_Call + 0xbc (0x5574cb253f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5574cb23a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #48: <unknown function> + 0x150582 (0x55ed37337582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #49: PyObject_Call + 0xbc (0x55ed37337f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55ed3731e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #51: _PyFunction_Vectorcall + 0x6c (0x5574cb247a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55ed3732ba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55ed37324007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55ed37335c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x5574cb240007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #54: <unknown function> + 0x211239 (0x55ed373f8239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #53: _PyObject_Call_Prepend + 0x69 (0x5574cb251c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #55: _PyObject_MakeTpCall + 0x26b (0x55ed37324a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #54: <unknown function> + 0x211239 (0x5574cb314239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #56: _PyEval_EvalFrameDefault + 0x5723 (0x55ed37320c53 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #57: <unknown function> + 0x150582 (0x55ed37337582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55ed3731c8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #59: <unknown function> + 0x150582 (0x55ed37337582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #60: PyObject_Call + 0xbc (0x55ed37337f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #55: PyObject_Call + 0x207 (0x5574cb254067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55ed3731e2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5574cb23a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #62: <unknown function> + 0x150582 (0x55ed37337582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #57: <unknown function> + 0x150582 (0x5574cb253582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: frame #63: PyObject_Call + 0xbc (0x55ed37337f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5574cb2388fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #59: <unknown function> + 0x150582 (0x5574cb253582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #60: PyObject_Call + 0xbc (0x5574cb253f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank7]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default2]:[rank2]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5574cb23a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #62: <unknown function> + 0x150582 (0x5574cb253582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: frame #63: PyObject_Call + 0xbc (0x5574cb253f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank2]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| W0704 00:06:34.813000 140695680661312 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 135980 closing signal SIGTERM | |
| W0704 00:06:34.813000 140695680661312 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 135981 closing signal SIGTERM | |
| W0704 00:06:34.814000 140695680661312 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 135982 closing signal SIGTERM | |
| W0704 00:06:34.814000 140695680661312 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 135983 closing signal SIGTERM | |
| W0704 00:06:34.815000 140695680661312 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 135984 closing signal SIGTERM | |
| W0704 00:06:34.816000 140695680661312 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 135985 closing signal SIGTERM | |
| W0704 00:06:34.816000 140695680661312 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 135986 closing signal SIGTERM | |
| E0704 00:06:35.940000 140695680661312 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 135979) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 | |
| Traceback (most recent call last): | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module> | |
| sys.exit(main()) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper | |
| return f(*args, **kwargs) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main | |
| run(args) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run | |
| elastic_launch( | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__ | |
| return launch_agent(self._config, self._entrypoint, list(args)) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent | |
| raise ChildFailedError( | |
| torch.distributed.elastic.multiprocessing.errors.ChildFailedError: | |
| ============================================================ | |
| /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED | |
| ------------------------------------------------------------ | |
| Failures: | |
| <NO_OTHER_FAILURES> | |
| ------------------------------------------------------------ | |
| Root Cause (first observed failure): | |
| [0]: | |
| time : 2024-07-04_00:06:34 | |
| host : ip-26-0-169-247.ec2.internal | |
| rank : 0 (local_rank: 0) | |
| exitcode : 1 (pid: 135979) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| ============================================================ | |
| srun: error: ip-26-0-169-247: task 0: Exited with exit code 1 | |
| Consider using `hf_transfer` for faster uploads. This solution comes with some limitations. See https://huggingface.co/docs/huggingface_hub/hf_transfer for more details. | |