| ======================== | |
| START TIME: Tue Jul 2 18:30:58 UTC 2024 | |
| python3 version = Python 3.10.14 | |
| ======================== | |
| The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well. | |
| Token is valid (permission: write). | |
| Your token has been saved to /admin/home/ferdinand_mom/.cache/huggingface/token | |
| Login successful | |
| Already on 'bench_cluster' | |
| M examples/config_tiny_llama.py | |
| M examples/config_tiny_llama.yaml | |
| M examples/train_tiny_llama.sh | |
| M src/nanotron/models/llama.py | |
| M src/nanotron/trainer.py | |
| Your branch is up to date with 'origin/bench_cluster'. | |
| Job status: RUNNING | |
| W0702 18:31:01.254000 140390681818944 torch/distributed/run.py:757] | |
| W0702 18:31:01.254000 140390681818944 torch/distributed/run.py:757] ***************************************** | |
| W0702 18:31:01.254000 140390681818944 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. | |
| W0702 18:31:01.254000 140390681818944 torch/distributed/run.py:757] ***************************************** | |
| W0702 18:31:01.268000 140027356682048 torch/distributed/run.py:757] | |
| W0702 18:31:01.268000 140027356682048 torch/distributed/run.py:757] ***************************************** | |
| W0702 18:31:01.268000 140027356682048 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. | |
| W0702 18:31:01.268000 140027356682048 torch/distributed/run.py:757] ***************************************** | |
| [default0]:07/02/2024 18:31:19 [WARNING|DP=0|PP=0|TP=0|ip-26-0-163-226]: [Vocab Size Padding] Padded vocab (size: 50257) with 1 dummy tokens (new size: 50258) | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: Config: | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: Config(general=GeneralArgs(project='bench_cluster', | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: run='%date_%jobid', | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: seed=42, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: step=None, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: consumed_train_samples=None, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: benchmark_csv_path=None, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: ignore_sanity_checks=True), | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: parallelism=ParallelismArgs(dp=1, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: pp=8, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: tp=2, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: pp_engine=<nanotron.parallel.pipeline_parallel.engine.OneForwardOneBackwardPipelineEngine object at 0x7efba2fb4910>, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: tp_mode=<TensorParallelLinearMode.REDUCE_SCATTER: 2>, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: tp_linear_async_communication=False, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: expert_parallel_size=1), | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: model=ModelArgs(model_config=LlamaConfig(bos_token_id=1, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: eos_token_id=2, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: hidden_act='silu', | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: hidden_size=2048, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: initializer_range=0.02, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: intermediate_size=4096, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: is_llama_config=True, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: max_position_embeddings=4096, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: num_attention_heads=32, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: num_hidden_layers=24, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: num_key_value_heads=32, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: pad_token_id=None, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: pretraining_tp=1, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: rms_norm_eps=1e-05, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: rope_scaling=None, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: rope_theta=10000.0, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: tie_word_embeddings=True, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: use_cache=True, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: vocab_size=50258), | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: init_method=RandomInit(std=0.025), | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: dtype=torch.bfloat16, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: make_vocab_size_divisible_by=1, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: ddp_bucket_cap_mb=25), | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: tokenizer=TokenizerArgs(tokenizer_name_or_path='openai-community/gpt2', | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: tokenizer_revision=None, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: tokenizer_max_length=None), | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: checkpoints=CheckpointsArgs(checkpoints_path=Path('/dev/null'), | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: checkpoint_interval=100000, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: save_initial_state=False, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: resume_checkpoint_path=None, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: checkpoints_path_is_shared_file_system=False), | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: logging=LoggingArgs(log_level='info', | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: log_level_replica='info', | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: iteration_step_info_interval=1), | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: tokens=TokensArgs(sequence_length=4096, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: train_steps=20, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: micro_batch_size=1, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: batch_accumulation_per_replica=1024, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: val_check_interval=-1, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: limit_val_batches=0, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: limit_test_batches=0), | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: optimizer=OptimizerArgs(optimizer_factory=AdamWOptimizerArgs(adam_eps=1e-08, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: adam_beta1=0.9, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: adam_beta2=0.95, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: torch_adam_is_fused=True, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: name='adamW'), | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: zero_stage=1, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: weight_decay=0.01, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: clip_grad=1.0, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: accumulate_grad_in_fp32=True, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: learning_rate_scheduler=LRSchedulerArgs(learning_rate=0.0001, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: lr_warmup_steps=1, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: lr_warmup_style='linear', | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: lr_decay_style='linear', | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: lr_decay_steps=19, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: lr_decay_starting_step=None, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: min_decay_lr=1e-05)), | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: data_stages=[DatasetStageArgs(name='Training Stage', | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: start_training_step=1, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: data=DataArgs(dataset=PretrainDatasetsArgs(hf_dataset_or_datasets='roneneldan/TinyStories', | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: hf_dataset_splits='train', | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: hf_dataset_config_name=None, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: dataset_processing_num_proc_per_process=64, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: dataset_overwrite_cache=False, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: text_column_name='text'), | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: seed=42, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: num_loading_workers=32))], | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: profiler=ProfilerArgs(profiler_export_path=Path('/fsx/ferdinandmom/ferdinand-hf/bench_cluster/results/llama-1B/16_GPUS/dp-1_tp-2_pp-8_mbz-1')), | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: lighteval=None) | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: Model Config: | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: LlamaConfig(bos_token_id=1, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: eos_token_id=2, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: hidden_act='silu', | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: hidden_size=2048, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: initializer_range=0.02, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: intermediate_size=4096, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: is_llama_config=True, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: max_position_embeddings=4096, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: num_attention_heads=32, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: num_hidden_layers=24, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: num_key_value_heads=32, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: pad_token_id=None, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: pretraining_tp=1, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: rms_norm_eps=1e-05, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: rope_scaling=None, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: rope_theta=10000.0, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: tie_word_embeddings=True, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: use_cache=True, | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: vocab_size=50258) | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: Building model.. | |
| [default0]:07/02/2024 18:31:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: Setting PP block ranks... | |
| [default3]:07/02/2024 18:31:34 [INFO|DP=0|PP=1|TP=1|ip-26-0-163-226]: Local number of parameters: 62.9M (120.02MiB) | |
| [default3]:07/02/2024 18:31:34 [INFO|DP=0|PP=1|TP=1|ip-26-0-163-226]: [After model building] Memory usage: 123.03MiB. Peak allocated: 125.06MiB Peak reserved: 138.00MiB | |
| [default3]:07/02/2024 18:31:34 [INFO|DP=0|PP=1|TP=1|ip-26-0-163-226]: No checkpoint path provided. | |
| [default1]:07/02/2024 18:31:34 [INFO|DP=0|PP=0|TP=1|ip-26-0-163-226]: Local number of parameters: 135M (258.19MiB) | |
| [default1]:07/02/2024 18:31:34 [INFO|DP=0|PP=0|TP=1|ip-26-0-163-226]: [After model building] Memory usage: 262.20MiB. Peak allocated: 264.23MiB Peak reserved: 280.00MiB | |
| [default1]:07/02/2024 18:31:34 [INFO|DP=0|PP=0|TP=1|ip-26-0-163-226]: No checkpoint path provided. | |
| [default2]:07/02/2024 18:31:34 [INFO|DP=0|PP=1|TP=0|ip-26-0-163-226]: Local number of parameters: 62.9M (120.02MiB) | |
| [default2]:07/02/2024 18:31:34 [INFO|DP=0|PP=1|TP=0|ip-26-0-163-226]: [After model building] Memory usage: 123.03MiB. Peak allocated: 125.06MiB Peak reserved: 138.00MiB | |
| [default2]:07/02/2024 18:31:34 [INFO|DP=0|PP=1|TP=0|ip-26-0-163-226]: No checkpoint path provided. | |
| [default0]:07/02/2024 18:31:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: Total number of parameters: 1.21G (2313.02MiB) | |
| [default0]:07/02/2024 18:31:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: Local number of parameters: 135M (258.19MiB) | |
| [default0]:07/02/2024 18:31:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: [After model building] Memory usage: 262.20MiB. Peak allocated: 264.23MiB Peak reserved: 280.00MiB | |
| [default0]:07/02/2024 18:31:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: No checkpoint path provided. | |
| [default0]:07/02/2024 18:31:34 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: Parametrizing model parameters using StandardParametrizator | |
| [default7]:07/02/2024 18:31:34 [INFO|DP=0|PP=3|TP=1|ip-26-0-163-226]: Local number of parameters: 83.9M (160.03MiB) | |
| [default7]:07/02/2024 18:31:34 [INFO|DP=0|PP=3|TP=1|ip-26-0-163-226]: [After model building] Memory usage: 164.04MiB. Peak allocated: 166.07MiB Peak reserved: 180.00MiB | |
| [default7]:07/02/2024 18:31:34 [INFO|DP=0|PP=3|TP=1|ip-26-0-163-226]: No checkpoint path provided. | |
| [default5]:07/02/2024 18:31:34 [INFO|DP=0|PP=2|TP=1|ip-26-0-163-226]: Local number of parameters: 62.9M (120.02MiB) | |
| [default5]:07/02/2024 18:31:34 [INFO|DP=0|PP=2|TP=1|ip-26-0-163-226]: [After model building] Memory usage: 123.03MiB. Peak allocated: 125.06MiB Peak reserved: 138.00MiB | |
| [default5]:07/02/2024 18:31:34 [INFO|DP=0|PP=2|TP=1|ip-26-0-163-226]: No checkpoint path provided. | |
| [default6]:07/02/2024 18:31:34 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-226]: Local number of parameters: 83.9M (160.03MiB) | |
| [default6]:07/02/2024 18:31:34 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-226]: [After model building] Memory usage: 164.04MiB. Peak allocated: 166.07MiB Peak reserved: 180.00MiB | |
| [default6]:07/02/2024 18:31:34 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-226]: No checkpoint path provided. | |
| [default4]:07/02/2024 18:31:34 [INFO|DP=0|PP=2|TP=0|ip-26-0-163-226]: Local number of parameters: 62.9M (120.02MiB) | |
| [default4]:07/02/2024 18:31:34 [INFO|DP=0|PP=2|TP=0|ip-26-0-163-226]: [After model building] Memory usage: 123.03MiB. Peak allocated: 125.06MiB Peak reserved: 138.00MiB | |
| [default4]:07/02/2024 18:31:34 [INFO|DP=0|PP=2|TP=0|ip-26-0-163-226]: No checkpoint path provided. | |
| [default2]:07/02/2024 18:31:34 [INFO|DP=0|PP=5|TP=0|ip-26-0-172-73]: Local number of parameters: 62.9M (120.02MiB) | |
| [default2]:07/02/2024 18:31:34 [INFO|DP=0|PP=5|TP=0|ip-26-0-172-73]: [After model building] Memory usage: 123.03MiB. Peak allocated: 125.06MiB Peak reserved: 138.00MiB | |
| [default0]:07/02/2024 18:31:34 [INFO|DP=0|PP=4|TP=0|ip-26-0-172-73]: Local number of parameters: 62.9M (120.02MiB) | |
| [default0]:07/02/2024 18:31:34 [INFO|DP=0|PP=4|TP=0|ip-26-0-172-73]: [After model building] Memory usage: 123.03MiB. Peak allocated: 125.06MiB Peak reserved: 138.00MiB | |
| [default0]:07/02/2024 18:31:34 [INFO|DP=0|PP=4|TP=0|ip-26-0-172-73]: No checkpoint path provided. | |
| [default4]:07/02/2024 18:31:34 [INFO|DP=0|PP=6|TP=0|ip-26-0-172-73]: Local number of parameters: 83.9M (160.03MiB) | |
| [default4]:07/02/2024 18:31:34 [INFO|DP=0|PP=6|TP=0|ip-26-0-172-73]: [After model building] Memory usage: 164.04MiB. Peak allocated: 166.07MiB Peak reserved: 180.00MiB | |
| [default4]:07/02/2024 18:31:34 [INFO|DP=0|PP=6|TP=0|ip-26-0-172-73]: No checkpoint path provided. | |
| [default2]:07/02/2024 18:31:34 [INFO|DP=0|PP=5|TP=0|ip-26-0-172-73]: No checkpoint path provided. | |
| [default6]:07/02/2024 18:31:34 [INFO|DP=0|PP=7|TP=0|ip-26-0-172-73]: Local number of parameters: 51.5M (98.16MiB) | |
| [default6]:07/02/2024 18:31:34 [INFO|DP=0|PP=7|TP=0|ip-26-0-172-73]: [After model building] Memory usage: 98.17MiB. Peak allocated: 98.18MiB Peak reserved: 102.00MiB | |
| [default1]:07/02/2024 18:31:34 [INFO|DP=0|PP=4|TP=1|ip-26-0-172-73]: Local number of parameters: 62.9M (120.02MiB) | |
| [default1]:07/02/2024 18:31:34 [INFO|DP=0|PP=4|TP=1|ip-26-0-172-73]: [After model building] Memory usage: 123.03MiB. Peak allocated: 125.06MiB Peak reserved: 138.00MiB | |
| [default1]:07/02/2024 18:31:34 [INFO|DP=0|PP=4|TP=1|ip-26-0-172-73]: No checkpoint path provided. | |
| [default6]:07/02/2024 18:31:34 [INFO|DP=0|PP=7|TP=0|ip-26-0-172-73]: No checkpoint path provided. | |
| [default5]:07/02/2024 18:31:34 [INFO|DP=0|PP=6|TP=1|ip-26-0-172-73]: Local number of parameters: 83.9M (160.03MiB) | |
| [default5]:07/02/2024 18:31:34 [INFO|DP=0|PP=6|TP=1|ip-26-0-172-73]: [After model building] Memory usage: 164.04MiB. Peak allocated: 166.07MiB Peak reserved: 180.00MiB | |
| [default5]:07/02/2024 18:31:34 [INFO|DP=0|PP=6|TP=1|ip-26-0-172-73]: No checkpoint path provided. | |
| [default3]:07/02/2024 18:31:34 [INFO|DP=0|PP=5|TP=1|ip-26-0-172-73]: Local number of parameters: 62.9M (120.02MiB) | |
| [default3]:07/02/2024 18:31:34 [INFO|DP=0|PP=5|TP=1|ip-26-0-172-73]: [After model building] Memory usage: 123.03MiB. Peak allocated: 125.06MiB Peak reserved: 138.00MiB | |
| [default3]:07/02/2024 18:31:34 [INFO|DP=0|PP=5|TP=1|ip-26-0-172-73]: No checkpoint path provided. | |
| [default7]:07/02/2024 18:31:34 [INFO|DP=0|PP=7|TP=1|ip-26-0-172-73]: Local number of parameters: 51.5M (98.16MiB) | |
| [default7]:07/02/2024 18:31:34 [INFO|DP=0|PP=7|TP=1|ip-26-0-172-73]: [After model building] Memory usage: 98.17MiB. Peak allocated: 98.18MiB Peak reserved: 102.00MiB | |
| [default7]:07/02/2024 18:31:34 [INFO|DP=0|PP=7|TP=1|ip-26-0-172-73]: No checkpoint path provided. | |
| [default0]:07/02/2024 18:31:36 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: [Optimizer Building] Using LearningRateForSP as learning rate | |
| [default0]:07/02/2024 18:31:36 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: [ZeRO sharding] Size of optimizer params per rank: | |
| [default0]:07/02/2024 18:31:36 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: [ZeRO sharding] DP Rank 0 has 135M out of 135M (100.00%) params' optimizer states | |
| [default0]:07/02/2024 18:31:36 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: [Training Plan] Stage Training Stage has 19 remaining training steps and has consumed 0 samples | |
| [default0]:07/02/2024 18:31:36 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: Using `datasets` library | |
| [default0]:07/02/2024 18:31:36 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: Loading tokenizer from openai-community/gpt2 and transformers/hf_hub versions ('4.41.2', '0.23.4') | |
| [default0]:07/02/2024 18:31:37 [WARNING|DP=0|PP=0|TP=0|ip-26-0-163-226]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default0]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default0]:07/02/2024 18:31:37 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: [Training Plan] There are 1 training stages | |
| [default0]:07/02/2024 18:31:37 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: [Stage Training Stage] start from step 1 | |
| [default0]:07/02/2024 18:31:37 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: | |
| [default0]:07/02/2024 18:31:37 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: [Start training] datetime: 2024-07-02 18:31:37.764476 | mbs: 1 | grad_accum: 1024 | global_batch_size: 1024 | sequence_length: 4096 | train_steps: 20 | start_iteration_step: 0 | consumed_train_samples: 0 | |
| [default0]:07/02/2024 18:31:37 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: Resuming training from stage Training Stage, it has trained for 0 samples and has 19 remaining train steps | |
| [default0]:07/02/2024 18:31:37 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: Memory usage: 1294.97MiB. Peak allocated 1294.97MiB. Peak reserved: 1316.00MiB | |
| [default3]:07/02/2024 18:31:37 [WARNING|DP=0|PP=1|TP=1|ip-26-0-163-226]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default1]:07/02/2024 18:31:37 [WARNING|DP=0|PP=0|TP=1|ip-26-0-163-226]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default2]:07/02/2024 18:31:37 [WARNING|DP=0|PP=1|TP=0|ip-26-0-163-226]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:07/02/2024 18:31:37 [WARNING|DP=0|PP=3|TP=1|ip-26-0-163-226]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default4]:07/02/2024 18:31:37 [WARNING|DP=0|PP=2|TP=0|ip-26-0-163-226]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default6]:07/02/2024 18:31:37 [WARNING|DP=0|PP=3|TP=0|ip-26-0-163-226]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default5]:07/02/2024 18:31:37 [WARNING|DP=0|PP=2|TP=1|ip-26-0-163-226]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default6]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default5]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default4]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default1]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default4]:07/02/2024 18:31:37 [WARNING|DP=0|PP=6|TP=0|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default1]:07/02/2024 18:31:37 [WARNING|DP=0|PP=4|TP=1|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default3]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default0]:07/02/2024 18:31:37 [WARNING|DP=0|PP=4|TP=0|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default2]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default6]:07/02/2024 18:31:37 [WARNING|DP=0|PP=7|TP=0|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default2]:07/02/2024 18:31:37 [WARNING|DP=0|PP=5|TP=0|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default5]:07/02/2024 18:31:37 [WARNING|DP=0|PP=6|TP=1|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default3]:07/02/2024 18:31:37 [WARNING|DP=0|PP=5|TP=1|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default6]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default2]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default0]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default5]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default1]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default3]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default4]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:07/02/2024 18:31:38 [WARNING|DP=0|PP=7|TP=1|ip-26-0-172-73]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default6]:07/02/2024 18:31:53 [WARNING|DP=0|PP=7|TP=0|ip-26-0-172-73]: Using the latest cached version of the dataset since roneneldan/TinyStories couldn't be found on the Hugging Face Hub | |
| [default6]:07/02/2024 18:31:53 [WARNING|DP=0|PP=7|TP=0|ip-26-0-172-73]: Found the latest cached dataset configuration 'default' at /admin/home/ferdinand_mom/.cache/roneneldan___tiny_stories/default/0.0.0/691b0d9bd48ade766778c940011ca1c549f6359b (last modified on Mon Jun 24 07:59:52 2024). | |
| [default6]:Using the latest cached version of the dataset since roneneldan/TinyStories couldn't be found on the Hugging Face Hub | |
| [default6]:Found the latest cached dataset configuration 'default' at /admin/home/ferdinand_mom/.cache/roneneldan___tiny_stories/default/0.0.0/691b0d9bd48ade766778c940011ca1c549f6359b (last modified on Mon Jun 24 07:59:52 2024). | |
| [default6]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default6]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default7]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default7]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default4]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default4]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default5]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default5]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default2]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default2]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default3]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default3]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default1]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default1]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default0]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default0]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default6]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default6]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default7]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default7]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default4]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default4]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default5]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default5]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default2]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default2]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default3]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default3]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default0]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default0]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default1]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default1]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default4]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions | |
| [default4]: warnings.warn( | |
| [default6]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions | |
| [default6]: warnings.warn( | |
| [default7]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions | |
| [default7]: warnings.warn( | |
| [default3]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions | |
| [default3]: warnings.warn( | |
| [default2]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions | |
| [default2]: warnings.warn( | |
| [default5]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions | |
| [default5]: warnings.warn( | |
| [default6]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions | |
| [default6]: warnings.warn( | |
| [default4]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions | |
| [default4]: warnings.warn( | |
| [default7]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions | |
| [default7]: warnings.warn( | |
| [default5]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions | |
| [default5]: warnings.warn( | |
| [default3]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions | |
| [default3]: warnings.warn( | |
| [default0]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions | |
| [default0]: warnings.warn( | |
| [default1]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions | |
| [default1]: warnings.warn( | |
| [default2]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions | |
| [default2]: warnings.warn( | |
| [default0]:07/02/2024 18:33:58 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: Memory usage: 1361.00MiB. Peak allocated 6703.59MiB. Peak reserved: 6804.00MiB | |
| [default1]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions | |
| [default1]: warnings.warn( | |
| [default0]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions | |
| [default0]: warnings.warn( | |
| [default0]:07/02/2024 18:34:06 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: Memory usage: 2393.78MiB. Peak allocated 2393.78MiB. Peak reserved: 7002.00MiB | |
| [default6]:07/02/2024 18:34:06 [INFO|DP=0|PP=7|TP=0|ip-26-0-172-73]: iteration: 1 / 20 | consumed_tokens: 4.19M | elapsed_time_per_iteration_ms: 133K | tokens_per_sec: 31.4K | tokens_per_sec_per_gpu: 1.96K | global_batch_size: 1.02K | lm_loss: 11.2 | lr: 0.0001 | model_tflops_per_gpu: 17.8 | hardware_tflops_per_gpu: 17.8 | grad_norm: 17.8 | cuda_memory_allocated: 994M | cuda_max_memory_reserved: 2.04G | hd_total_memory_tb: 312G | hd_used_memory_tb: 65.5G | hd_free_memory_tb: 247G | |
| [default0]:07/02/2024 18:35:43 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: Memory usage: 2393.78MiB. Peak allocated 7570.61MiB. Peak reserved: 7778.00MiB | |
| [default0]:07/02/2024 18:35:43 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: Memory usage: 2393.78MiB. Peak allocated 2393.80MiB. Peak reserved: 7778.00MiB | |
| [default6]:07/02/2024 18:35:43 [INFO|DP=0|PP=7|TP=0|ip-26-0-172-73]: iteration: 2 / 20 | consumed_tokens: 8.39M | elapsed_time_per_iteration_ms: 96.4K | tokens_per_sec: 43.5K | tokens_per_sec_per_gpu: 2.72K | global_batch_size: 1.02K | lm_loss: 11.2 | lr: 9.53e-05 | model_tflops_per_gpu: 24.7 | hardware_tflops_per_gpu: 24.7 | grad_norm: 17.8 | cuda_memory_allocated: 994M | cuda_max_memory_reserved: 2.04G | hd_total_memory_tb: 312G | hd_used_memory_tb: 65.5G | hd_free_memory_tb: 247G | |
| [default0]:07/02/2024 18:37:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: Memory usage: 2393.78MiB. Peak allocated 7570.61MiB. Peak reserved: 7850.00MiB | |
| [default0]:07/02/2024 18:37:23 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: Memory usage: 2393.78MiB. Peak allocated 2393.80MiB. Peak reserved: 7850.00MiB | |
| [default0]:STAGE:2024-07-02 18:37:23 2980592:2980592 ActivityProfilerController.cpp:314] Completed Stage: Warm Up | |
| [default6]:07/02/2024 18:37:23 [INFO|DP=0|PP=7|TP=0|ip-26-0-172-73]: iteration: 3 / 20 | consumed_tokens: 12.6M | elapsed_time_per_iteration_ms: 100K | tokens_per_sec: 42K | tokens_per_sec_per_gpu: 2.62K | global_batch_size: 1.02K | lm_loss: 9.62 | lr: 9.05e-05 | model_tflops_per_gpu: 23.8 | hardware_tflops_per_gpu: 23.8 | grad_norm: 21.7 | cuda_memory_allocated: 994M | cuda_max_memory_reserved: 2.04G | hd_total_memory_tb: 312G | hd_used_memory_tb: 65.5G | hd_free_memory_tb: 247G | |
| [default0]:07/02/2024 18:39:00 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: Memory usage: 2393.78MiB. Peak allocated 7570.61MiB. Peak reserved: 7850.00MiB | |
| [default0]:07/02/2024 18:39:00 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: Memory usage: 2393.78MiB. Peak allocated 2393.80MiB. Peak reserved: 7850.00MiB | |
| [default6]:07/02/2024 18:39:00 [INFO|DP=0|PP=7|TP=0|ip-26-0-172-73]: iteration: 4 / 20 | consumed_tokens: 16.8M | elapsed_time_per_iteration_ms: 97.2K | tokens_per_sec: 43.1K | tokens_per_sec_per_gpu: 2.7K | global_batch_size: 1.02K | lm_loss: 10.4 | lr: 8.58e-05 | model_tflops_per_gpu: 24.5 | hardware_tflops_per_gpu: 24.5 | grad_norm: 45.6 | cuda_memory_allocated: 994M | cuda_max_memory_reserved: 2.04G | hd_total_memory_tb: 312G | hd_used_memory_tb: 65.5G | hd_free_memory_tb: 247G | |
| [default0]:07/02/2024 18:40:27 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-226]: Memory usage: 2393.78MiB. Peak allocated 7570.61MiB. Peak reserved: 7850.00MiB | |
| [default6]:07/02/2024 18:40:27 [INFO|DP=0|PP=7|TP=0|ip-26-0-172-73]: iteration: 5 / 20 | consumed_tokens: 21M | elapsed_time_per_iteration_ms: 87.3K | tokens_per_sec: 48K | tokens_per_sec_per_gpu: 3K | global_batch_size: 1.02K | lm_loss: 9.43 | lr: 8.11e-05 | model_tflops_per_gpu: 27.2 | hardware_tflops_per_gpu: 27.2 | grad_norm: 11.2 | |
| [default6]:07/02/2024 18:41:59 [INFO|DP=0|PP=7|TP=0|ip-26-0-172-73]: iteration: 6 / 20 | consumed_tokens: 25.2M | elapsed_time_per_iteration_ms: 91.7K | tokens_per_sec: 45.8K | tokens_per_sec_per_gpu: 2.86K | global_batch_size: 1.02K | lm_loss: 9.37 | lr: 7.63e-05 | model_tflops_per_gpu: 25.9 | hardware_tflops_per_gpu: 25.9 | grad_norm: 7.68 | |
| [default0]:STAGE:2024-07-02 18:43:52 2980592:2980592 ActivityProfilerController.cpp:320] Completed Stage: Collection | |
| [default0]:STAGE:2024-07-02 18:44:10 2980592:2980592 ActivityProfilerController.cpp:324] Completed Stage: Post Processing | |
| [default7]:[rank15]:[E ProcessGroupNCCL.cpp:563] [Rank 7] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=36867, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600002 milliseconds before timing out. | |
| [default6]:[rank14]:[E ProcessGroupNCCL.cpp:563] [Rank 7] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=36867, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600028 milliseconds before timing out. | |
| [default1]:[rank1]:[E ProcessGroupNCCL.cpp:563] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=208913, OpType=_REDUCE_SCATTER_BASE, NumelIn=8388608, NumelOut=4194304, Timeout(ms)=600000) ran for 600060 milliseconds before timing out. | |
| [default6]:[rank6]:[E ProcessGroupNCCL.cpp:563] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600073 milliseconds before timing out. | |
| [default3]:[rank3]:[E ProcessGroupNCCL.cpp:563] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600035 milliseconds before timing out. | |
| [default4]:[rank4]:[E ProcessGroupNCCL.cpp:563] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600072 milliseconds before timing out. | |
| [default7]:[rank7]:[E ProcessGroupNCCL.cpp:563] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600058 milliseconds before timing out. | |
| [default1]:[rank9]:[E ProcessGroupNCCL.cpp:563] [Rank 4] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600073 milliseconds before timing out. | |
| [default0]:[rank8]:[E ProcessGroupNCCL.cpp:563] [Rank 4] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600058 milliseconds before timing out. | |
| [default2]:[rank10]:[E ProcessGroupNCCL.cpp:563] [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600051 milliseconds before timing out. | |
| [default4]:[rank12]:[E ProcessGroupNCCL.cpp:563] [Rank 6] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=92163, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600063 milliseconds before timing out. | |
| [default5]:[rank13]:[E ProcessGroupNCCL.cpp:563] [Rank 6] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=92163, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600038 milliseconds before timing out. | |
| [default3]:[rank11]:[E ProcessGroupNCCL.cpp:563] [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600072 milliseconds before timing out. | |
| [default2]:[rank2]:[E ProcessGroupNCCL.cpp:563] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600098 milliseconds before timing out. | |
| [default5]:[rank5]:[E ProcessGroupNCCL.cpp:563] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600091 milliseconds before timing out. | |
| [default5]:[rank13]: Traceback (most recent call last): | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default5]:[rank13]: trainer.train(dataloader) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default5]:[rank13]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default5]:[rank13]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default5]:[rank13]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default5]:[rank13]: output = model(**micro_batch) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank13]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank13]: return forward_call(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default5]:[rank13]: sharded_logits = self.model( | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank13]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank13]: return forward_call(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default5]:[rank13]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default5]:[rank13]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank13]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank13]: return forward_call(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default5]:[rank13]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default5]:[rank13]: pipeline_state.run_communication() | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default5]:[rank13]: recv_activation_tensor = recv_activation() | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default5]:[rank13]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default5]:[rank13]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default5]:[rank13]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 269, in _recv_meta | |
| [default5]:[rank13]: dist.recv( | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default5]:[rank13]: return func(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default5]:[rank13]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default5]:[rank13]: torch.distributed.DistBackendError: NCCL communicator was aborted on rank 1. | |
| [default6]:[rank14]: Traceback (most recent call last): | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default6]:[rank14]: trainer.train(dataloader) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default6]:[rank14]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default6]:[rank14]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default6]:[rank14]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default6]:[rank14]: output = model(**micro_batch) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank14]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank14]: return forward_call(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default6]:[rank14]: sharded_logits = self.model( | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank14]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank14]: return forward_call(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default6]:[rank14]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 782, in forward_with_hidden_states | |
| [default6]:[rank14]: hidden_states = self.final_layer_norm(input=hidden_encoder_states["hidden_states"])["hidden_states"] | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank14]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank14]: return forward_call(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default6]:[rank14]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default6]:[rank14]: pipeline_state.run_communication() | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default6]:[rank14]: recv_activation_tensor = recv_activation() | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default6]:[rank14]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default6]:[rank14]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default6]:[rank14]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 269, in _recv_meta | |
| [default6]:[rank14]: dist.recv( | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default6]:[rank14]: return func(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default6]:[rank14]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default6]:[rank14]: torch.distributed.DistBackendError: NCCL communicator was aborted on rank 1. | |
| [default6]:[rank6]: Traceback (most recent call last): | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default6]:[rank6]: trainer.train(dataloader) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default6]:[rank6]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default6]:[rank6]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default6]:[rank6]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default6]:[rank6]: output = model(**micro_batch) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank6]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank6]: return forward_call(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default6]:[rank6]: sharded_logits = self.model( | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank6]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank6]: return forward_call(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default6]:[rank6]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default6]:[rank6]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank6]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank6]: return forward_call(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default6]:[rank6]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default6]:[rank6]: pipeline_state.run_communication() | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default6]:[rank6]: recv_activation_tensor = recv_activation() | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default6]:[rank6]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default6]:[rank6]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default6]:[rank6]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 269, in _recv_meta | |
| [default6]:[rank6]: dist.recv( | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default6]:[rank6]: return func(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default6]:[rank6]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default6]:[rank6]: torch.distributed.DistBackendError: NCCL communicator was aborted on rank 1. | |
| [default7]:[rank7]: Traceback (most recent call last): | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default7]:[rank7]: trainer.train(dataloader) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default7]:[rank7]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default7]:[rank7]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default7]:[rank7]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default7]:[rank7]: output = model(**micro_batch) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank7]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank7]: return forward_call(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default7]:[rank7]: sharded_logits = self.model( | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank7]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank7]: return forward_call(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default7]:[rank7]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default7]:[rank7]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank7]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank7]: return forward_call(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default7]:[rank7]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default7]:[rank7]: pipeline_state.run_communication() | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default7]:[rank7]: recv_activation_tensor = recv_activation() | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default7]:[rank7]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default7]:[rank7]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default7]:[rank7]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 269, in _recv_meta | |
| [default7]:[rank7]: dist.recv( | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default7]:[rank7]: return func(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default7]:[rank7]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default7]:[rank7]: torch.distributed.DistBackendError: NCCL communicator was aborted on rank 1. | |
| [default3]:[rank11]: Traceback (most recent call last): | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default3]:[rank11]: trainer.train(dataloader) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default3]:[rank11]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default3]:[rank11]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default3]:[rank11]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default3]:[rank11]: output = model(**micro_batch) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank11]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank11]: return forward_call(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default3]:[rank11]: sharded_logits = self.model( | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank11]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank11]: return forward_call(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default3]:[rank11]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default3]:[rank11]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank11]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank11]: return forward_call(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default3]:[rank11]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default3]:[rank11]: pipeline_state.run_communication() | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default3]:[rank11]: recv_activation_tensor = recv_activation() | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default3]:[rank11]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default3]:[rank11]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default3]:[rank11]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 269, in _recv_meta | |
| [default3]:[rank11]: dist.recv( | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default3]:[rank11]: return func(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default3]:[rank11]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default3]:[rank11]: torch.distributed.DistBackendError: NCCL communicator was aborted on rank 1. | |
| [default2]:[rank2]: Traceback (most recent call last): | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default2]:[rank2]: trainer.train(dataloader) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default2]:[rank2]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default2]:[rank2]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default2]:[rank2]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default2]:[rank2]: output = model(**micro_batch) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank2]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank2]: return forward_call(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default2]:[rank2]: sharded_logits = self.model( | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank2]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank2]: return forward_call(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default2]:[rank2]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default2]:[rank2]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank2]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank2]: return forward_call(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default2]:[rank2]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default2]:[rank2]: pipeline_state.run_communication() | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default2]:[rank2]: recv_activation_tensor = recv_activation() | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default2]:[rank2]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default2]:[rank2]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default2]:[rank2]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 269, in _recv_meta | |
| [default2]:[rank2]: dist.recv( | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default2]:[rank2]: return func(*args, **kwargs) | |
| [default2]:[rank2]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default2]:[rank2]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default2]:[rank2]: torch.distributed.DistBackendError: NCCL communicator was aborted on rank 1. | |
| [default7]:[rank15]: Traceback (most recent call last): | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default7]:[rank15]: trainer.train(dataloader) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default7]:[rank15]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default7]:[rank15]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default7]:[rank15]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default7]:[rank15]: output = model(**micro_batch) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank15]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank15]: return forward_call(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default7]:[rank15]: sharded_logits = self.model( | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank15]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank15]: return forward_call(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default7]:[rank15]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 782, in forward_with_hidden_states | |
| [default7]:[rank15]: hidden_states = self.final_layer_norm(input=hidden_encoder_states["hidden_states"])["hidden_states"] | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank15]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank15]: return forward_call(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default7]:[rank15]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default7]:[rank15]: pipeline_state.run_communication() | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default7]:[rank15]: recv_activation_tensor = recv_activation() | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default7]:[rank15]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default7]:[rank15]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default7]:[rank15]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 269, in _recv_meta | |
| [default7]:[rank15]: dist.recv( | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default7]:[rank15]: return func(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default7]:[rank15]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default7]:[rank15]: torch.distributed.DistBackendError: NCCL communicator was aborted on rank 1. | |
| [default5]:[rank5]: Traceback (most recent call last): | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default5]:[rank5]: trainer.train(dataloader) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default5]:[rank5]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default5]:[rank5]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default5]:[rank5]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default5]:[rank5]: output = model(**micro_batch) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank5]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank5]: return forward_call(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default5]:[rank5]: sharded_logits = self.model( | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank5]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank5]: return forward_call(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default5]:[rank5]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default5]:[rank5]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank5]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank5]: return forward_call(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default5]:[rank5]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default5]:[rank5]: pipeline_state.run_communication() | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default5]:[rank5]: recv_activation_tensor = recv_activation() | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default5]:[rank5]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default5]:[rank5]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default5]:[rank5]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 269, in _recv_meta | |
| [default5]:[rank5]: dist.recv( | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default5]:[rank5]: return func(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default5]:[rank5]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default5]:[rank5]: torch.distributed.DistBackendError: NCCL communicator was aborted on rank 1. | |
| [default3]:[rank3]: Traceback (most recent call last): | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default3]:[rank3]: trainer.train(dataloader) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default3]:[rank3]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default3]:[rank3]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default3]:[rank3]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default3]:[rank3]: output = model(**micro_batch) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank3]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank3]: return forward_call(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default3]:[rank3]: sharded_logits = self.model( | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank3]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank3]: return forward_call(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default3]:[rank3]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default3]:[rank3]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank3]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank3]: return forward_call(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default3]:[rank3]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default3]:[rank3]: pipeline_state.run_communication() | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default3]:[rank3]: recv_activation_tensor = recv_activation() | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default3]:[rank3]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default3]:[rank3]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default3]:[rank3]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 269, in _recv_meta | |
| [default3]:[rank3]: dist.recv( | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default3]:[rank3]: return func(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default3]:[rank3]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default3]:[rank3]: torch.distributed.DistBackendError: NCCL communicator was aborted on rank 1. | |
| [default2]:[rank10]: Traceback (most recent call last): | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default2]:[rank10]: trainer.train(dataloader) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default2]:[rank10]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default2]:[rank10]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default2]:[rank10]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default2]:[rank10]: output = model(**micro_batch) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank10]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank10]: return forward_call(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default2]:[rank10]: sharded_logits = self.model( | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank10]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank10]: return forward_call(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default2]:[rank10]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default2]:[rank10]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank10]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank10]: return forward_call(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default2]:[rank10]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default2]:[rank10]: pipeline_state.run_communication() | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default2]:[rank10]: recv_activation_tensor = recv_activation() | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default2]:[rank10]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default2]:[rank10]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default2]:[rank10]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 269, in _recv_meta | |
| [default2]:[rank10]: dist.recv( | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default2]:[rank10]: return func(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default2]:[rank10]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default2]:[rank10]: torch.distributed.DistBackendError: NCCL communicator was aborted on rank 1. | |
| [default4]:[rank12]: Traceback (most recent call last): | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default4]:[rank12]: trainer.train(dataloader) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default4]:[rank12]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default4]:[rank12]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default4]:[rank12]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default4]:[rank12]: output = model(**micro_batch) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank12]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank12]: return forward_call(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default4]:[rank12]: sharded_logits = self.model( | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank12]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank12]: return forward_call(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default4]:[rank12]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default4]:[rank12]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank12]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank12]: return forward_call(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default4]:[rank12]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default4]:[rank12]: pipeline_state.run_communication() | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default4]:[rank12]: recv_activation_tensor = recv_activation() | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default4]:[rank12]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default4]:[rank12]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default4]:[rank12]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 269, in _recv_meta | |
| [default4]:[rank12]: dist.recv( | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default4]:[rank12]: return func(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default4]:[rank12]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default4]:[rank12]: torch.distributed.DistBackendError: NCCL communicator was aborted on rank 1. | |
| [default4]:[rank4]: Traceback (most recent call last): | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default4]:[rank4]: trainer.train(dataloader) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default4]:[rank4]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default4]:[rank4]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default4]:[rank4]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default4]:[rank4]: output = model(**micro_batch) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank4]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank4]: return forward_call(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default4]:[rank4]: sharded_logits = self.model( | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank4]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank4]: return forward_call(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default4]:[rank4]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default4]:[rank4]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default4]:[rank4]: return self._call_impl(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default4]:[rank4]: return forward_call(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default4]:[rank4]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default4]:[rank4]: pipeline_state.run_communication() | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default4]:[rank4]: recv_activation_tensor = recv_activation() | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default4]:[rank4]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default4]:[rank4]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default4]:[rank4]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 269, in _recv_meta | |
| [default4]:[rank4]: dist.recv( | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default4]:[rank4]: return func(*args, **kwargs) | |
| [default4]:[rank4]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default4]:[rank4]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default4]:[rank4]: torch.distributed.DistBackendError: NCCL communicator was aborted on rank 1. | |
| [default0]:[rank8]: Traceback (most recent call last): | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default0]:[rank8]: trainer.train(dataloader) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default0]:[rank8]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default0]:[rank8]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default0]:[rank8]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default0]:[rank8]: output = model(**micro_batch) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank8]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank8]: return forward_call(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default0]:[rank8]: sharded_logits = self.model( | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank8]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank8]: return forward_call(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default0]:[rank8]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default0]:[rank8]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank8]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank8]: return forward_call(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default0]:[rank8]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default0]:[rank8]: pipeline_state.run_communication() | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default0]:[rank8]: recv_activation_tensor = recv_activation() | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default0]:[rank8]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default0]:[rank8]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default0]:[rank8]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 269, in _recv_meta | |
| [default0]:[rank8]: dist.recv( | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default0]:[rank8]: return func(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default0]:[rank8]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default0]:[rank8]: torch.distributed.DistBackendError: NCCL communicator was aborted on rank 1. | |
| [default1]:[rank9]: Traceback (most recent call last): | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default1]:[rank9]: trainer.train(dataloader) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default1]:[rank9]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default1]:[rank9]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default1]:[rank9]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default1]:[rank9]: output = model(**micro_batch) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank9]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank9]: return forward_call(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default1]:[rank9]: sharded_logits = self.model( | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank9]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank9]: return forward_call(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default1]:[rank9]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default1]:[rank9]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank9]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank9]: return forward_call(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default1]:[rank9]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default1]:[rank9]: pipeline_state.run_communication() | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default1]:[rank9]: recv_activation_tensor = recv_activation() | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default1]:[rank9]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default1]:[rank9]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default1]:[rank9]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 269, in _recv_meta | |
| [default1]:[rank9]: dist.recv( | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default1]:[rank9]: return func(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default1]:[rank9]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default1]:[rank9]: torch.distributed.DistBackendError: NCCL communicator was aborted on rank 1. | |
| [default6]:[rank14]:[E ProcessGroupNCCL.cpp:1537] [PG 4 Rank 7] Timeout at NCCL work: 36867, last enqueued NCCL work: 36867, last completed NCCL work: 36866. | |
| [default6]:[rank14]:[E ProcessGroupNCCL.cpp:577] [Rank 7] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. | |
| [default6]:[rank14]:[E ProcessGroupNCCL.cpp:583] [Rank 7] To avoid data inconsistency, we are taking the entire process down. | |
| [default6]:[rank14]:[E ProcessGroupNCCL.cpp:1414] [PG 4 Rank 7] Process group watchdog thread terminated with exception: [Rank 7] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=36867, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600028 milliseconds before timing out. | |
| [default6]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default6]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f72a3963897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default6]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f72a4c3cc62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7f72a4c41a80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f72a4c42dcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:frame #4: <unknown function> + 0xd3e95 (0x7f72f06dbe95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default6]:frame #5: <unknown function> + 0x8609 (0x7f72f5722609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default6]:frame #6: clone + 0x43 (0x7f72f54ed353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default6]: | |
| [default6]:terminate called after throwing an instance of 'c10::DistBackendError' | |
| [default6]: what(): [PG 4 Rank 7] Process group watchdog thread terminated with exception: [Rank 7] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=36867, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600028 milliseconds before timing out. | |
| [default6]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default6]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f72a3963897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default6]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f72a4c3cc62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7f72a4c41a80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f72a4c42dcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:frame #4: <unknown function> + 0xd3e95 (0x7f72f06dbe95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default6]:frame #5: <unknown function> + 0x8609 (0x7f72f5722609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default6]:frame #6: clone + 0x43 (0x7f72f54ed353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default6]: | |
| [default6]:Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first): | |
| [default6]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f72a3963897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default6]:frame #1: <unknown function> + 0xe32119 (0x7f72a48c6119 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:frame #2: <unknown function> + 0xd3e95 (0x7f72f06dbe95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default6]:frame #3: <unknown function> + 0x8609 (0x7f72f5722609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default6]:frame #4: clone + 0x43 (0x7f72f54ed353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default6]: | |
| [default7]:[rank15]:[E ProcessGroupNCCL.cpp:1537] [PG 4 Rank 7] Timeout at NCCL work: 36867, last enqueued NCCL work: 36867, last completed NCCL work: 36866. | |
| [default7]:[rank15]:[E ProcessGroupNCCL.cpp:577] [Rank 7] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. | |
| [default7]:[rank15]:[E ProcessGroupNCCL.cpp:583] [Rank 7] To avoid data inconsistency, we are taking the entire process down. | |
| [default7]:[rank15]:[E ProcessGroupNCCL.cpp:1414] [PG 4 Rank 7] Process group watchdog thread terminated with exception: [Rank 7] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=36867, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600002 milliseconds before timing out. | |
| [default7]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default7]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f5e29147897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default7]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f5e2a420c62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7f5e2a425a80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f5e2a426dcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:frame #4: <unknown function> + 0xd3e95 (0x7f5e75ebfe95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default7]:frame #5: <unknown function> + 0x8609 (0x7f5e7af06609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default7]:frame #6: clone + 0x43 (0x7f5e7acd1353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default7]: | |
| [default7]:terminate called after throwing an instance of 'c10::DistBackendError' | |
| [default7]: what(): [PG 4 Rank 7] Process group watchdog thread terminated with exception: [Rank 7] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=36867, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600002 milliseconds before timing out. | |
| [default7]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default7]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f5e29147897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default7]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f5e2a420c62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7f5e2a425a80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f5e2a426dcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:frame #4: <unknown function> + 0xd3e95 (0x7f5e75ebfe95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default7]:frame #5: <unknown function> + 0x8609 (0x7f5e7af06609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default7]:frame #6: clone + 0x43 (0x7f5e7acd1353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default7]: | |
| [default7]:Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first): | |
| [default7]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f5e29147897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default7]:frame #1: <unknown function> + 0xe32119 (0x7f5e2a0aa119 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:frame #2: <unknown function> + 0xd3e95 (0x7f5e75ebfe95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default7]:frame #3: <unknown function> + 0x8609 (0x7f5e7af06609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default7]:frame #4: clone + 0x43 (0x7f5e7acd1353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default7]: | |
| [default6]:[rank6]:[E ProcessGroupNCCL.cpp:1537] [PG 4 Rank 3] Timeout at NCCL work: 110595, last enqueued NCCL work: 110595, last completed NCCL work: 110594. | |
| [default6]:[rank6]:[E ProcessGroupNCCL.cpp:577] [Rank 3] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. | |
| [default6]:[rank6]:[E ProcessGroupNCCL.cpp:583] [Rank 3] To avoid data inconsistency, we are taking the entire process down. | |
| [default6]:[rank6]:[E ProcessGroupNCCL.cpp:1414] [PG 4 Rank 3] Process group watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600073 milliseconds before timing out. | |
| [default6]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default6]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fad588ee897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default6]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7fad59bc7c62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7fad59bcca80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fad59bcddcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:frame #4: <unknown function> + 0xd3e95 (0x7fada5666e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default6]:frame #5: <unknown function> + 0x8609 (0x7fadaa6ad609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default6]:frame #6: clone + 0x43 (0x7fadaa478353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default6]: | |
| [default6]:terminate called after throwing an instance of 'c10::DistBackendError' | |
| [default6]: what(): [PG 4 Rank 3] Process group watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600073 milliseconds before timing out. | |
| [default6]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default6]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fad588ee897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default6]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7fad59bc7c62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7fad59bcca80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fad59bcddcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:frame #4: <unknown function> + 0xd3e95 (0x7fada5666e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default6]:frame #5: <unknown function> + 0x8609 (0x7fadaa6ad609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default6]:frame #6: clone + 0x43 (0x7fadaa478353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default6]: | |
| [default6]:Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first): | |
| [default6]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fad588ee897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default6]:frame #1: <unknown function> + 0xe32119 (0x7fad59851119 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:frame #2: <unknown function> + 0xd3e95 (0x7fada5666e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default6]:frame #3: <unknown function> + 0x8609 (0x7fadaa6ad609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default6]:frame #4: clone + 0x43 (0x7fadaa478353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default6]: | |
| [default7]:[rank7]:[E ProcessGroupNCCL.cpp:1537] [PG 4 Rank 3] Timeout at NCCL work: 110595, last enqueued NCCL work: 110595, last completed NCCL work: 110594. | |
| [default7]:[rank7]:[E ProcessGroupNCCL.cpp:577] [Rank 3] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. | |
| [default7]:[rank7]:[E ProcessGroupNCCL.cpp:583] [Rank 3] To avoid data inconsistency, we are taking the entire process down. | |
| [default7]:[rank7]:[E ProcessGroupNCCL.cpp:1414] [PG 4 Rank 3] Process group watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600058 milliseconds before timing out. | |
| [default7]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default7]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fc836ba0897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default7]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7fc837e79c62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7fc837e7ea80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fc837e7fdcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:frame #4: <unknown function> + 0xd3e95 (0x7fc883918e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default7]:frame #5: <unknown function> + 0x8609 (0x7fc88895f609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default7]:frame #6: clone + 0x43 (0x7fc88872a353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default7]: | |
| [default7]:terminate called after throwing an instance of 'c10::DistBackendError' | |
| [default7]: what(): [PG 4 Rank 3] Process group watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600058 milliseconds before timing out. | |
| [default7]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default7]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fc836ba0897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default7]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7fc837e79c62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7fc837e7ea80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fc837e7fdcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:frame #4: <unknown function> + 0xd3e95 (0x7fc883918e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default7]:frame #5: <unknown function> + 0x8609 (0x7fc88895f609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default7]:frame #6: clone + 0x43 (0x7fc88872a353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default7]: | |
| [default7]:Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first): | |
| [default7]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fc836ba0897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default7]:frame #1: <unknown function> + 0xe32119 (0x7fc837b03119 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:frame #2: <unknown function> + 0xd3e95 (0x7fc883918e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default7]:frame #3: <unknown function> + 0x8609 (0x7fc88895f609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default7]:frame #4: clone + 0x43 (0x7fc88872a353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default7]: | |
| [default5]:[rank5]:[E ProcessGroupNCCL.cpp:1537] [PG 4 Rank 2] Timeout at NCCL work: 110595, last enqueued NCCL work: 110595, last completed NCCL work: 110594. | |
| [default5]:[rank5]:[E ProcessGroupNCCL.cpp:577] [Rank 2] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. | |
| [default5]:[rank5]:[E ProcessGroupNCCL.cpp:583] [Rank 2] To avoid data inconsistency, we are taking the entire process down. | |
| [default5]:[rank5]:[E ProcessGroupNCCL.cpp:1414] [PG 4 Rank 2] Process group watchdog thread terminated with exception: [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600091 milliseconds before timing out. | |
| [default5]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default5]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f1d1d40f897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default5]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f1d1e6e8c62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7f1d1e6eda80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f1d1e6eedcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:frame #4: <unknown function> + 0xd3e95 (0x7f1d6a187e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default5]:frame #5: <unknown function> + 0x8609 (0x7f1d6f1ce609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default5]:frame #6: clone + 0x43 (0x7f1d6ef99353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default5]: | |
| [default5]:terminate called after throwing an instance of 'c10::DistBackendError' | |
| [default5]: what(): [PG 4 Rank 2] Process group watchdog thread terminated with exception: [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600091 milliseconds before timing out. | |
| [default5]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default5]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f1d1d40f897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default5]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f1d1e6e8c62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7f1d1e6eda80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f1d1e6eedcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:frame #4: <unknown function> + 0xd3e95 (0x7f1d6a187e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default5]:frame #5: <unknown function> + 0x8609 (0x7f1d6f1ce609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default5]:frame #6: clone + 0x43 (0x7f1d6ef99353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default5]: | |
| [default5]:Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first): | |
| [default5]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f1d1d40f897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default5]:frame #1: <unknown function> + 0xe32119 (0x7f1d1e372119 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:frame #2: <unknown function> + 0xd3e95 (0x7f1d6a187e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default5]:frame #3: <unknown function> + 0x8609 (0x7f1d6f1ce609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default5]:frame #4: clone + 0x43 (0x7f1d6ef99353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default5]: | |
| [default5]:[rank13]:[E ProcessGroupNCCL.cpp:1537] [PG 4 Rank 6] Timeout at NCCL work: 92163, last enqueued NCCL work: 92163, last completed NCCL work: 92162. | |
| [default5]:[rank13]:[E ProcessGroupNCCL.cpp:577] [Rank 6] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. | |
| [default5]:[rank13]:[E ProcessGroupNCCL.cpp:583] [Rank 6] To avoid data inconsistency, we are taking the entire process down. | |
| [default5]:[rank13]:[E ProcessGroupNCCL.cpp:1414] [PG 4 Rank 6] Process group watchdog thread terminated with exception: [Rank 6] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=92163, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600038 milliseconds before timing out. | |
| [default5]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default5]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f5b7b3c1897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default5]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f5b7c69ac62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7f5b7c69fa80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f5b7c6a0dcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:frame #4: <unknown function> + 0xd3e95 (0x7f5bc8139e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default5]:frame #5: <unknown function> + 0x8609 (0x7f5bcd180609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default5]:frame #6: clone + 0x43 (0x7f5bccf4b353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default5]: | |
| [default5]:terminate called after throwing an instance of 'c10::DistBackendError' | |
| [default5]: what(): [PG 4 Rank 6] Process group watchdog thread terminated with exception: [Rank 6] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=92163, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600038 milliseconds before timing out. | |
| [default5]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default5]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f5b7b3c1897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default5]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f5b7c69ac62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7f5b7c69fa80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f5b7c6a0dcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:frame #4: <unknown function> + 0xd3e95 (0x7f5bc8139e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default5]:frame #5: <unknown function> + 0x8609 (0x7f5bcd180609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default5]:frame #6: clone + 0x43 (0x7f5bccf4b353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default5]: | |
| [default5]:Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first): | |
| [default5]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f5b7b3c1897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default5]:frame #1: <unknown function> + 0xe32119 (0x7f5b7c324119 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:frame #2: <unknown function> + 0xd3e95 (0x7f5bc8139e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default5]:frame #3: <unknown function> + 0x8609 (0x7f5bcd180609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default5]:frame #4: clone + 0x43 (0x7f5bccf4b353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default5]: | |
| [default4]:[rank4]:[E ProcessGroupNCCL.cpp:1537] [PG 4 Rank 2] Timeout at NCCL work: 110595, last enqueued NCCL work: 110595, last completed NCCL work: 110594. | |
| [default4]:[rank4]:[E ProcessGroupNCCL.cpp:577] [Rank 2] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. | |
| [default4]:[rank4]:[E ProcessGroupNCCL.cpp:583] [Rank 2] To avoid data inconsistency, we are taking the entire process down. | |
| [default4]:[rank4]:[E ProcessGroupNCCL.cpp:1414] [PG 4 Rank 2] Process group watchdog thread terminated with exception: [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600072 milliseconds before timing out. | |
| [default4]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default4]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fbe42eb1897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default4]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7fbe4418ac62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7fbe4418fa80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fbe44190dcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:frame #4: <unknown function> + 0xd3e95 (0x7fbe8fc29e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default4]:frame #5: <unknown function> + 0x8609 (0x7fbe94c70609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default4]:frame #6: clone + 0x43 (0x7fbe94a3b353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default4]: | |
| [default4]:terminate called after throwing an instance of 'c10::DistBackendError' | |
| [default4]: what(): [PG 4 Rank 2] Process group watchdog thread terminated with exception: [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600072 milliseconds before timing out. | |
| [default4]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default4]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fbe42eb1897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default4]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7fbe4418ac62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7fbe4418fa80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fbe44190dcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:frame #4: <unknown function> + 0xd3e95 (0x7fbe8fc29e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default4]:frame #5: <unknown function> + 0x8609 (0x7fbe94c70609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default4]:frame #6: clone + 0x43 (0x7fbe94a3b353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default4]: | |
| [default4]:Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first): | |
| [default4]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fbe42eb1897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default4]:frame #1: <unknown function> + 0xe32119 (0x7fbe43e14119 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:frame #2: <unknown function> + 0xd3e95 (0x7fbe8fc29e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default4]:frame #3: <unknown function> + 0x8609 (0x7fbe94c70609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default4]:frame #4: clone + 0x43 (0x7fbe94a3b353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default4]: | |
| [default4]:[rank12]:[E ProcessGroupNCCL.cpp:1537] [PG 4 Rank 6] Timeout at NCCL work: 92163, last enqueued NCCL work: 92163, last completed NCCL work: 92162. | |
| [default4]:[rank12]:[E ProcessGroupNCCL.cpp:577] [Rank 6] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. | |
| [default4]:[rank12]:[E ProcessGroupNCCL.cpp:583] [Rank 6] To avoid data inconsistency, we are taking the entire process down. | |
| [default4]:[rank12]:[E ProcessGroupNCCL.cpp:1414] [PG 4 Rank 6] Process group watchdog thread terminated with exception: [Rank 6] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=92163, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600063 milliseconds before timing out. | |
| [default4]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default4]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fbe9d36f897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default4]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7fbe9e648c62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7fbe9e64da80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fbe9e64edcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:frame #4: <unknown function> + 0xd3e95 (0x7fbeea0e7e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default4]:frame #5: <unknown function> + 0x8609 (0x7fbeef12e609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default4]:frame #6: clone + 0x43 (0x7fbeeeef9353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default4]: | |
| [default4]:terminate called after throwing an instance of 'c10::DistBackendError' | |
| [default4]: what(): [PG 4 Rank 6] Process group watchdog thread terminated with exception: [Rank 6] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=92163, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600063 milliseconds before timing out. | |
| [default4]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default4]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fbe9d36f897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default4]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7fbe9e648c62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7fbe9e64da80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fbe9e64edcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:frame #4: <unknown function> + 0xd3e95 (0x7fbeea0e7e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default4]:frame #5: <unknown function> + 0x8609 (0x7fbeef12e609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default4]:frame #6: clone + 0x43 (0x7fbeeeef9353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default4]: | |
| [default4]:Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first): | |
| [default4]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fbe9d36f897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default4]:frame #1: <unknown function> + 0xe32119 (0x7fbe9e2d2119 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default4]:frame #2: <unknown function> + 0xd3e95 (0x7fbeea0e7e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default4]:frame #3: <unknown function> + 0x8609 (0x7fbeef12e609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default4]:frame #4: clone + 0x43 (0x7fbeeeef9353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default4]: | |
| [default2]:[rank10]:[E ProcessGroupNCCL.cpp:1537] [PG 4 Rank 5] Timeout at NCCL work: 110595, last enqueued NCCL work: 110595, last completed NCCL work: 110594. | |
| [default2]:[rank10]:[E ProcessGroupNCCL.cpp:577] [Rank 5] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. | |
| [default2]:[rank10]:[E ProcessGroupNCCL.cpp:583] [Rank 5] To avoid data inconsistency, we are taking the entire process down. | |
| [default2]:[rank10]:[E ProcessGroupNCCL.cpp:1414] [PG 4 Rank 5] Process group watchdog thread terminated with exception: [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600051 milliseconds before timing out. | |
| [default2]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default2]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f1f259bb897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default2]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f1f26c94c62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7f1f26c99a80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f1f26c9adcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:frame #4: <unknown function> + 0xd3e95 (0x7f1f72733e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default2]:frame #5: <unknown function> + 0x8609 (0x7f1f7777a609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default2]:frame #6: clone + 0x43 (0x7f1f77545353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default2]: | |
| [default2]:terminate called after throwing an instance of 'c10::DistBackendError' | |
| [default2]: what(): [PG 4 Rank 5] Process group watchdog thread terminated with exception: [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600051 milliseconds before timing out. | |
| [default2]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default2]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f1f259bb897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default2]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f1f26c94c62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7f1f26c99a80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f1f26c9adcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:frame #4: <unknown function> + 0xd3e95 (0x7f1f72733e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default2]:frame #5: <unknown function> + 0x8609 (0x7f1f7777a609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default2]:frame #6: clone + 0x43 (0x7f1f77545353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default2]: | |
| [default2]:Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first): | |
| [default2]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f1f259bb897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default2]:frame #1: <unknown function> + 0xe32119 (0x7f1f2691e119 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:frame #2: <unknown function> + 0xd3e95 (0x7f1f72733e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default2]:frame #3: <unknown function> + 0x8609 (0x7f1f7777a609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default2]:frame #4: clone + 0x43 (0x7f1f77545353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default2]: | |
| [default3]:[rank11]:[E ProcessGroupNCCL.cpp:1537] [PG 4 Rank 5] Timeout at NCCL work: 110595, last enqueued NCCL work: 110595, last completed NCCL work: 110594. | |
| [default3]:[rank11]:[E ProcessGroupNCCL.cpp:577] [Rank 5] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. | |
| [default3]:[rank11]:[E ProcessGroupNCCL.cpp:583] [Rank 5] To avoid data inconsistency, we are taking the entire process down. | |
| [default3]:[rank11]:[E ProcessGroupNCCL.cpp:1414] [PG 4 Rank 5] Process group watchdog thread terminated with exception: [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600072 milliseconds before timing out. | |
| [default3]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default3]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fc90125d897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default3]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7fc902536c62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7fc90253ba80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fc90253cdcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:frame #4: <unknown function> + 0xd3e95 (0x7fc94dfd5e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default3]:frame #5: <unknown function> + 0x8609 (0x7fc95301c609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default3]:frame #6: clone + 0x43 (0x7fc952de7353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default3]: | |
| [default3]:terminate called after throwing an instance of 'c10::DistBackendError' | |
| [default3]: what(): [PG 4 Rank 5] Process group watchdog thread terminated with exception: [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600072 milliseconds before timing out. | |
| [default3]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default3]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fc90125d897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default3]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7fc902536c62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7fc90253ba80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fc90253cdcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:frame #4: <unknown function> + 0xd3e95 (0x7fc94dfd5e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default3]:frame #5: <unknown function> + 0x8609 (0x7fc95301c609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default3]:frame #6: clone + 0x43 (0x7fc952de7353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default3]: | |
| [default3]:Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first): | |
| [default3]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fc90125d897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default3]:frame #1: <unknown function> + 0xe32119 (0x7fc9021c0119 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:frame #2: <unknown function> + 0xd3e95 (0x7fc94dfd5e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default3]:frame #3: <unknown function> + 0x8609 (0x7fc95301c609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default3]:frame #4: clone + 0x43 (0x7fc952de7353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default3]: | |
| [default3]:[rank3]:[E ProcessGroupNCCL.cpp:1537] [PG 4 Rank 1] Timeout at NCCL work: 110595, last enqueued NCCL work: 110595, last completed NCCL work: 110594. | |
| [default3]:[rank3]:[E ProcessGroupNCCL.cpp:577] [Rank 1] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. | |
| [default3]:[rank3]:[E ProcessGroupNCCL.cpp:583] [Rank 1] To avoid data inconsistency, we are taking the entire process down. | |
| [default3]:[rank3]:[E ProcessGroupNCCL.cpp:1414] [PG 4 Rank 1] Process group watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600035 milliseconds before timing out. | |
| [default3]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default3]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7feeb2a8d897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default3]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7feeb3d66c62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7feeb3d6ba80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7feeb3d6cdcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:frame #4: <unknown function> + 0xd3e95 (0x7feeff805e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default3]:frame #5: <unknown function> + 0x8609 (0x7fef0484c609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default3]:frame #6: clone + 0x43 (0x7fef04617353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default3]: | |
| [default3]:terminate called after throwing an instance of 'c10::DistBackendError' | |
| [default3]: what(): [PG 4 Rank 1] Process group watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600035 milliseconds before timing out. | |
| [default3]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default3]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7feeb2a8d897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default3]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7feeb3d66c62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7feeb3d6ba80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7feeb3d6cdcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:frame #4: <unknown function> + 0xd3e95 (0x7feeff805e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default3]:frame #5: <unknown function> + 0x8609 (0x7fef0484c609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default3]:frame #6: clone + 0x43 (0x7fef04617353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default3]: | |
| [default3]:Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first): | |
| [default3]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7feeb2a8d897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default3]:frame #1: <unknown function> + 0xe32119 (0x7feeb39f0119 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:frame #2: <unknown function> + 0xd3e95 (0x7feeff805e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default3]:frame #3: <unknown function> + 0x8609 (0x7fef0484c609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default3]:frame #4: clone + 0x43 (0x7fef04617353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default3]: | |
| [default2]:[rank2]:[E ProcessGroupNCCL.cpp:1537] [PG 4 Rank 1] Timeout at NCCL work: 110595, last enqueued NCCL work: 110595, last completed NCCL work: 110594. | |
| [default2]:[rank2]:[E ProcessGroupNCCL.cpp:577] [Rank 1] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. | |
| [default2]:[rank2]:[E ProcessGroupNCCL.cpp:583] [Rank 1] To avoid data inconsistency, we are taking the entire process down. | |
| [default2]:[rank2]:[E ProcessGroupNCCL.cpp:1414] [PG 4 Rank 1] Process group watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600098 milliseconds before timing out. | |
| [default2]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default2]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f790735f897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default2]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f7908638c62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7f790863da80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f790863edcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:frame #4: <unknown function> + 0xd3e95 (0x7f79540d7e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default2]:frame #5: <unknown function> + 0x8609 (0x7f795911e609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default2]:frame #6: clone + 0x43 (0x7f7958ee9353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default2]: | |
| [default2]:terminate called after throwing an instance of 'c10::DistBackendError' | |
| [default2]: what(): [PG 4 Rank 1] Process group watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600098 milliseconds before timing out. | |
| [default2]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default2]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f790735f897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default2]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f7908638c62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7f790863da80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f790863edcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:frame #4: <unknown function> + 0xd3e95 (0x7f79540d7e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default2]:frame #5: <unknown function> + 0x8609 (0x7f795911e609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default2]:frame #6: clone + 0x43 (0x7f7958ee9353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default2]: | |
| [default2]:Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first): | |
| [default2]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f790735f897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default2]:frame #1: <unknown function> + 0xe32119 (0x7f79082c2119 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:frame #2: <unknown function> + 0xd3e95 (0x7f79540d7e95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default2]:frame #3: <unknown function> + 0x8609 (0x7f795911e609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default2]:frame #4: clone + 0x43 (0x7f7958ee9353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default2]: | |
| [default0]:[rank8]:[E ProcessGroupNCCL.cpp:1537] [PG 4 Rank 4] Timeout at NCCL work: 110595, last enqueued NCCL work: 110595, last completed NCCL work: 110594. | |
| [default0]:[rank8]:[E ProcessGroupNCCL.cpp:577] [Rank 4] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. | |
| [default0]:[rank8]:[E ProcessGroupNCCL.cpp:583] [Rank 4] To avoid data inconsistency, we are taking the entire process down. | |
| [default0]:[rank8]:[E ProcessGroupNCCL.cpp:1414] [PG 4 Rank 4] Process group watchdog thread terminated with exception: [Rank 4] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600058 milliseconds before timing out. | |
| [default0]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7ff26baa7897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default0]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7ff26cd80c62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default0]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7ff26cd85a80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default0]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7ff26cd86dcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default0]:frame #4: <unknown function> + 0xd3e95 (0x7ff2b881fe95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default0]:frame #5: <unknown function> + 0x8609 (0x7ff2bd866609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default0]:frame #6: clone + 0x43 (0x7ff2bd631353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default0]: | |
| [default0]:terminate called after throwing an instance of 'c10::DistBackendError' | |
| [default0]: what(): [PG 4 Rank 4] Process group watchdog thread terminated with exception: [Rank 4] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=110595, OpType=RECV, NumelIn=7, NumelOut=7, Timeout(ms)=600000) ran for 600058 milliseconds before timing out. | |
| [default0]:Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first): | |
| [default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7ff26baa7897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default0]:frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7ff26cd80c62 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default0]:frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7ff26cd85a80 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default0]:frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7ff26cd86dcc in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default0]:frame #4: <unknown function> + 0xd3e95 (0x7ff2b881fe95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default0]:frame #5: <unknown function> + 0x8609 (0x7ff2bd866609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default0]:frame #6: clone + 0x43 (0x7ff2bd631353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default0]: | |
| [default0]:Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first): | |
| [default0]:frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7ff26baa7897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default0]:frame #1: <unknown function> + 0xe32119 (0x7ff26ca0a119 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default0]:frame #2: <unknown function> + 0xd3e95 (0x7ff2b881fe95 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/../lib/libstdc++.so.6) | |
| [default0]:frame #3: <unknown function> + 0x8609 (0x7ff2bd866609 in /lib/x86_64-linux-gnu/libpthread.so.0) | |
| [default0]:frame #4: clone + 0x43 (0x7ff2bd631353 in /lib/x86_64-linux-gnu/libc.so.6) | |
| [default0]: | |
| W0702 18:52:03.689000 140027356682048 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2980592 closing signal SIGTERM | |
| W0702 18:52:03.692000 140027356682048 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2980593 closing signal SIGTERM | |
| W0702 18:52:03.694000 140027356682048 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2980594 closing signal SIGTERM | |
| W0702 18:52:03.754000 140390681818944 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 659126 closing signal SIGTERM | |
| W0702 18:52:03.755000 140390681818944 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 659127 closing signal SIGTERM | |
| W0702 18:52:03.755000 140390681818944 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 659128 closing signal SIGTERM | |
| E0702 18:52:04.981000 140390681818944 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: -6) local_rank: 3 (pid: 659129) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 | |
| Traceback (most recent call last): | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module> | |
| sys.exit(main()) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper | |
| return f(*args, **kwargs) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main | |
| run(args) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run | |
| elastic_launch( | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__ | |
| return launch_agent(self._config, self._entrypoint, list(args)) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent | |
| raise ChildFailedError( | |
| torch.distributed.elastic.multiprocessing.errors.ChildFailedError: | |
| ============================================================ | |
| /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED | |
| ------------------------------------------------------------ | |
| Failures: | |
| [1]: | |
| time : 2024-07-02_18:52:03 | |
| host : ip-26-0-172-73.ec2.internal | |
| rank : 12 (local_rank: 4) | |
| exitcode : -6 (pid: 659130) | |
| error_file: <N/A> | |
| traceback : Signal 6 (SIGABRT) received by PID 659130 | |
| [2]: | |
| time : 2024-07-02_18:52:03 | |
| host : ip-26-0-172-73.ec2.internal | |
| rank : 13 (local_rank: 5) | |
| exitcode : -6 (pid: 659131) | |
| error_file: <N/A> | |
| traceback : Signal 6 (SIGABRT) received by PID 659131 | |
| [3]: | |
| time : 2024-07-02_18:52:03 | |
| host : ip-26-0-172-73.ec2.internal | |
| rank : 14 (local_rank: 6) | |
| exitcode : -6 (pid: 659132) | |
| error_file: <N/A> | |
| traceback : Signal 6 (SIGABRT) received by PID 659132 | |
| [4]: | |
| time : 2024-07-02_18:52:03 | |
| host : ip-26-0-172-73.ec2.internal | |
| rank : 15 (local_rank: 7) | |
| exitcode : -6 (pid: 659133) | |
| error_file: <N/A> | |
| traceback : Signal 6 (SIGABRT) received by PID 659133 | |
| ------------------------------------------------------------ | |
| Root Cause (first observed failure): | |
| [0]: | |
| time : 2024-07-02_18:52:03 | |
| host : ip-26-0-172-73.ec2.internal | |
| rank : 11 (local_rank: 3) | |
| exitcode : -6 (pid: 659129) | |
| error_file: <N/A> | |
| traceback : Signal 6 (SIGABRT) received by PID 659129 | |
| ============================================================ | |
| srun: error: ip-26-0-172-73: task 1: Exited with exit code 1 | |
| E0702 18:52:09.127000 140027356682048 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: -6) local_rank: 3 (pid: 2980595) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 | |
| Traceback (most recent call last): | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module> | |
| sys.exit(main()) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper | |
| return f(*args, **kwargs) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main | |
| run(args) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run | |
| elastic_launch( | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__ | |
| return launch_agent(self._config, self._entrypoint, list(args)) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent | |
| raise ChildFailedError( | |
| torch.distributed.elastic.multiprocessing.errors.ChildFailedError: | |
| ============================================================ | |
| /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED | |
| ------------------------------------------------------------ | |
| Failures: | |
| [1]: | |
| time : 2024-07-02_18:52:03 | |
| host : ip-26-0-163-226.ec2.internal | |
| rank : 4 (local_rank: 4) | |
| exitcode : -6 (pid: 2980596) | |
| error_file: <N/A> | |
| traceback : Signal 6 (SIGABRT) received by PID 2980596 | |
| [2]: | |
| time : 2024-07-02_18:52:03 | |
| host : ip-26-0-163-226.ec2.internal | |
| rank : 5 (local_rank: 5) | |
| exitcode : -6 (pid: 2980597) | |
| error_file: <N/A> | |
| traceback : Signal 6 (SIGABRT) received by PID 2980597 | |
| [3]: | |
| time : 2024-07-02_18:52:03 | |
| host : ip-26-0-163-226.ec2.internal | |
| rank : 6 (local_rank: 6) | |
| exitcode : -6 (pid: 2980598) | |
| error_file: <N/A> | |
| traceback : Signal 6 (SIGABRT) received by PID 2980598 | |
| [4]: | |
| time : 2024-07-02_18:52:03 | |
| host : ip-26-0-163-226.ec2.internal | |
| rank : 7 (local_rank: 7) | |
| exitcode : -6 (pid: 2980599) | |
| error_file: <N/A> | |
| traceback : Signal 6 (SIGABRT) received by PID 2980599 | |
| ------------------------------------------------------------ | |
| Root Cause (first observed failure): | |
| [0]: | |
| time : 2024-07-02_18:52:03 | |
| host : ip-26-0-163-226.ec2.internal | |
| rank : 3 (local_rank: 3) | |
| exitcode : -6 (pid: 2980595) | |
| error_file: <N/A> | |
| traceback : Signal 6 (SIGABRT) received by PID 2980595 | |
| ============================================================ | |
| srun: error: ip-26-0-163-226: task 0: Exited with exit code 1 | |
| Consider using `hf_transfer` for faster uploads. This solution comes with some limitations. See https://huggingface.co/docs/huggingface_hub/hf_transfer for more details. | |