| ======================== | |
| START TIME: Tue Jul 2 16:32:00 UTC 2024 | |
| python3 version = Python 3.10.14 | |
| ======================== | |
| The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well. | |
| Token is valid (permission: write). | |
| Your token has been saved to /admin/home/ferdinand_mom/.cache/huggingface/token | |
| Login successful | |
| Already on 'bench_cluster' | |
| M examples/config_tiny_llama.py | |
| M examples/config_tiny_llama.yaml | |
| M examples/train_tiny_llama.sh | |
| M src/nanotron/models/llama.py | |
| M src/nanotron/trainer.py | |
| Your branch is up to date with 'origin/bench_cluster'. | |
| Job status: RUNNING | |
| W0702 16:32:02.522000 140456080881472 torch/distributed/run.py:757] | |
| W0702 16:32:02.522000 140456080881472 torch/distributed/run.py:757] ***************************************** | |
| W0702 16:32:02.522000 140456080881472 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. | |
| W0702 16:32:02.522000 140456080881472 torch/distributed/run.py:757] ***************************************** | |
| W0702 16:32:02.538000 139995769620288 torch/distributed/run.py:757] | |
| W0702 16:32:02.538000 139995769620288 torch/distributed/run.py:757] ***************************************** | |
| W0702 16:32:02.538000 139995769620288 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. | |
| W0702 16:32:02.538000 139995769620288 torch/distributed/run.py:757] ***************************************** | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Config: | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Config(general=GeneralArgs(project='bench_cluster', | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: run='%date_%jobid', | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: seed=42, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: step=None, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: consumed_train_samples=None, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: benchmark_csv_path=None, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: ignore_sanity_checks=True), | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: parallelism=ParallelismArgs(dp=8, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: pp=2, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: tp=1, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: pp_engine=<nanotron.parallel.pipeline_parallel.engine.OneForwardOneBackwardPipelineEngine object at 0x7fa22cf206a0>, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: tp_mode=<TensorParallelLinearMode.REDUCE_SCATTER: 2>, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: tp_linear_async_communication=False, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: expert_parallel_size=1), | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: model=ModelArgs(model_config=LlamaConfig(bos_token_id=1, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: eos_token_id=2, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: hidden_act='silu', | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: hidden_size=2048, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: initializer_range=0.02, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: intermediate_size=4096, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: is_llama_config=True, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: max_position_embeddings=4096, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: num_attention_heads=32, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: num_hidden_layers=24, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: num_key_value_heads=32, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: pad_token_id=None, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: pretraining_tp=1, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: rms_norm_eps=1e-05, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: rope_scaling=None, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: rope_theta=10000.0, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: tie_word_embeddings=True, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: use_cache=True, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: vocab_size=50257), | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: init_method=RandomInit(std=0.025), | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: dtype=torch.bfloat16, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: make_vocab_size_divisible_by=1, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: ddp_bucket_cap_mb=25), | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: tokenizer=TokenizerArgs(tokenizer_name_or_path='openai-community/gpt2', | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: tokenizer_revision=None, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: tokenizer_max_length=None), | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: checkpoints=CheckpointsArgs(checkpoints_path=Path('/dev/null'), | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: checkpoint_interval=100000, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: save_initial_state=False, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: resume_checkpoint_path=None, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: checkpoints_path_is_shared_file_system=False), | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: logging=LoggingArgs(log_level='info', | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: log_level_replica='info', | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: iteration_step_info_interval=1), | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: tokens=TokensArgs(sequence_length=4096, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: train_steps=20, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: micro_batch_size=1, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: batch_accumulation_per_replica=128, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: val_check_interval=-1, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: limit_val_batches=0, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: limit_test_batches=0), | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: optimizer=OptimizerArgs(optimizer_factory=AdamWOptimizerArgs(adam_eps=1e-08, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: adam_beta1=0.9, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: adam_beta2=0.95, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: torch_adam_is_fused=True, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: name='adamW'), | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: zero_stage=1, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: weight_decay=0.01, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: clip_grad=1.0, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: accumulate_grad_in_fp32=True, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: learning_rate_scheduler=LRSchedulerArgs(learning_rate=0.0001, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: lr_warmup_steps=1, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: lr_warmup_style='linear', | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: lr_decay_style='linear', | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: lr_decay_steps=19, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: lr_decay_starting_step=None, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: min_decay_lr=1e-05)), | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: data_stages=[DatasetStageArgs(name='Training Stage', | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: start_training_step=1, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: data=DataArgs(dataset=PretrainDatasetsArgs(hf_dataset_or_datasets='roneneldan/TinyStories', | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: hf_dataset_splits='train', | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: hf_dataset_config_name=None, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: dataset_processing_num_proc_per_process=64, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: dataset_overwrite_cache=False, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: text_column_name='text'), | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: seed=42, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: num_loading_workers=32))], | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: profiler=ProfilerArgs(profiler_export_path=Path('/fsx/ferdinandmom/ferdinand-hf/bench_cluster/results/llama-1B/16_GPUS/dp-8_tp-1_pp-2_mbz-1')), | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: lighteval=None) | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Model Config: | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: LlamaConfig(bos_token_id=1, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: eos_token_id=2, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: hidden_act='silu', | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: hidden_size=2048, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: initializer_range=0.02, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: intermediate_size=4096, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: is_llama_config=True, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: max_position_embeddings=4096, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: num_attention_heads=32, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: num_hidden_layers=24, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: num_key_value_heads=32, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: pad_token_id=None, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: pretraining_tp=1, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: rms_norm_eps=1e-05, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: rope_scaling=None, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: rope_theta=10000.0, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: tie_word_embeddings=True, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: use_cache=True, | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: vocab_size=50257) | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Building model.. | |
| [default0]:07/02/2024 16:32:19 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Setting PP block ranks... | |
| [default1]:07/02/2024 16:32:30 [INFO|DP=1|PP=1|TP=0|ip-26-0-171-56]: No checkpoint path provided. | |
| [default0]:07/02/2024 16:32:30 [INFO|DP=0|PP=1|TP=0|ip-26-0-171-56]: Local number of parameters: 522M (996.40MiB) | |
| [default0]:07/02/2024 16:32:30 [INFO|DP=0|PP=1|TP=0|ip-26-0-171-56]: [After model building] Memory usage: 1006.41MiB. Peak allocated: 1008.44MiB Peak reserved: 1032.00MiB | |
| [default0]:07/02/2024 16:32:30 [INFO|DP=0|PP=1|TP=0|ip-26-0-171-56]: No checkpoint path provided. | |
| [default3]:07/02/2024 16:32:30 [INFO|DP=3|PP=1|TP=0|ip-26-0-171-56]: No checkpoint path provided. | |
| [default0]:07/02/2024 16:32:30 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Total number of parameters: 1.21G (2312.82MiB) | |
| [default0]:07/02/2024 16:32:30 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Local number of parameters: 690M (1316.43MiB) | |
| [default0]:07/02/2024 16:32:30 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [After model building] Memory usage: 1330.44MiB. Peak allocated: 1332.47MiB Peak reserved: 1364.00MiB | |
| [default0]:07/02/2024 16:32:30 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: No checkpoint path provided. | |
| [default0]:07/02/2024 16:32:30 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Parametrizing model parameters using StandardParametrizator | |
| [default1]:07/02/2024 16:32:30 [INFO|DP=1|PP=0|TP=0|ip-26-0-160-225]: No checkpoint path provided. | |
| [default3]:07/02/2024 16:32:30 [INFO|DP=3|PP=0|TP=0|ip-26-0-160-225]: No checkpoint path provided. | |
| [default7]:07/02/2024 16:32:30 [INFO|DP=7|PP=1|TP=0|ip-26-0-171-56]: No checkpoint path provided. | |
| [default6]:07/02/2024 16:32:30 [INFO|DP=6|PP=1|TP=0|ip-26-0-171-56]: No checkpoint path provided. | |
| [default2]:07/02/2024 16:32:30 [INFO|DP=2|PP=1|TP=0|ip-26-0-171-56]: No checkpoint path provided. | |
| [default2]:07/02/2024 16:32:30 [INFO|DP=2|PP=0|TP=0|ip-26-0-160-225]: No checkpoint path provided. | |
| [default7]:07/02/2024 16:32:30 [INFO|DP=7|PP=0|TP=0|ip-26-0-160-225]: No checkpoint path provided. | |
| [default6]:07/02/2024 16:32:30 [INFO|DP=6|PP=0|TP=0|ip-26-0-160-225]: No checkpoint path provided. | |
| [default5]:07/02/2024 16:32:30 [INFO|DP=5|PP=1|TP=0|ip-26-0-171-56]: No checkpoint path provided. | |
| [default4]:07/02/2024 16:32:30 [INFO|DP=4|PP=1|TP=0|ip-26-0-171-56]: No checkpoint path provided. | |
| [default5]:07/02/2024 16:32:30 [INFO|DP=5|PP=0|TP=0|ip-26-0-160-225]: No checkpoint path provided. | |
| [default4]:07/02/2024 16:32:30 [INFO|DP=4|PP=0|TP=0|ip-26-0-160-225]: No checkpoint path provided. | |
| [default0]:07/02/2024 16:32:37 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [Optimizer Building] Using LearningRateForSP as learning rate | |
| [default0]:07/02/2024 16:32:37 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [ZeRO sharding] Size of optimizer params per rank: | |
| [default0]:07/02/2024 16:32:37 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [ZeRO sharding] DP Rank 0 has 86.3M out of 690M (12.50%) params' optimizer states | |
| [default0]:07/02/2024 16:32:37 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [ZeRO sharding] DP Rank 1 has 86.3M out of 690M (12.50%) params' optimizer states | |
| [default0]:07/02/2024 16:32:37 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [ZeRO sharding] DP Rank 2 has 86.3M out of 690M (12.50%) params' optimizer states | |
| [default0]:07/02/2024 16:32:37 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [ZeRO sharding] DP Rank 3 has 86.3M out of 690M (12.50%) params' optimizer states | |
| [default0]:07/02/2024 16:32:37 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [ZeRO sharding] DP Rank 4 has 86.3M out of 690M (12.50%) params' optimizer states | |
| [default0]:07/02/2024 16:32:37 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [ZeRO sharding] DP Rank 5 has 86.3M out of 690M (12.50%) params' optimizer states | |
| [default0]:07/02/2024 16:32:37 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [ZeRO sharding] DP Rank 6 has 86.3M out of 690M (12.50%) params' optimizer states | |
| [default0]:07/02/2024 16:32:37 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [ZeRO sharding] DP Rank 7 has 86.3M out of 690M (12.50%) params' optimizer states | |
| [default0]:07/02/2024 16:32:39 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [Training Plan] Stage Training Stage has 19 remaining training steps and has consumed 0 samples | |
| [default0]:07/02/2024 16:32:39 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Using `datasets` library | |
| [default0]:07/02/2024 16:32:39 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Loading tokenizer from openai-community/gpt2 and transformers/hf_hub versions ('4.41.2', '0.23.4') | |
| [default0]:07/02/2024 16:32:39 [WARNING|DP=0|PP=0|TP=0|ip-26-0-160-225]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default0]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default0]:07/02/2024 16:32:40 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [Training Plan] There are 1 training stages | |
| [default0]:07/02/2024 16:32:40 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [Stage Training Stage] start from step 1 | |
| [default0]:07/02/2024 16:32:40 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: | |
| [default0]:07/02/2024 16:32:40 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: [Start training] datetime: 2024-07-02 16:32:40.075618 | mbs: 1 | grad_accum: 128 | global_batch_size: 1024 | sequence_length: 4096 | train_steps: 20 | start_iteration_step: 0 | consumed_train_samples: 0 | |
| [default0]:07/02/2024 16:32:40 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Resuming training from stage Training Stage, it has trained for 0 samples and has 19 remaining train steps | |
| [default0]:07/02/2024 16:32:40 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Memory usage: 4293.29MiB. Peak allocated 4293.29MiB. Peak reserved: 4328.00MiB | |
| [default0]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default0]:07/02/2024 16:32:40 [WARNING|DP=0|PP=1|TP=0|ip-26-0-171-56]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default5]:07/02/2024 16:32:40 [WARNING|DP=5|PP=1|TP=0|ip-26-0-171-56]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:07/02/2024 16:32:40 [WARNING|DP=7|PP=1|TP=0|ip-26-0-171-56]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default6]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default1]:07/02/2024 16:32:40 [WARNING|DP=1|PP=1|TP=0|ip-26-0-171-56]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default6]:07/02/2024 16:32:40 [WARNING|DP=6|PP=1|TP=0|ip-26-0-171-56]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default2]:07/02/2024 16:32:40 [WARNING|DP=2|PP=1|TP=0|ip-26-0-171-56]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default3]:07/02/2024 16:32:40 [WARNING|DP=3|PP=1|TP=0|ip-26-0-171-56]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default1]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default5]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default3]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default2]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default2]:07/02/2024 16:32:40 [WARNING|DP=2|PP=0|TP=0|ip-26-0-160-225]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default4]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default4]:07/02/2024 16:32:40 [WARNING|DP=4|PP=0|TP=0|ip-26-0-160-225]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default2]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default5]:07/02/2024 16:32:40 [WARNING|DP=5|PP=0|TP=0|ip-26-0-160-225]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default3]:07/02/2024 16:32:40 [WARNING|DP=3|PP=0|TP=0|ip-26-0-160-225]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default5]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default3]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default6]:07/02/2024 16:32:40 [WARNING|DP=6|PP=0|TP=0|ip-26-0-160-225]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default4]:07/02/2024 16:32:40 [WARNING|DP=4|PP=1|TP=0|ip-26-0-171-56]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:07/02/2024 16:32:40 [WARNING|DP=7|PP=0|TP=0|ip-26-0-160-225]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default7]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default4]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default6]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default1]:Repo card metadata block was not found. Setting CardData to empty. | |
| [default1]:07/02/2024 16:32:40 [WARNING|DP=1|PP=0|TP=0|ip-26-0-160-225]: Repo card metadata block was not found. Setting CardData to empty. | |
| [default0]:[rank0]: OSError: [Errno 122] Disk quota exceeded | |
| [default0]: | |
| [default0]:[rank0]: During handling of the above exception, another exception occurred: | |
| [default0]: | |
| [default0]:[rank0]: Traceback (most recent call last): | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default0]:[rank0]: trainer.train(dataloader) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default0]:[rank0]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default0]:[rank0]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default0]:[rank0]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default0]:[rank0]: output = model(**micro_batch) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank0]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank0]: return forward_call(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default0]:[rank0]: sharded_logits = self.model( | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank0]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank0]: return forward_call(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default0]:[rank0]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default0]:[rank0]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank0]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank0]: return forward_call(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward | |
| [default0]:[rank0]: output = self.pp_block(**new_kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank0]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank0]: return forward_call(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward | |
| [default0]:[rank0]: output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank0]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank0]: return forward_call(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 566, in forward | |
| [default0]:[rank0]: query_states, key_value_states = self.flash_rotary_embedding(query_states, kv=key_value_states) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank0]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank0]: return forward_call(*args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/layers/rotary.py", line 457, in forward | |
| [default0]:[rank0]: q = apply_rotary_emb_func( | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/layers/rotary.py", line 122, in apply_rotary_emb | |
| [default0]:[rank0]: return ApplyRotaryEmb.apply( | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/function.py", line 598, in apply | |
| [default0]:[rank0]: return super().apply(*args, **kwargs) # type: ignore[misc] | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/layers/rotary.py", line 48, in forward | |
| [default0]:[rank0]: out = apply_rotary( | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/rotary.py", line 202, in apply_rotary | |
| [default0]:[rank0]: rotary_kernel[grid]( | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/jit.py", line 167, in <lambda> | |
| [default0]:[rank0]: return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/jit.py", line 416, in run | |
| [default0]:[rank0]: self.cache[device][key] = compile( | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/compiler/compiler.py", line 194, in compile | |
| [default0]:[rank0]: metadata_group[f"{src.name}.{ext}"] = fn_cache_manager.put(next_module, f"{src.name}.{ext}") | |
| [default0]:[rank0]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/cache.py", line 123, in put | |
| [default0]:[rank0]: with open(temp_path, mode) as f: | |
| [default0]:[rank0]: OSError: [Errno 122] Disk quota exceeded | |
| [default7]:[rank7]: OSError: [Errno 122] Disk quota exceeded | |
| [default7]: | |
| [default7]:[rank7]: During handling of the above exception, another exception occurred: | |
| [default7]: | |
| [default7]:[rank7]: Traceback (most recent call last): | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default7]:[rank7]: trainer.train(dataloader) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default7]:[rank7]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default7]:[rank7]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default7]:[rank7]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default7]:[rank7]: output = model(**micro_batch) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank7]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank7]: return forward_call(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default7]:[rank7]: sharded_logits = self.model( | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank7]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank7]: return forward_call(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default7]:[rank7]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default7]:[rank7]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank7]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank7]: return forward_call(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward | |
| [default7]:[rank7]: output = self.pp_block(**new_kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank7]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank7]: return forward_call(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 629, in forward | |
| [default7]:[rank7]: hidden_states = self.input_layernorm(hidden_states) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank7]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank7]: return forward_call(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/nn/layer_norm.py", line 42, in forward | |
| [default7]:[rank7]: return layer_norm_fn( | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py", line 875, in layer_norm_fn | |
| [default7]:[rank7]: return LayerNormFn.apply( | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/function.py", line 598, in apply | |
| [default7]:[rank7]: return super().apply(*args, **kwargs) # type: ignore[misc] | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py", line 748, in forward | |
| [default7]:[rank7]: y, y1, mean, rstd, residual_out, seeds, dropout_mask, dropout_mask1 = _layer_norm_fwd( | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py", line 335, in _layer_norm_fwd | |
| [default7]:[rank7]: _layer_norm_fwd_1pass_kernel[(M,)]( | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/jit.py", line 167, in <lambda> | |
| [default7]:[rank7]: return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 143, in run | |
| [default7]:[rank7]: timings = {config: self._bench(*args, config=config, **kwargs) for config in pruned_configs} | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 143, in <dictcomp> | |
| [default7]:[rank7]: timings = {config: self._bench(*args, config=config, **kwargs) for config in pruned_configs} | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 122, in _bench | |
| [default7]:[rank7]: return do_bench(kernel_call, warmup=self.warmup, rep=self.rep, quantiles=(0.5, 0.2, 0.8)) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/testing.py", line 102, in do_bench | |
| [default7]:[rank7]: fn() | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 110, in kernel_call | |
| [default7]:[rank7]: self.fn.run( | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 305, in run | |
| [default7]:[rank7]: return self.fn.run(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 305, in run | |
| [default7]:[rank7]: return self.fn.run(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 305, in run | |
| [default7]:[rank7]: return self.fn.run(*args, **kwargs) | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/jit.py", line 416, in run | |
| [default7]:[rank7]: self.cache[device][key] = compile( | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/compiler/compiler.py", line 194, in compile | |
| [default7]:[rank7]: metadata_group[f"{src.name}.{ext}"] = fn_cache_manager.put(next_module, f"{src.name}.{ext}") | |
| [default7]:[rank7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/cache.py", line 123, in put | |
| [default7]:[rank7]: with open(temp_path, mode) as f: | |
| [default7]:[rank7]: OSError: [Errno 122] Disk quota exceeded | |
| [default3]:[rank3]: OSError: [Errno 122] Disk quota exceeded | |
| [default3]: | |
| [default3]:[rank3]: During handling of the above exception, another exception occurred: | |
| [default3]: | |
| [default3]:[rank3]: Traceback (most recent call last): | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default3]:[rank3]: trainer.train(dataloader) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default3]:[rank3]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default3]:[rank3]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default3]:[rank3]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default3]:[rank3]: output = model(**micro_batch) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank3]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank3]: return forward_call(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default3]:[rank3]: sharded_logits = self.model( | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank3]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank3]: return forward_call(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default3]:[rank3]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default3]:[rank3]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank3]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank3]: return forward_call(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward | |
| [default3]:[rank3]: output = self.pp_block(**new_kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank3]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank3]: return forward_call(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 629, in forward | |
| [default3]:[rank3]: hidden_states = self.input_layernorm(hidden_states) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank3]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank3]: return forward_call(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/nn/layer_norm.py", line 42, in forward | |
| [default3]:[rank3]: return layer_norm_fn( | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py", line 875, in layer_norm_fn | |
| [default3]:[rank3]: return LayerNormFn.apply( | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/function.py", line 598, in apply | |
| [default3]:[rank3]: return super().apply(*args, **kwargs) # type: ignore[misc] | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py", line 748, in forward | |
| [default3]:[rank3]: y, y1, mean, rstd, residual_out, seeds, dropout_mask, dropout_mask1 = _layer_norm_fwd( | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py", line 335, in _layer_norm_fwd | |
| [default3]:[rank3]: _layer_norm_fwd_1pass_kernel[(M,)]( | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/jit.py", line 167, in <lambda> | |
| [default3]:[rank3]: return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 143, in run | |
| [default3]:[rank3]: timings = {config: self._bench(*args, config=config, **kwargs) for config in pruned_configs} | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 143, in <dictcomp> | |
| [default3]:[rank3]: timings = {config: self._bench(*args, config=config, **kwargs) for config in pruned_configs} | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 122, in _bench | |
| [default3]:[rank3]: return do_bench(kernel_call, warmup=self.warmup, rep=self.rep, quantiles=(0.5, 0.2, 0.8)) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/testing.py", line 102, in do_bench | |
| [default3]:[rank3]: fn() | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 110, in kernel_call | |
| [default3]:[rank3]: self.fn.run( | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 305, in run | |
| [default3]:[rank3]: return self.fn.run(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 305, in run | |
| [default3]:[rank3]: return self.fn.run(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 305, in run | |
| [default3]:[rank3]: return self.fn.run(*args, **kwargs) | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/jit.py", line 416, in run | |
| [default3]:[rank3]: self.cache[device][key] = compile( | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/compiler/compiler.py", line 194, in compile | |
| [default3]:[rank3]: metadata_group[f"{src.name}.{ext}"] = fn_cache_manager.put(next_module, f"{src.name}.{ext}") | |
| [default3]:[rank3]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/cache.py", line 123, in put | |
| [default3]:[rank3]: with open(temp_path, mode) as f: | |
| [default3]:[rank3]: OSError: [Errno 122] Disk quota exceeded | |
| [default6]:[rank6]: OSError: [Errno 122] Disk quota exceeded | |
| [default6]: | |
| [default6]:[rank6]: During handling of the above exception, another exception occurred: | |
| [default6]: | |
| [default6]:[rank6]: Traceback (most recent call last): | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default6]:[rank6]: trainer.train(dataloader) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default6]:[rank6]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default6]:[rank6]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default6]:[rank6]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default6]:[rank6]: output = model(**micro_batch) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank6]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank6]: return forward_call(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default6]:[rank6]: sharded_logits = self.model( | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank6]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank6]: return forward_call(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default6]:[rank6]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default6]:[rank6]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank6]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank6]: return forward_call(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward | |
| [default6]:[rank6]: output = self.pp_block(**new_kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank6]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank6]: return forward_call(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 629, in forward | |
| [default6]:[rank6]: hidden_states = self.input_layernorm(hidden_states) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank6]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank6]: return forward_call(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/nn/layer_norm.py", line 42, in forward | |
| [default6]:[rank6]: return layer_norm_fn( | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py", line 875, in layer_norm_fn | |
| [default6]:[rank6]: return LayerNormFn.apply( | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/function.py", line 598, in apply | |
| [default6]:[rank6]: return super().apply(*args, **kwargs) # type: ignore[misc] | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py", line 748, in forward | |
| [default6]:[rank6]: y, y1, mean, rstd, residual_out, seeds, dropout_mask, dropout_mask1 = _layer_norm_fwd( | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py", line 335, in _layer_norm_fwd | |
| [default6]:[rank6]: _layer_norm_fwd_1pass_kernel[(M,)]( | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/jit.py", line 167, in <lambda> | |
| [default6]:[rank6]: return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 143, in run | |
| [default6]:[rank6]: timings = {config: self._bench(*args, config=config, **kwargs) for config in pruned_configs} | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 143, in <dictcomp> | |
| [default6]:[rank6]: timings = {config: self._bench(*args, config=config, **kwargs) for config in pruned_configs} | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 122, in _bench | |
| [default6]:[rank6]: return do_bench(kernel_call, warmup=self.warmup, rep=self.rep, quantiles=(0.5, 0.2, 0.8)) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/testing.py", line 102, in do_bench | |
| [default6]:[rank6]: fn() | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 110, in kernel_call | |
| [default6]:[rank6]: self.fn.run( | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 305, in run | |
| [default6]:[rank6]: return self.fn.run(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 305, in run | |
| [default6]:[rank6]: return self.fn.run(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 305, in run | |
| [default6]:[rank6]: return self.fn.run(*args, **kwargs) | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/jit.py", line 416, in run | |
| [default6]:[rank6]: self.cache[device][key] = compile( | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/compiler/compiler.py", line 194, in compile | |
| [default6]:[rank6]: metadata_group[f"{src.name}.{ext}"] = fn_cache_manager.put(next_module, f"{src.name}.{ext}") | |
| [default6]:[rank6]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/cache.py", line 123, in put | |
| [default6]:[rank6]: with open(temp_path, mode) as f: | |
| [default6]:[rank6]: OSError: [Errno 122] Disk quota exceeded | |
| [default5]:[rank5]: OSError: [Errno 122] Disk quota exceeded | |
| [default5]: | |
| [default5]:[rank5]: During handling of the above exception, another exception occurred: | |
| [default5]: | |
| [default5]:[rank5]: Traceback (most recent call last): | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default5]:[rank5]: trainer.train(dataloader) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default5]:[rank5]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default5]:[rank5]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default5]:[rank5]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default5]:[rank5]: output = model(**micro_batch) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank5]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank5]: return forward_call(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default5]:[rank5]: sharded_logits = self.model( | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank5]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank5]: return forward_call(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default5]:[rank5]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default5]:[rank5]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank5]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank5]: return forward_call(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward | |
| [default5]:[rank5]: output = self.pp_block(**new_kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank5]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank5]: return forward_call(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 629, in forward | |
| [default5]:[rank5]: hidden_states = self.input_layernorm(hidden_states) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank5]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank5]: return forward_call(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/nn/layer_norm.py", line 42, in forward | |
| [default5]:[rank5]: return layer_norm_fn( | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py", line 875, in layer_norm_fn | |
| [default5]:[rank5]: return LayerNormFn.apply( | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/function.py", line 598, in apply | |
| [default5]:[rank5]: return super().apply(*args, **kwargs) # type: ignore[misc] | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py", line 748, in forward | |
| [default5]:[rank5]: y, y1, mean, rstd, residual_out, seeds, dropout_mask, dropout_mask1 = _layer_norm_fwd( | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py", line 335, in _layer_norm_fwd | |
| [default5]:[rank5]: _layer_norm_fwd_1pass_kernel[(M,)]( | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/jit.py", line 167, in <lambda> | |
| [default5]:[rank5]: return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 143, in run | |
| [default5]:[rank5]: timings = {config: self._bench(*args, config=config, **kwargs) for config in pruned_configs} | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 143, in <dictcomp> | |
| [default5]:[rank5]: timings = {config: self._bench(*args, config=config, **kwargs) for config in pruned_configs} | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 122, in _bench | |
| [default5]:[rank5]: return do_bench(kernel_call, warmup=self.warmup, rep=self.rep, quantiles=(0.5, 0.2, 0.8)) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/testing.py", line 102, in do_bench | |
| [default5]:[rank5]: fn() | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 110, in kernel_call | |
| [default5]:[rank5]: self.fn.run( | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 305, in run | |
| [default5]:[rank5]: return self.fn.run(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 305, in run | |
| [default5]:[rank5]: return self.fn.run(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 305, in run | |
| [default5]:[rank5]: return self.fn.run(*args, **kwargs) | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/jit.py", line 416, in run | |
| [default5]:[rank5]: self.cache[device][key] = compile( | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/compiler/compiler.py", line 194, in compile | |
| [default5]:[rank5]: metadata_group[f"{src.name}.{ext}"] = fn_cache_manager.put(next_module, f"{src.name}.{ext}") | |
| [default5]:[rank5]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/cache.py", line 123, in put | |
| [default5]:[rank5]: with open(temp_path, mode) as f: | |
| [default5]:[rank5]: OSError: [Errno 122] Disk quota exceeded | |
| [default0]:[rank8]: Traceback (most recent call last): | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default0]:[rank8]: trainer.train(dataloader) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default0]:[rank8]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default0]:[rank8]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default0]:[rank8]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default0]:[rank8]: output = model(**micro_batch) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank8]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank8]: return forward_call(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default0]:[rank8]: sharded_logits = self.model( | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank8]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank8]: return forward_call(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default0]:[rank8]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default0]:[rank8]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default0]:[rank8]: return self._call_impl(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default0]:[rank8]: return forward_call(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default0]:[rank8]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default0]:[rank8]: pipeline_state.run_communication() | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default0]:[rank8]: recv_activation_tensor = recv_activation() | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default0]:[rank8]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default0]:[rank8]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default0]:[rank8]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default0]:[rank8]: dist.recv( | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default0]:[rank8]: return func(*args, **kwargs) | |
| [default0]:[rank8]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default0]:[rank8]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default0]:[rank8]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer | |
| [default0]:[rank8]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default0]:[rank8]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fb1c81f7897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default0]:[rank8]: frame #1: <unknown function> + 0x5b3a23e (0x7fb201d1423e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fb201d0ec87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fb201d0ef82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fb201d0ffd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb201cc4371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb201cc4371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb201cc4371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb201cc4371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fb1c94d1189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default0]:[rank8]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fb1c94d8610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default0]:[rank8]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fb1c94f7978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default0]:[rank8]: frame #12: <unknown function> + 0x5adc309 (0x7fb201cb6309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #13: <unknown function> + 0x5ae6f10 (0x7fb201cc0f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fb201cc0fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #15: <unknown function> + 0x5124446 (0x7fb2012fe446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #16: <unknown function> + 0x1acf4b8 (0x7fb1fdca94b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #17: <unknown function> + 0x5aee004 (0x7fb201cc8004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #18: <unknown function> + 0x5af36b5 (0x7fb201ccd6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default0]:[rank8]: frame #19: <unknown function> + 0xd2631e (0x7fb2148b731e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default0]:[rank8]: frame #20: <unknown function> + 0x47def4 (0x7fb21400eef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default0]:[rank8]: frame #21: <unknown function> + 0x1445a6 (0x55b90acb25a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55b90acaba6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #23: <unknown function> + 0x150866 (0x55b90acbe866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55b90aca7142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55b90acb2a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #26: PyObject_Call + 0xbc (0x55b90acbef1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55b90aca52b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55b90acb2a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55b90aca38fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #30: <unknown function> + 0x150582 (0x55b90acbe582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55b90aca38fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #32: <unknown function> + 0x150582 (0x55b90acbe582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55b90aca38fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #34: <unknown function> + 0x150582 (0x55b90acbe582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55b90aca38fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55b90acaaf50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55b90acbcc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #38: <unknown function> + 0x211239 (0x55b90ad7f239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55b90acaba6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55b90aca73e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55b90acb2a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55b90aca2c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55b90acb2a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55b90aca38fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #45: <unknown function> + 0x150582 (0x55b90acbe582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #46: PyObject_Call + 0xbc (0x55b90acbef1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55b90aca52b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #48: <unknown function> + 0x150582 (0x55b90acbe582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #49: PyObject_Call + 0xbc (0x55b90acbef1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55b90aca52b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55b90acb2a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55b90acab007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55b90acbcc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #54: <unknown function> + 0x211239 (0x55b90ad7f239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #55: PyObject_Call + 0x207 (0x55b90acbf067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55b90aca52b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #57: <unknown function> + 0x150582 (0x55b90acbe582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55b90aca38fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #59: <unknown function> + 0x150582 (0x55b90acbe582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #60: PyObject_Call + 0xbc (0x55b90acbef1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55b90aca52b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #62: <unknown function> + 0x150582 (0x55b90acbe582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: frame #63: PyObject_Call + 0xbc (0x55b90acbef1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default0]:[rank8]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default1]:[rank9]: Traceback (most recent call last): | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default1]:[rank9]: trainer.train(dataloader) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default1]:[rank9]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default1]:[rank9]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default1]:[rank9]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default1]:[rank9]: output = model(**micro_batch) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank9]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank9]: return forward_call(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default1]:[rank9]: sharded_logits = self.model( | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank9]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank9]: return forward_call(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default1]:[rank9]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default1]:[rank9]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank9]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank9]: return forward_call(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default1]:[rank9]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default1]:[rank9]: pipeline_state.run_communication() | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default1]:[rank9]: recv_activation_tensor = recv_activation() | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default1]:[rank9]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default1]:[rank9]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default1]:[rank9]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default1]:[rank9]: dist.recv( | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default1]:[rank9]: return func(*args, **kwargs) | |
| [default1]:[rank9]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default1]:[rank9]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default1]:[rank9]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer | |
| [default1]:[rank9]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default1]:[rank9]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f6926558897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default1]:[rank9]: frame #1: <unknown function> + 0x5b3a23e (0x7f696007523e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f696006fc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f696006ff82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f6960070fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6960025371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6960025371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6960025371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6960025371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f6927832189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default1]:[rank9]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f6927839610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default1]:[rank9]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f6927858978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default1]:[rank9]: frame #12: <unknown function> + 0x5adc309 (0x7f6960017309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #13: <unknown function> + 0x5ae6f10 (0x7f6960021f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f6960021fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #15: <unknown function> + 0x5124446 (0x7f695f65f446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #16: <unknown function> + 0x1acf4b8 (0x7f695c00a4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #17: <unknown function> + 0x5aee004 (0x7f6960029004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #18: <unknown function> + 0x5af36b5 (0x7f696002e6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default1]:[rank9]: frame #19: <unknown function> + 0xd2631e (0x7f6972c1831e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default1]:[rank9]: frame #20: <unknown function> + 0x47def4 (0x7f697236fef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default1]:[rank9]: frame #21: <unknown function> + 0x1445a6 (0x560d2166c5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #22: _PyObject_MakeTpCall + 0x26b (0x560d21665a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #23: <unknown function> + 0x150866 (0x560d21678866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x560d21661142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #25: _PyFunction_Vectorcall + 0x6c (0x560d2166ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #26: PyObject_Call + 0xbc (0x560d21678f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x560d2165f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #28: _PyFunction_Vectorcall + 0x6c (0x560d2166ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x560d2165d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #30: <unknown function> + 0x150582 (0x560d21678582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x560d2165d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #32: <unknown function> + 0x150582 (0x560d21678582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x560d2165d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #34: <unknown function> + 0x150582 (0x560d21678582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x560d2165d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x560d21664f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #37: _PyObject_Call_Prepend + 0x69 (0x560d21676c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #38: <unknown function> + 0x211239 (0x560d21739239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #39: _PyObject_MakeTpCall + 0x26b (0x560d21665a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x560d216613e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #41: _PyFunction_Vectorcall + 0x6c (0x560d2166ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x560d2165cc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #43: _PyFunction_Vectorcall + 0x6c (0x560d2166ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x560d2165d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #45: <unknown function> + 0x150582 (0x560d21678582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #46: PyObject_Call + 0xbc (0x560d21678f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x560d2165f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #48: <unknown function> + 0x150582 (0x560d21678582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #49: PyObject_Call + 0xbc (0x560d21678f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x560d2165f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #51: _PyFunction_Vectorcall + 0x6c (0x560d2166ca2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x560d21665007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #53: _PyObject_Call_Prepend + 0x69 (0x560d21676c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #54: <unknown function> + 0x211239 (0x560d21739239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #55: PyObject_Call + 0x207 (0x560d21679067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x560d2165f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #57: <unknown function> + 0x150582 (0x560d21678582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x560d2165d8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #59: <unknown function> + 0x150582 (0x560d21678582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #60: PyObject_Call + 0xbc (0x560d21678f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x560d2165f2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #62: <unknown function> + 0x150582 (0x560d21678582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: frame #63: PyObject_Call + 0xbc (0x560d21678f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default1]:[rank9]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default3]:[rank11]: Traceback (most recent call last): | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default3]:[rank11]: trainer.train(dataloader) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default3]:[rank11]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default3]:[rank11]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default3]:[rank11]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default3]:[rank11]: output = model(**micro_batch) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank11]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank11]: return forward_call(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default3]:[rank11]: sharded_logits = self.model( | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank11]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank11]: return forward_call(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default3]:[rank11]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default3]:[rank11]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default3]:[rank11]: return self._call_impl(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default3]:[rank11]: return forward_call(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default3]:[rank11]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default3]:[rank11]: pipeline_state.run_communication() | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default3]:[rank11]: recv_activation_tensor = recv_activation() | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default3]:[rank11]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default3]:[rank11]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default3]:[rank11]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default3]:[rank11]: dist.recv( | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default3]:[rank11]: return func(*args, **kwargs) | |
| [default3]:[rank11]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default3]:[rank11]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default3]:[rank11]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer | |
| [default3]:[rank11]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default3]:[rank11]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f10b9f26897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default3]:[rank11]: frame #1: <unknown function> + 0x5b3a23e (0x7f10f3a4323e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f10f3a3dc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f10f3a3df82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f10f3a3efd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f10f39f3371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f10f39f3371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f10f39f3371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f10f39f3371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f10bb200189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:[rank11]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f10bb207610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:[rank11]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f10bb226978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default3]:[rank11]: frame #12: <unknown function> + 0x5adc309 (0x7f10f39e5309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #13: <unknown function> + 0x5ae6f10 (0x7f10f39eff10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f10f39effa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #15: <unknown function> + 0x5124446 (0x7f10f302d446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #16: <unknown function> + 0x1acf4b8 (0x7f10ef9d84b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #17: <unknown function> + 0x5aee004 (0x7f10f39f7004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #18: <unknown function> + 0x5af36b5 (0x7f10f39fc6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default3]:[rank11]: frame #19: <unknown function> + 0xd2631e (0x7f11065e631e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default3]:[rank11]: frame #20: <unknown function> + 0x47def4 (0x7f1105d3def4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default3]:[rank11]: frame #21: <unknown function> + 0x1445a6 (0x562eff4e75a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #22: _PyObject_MakeTpCall + 0x26b (0x562eff4e0a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #23: <unknown function> + 0x150866 (0x562eff4f3866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x562eff4dc142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #25: _PyFunction_Vectorcall + 0x6c (0x562eff4e7a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #26: PyObject_Call + 0xbc (0x562eff4f3f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x562eff4da2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #28: _PyFunction_Vectorcall + 0x6c (0x562eff4e7a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x562eff4d88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #30: <unknown function> + 0x150582 (0x562eff4f3582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x562eff4d88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #32: <unknown function> + 0x150582 (0x562eff4f3582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x562eff4d88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #34: <unknown function> + 0x150582 (0x562eff4f3582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x562eff4d88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x562eff4dff50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #37: _PyObject_Call_Prepend + 0x69 (0x562eff4f1c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #38: <unknown function> + 0x211239 (0x562eff5b4239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #39: _PyObject_MakeTpCall + 0x26b (0x562eff4e0a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x562eff4dc3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #41: _PyFunction_Vectorcall + 0x6c (0x562eff4e7a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x562eff4d7c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #43: _PyFunction_Vectorcall + 0x6c (0x562eff4e7a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x562eff4d88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #45: <unknown function> + 0x150582 (0x562eff4f3582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #46: PyObject_Call + 0xbc (0x562eff4f3f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x562eff4da2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #48: <unknown function> + 0x150582 (0x562eff4f3582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #49: PyObject_Call + 0xbc (0x562eff4f3f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x562eff4da2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #51: _PyFunction_Vectorcall + 0x6c (0x562eff4e7a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x562eff4e0007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #53: _PyObject_Call_Prepend + 0x69 (0x562eff4f1c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #54: <unknown function> + 0x211239 (0x562eff5b4239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #55: PyObject_Call + 0x207 (0x562eff4f4067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x562eff4da2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #57: <unknown function> + 0x150582 (0x562eff4f3582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x562eff4d88fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #59: <unknown function> + 0x150582 (0x562eff4f3582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #60: PyObject_Call + 0xbc (0x562eff4f3f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x562eff4da2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #62: <unknown function> + 0x150582 (0x562eff4f3582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: frame #63: PyObject_Call + 0xbc (0x562eff4f3f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default3]:[rank11]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default2]:[rank10]: Traceback (most recent call last): | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default2]:[rank10]: trainer.train(dataloader) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default2]:[rank10]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default2]:[rank10]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default2]:[rank10]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default2]:[rank10]: output = model(**micro_batch) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank10]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank10]: return forward_call(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default2]:[rank10]: sharded_logits = self.model( | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank10]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank10]: return forward_call(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default2]:[rank10]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default2]:[rank10]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default2]:[rank10]: return self._call_impl(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default2]:[rank10]: return forward_call(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default2]:[rank10]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default2]:[rank10]: pipeline_state.run_communication() | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default2]:[rank10]: recv_activation_tensor = recv_activation() | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default2]:[rank10]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default2]:[rank10]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default2]:[rank10]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default2]:[rank10]: dist.recv( | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default2]:[rank10]: return func(*args, **kwargs) | |
| [default2]:[rank10]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default2]:[rank10]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default2]:[rank10]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer | |
| [default2]:[rank10]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default2]:[rank10]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fe56b84c897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default2]:[rank10]: frame #1: <unknown function> + 0x5b3a23e (0x7fe5a536923e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fe5a5363c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fe5a5363f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fe5a5364fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe5a5319371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe5a5319371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe5a5319371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe5a5319371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fe56cb26189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:[rank10]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fe56cb2d610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:[rank10]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fe56cb4c978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default2]:[rank10]: frame #12: <unknown function> + 0x5adc309 (0x7fe5a530b309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #13: <unknown function> + 0x5ae6f10 (0x7fe5a5315f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fe5a5315fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #15: <unknown function> + 0x5124446 (0x7fe5a4953446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #16: <unknown function> + 0x1acf4b8 (0x7fe5a12fe4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #17: <unknown function> + 0x5aee004 (0x7fe5a531d004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #18: <unknown function> + 0x5af36b5 (0x7fe5a53226b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default2]:[rank10]: frame #19: <unknown function> + 0xd2631e (0x7fe5b7f0c31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default2]:[rank10]: frame #20: <unknown function> + 0x47def4 (0x7fe5b7663ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default2]:[rank10]: frame #21: <unknown function> + 0x1445a6 (0x55e086f675a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55e086f60a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #23: <unknown function> + 0x150866 (0x55e086f73866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55e086f5c142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55e086f67a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #26: PyObject_Call + 0xbc (0x55e086f73f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55e086f5a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55e086f67a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55e086f588fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #30: <unknown function> + 0x150582 (0x55e086f73582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55e086f588fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #32: <unknown function> + 0x150582 (0x55e086f73582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55e086f588fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #34: <unknown function> + 0x150582 (0x55e086f73582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55e086f588fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55e086f5ff50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55e086f71c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #38: <unknown function> + 0x211239 (0x55e087034239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55e086f60a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55e086f5c3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55e086f67a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55e086f57c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55e086f67a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55e086f588fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #45: <unknown function> + 0x150582 (0x55e086f73582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #46: PyObject_Call + 0xbc (0x55e086f73f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55e086f5a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #48: <unknown function> + 0x150582 (0x55e086f73582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #49: PyObject_Call + 0xbc (0x55e086f73f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55e086f5a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55e086f67a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55e086f60007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55e086f71c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #54: <unknown function> + 0x211239 (0x55e087034239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #55: PyObject_Call + 0x207 (0x55e086f74067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55e086f5a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #57: <unknown function> + 0x150582 (0x55e086f73582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55e086f588fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #59: <unknown function> + 0x150582 (0x55e086f73582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #60: PyObject_Call + 0xbc (0x55e086f73f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55e086f5a2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #62: <unknown function> + 0x150582 (0x55e086f73582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: frame #63: PyObject_Call + 0xbc (0x55e086f73f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default2]:[rank10]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default1]:[rank1]: OSError: [Errno 122] Disk quota exceeded | |
| [default1]: | |
| [default1]:[rank1]: During handling of the above exception, another exception occurred: | |
| [default1]: | |
| [default1]:[rank1]: Traceback (most recent call last): | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default1]:[rank1]: trainer.train(dataloader) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default1]:[rank1]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default1]:[rank1]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 252, in train_batch_iter | |
| [default1]:[rank1]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default1]:[rank1]: output = model(**micro_batch) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default1]:[rank1]: sharded_logits = self.model( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default1]:[rank1]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default1]:[rank1]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 151, in forward | |
| [default1]:[rank1]: output = self.pp_block(**new_kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 631, in forward | |
| [default1]:[rank1]: output = self.attn(hidden_states=hidden_states, sequence_mask=sequence_mask) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 566, in forward | |
| [default1]:[rank1]: query_states, key_value_states = self.flash_rotary_embedding(query_states, kv=key_value_states) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default1]:[rank1]: return self._call_impl(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default1]:[rank1]: return forward_call(*args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/layers/rotary.py", line 457, in forward | |
| [default1]:[rank1]: q = apply_rotary_emb_func( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/layers/rotary.py", line 122, in apply_rotary_emb | |
| [default1]:[rank1]: return ApplyRotaryEmb.apply( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/function.py", line 598, in apply | |
| [default1]:[rank1]: return super().apply(*args, **kwargs) # type: ignore[misc] | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/layers/rotary.py", line 48, in forward | |
| [default1]:[rank1]: out = apply_rotary( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/rotary.py", line 202, in apply_rotary | |
| [default1]:[rank1]: rotary_kernel[grid]( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/jit.py", line 167, in <lambda> | |
| [default1]:[rank1]: return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/jit.py", line 416, in run | |
| [default1]:[rank1]: self.cache[device][key] = compile( | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/compiler/compiler.py", line 194, in compile | |
| [default1]:[rank1]: metadata_group[f"{src.name}.{ext}"] = fn_cache_manager.put(next_module, f"{src.name}.{ext}") | |
| [default1]:[rank1]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/cache.py", line 123, in put | |
| [default1]:[rank1]: with open(temp_path, mode) as f: | |
| [default1]:[rank1]: OSError: [Errno 122] Disk quota exceeded | |
| [default6]:[rank14]: Traceback (most recent call last): | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default6]:[rank14]: trainer.train(dataloader) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default6]:[rank14]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default6]:[rank14]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default6]:[rank14]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default6]:[rank14]: output = model(**micro_batch) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank14]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank14]: return forward_call(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default6]:[rank14]: sharded_logits = self.model( | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank14]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank14]: return forward_call(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default6]:[rank14]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default6]:[rank14]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default6]:[rank14]: return self._call_impl(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default6]:[rank14]: return forward_call(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default6]:[rank14]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default6]:[rank14]: pipeline_state.run_communication() | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default6]:[rank14]: recv_activation_tensor = recv_activation() | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default6]:[rank14]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default6]:[rank14]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default6]:[rank14]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default6]:[rank14]: dist.recv( | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default6]:[rank14]: return func(*args, **kwargs) | |
| [default6]:[rank14]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default6]:[rank14]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default6]:[rank14]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer | |
| [default6]:[rank14]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default6]:[rank14]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f2d2ba67897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default6]:[rank14]: frame #1: <unknown function> + 0x5b3a23e (0x7f2d6558423e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f2d6557ec87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f2d6557ef82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f2d6557ffd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2d65534371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2d65534371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2d65534371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2d65534371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f2d2cd41189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:[rank14]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f2d2cd48610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:[rank14]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f2d2cd67978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default6]:[rank14]: frame #12: <unknown function> + 0x5adc309 (0x7f2d65526309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #13: <unknown function> + 0x5ae6f10 (0x7f2d65530f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f2d65530fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #15: <unknown function> + 0x5124446 (0x7f2d64b6e446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #16: <unknown function> + 0x1acf4b8 (0x7f2d615194b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #17: <unknown function> + 0x5aee004 (0x7f2d65538004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #18: <unknown function> + 0x5af36b5 (0x7f2d6553d6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default6]:[rank14]: frame #19: <unknown function> + 0xd2631e (0x7f2d7812731e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default6]:[rank14]: frame #20: <unknown function> + 0x47def4 (0x7f2d7787eef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default6]:[rank14]: frame #21: <unknown function> + 0x1445a6 (0x5605aa5c05a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #22: _PyObject_MakeTpCall + 0x26b (0x5605aa5b9a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #23: <unknown function> + 0x150866 (0x5605aa5cc866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x5605aa5b5142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #25: _PyFunction_Vectorcall + 0x6c (0x5605aa5c0a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #26: PyObject_Call + 0xbc (0x5605aa5ccf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x5605aa5b32b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #28: _PyFunction_Vectorcall + 0x6c (0x5605aa5c0a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x5605aa5b18fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #30: <unknown function> + 0x150582 (0x5605aa5cc582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x5605aa5b18fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #32: <unknown function> + 0x150582 (0x5605aa5cc582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x5605aa5b18fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #34: <unknown function> + 0x150582 (0x5605aa5cc582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x5605aa5b18fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x5605aa5b8f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #37: _PyObject_Call_Prepend + 0x69 (0x5605aa5cac39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #38: <unknown function> + 0x211239 (0x5605aa68d239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #39: _PyObject_MakeTpCall + 0x26b (0x5605aa5b9a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x5605aa5b53e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #41: _PyFunction_Vectorcall + 0x6c (0x5605aa5c0a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x5605aa5b0c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #43: _PyFunction_Vectorcall + 0x6c (0x5605aa5c0a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x5605aa5b18fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #45: <unknown function> + 0x150582 (0x5605aa5cc582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #46: PyObject_Call + 0xbc (0x5605aa5ccf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x5605aa5b32b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #48: <unknown function> + 0x150582 (0x5605aa5cc582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #49: PyObject_Call + 0xbc (0x5605aa5ccf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x5605aa5b32b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #51: _PyFunction_Vectorcall + 0x6c (0x5605aa5c0a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x5605aa5b9007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #53: _PyObject_Call_Prepend + 0x69 (0x5605aa5cac39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #54: <unknown function> + 0x211239 (0x5605aa68d239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #55: PyObject_Call + 0x207 (0x5605aa5cd067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x5605aa5b32b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #57: <unknown function> + 0x150582 (0x5605aa5cc582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x5605aa5b18fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #59: <unknown function> + 0x150582 (0x5605aa5cc582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #60: PyObject_Call + 0xbc (0x5605aa5ccf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x5605aa5b32b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #62: <unknown function> + 0x150582 (0x5605aa5cc582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: frame #63: PyObject_Call + 0xbc (0x5605aa5ccf1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default6]:[rank14]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default7]:[rank15]: Traceback (most recent call last): | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default7]:[rank15]: trainer.train(dataloader) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default7]:[rank15]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default7]:[rank15]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default7]:[rank15]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default7]:[rank15]: output = model(**micro_batch) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank15]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank15]: return forward_call(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default7]:[rank15]: sharded_logits = self.model( | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank15]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank15]: return forward_call(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default7]:[rank15]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default7]:[rank15]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default7]:[rank15]: return self._call_impl(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default7]:[rank15]: return forward_call(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default7]:[rank15]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default7]:[rank15]: pipeline_state.run_communication() | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default7]:[rank15]: recv_activation_tensor = recv_activation() | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default7]:[rank15]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default7]:[rank15]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default7]:[rank15]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default7]:[rank15]: dist.recv( | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default7]:[rank15]: return func(*args, **kwargs) | |
| [default7]:[rank15]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default7]:[rank15]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default7]:[rank15]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer | |
| [default7]:[rank15]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default7]:[rank15]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fc298c58897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default7]:[rank15]: frame #1: <unknown function> + 0x5b3a23e (0x7fc2d277523e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7fc2d276fc87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7fc2d276ff82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7fc2d2770fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc2d2725371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc2d2725371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc2d2725371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc2d2725371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7fc299f32189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:[rank15]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7fc299f39610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:[rank15]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7fc299f58978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default7]:[rank15]: frame #12: <unknown function> + 0x5adc309 (0x7fc2d2717309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #13: <unknown function> + 0x5ae6f10 (0x7fc2d2721f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #14: <unknown function> + 0x5ae6fa5 (0x7fc2d2721fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #15: <unknown function> + 0x5124446 (0x7fc2d1d5f446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #16: <unknown function> + 0x1acf4b8 (0x7fc2ce70a4b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #17: <unknown function> + 0x5aee004 (0x7fc2d2729004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #18: <unknown function> + 0x5af36b5 (0x7fc2d272e6b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default7]:[rank15]: frame #19: <unknown function> + 0xd2631e (0x7fc2e531831e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default7]:[rank15]: frame #20: <unknown function> + 0x47def4 (0x7fc2e4a6fef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default7]:[rank15]: frame #21: <unknown function> + 0x1445a6 (0x55b205fdb5a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #22: _PyObject_MakeTpCall + 0x26b (0x55b205fd4a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #23: <unknown function> + 0x150866 (0x55b205fe7866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x55b205fd0142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #25: _PyFunction_Vectorcall + 0x6c (0x55b205fdba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #26: PyObject_Call + 0xbc (0x55b205fe7f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x55b205fce2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #28: _PyFunction_Vectorcall + 0x6c (0x55b205fdba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x55b205fcc8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #30: <unknown function> + 0x150582 (0x55b205fe7582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x55b205fcc8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #32: <unknown function> + 0x150582 (0x55b205fe7582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x55b205fcc8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #34: <unknown function> + 0x150582 (0x55b205fe7582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x55b205fcc8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x55b205fd3f50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #37: _PyObject_Call_Prepend + 0x69 (0x55b205fe5c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #38: <unknown function> + 0x211239 (0x55b2060a8239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #39: _PyObject_MakeTpCall + 0x26b (0x55b205fd4a6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x55b205fd03e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #41: _PyFunction_Vectorcall + 0x6c (0x55b205fdba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x55b205fcbc5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #43: _PyFunction_Vectorcall + 0x6c (0x55b205fdba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x55b205fcc8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #45: <unknown function> + 0x150582 (0x55b205fe7582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #46: PyObject_Call + 0xbc (0x55b205fe7f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x55b205fce2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #48: <unknown function> + 0x150582 (0x55b205fe7582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #49: PyObject_Call + 0xbc (0x55b205fe7f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x55b205fce2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #51: _PyFunction_Vectorcall + 0x6c (0x55b205fdba2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x55b205fd4007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #53: _PyObject_Call_Prepend + 0x69 (0x55b205fe5c39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #54: <unknown function> + 0x211239 (0x55b2060a8239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #55: PyObject_Call + 0x207 (0x55b205fe8067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x55b205fce2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #57: <unknown function> + 0x150582 (0x55b205fe7582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x55b205fcc8fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #59: <unknown function> + 0x150582 (0x55b205fe7582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #60: PyObject_Call + 0xbc (0x55b205fe7f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x55b205fce2b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #62: <unknown function> + 0x150582 (0x55b205fe7582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: frame #63: PyObject_Call + 0xbc (0x55b205fe7f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default7]:[rank15]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default5]:[rank13]: Traceback (most recent call last): | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default5]:[rank13]: trainer.train(dataloader) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default5]:[rank13]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default5]:[rank13]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 278, in train_batch_iter | |
| [default5]:[rank13]: output = self.forward(context=context, state=state, micro_batch=micro_batch, model=model) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 44, in forward | |
| [default5]:[rank13]: output = model(**micro_batch) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank13]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank13]: return forward_call(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 891, in forward | |
| [default5]:[rank13]: sharded_logits = self.model( | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank13]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank13]: return forward_call(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 764, in forward | |
| [default5]:[rank13]: return self.forward_with_hidden_states(input_ids=input_ids, input_mask=input_mask)[0] | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 780, in forward_with_hidden_states | |
| [default5]:[rank13]: hidden_encoder_states = encoder_block(**hidden_encoder_states) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl | |
| [default5]:[rank13]: return self._call_impl(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl | |
| [default5]:[rank13]: return forward_call(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 126, in forward | |
| [default5]:[rank13]: new_kwargs[name] = recv_from_pipeline_state_buffer( | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/functional.py", line 117, in recv_from_pipeline_state_buffer | |
| [default5]:[rank13]: pipeline_state.run_communication() | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 150, in run_communication | |
| [default5]:[rank13]: recv_activation_tensor = recv_activation() | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/state.py", line 31, in __call__ | |
| [default5]:[rank13]: return self.p2p.recv_tensors(num_tensors=1, from_rank=self.from_rank)[0] | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 353, in recv_tensors | |
| [default5]:[rank13]: buffers, futures = self.irecv_tensors(num_tensors=num_tensors, from_rank=from_rank, tag=tag) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 326, in irecv_tensors | |
| [default5]:[rank13]: meta = self._recv_meta(from_rank=from_rank, tag=tag) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/p2p.py", line 246, in _recv_meta | |
| [default5]:[rank13]: dist.recv( | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper | |
| [default5]:[rank13]: return func(*args, **kwargs) | |
| [default5]:[rank13]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1932, in recv | |
| [default5]:[rank13]: pg.recv([tensor], group_src_rank, tag).wait() | |
| [default5]:[rank13]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0:1', but store->get('0:1') got error: Connection reset by peer | |
| [default5]:[rank13]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:672 (most recent call first): | |
| [default5]:[rank13]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f041942f897 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libc10.so) | |
| [default5]:[rank13]: frame #1: <unknown function> + 0x5b3a23e (0x7f0452f4c23e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef<std::string>, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0x2c7 (0x7f0452f46c87 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x32 (0x7f0452f46f82 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #4: c10d::TCPStore::get(std::string const&) + 0xa1 (0x7f0452f47fd1 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0452efc371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0452efc371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0452efc371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #8: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0452efc371 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xa9 (0x7f041a709189 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:[rank13]: frame #10: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0xc50 (0x7f041a710610 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:[rank13]: frame #11: c10d::ProcessGroupNCCL::recv(std::vector<at::Tensor, std::allocator<at::Tensor> >&, int, int) + 0x5f8 (0x7f041a72f978 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) | |
| [default5]:[rank13]: frame #12: <unknown function> + 0x5adc309 (0x7f0452eee309 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #13: <unknown function> + 0x5ae6f10 (0x7f0452ef8f10 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #14: <unknown function> + 0x5ae6fa5 (0x7f0452ef8fa5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #15: <unknown function> + 0x5124446 (0x7f0452536446 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #16: <unknown function> + 0x1acf4b8 (0x7f044eee14b8 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #17: <unknown function> + 0x5aee004 (0x7f0452f00004 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #18: <unknown function> + 0x5af36b5 (0x7f0452f056b5 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) | |
| [default5]:[rank13]: frame #19: <unknown function> + 0xd2631e (0x7f0465aef31e in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default5]:[rank13]: frame #20: <unknown function> + 0x47def4 (0x7f0465246ef4 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/lib/libtorch_python.so) | |
| [default5]:[rank13]: frame #21: <unknown function> + 0x1445a6 (0x560b300955a6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #22: _PyObject_MakeTpCall + 0x26b (0x560b3008ea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #23: <unknown function> + 0x150866 (0x560b300a1866 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #24: _PyEval_EvalFrameDefault + 0x4c12 (0x560b3008a142 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #25: _PyFunction_Vectorcall + 0x6c (0x560b30095a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #26: PyObject_Call + 0xbc (0x560b300a1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #27: _PyEval_EvalFrameDefault + 0x2d83 (0x560b300882b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #28: _PyFunction_Vectorcall + 0x6c (0x560b30095a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #29: _PyEval_EvalFrameDefault + 0x13ca (0x560b300868fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #30: <unknown function> + 0x150582 (0x560b300a1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #31: _PyEval_EvalFrameDefault + 0x13ca (0x560b300868fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #32: <unknown function> + 0x150582 (0x560b300a1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #33: _PyEval_EvalFrameDefault + 0x13ca (0x560b300868fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #34: <unknown function> + 0x150582 (0x560b300a1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #35: _PyEval_EvalFrameDefault + 0x13ca (0x560b300868fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #36: _PyObject_FastCallDictTstate + 0xd0 (0x560b3008df50 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #37: _PyObject_Call_Prepend + 0x69 (0x560b3009fc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #38: <unknown function> + 0x211239 (0x560b30162239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #39: _PyObject_MakeTpCall + 0x26b (0x560b3008ea6b in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #40: _PyEval_EvalFrameDefault + 0x4eb6 (0x560b3008a3e6 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #41: _PyFunction_Vectorcall + 0x6c (0x560b30095a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #42: _PyEval_EvalFrameDefault + 0x72c (0x560b30085c5c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #43: _PyFunction_Vectorcall + 0x6c (0x560b30095a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #44: _PyEval_EvalFrameDefault + 0x13ca (0x560b300868fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #45: <unknown function> + 0x150582 (0x560b300a1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #46: PyObject_Call + 0xbc (0x560b300a1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #47: _PyEval_EvalFrameDefault + 0x2d83 (0x560b300882b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #48: <unknown function> + 0x150582 (0x560b300a1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #49: PyObject_Call + 0xbc (0x560b300a1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #50: _PyEval_EvalFrameDefault + 0x2d83 (0x560b300882b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #51: _PyFunction_Vectorcall + 0x6c (0x560b30095a2c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #52: _PyObject_FastCallDictTstate + 0x187 (0x560b3008e007 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #53: _PyObject_Call_Prepend + 0x69 (0x560b3009fc39 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #54: <unknown function> + 0x211239 (0x560b30162239 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #55: PyObject_Call + 0x207 (0x560b300a2067 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #56: _PyEval_EvalFrameDefault + 0x2d83 (0x560b300882b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #57: <unknown function> + 0x150582 (0x560b300a1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #58: _PyEval_EvalFrameDefault + 0x13ca (0x560b300868fa in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #59: <unknown function> + 0x150582 (0x560b300a1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #60: PyObject_Call + 0xbc (0x560b300a1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #61: _PyEval_EvalFrameDefault + 0x2d83 (0x560b300882b3 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #62: <unknown function> + 0x150582 (0x560b300a1582 in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: frame #63: PyObject_Call + 0xbc (0x560b300a1f1c in /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10) | |
| [default5]:[rank13]: . This may indicate a possible application crash on rank 0 or a network set up issue. | |
| [default7]:Exception in thread Thread-2 (_pin_memory_loop): | |
| [default7]:Traceback (most recent call last): | |
| [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/threading.py", line 1016, in _bootstrap_inner | |
| [default7]: self.run() | |
| [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/threading.py", line 953, in run | |
| [default7]: self._target(*self._args, **self._kwargs) | |
| [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 54, in _pin_memory_loop | |
| [default7]: do_one_step() | |
| [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 31, in do_one_step | |
| [default7]: r = in_queue.get(timeout=MP_STATUS_CHECK_INTERVAL) | |
| [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/queues.py", line 122, in get | |
| [default7]: return _ForkingPickler.loads(res) | |
| [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/multiprocessing/reductions.py", line 495, in rebuild_storage_fd | |
| [default7]: fd = df.detach() | |
| [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/resource_sharer.py", line 57, in detach | |
| [default7]: with _resource_sharer.get_connection(self._id) as conn: | |
| [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/resource_sharer.py", line 86, in get_connection | |
| [default7]: c = Client(address, authkey=process.current_process().authkey) | |
| [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 508, in Client | |
| [default7]: answer_challenge(c, authkey) | |
| [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 757, in answer_challenge | |
| [default7]: response = connection.recv_bytes(256) # reject large message | |
| [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes | |
| [default7]: buf = self._recv_bytes(maxlength) | |
| [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes | |
| [default7]: buf = self._recv(4) | |
| [default7]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/multiprocessing/connection.py", line 379, in _recv | |
| [default7]: chunk = read(handle, remaining) | |
| [default7]:ConnectionResetError: [Errno 104] Connection reset by peer | |
| [default4]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.) | |
| [default4]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default4]:[rank12]: OSError: [Errno 122] Disk quota exceeded | |
| [default4]: | |
| [default4]:[rank12]: During handling of the above exception, another exception occurred: | |
| [default4]: | |
| [default4]:[rank12]: Traceback (most recent call last): | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 237, in <module> | |
| [default4]:[rank12]: trainer.train(dataloader) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 429, in train | |
| [default4]:[rank12]: outputs, loss_avg = self.training_step(dataloader=self.current_dataloader) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 462, in training_step | |
| [default4]:[rank12]: outputs = self.pipeline_engine.train_batch_iter( | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 295, in train_batch_iter | |
| [default4]:[rank12]: self.backward(context=context, state=state, grad_accumulator=grad_accumulator) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/engine.py", line 86, in backward | |
| [default4]:[rank12]: grad_accumulator.backward(sum(activations)) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/optim/gradient_accumulator.py", line 205, in backward | |
| [default4]:[rank12]: result = loss.backward() | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/_tensor.py", line 525, in backward | |
| [default4]:[rank12]: torch.autograd.backward( | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/__init__.py", line 267, in backward | |
| [default4]:[rank12]: _engine_run_backward( | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py", line 744, in _engine_run_backward | |
| [default4]:[rank12]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/function.py", line 301, in apply | |
| [default4]:[rank12]: return user_fn(self, *args) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py", line 821, in backward | |
| [default4]:[rank12]: dx, dw, db, dresidual_in, dx1, dw1, db1 = _layer_norm_bwd( | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py", line 643, in _layer_norm_bwd | |
| [default4]:[rank12]: _layer_norm_bwd_kernel[grid]( | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/jit.py", line 167, in <lambda> | |
| [default4]:[rank12]: return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 143, in run | |
| [default4]:[rank12]: timings = {config: self._bench(*args, config=config, **kwargs) for config in pruned_configs} | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 143, in <dictcomp> | |
| [default4]:[rank12]: timings = {config: self._bench(*args, config=config, **kwargs) for config in pruned_configs} | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 122, in _bench | |
| [default4]:[rank12]: return do_bench(kernel_call, warmup=self.warmup, rep=self.rep, quantiles=(0.5, 0.2, 0.8)) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/testing.py", line 102, in do_bench | |
| [default4]:[rank12]: fn() | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 110, in kernel_call | |
| [default4]:[rank12]: self.fn.run( | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 305, in run | |
| [default4]:[rank12]: return self.fn.run(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 305, in run | |
| [default4]:[rank12]: return self.fn.run(*args, **kwargs) | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 305, in run | |
| [default4]:[rank12]: return self.fn.run(*args, **kwargs) | |
| [default4]:[rank12]: [Previous line repeated 2 more times] | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/jit.py", line 416, in run | |
| [default4]:[rank12]: self.cache[device][key] = compile( | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/compiler/compiler.py", line 194, in compile | |
| [default4]:[rank12]: metadata_group[f"{src.name}.{ext}"] = fn_cache_manager.put(next_module, f"{src.name}.{ext}") | |
| [default4]:[rank12]: File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/triton/runtime/cache.py", line 123, in put | |
| [default4]:[rank12]: with open(temp_path, mode) as f: | |
| [default4]:[rank12]: OSError: [Errno 122] Disk quota exceeded | |
| W0702 16:32:48.645000 139995769620288 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3296723 closing signal SIGTERM | |
| W0702 16:32:48.645000 139995769620288 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3296724 closing signal SIGTERM | |
| W0702 16:32:48.646000 139995769620288 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3296725 closing signal SIGTERM | |
| W0702 16:32:48.646000 139995769620288 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3296727 closing signal SIGTERM | |
| W0702 16:32:48.647000 139995769620288 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3296728 closing signal SIGTERM | |
| W0702 16:32:48.650000 139995769620288 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3296729 closing signal SIGTERM | |
| W0702 16:32:48.651000 139995769620288 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3296730 closing signal SIGTERM | |
| W0702 16:32:48.656000 140456080881472 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1391321 closing signal SIGTERM | |
| W0702 16:32:48.660000 140456080881472 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1391323 closing signal SIGTERM | |
| E0702 16:32:49.876000 140456080881472 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 1391319) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 | |
| Traceback (most recent call last): | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module> | |
| sys.exit(main()) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper | |
| return f(*args, **kwargs) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main | |
| run(args) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run | |
| elastic_launch( | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__ | |
| return launch_agent(self._config, self._entrypoint, list(args)) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent | |
| raise ChildFailedError( | |
| torch.distributed.elastic.multiprocessing.errors.ChildFailedError: | |
| ============================================================ | |
| /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED | |
| ------------------------------------------------------------ | |
| Failures: | |
| [1]: | |
| time : 2024-07-02_16:32:48 | |
| host : ip-26-0-160-225.ec2.internal | |
| rank : 1 (local_rank: 1) | |
| exitcode : 1 (pid: 1391320) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| [2]: | |
| time : 2024-07-02_16:32:48 | |
| host : ip-26-0-160-225.ec2.internal | |
| rank : 3 (local_rank: 3) | |
| exitcode : 1 (pid: 1391322) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| [3]: | |
| time : 2024-07-02_16:32:48 | |
| host : ip-26-0-160-225.ec2.internal | |
| rank : 5 (local_rank: 5) | |
| exitcode : 1 (pid: 1391324) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| [4]: | |
| time : 2024-07-02_16:32:48 | |
| host : ip-26-0-160-225.ec2.internal | |
| rank : 6 (local_rank: 6) | |
| exitcode : 1 (pid: 1391325) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| [5]: | |
| time : 2024-07-02_16:32:48 | |
| host : ip-26-0-160-225.ec2.internal | |
| rank : 7 (local_rank: 7) | |
| exitcode : 1 (pid: 1391326) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| ------------------------------------------------------------ | |
| Root Cause (first observed failure): | |
| [0]: | |
| time : 2024-07-02_16:32:48 | |
| host : ip-26-0-160-225.ec2.internal | |
| rank : 0 (local_rank: 0) | |
| exitcode : 1 (pid: 1391319) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| ============================================================ | |
| E0702 16:32:50.168000 139995769620288 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 3 (pid: 3296726) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 | |
| W0702 16:32:50.175000 139995769620288 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-171-56.ec2.internal_3296654_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError. | |
| W0702 16:32:50.201000 139995769620288 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-171-56.ec2.internal_3296654_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError. | |
| W0702 16:32:50.210000 139995769620288 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1203] The node 'ip-26-0-171-56.ec2.internal_3296654_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError. | |
| Traceback (most recent call last): | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in <module> | |
| sys.exit(main()) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper | |
| return f(*args, **kwargs) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main | |
| run(args) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run | |
| elastic_launch( | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__ | |
| return launch_agent(self._config, self._entrypoint, list(args)) | |
| File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent | |
| raise ChildFailedError( | |
| torch.distributed.elastic.multiprocessing.errors.ChildFailedError: | |
| ============================================================ | |
| /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED | |
| ------------------------------------------------------------ | |
| Failures: | |
| <NO_OTHER_FAILURES> | |
| ------------------------------------------------------------ | |
| Root Cause (first observed failure): | |
| [0]: | |
| time : 2024-07-02_16:32:48 | |
| host : ip-26-0-171-56.ec2.internal | |
| rank : 11 (local_rank: 3) | |
| exitcode : 1 (pid: 3296726) | |
| error_file: <N/A> | |
| traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html | |
| ============================================================ | |
| srun: error: ip-26-0-160-225: task 0: Exited with exit code 1 | |
| srun: error: ip-26-0-171-56: task 1: Exited with exit code 1 | |
| Consider using `hf_transfer` for faster uploads. This solution comes with some limitations. See https://huggingface.co/docs/huggingface_hub/hf_transfer for more details. | |