======================== START TIME: Tue Jul 2 23:39:36 UTC 2024 python3 version = Python 3.10.14 ======================== The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well. Token is valid (permission: write). Your token has been saved to /admin/home/ferdinand_mom/.cache/huggingface/token Login successful Already on 'bench_cluster' M examples/config_tiny_llama.py M examples/config_tiny_llama.yaml M examples/train_tiny_llama.sh M src/nanotron/models/llama.py M src/nanotron/trainer.py Your branch is up to date with 'origin/bench_cluster'. Job status: RUNNING W0702 23:39:39.145000 140306096875328 torch/distributed/run.py:757] W0702 23:39:39.145000 140306096875328 torch/distributed/run.py:757] ***************************************** W0702 23:39:39.145000 140306096875328 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. W0702 23:39:39.145000 140306096875328 torch/distributed/run.py:757] ***************************************** W0702 23:39:39.147000 139649365083968 torch/distributed/run.py:757] W0702 23:39:39.147000 139649365083968 torch/distributed/run.py:757] ***************************************** W0702 23:39:39.147000 139649365083968 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. W0702 23:39:39.147000 139649365083968 torch/distributed/run.py:757] ***************************************** W0702 23:39:39.147000 139987077363520 torch/distributed/run.py:757] W0702 23:39:39.147000 139987077363520 torch/distributed/run.py:757] ***************************************** W0702 23:39:39.147000 139987077363520 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. W0702 23:39:39.147000 139987077363520 torch/distributed/run.py:757] ***************************************** W0702 23:39:39.148000 140698504841024 torch/distributed/run.py:757] W0702 23:39:39.148000 140698504841024 torch/distributed/run.py:757] ***************************************** W0702 23:39:39.148000 140698504841024 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. W0702 23:39:39.148000 140698504841024 torch/distributed/run.py:757] ***************************************** W0702 23:39:39.148000 140094763059008 torch/distributed/run.py:757] W0702 23:39:39.148000 140094763059008 torch/distributed/run.py:757] ***************************************** W0702 23:39:39.148000 140094763059008 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. W0702 23:39:39.148000 140094763059008 torch/distributed/run.py:757] ***************************************** W0702 23:39:39.151000 139788966299456 torch/distributed/run.py:757] W0702 23:39:39.151000 139788966299456 torch/distributed/run.py:757] ***************************************** W0702 23:39:39.151000 139788966299456 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. W0702 23:39:39.151000 139788966299456 torch/distributed/run.py:757] ***************************************** W0702 23:39:39.153000 139814066579264 torch/distributed/run.py:757] W0702 23:39:39.153000 139814066579264 torch/distributed/run.py:757] ***************************************** W0702 23:39:39.153000 139814066579264 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. W0702 23:39:39.153000 139814066579264 torch/distributed/run.py:757] ***************************************** W0702 23:39:39.175000 140156427769664 torch/distributed/run.py:757] W0702 23:39:39.175000 140156427769664 torch/distributed/run.py:757] ***************************************** W0702 23:39:39.175000 140156427769664 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. W0702 23:39:39.175000 140156427769664 torch/distributed/run.py:757] ***************************************** [default0]:07/02/2024 23:39:59 [WARNING|DP=0|PP=0|TP=0|ip-26-0-160-225]: [Vocab Size Padding] Padded vocab (size: 50257) with 47 dummy tokens (new size: 50304) [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Config: [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Config(general=GeneralArgs(project='bench_cluster', [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: run='%date_%jobid', [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: seed=42, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: step=None, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: consumed_train_samples=None, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: benchmark_csv_path=None, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: ignore_sanity_checks=True), [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: parallelism=ParallelismArgs(dp=1, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: pp=1, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: tp=64, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: pp_engine=, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: tp_mode=, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: tp_linear_async_communication=False, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: expert_parallel_size=1), [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: model=ModelArgs(model_config=LlamaConfig(bos_token_id=1, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: eos_token_id=2, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: hidden_act='silu', [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: hidden_size=2048, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: initializer_range=0.02, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: intermediate_size=4096, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: is_llama_config=True, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: max_position_embeddings=4096, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: num_attention_heads=32, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: num_hidden_layers=24, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: num_key_value_heads=32, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: pad_token_id=None, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: pretraining_tp=1, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: rms_norm_eps=1e-05, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: rope_scaling=None, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: rope_theta=10000.0, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: tie_word_embeddings=True, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: use_cache=True, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: vocab_size=50304), [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: init_method=RandomInit(std=0.025), [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: dtype=torch.bfloat16, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: make_vocab_size_divisible_by=1, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: ddp_bucket_cap_mb=25), [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: tokenizer=TokenizerArgs(tokenizer_name_or_path='openai-community/gpt2', [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: tokenizer_revision=None, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: tokenizer_max_length=None), [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: checkpoints=CheckpointsArgs(checkpoints_path=Path('/dev/null'), [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: checkpoint_interval=100000, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: save_initial_state=False, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: resume_checkpoint_path=None, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: checkpoints_path_is_shared_file_system=False), [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: logging=LoggingArgs(log_level='info', [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: log_level_replica='info', [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: iteration_step_info_interval=1), [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: tokens=TokensArgs(sequence_length=4096, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: train_steps=20, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: micro_batch_size=16, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: batch_accumulation_per_replica=64, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: val_check_interval=-1, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: limit_val_batches=0, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: limit_test_batches=0), [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: optimizer=OptimizerArgs(optimizer_factory=AdamWOptimizerArgs(adam_eps=1e-08, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: adam_beta1=0.9, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: adam_beta2=0.95, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: torch_adam_is_fused=True, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: name='adamW'), [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: zero_stage=1, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: weight_decay=0.01, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: clip_grad=1.0, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: accumulate_grad_in_fp32=True, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: learning_rate_scheduler=LRSchedulerArgs(learning_rate=0.0001, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: lr_warmup_steps=1, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: lr_warmup_style='linear', [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: lr_decay_style='linear', [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: lr_decay_steps=19, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: lr_decay_starting_step=None, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: min_decay_lr=1e-05)), [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: data_stages=[DatasetStageArgs(name='Training Stage', [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: start_training_step=1, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: data=DataArgs(dataset=PretrainDatasetsArgs(hf_dataset_or_datasets='roneneldan/TinyStories', [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: hf_dataset_splits='train', [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: hf_dataset_config_name=None, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: dataset_processing_num_proc_per_process=64, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: dataset_overwrite_cache=False, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: text_column_name='text'), [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: seed=42, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: num_loading_workers=0))], [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: profiler=ProfilerArgs(profiler_export_path=Path('/fsx/ferdinandmom/ferdinand-hf/bench_cluster/results/llama-1B/64_GPUS/dp-1_tp-64_pp-1_mbz-16')), [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: lighteval=None) [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Model Config: [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: LlamaConfig(bos_token_id=1, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: eos_token_id=2, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: hidden_act='silu', [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: hidden_size=2048, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: initializer_range=0.02, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: intermediate_size=4096, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: is_llama_config=True, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: max_position_embeddings=4096, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: num_attention_heads=32, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: num_hidden_layers=24, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: num_key_value_heads=32, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: pad_token_id=None, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: pretraining_tp=1, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: rms_norm_eps=1e-05, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: rope_scaling=None, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: rope_theta=10000.0, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: tie_word_embeddings=True, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: use_cache=True, [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: vocab_size=50304) [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Building model.. [default0]:07/02/2024 23:39:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-160-225]: Setting PP block ranks... [default0]:[rank0]: Traceback (most recent call last): [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default0]:[rank0]: trainer = DistributedTrainer(config_file) [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default0]:[rank0]: self.model = self.init_model() # Defines self.model [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default0]:[rank0]: model = self._init_model_instance() [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default0]:[rank0]: model = self._init_model( [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default0]:[rank0]: model = build_model( [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default0]:[rank0]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default0]:[rank0]: self.pp_block = self.module_builder(**self.module_kwargs) [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default0]:[rank0]: self.attn = CausalSelfAttention( [default0]:[rank0]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default0]:[rank0]: config.num_attention_heads % tp_pg.size() == 0 [default0]:[rank0]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default4]:[rank4]: Traceback (most recent call last): [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default4]:[rank4]: trainer = DistributedTrainer(config_file) [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default4]:[rank4]: self.model = self.init_model() # Defines self.model [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default4]:[rank4]: model = self._init_model_instance() [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default4]:[rank4]: model = self._init_model( [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default4]:[rank4]: model = build_model( [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default4]:[rank4]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default4]:[rank4]: self.pp_block = self.module_builder(**self.module_kwargs) [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default4]:[rank4]: self.attn = CausalSelfAttention( [default4]:[rank4]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default4]:[rank4]: config.num_attention_heads % tp_pg.size() == 0 [default4]:[rank4]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default6]:[rank6]: Traceback (most recent call last): [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default6]:[rank6]: trainer = DistributedTrainer(config_file) [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default6]:[rank6]: self.model = self.init_model() # Defines self.model [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default6]:[rank6]: model = self._init_model_instance() [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default6]:[rank6]: model = self._init_model( [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default6]:[rank6]: model = build_model( [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default6]:[rank6]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default6]:[rank6]: self.pp_block = self.module_builder(**self.module_kwargs) [default3]:[rank3]: Traceback (most recent call last): [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default6]:[rank6]: self.attn = CausalSelfAttention( [default6]:[rank6]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default6]:[rank6]: config.num_attention_heads % tp_pg.size() == 0 [default3]:[rank3]: trainer = DistributedTrainer(config_file) [default6]:[rank6]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default3]:[rank3]: self.model = self.init_model() # Defines self.model [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default3]:[rank3]: model = self._init_model_instance() [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default3]:[rank3]: model = self._init_model( [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default3]:[rank3]: model = build_model( [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default3]:[rank3]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default3]:[rank3]: self.pp_block = self.module_builder(**self.module_kwargs) [default1]:[rank33]: Traceback (most recent call last): [default1]:[rank33]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default1]:[rank33]: trainer = DistributedTrainer(config_file) [default1]:[rank33]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default1]:[rank33]: self.model = self.init_model() # Defines self.model [default1]:[rank33]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default1]:[rank33]: model = self._init_model_instance() [default1]:[rank33]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default1]:[rank33]: model = self._init_model( [default1]:[rank33]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default1]:[rank33]: model = build_model( [def[default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default3]:[rank3]: self.attn = CausalSelfAttention( [default3]:[rank3]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default3]:[rank3]: config.num_attention_heads % tp_pg.size() == 0 [default3]:[rank3]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). ault1]:[rank33]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default1]:[rank33]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default1]:[rank33]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default1]:[rank33]: self.pp_block = self.module_builder(**self.module_kwargs) [default1]:[rank33]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default1]:[rank33]: self.attn = CausalSelfAttention( [default1]:[rank33]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default1]:[rank33]: config.num_attention_heads % tp_pg.size() == 0 [default1]:[rank33]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default3]:[rank35]: Traceback (most recent call last): [default3]:[rank35]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default3]:[rank35]: trainer = DistributedTrainer(config_file) [default3]:[rank35]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default3]:[rank35]: self.model = self.init_model() # Defines self.model [default3]:[rank35]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default3]:[rank35]: model = self._init_model_instance() [default3]:[rank35]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default3]:[rank35]: model = self._init_model( [default3]:[rank35]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default3]:[rank35]: model = build_model( [default3]:[rank35]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default3]:[rank35]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default3]:[rank35]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default3]:[rank35]: self.pp_block = self.module_builder(**self.module_kwargs) [default3]:[rank35]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default3]:[rank35]: self.attn = CausalSelfAttention( [default3]:[rank35]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default3]:[rank35]: config.num_attention_heads % tp_pg.size() == 0 [default3]:[rank35]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default7]:[rank7]: Traceback (most recent call last): [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default5]:[rank5]: Traceback (most recent call last): [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default7]:[rank7]: trainer = DistributedTrainer(config_file) [default2]:[rank2]: Traceback (most recent call last): [default5]:[rank5]: trainer = DistributedTrainer(config_file) [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default4]:[rank36]: Traceback (most recent call last): [default4]:[rank36]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default4]:[rank36]: trainer = DistributedTrainer(config_file) [default4]:[rank36]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default4]:[rank36]: self.model = self.init_model() # Defines self.model [default4]:[rank36]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default4]:[rank36]: model = self._init_model_instance() [default4]:[rank36]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default4]:[rank36]: model = self._init_model( [default4]:[rank36]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default4]:[rank36]: model = build_model( [def[default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default5]:[rank5]: self.model = self.init_model() # Defines self.model ault4]:[rank36]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default4]:[rank36]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default4]:[rank36]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default4]:[rank36]: self.pp_block = self.module_builder(**self.module_kwargs) [default4]:[rank36]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default4]:[rank36]: self.attn = CausalSelfAttention( [default4]:[rank36]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default1]:[rank1]: Traceback (most recent call last): [default4]:[rank36]: config.num_attention_heads % tp_pg.size() == 0 [default4]:[rank36]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default0]:[rank32]: Traceback (most recent call last): [default0]:[rank32]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default0]:[rank32]: trainer = DistributedTrainer(config_file) [default0]:[rank32]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default0]:[rank32]: self.model = self.init_model() # Defines self.model [default1]:[rank1]: trainer = DistributedTrainer(config_file) [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default0]:[rank32]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default0]:[rank32]: model = self._init_model_instance() [default0]:[rank32]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default0]:[rank32]: model = self._init_model( [default0]:[rank32]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default0]:[rank32]: model = build_model( [default0]:[rank32]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default0]:[rank32]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default7]:[rank7]: self.model = self.init_model() # Defines self.model [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default2]:[rank2]: trainer = DistributedTrainer(config_file) [default5]:[rank5]: model = self._init_model_instance() [default0]:[rank32]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default5]:[rank37]: Traceback (most recent call last): [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default5]:[rank37]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default7]:[rank39]: Traceback (most recent call last): [default1]:[rank1]: self.model = self.init_model() # Defines self.model [default7]:[rank39]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default7]:[rank7]: model = self._init_model_instance() [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default5]:[rank37]: trainer = DistributedTrainer(config_file) [default7]:[rank39]: trainer = DistributedTrainer(config_file) [default7]:[rank39]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default7]:[rank55]: Traceback (most recent call last): [default7]:[rank55]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default7]:[rank55]: trainer = DistributedTrainer(config_file) [default7]:[rank55]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default7]:[rank55]: self.model = self.init_model() # Defines self.model [default7]:[rank55]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default7]:[rank55]: model = self._init_model_instance() [default7]:[rank55]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default7]:[rank55]: model = self._init_model( [default7]:[rank55]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default7]:[rank55]: model = build_model( [def[default5]:[rank5]: model = self._init_model( [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default1]:[rank1]: model = self._init_model_instance() [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default2]:[rank2]: self.model = self.init_model() # Defines self.model [default7]:[rank7]: model = self._init_model( [default5]:[rank5]: model = build_model( [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default5]:[rank37]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default7]:[rank39]: self.model = self.init_model() # Defines self.model [default6]:[rank38]: Traceback (most recent call last): [default6]:[rank38]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default0]:[rank32]: self.pp_block = self.module_builder(**self.module_kwargs) [default6]:[rank38]: trainer = DistributedTrainer(config_file) [default1]:[rank1]: model = self._init_model( [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default6]:[rank38]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default5]:[rank37]: self.model = self.init_model() # Defines self.model [default7]:[rank39]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default6]:[rank38]: self.model = self.init_model() # Defines self.model [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default5]:[rank37]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default5]:[rank37]: model = self._init_model_instance() ault7]:[rank55]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default7]:[rank55]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default7]:[rank55]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default7]:[rank55]: self.pp_block = self.module_builder(**self.module_kwargs) [default7]:[rank55]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default5]:[rank5]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default5]:[rank37]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default7]:[rank55]: self.attn = CausalSelfAttention( [default7]:[rank55]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default7]:[rank55]: config.num_attention_heads % tp_pg.size() == 0 [default7]:[rank55]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default0]:[rank32]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default5]:[rank37]: model = self._init_model( [default0]:[rank48]: Traceback (most recent call last): [default0]:[rank48]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default0]:[rank48]: trainer = DistributedTrainer(config_file) [default0]:[rank48]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default7]:[rank7]: model = build_model( [default6]:[rank38]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default0]:[rank48]: self.model = self.init_model() # Defines self.model [default0]:[rank48]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default0]:[rank48]: model = self._init_model_instance() [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default5]:[rank5]: self.pp_block = self.module_builder(**self.module_kwargs) [default0]:[rank48]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default0]:[rank48]: model = self._init_model( [default0]:[rank48]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default0]:[rank48]: model = build_model( [default0]:[rank48]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default0]:[rank48]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default0]:[rank48]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default0]:[rank32]: self.attn = CausalSelfAttention( [default0]:[rank32]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default0]:[rank48]: self.pp_block = self.module_builder(**self.module_kwargs) [default0]:[rank48]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default0]:[rank48]: self.attn = CausalSelfAttention( [default0]:[rank48]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default7]:[rank7]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default7]:[rank39]: model = self._init_model_instance() [default6]:[rank38]: model = self._init_model_instance() [default0]:[rank32]: config.num_attention_heads % tp_pg.size() == 0 [default0]:[rank48]: config.num_attention_heads % tp_pg.size() == 0 [default0]:[rank48]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default1]:[rank1]: model = build_model( [default0]:[rank32]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default5]:[rank5]: self.attn = CausalSelfAttention( [default5]:[rank37]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default5]:[rank37]: model = build_model( [default7]:[rank39]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default1]:[rank49]: Traceback (most recent call last): [default1]:[rank49]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default2]:[rank2]: model = self._init_model_instance() [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default1]:[rank49]: trainer = DistributedTrainer(config_file) [default1]:[rank49]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default1]:[rank1]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default5]:[rank37]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default1]:[rank49]: self.model = self.init_model() # Defines self.model [default1]:[rank49]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default1]:[rank49]: model = self._init_model_instance() [default1]:[rank49]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default1]:[rank49]: model = self._init_model( [default1]:[rank49]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default1]:[rank1]: self.pp_block = self.module_builder(**self.module_kwargs) [default5]:[rank5]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default7]:[rank7]: self.pp_block = self.module_builder(**self.module_kwargs) [default7]:[rank39]: model = self._init_model( [default5]:[rank37]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default1]:[rank49]: model = build_model( [default1]:[rank49]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default1]:[rank49]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default7]:[rank39]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default7]:[rank39]: model = build_model( [default7]:[rank39]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default1]:[rank49]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default1]:[rank49]: self.pp_block = self.module_builder(**self.module_kwargs) [default1]:[rank49]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default1]:[rank49]: self.attn = CausalSelfAttention( [default5]:[rank5]: config.num_attention_heads % tp_pg.size() == 0 [default6]:[rank38]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default1]:[rank49]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default1]:[rank49]: config.num_attention_heads % tp_pg.size() == 0 [default1]:[rank1]: self.attn = CausalSelfAttention( [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default2]:[rank2]: model = self._init_model( [default5]:[rank5]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default1]:[rank1]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default6]:[rank38]: model = self._init_model( [default7]:[rank39]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default1]:[rank49]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default7]:[rank7]: self.attn = CausalSelfAttention( [default5]:[rank37]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default5]:[rank37]: self.pp_block = self.module_builder(**self.module_kwargs) [default6]:[rank38]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default7]:[rank39]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default7]:[rank39]: self.pp_block = self.module_builder(**self.module_kwargs) [default5]:[rank37]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default6]:[rank38]: model = build_model( [default5]:[rank37]: self.attn = CausalSelfAttention( [default7]:[rank39]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default6]:[rank38]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default6]:[rank38]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default5]:[rank37]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default7]:[rank39]: self.attn = CausalSelfAttention( [default6]:[rank38]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default5]:[rank37]: config.num_attention_heads % tp_pg.size() == 0 [default7]:[rank39]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default6]:[rank38]: self.pp_block = self.module_builder(**self.module_kwargs) [default5]:[rank37]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default4]:[rank28]: Traceback (most recent call last): [default4]:[rank28]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default4]:[rank28]: trainer = DistributedTrainer(config_file) [default4]:[rank28]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default4]:[rank28]: self.model = self.init_model() # Defines self.model [default4]:[rank28]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default4]:[rank28]: model = self._init_model_instance() [default4]:[rank28]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default4]:[rank28]: model = self._init_model( [default4]:[rank28]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default4]:[rank28]: model = build_model( [def[default4]:[rank12]: Traceback (most recent call last): [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default4]:[rank12]: trainer = DistributedTrainer(config_file) [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default4]:[rank12]: self.model = self.init_model() # Defines self.model [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default4]:[rank12]: model = self._init_model_instance() [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default4]:[rank12]: model = self._init_model( [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default4]:[rank12]: model = build_model( [def[default6]:[rank38]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default7]:[rank39]: config.num_attention_heads % tp_pg.size() == 0 [default7]:[rank39]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default6]:[rank38]: self.attn = CausalSelfAttention( [default6]:[rank38]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default6]:[rank38]: config.num_attention_heads % tp_pg.size() == 0 [default6]:[rank38]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). ault4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default4]:[rank12]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) ault4]:[rank28]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default4]:[rank28]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default4]:[rank28]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default4]:[rank28]: self.pp_block = self.module_builder(**self.module_kwargs) [default4]:[rank28]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default4]:[rank28]: self.attn = CausalSelfAttention( [default4]:[rank28]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default4]:[rank12]: self.pp_block = self.module_builder(**self.module_kwargs) [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default4]:[rank12]: self.attn = CausalSelfAttention( [default4]:[rank12]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default4]:[rank12]: config.num_attention_heads % tp_pg.size() == 0 [default4]:[rank12]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default4]:[rank28]: config.num_attention_heads % tp_pg.size() == 0 [default4]:[rank28]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default1]:[rank25]: Traceback (most recent call last): [default1]:[rank25]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default1]:[rank25]: trainer = DistributedTrainer(config_file) [default1]:[rank25]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default1]:[rank25]: self.model = self.init_model() # Defines self.model [default1]:[rank25]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default1]:[rank25]: model = self._init_model_instance() [default1]:[rank25]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default1]:[rank25]: model = self._init_model( [default1]:[rank25]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default1]:[rank25]: model = build_model( [default1]:[rank25]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default1]:[rank25]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default1]:[rank25]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default1]:[rank25]: self.pp_block = self.module_builder(**self.module_kwargs) [default1]:[rank25]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default1]:[rank25]: self.attn = CausalSelfAttention( [default1]:[rank25]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default1]:[rank25]: config.num_attention_heads % tp_pg.size() == 0 [default1]:[rank25]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default5]:[rank13]: Traceback (most recent call last): [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default5]:[rank13]: trainer = DistributedTrainer(config_file) [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default5]:[rank13]: self.model = self.init_model() # Defines self.model [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default5]:[rank13]: model = self._init_model_instance() [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default5]:[rank13]: model = self._init_model( [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default5]:[rank13]: model = build_model( [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default5]:[rank13]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default5]:[rank13]: self.pp_block = self.module_builder(**self.module_kwargs) [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default5]:[rank13]: self.attn = CausalSelfAttention( [default5]:[rank13]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default5]:[rank13]: config.num_attention_heads % tp_pg.size() == 0 [default5]:[rank13]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default0]:[rank8]: Traceback (most recent call last): [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default0]:[rank8]: trainer = DistributedTrainer(config_file) [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default0]:[rank8]: self.model = self.init_model() # Defines self.model [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default0]:[rank8]: model = self._init_model_instance() [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default0]:[rank8]: model = self._init_model( [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default4]:[rank52]: Traceback (most recent call last): [default4]:[rank52]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default4]:[rank52]: trainer = DistributedTrainer(config_file) [default4]:[rank52]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default4]:[rank52]: self.model = self.init_model() # Defines self.model [default4]:[rank52]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default4]:[rank52]: model = self._init_model_instance() [default4]:[rank52]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default4]:[rank52]: model = self._init_model( [default4]:[rank52]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default4]:[rank52]: model = build_model( [def[default0]:[rank8]: model = build_model( [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default0]:[rank8]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank ault4]:[rank52]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default4]:[rank52]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default4]:[rank52]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default4]:[rank52]: self.pp_block = self.module_builder(**self.module_kwargs) [default4]:[rank52]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default4]:[rank52]: self.attn = CausalSelfAttention( [default4]:[rank52]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default4]:[rank52]: config.num_attention_heads % tp_pg.size() == 0 [default4]:[rank52]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default0]:[rank8]: self.pp_block = self.module_builder(**self.module_kwargs) [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default0]:[rank8]: self.attn = CausalSelfAttention( [default6]:[rank14]: Traceback (most recent call last): [default1]:[rank9]: Traceback (most recent call last): [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default1]:[rank9]: trainer = DistributedTrainer(config_file) [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default1]:[rank9]: self.model = self.init_model() # Defines self.model [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default1]:[rank9]: model = self._init_model_instance() [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default0]:[rank8]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default6]:[rank14]: trainer = DistributedTrainer(config_file) [default1]:[rank9]: model = self._init_model( [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default0]:[rank8]: config.num_attention_heads % tp_pg.size() == 0 [default6]:[rank14]: self.model = self.init_model() # Defines self.model [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default1]:[rank9]: model = build_model( [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default0]:[rank8]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default6]:[rank14]: model = self._init_model_instance() [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default1]:[rank9]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default6]:[rank14]: model = self._init_model( [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default6]:[rank14]: model = build_model( [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default6]:[rank14]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default6]:[rank14]: self.pp_block = self.module_builder(**self.module_kwargs) [default1]:[rank9]: self.pp_block = self.module_builder(**self.module_kwargs) [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default1]:[rank9]: self.attn = CausalSelfAttention( [default6]:[rank14]: self.attn = CausalSelfAttention( [default1]:[rank9]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default1]:[rank9]: config.num_attention_heads % tp_pg.size() == 0 [default6]:[rank14]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default6]:[rank14]: config.num_attention_heads % tp_pg.size() == 0 [default6]:[rank14]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default1]:[rank9]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default1]:[rank1]: config.num_attention_heads % tp_pg.size() == 0 [default7]:[rank7]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default1]:[rank1]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default2]:[rank2]: model = build_model( [default7]:[rank7]: config.num_attention_heads % tp_pg.size() == 0 [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default7]:[rank7]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default2]:[rank2]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default2]:[rank2]: self.pp_block = self.module_builder(**self.module_kwargs) [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default2]:[rank2]: self.attn = CausalSelfAttention( [default2]:[rank2]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default2]:[rank2]: config.num_attention_heads % tp_pg.size() == 0 [default2]:[rank2]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default2]:[rank26]: Traceback (most recent call last): [default2]:[rank26]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default2]:[rank26]: trainer = DistributedTrainer(config_file) [default2]:[rank26]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default2]:[rank26]: self.model = self.init_model() # Defines self.model [default2]:[rank26]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default2]:[rank26]: model = self._init_model_instance() [default2]:[rank26]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default2]:[rank26]: model = self._init_model( [default2]:[rank26]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default2]:[rank26]: model = build_model( [default2]:[rank26]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default2]:[rank26]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default2]:[rank26]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default2]:[rank26]: self.pp_block = self.module_builder(**self.module_kwargs) [default2]:[rank26]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default2]:[rank26]: self.attn = CausalSelfAttention( [default2]:[rank26]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default2]:[rank26]: config.num_attention_heads % tp_pg.size() == 0 [default2]:[rank26]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default5]:[rank29]: Traceback (most recent call last): [default5]:[rank29]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default5]:[rank29]: trainer = DistributedTrainer(config_file) [default5]:[rank29]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default5]:[rank29]: self.model = self.init_model() # Defines self.model [default5]:[rank29]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default5]:[rank29]: model = self._init_model_instance() [default5]:[rank29]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default5]:[rank29]: model = self._init_model( [default5]:[rank29]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default5]:[rank29]: model = build_model( [default5]:[rank29]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default5]:[rank29]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default5]:[rank29]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default5]:[rank29]: self.pp_block = self.module_builder(**self.module_kwargs) [default5]:[rank29]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default5]:[rank29]: self.attn = CausalSelfAttention( [default5]:[rank29]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default5]:[rank29]: config.num_attention_heads % tp_pg.size() == 0 [default5]:[rank29]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default0]:[rank24]: Traceback (most recent call last): [default0]:[rank24]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default0]:[rank24]: trainer = DistributedTrainer(config_file) [default0]:[rank24]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default0]:[rank24]: self.model = self.init_model() # Defines self.model [default0]:[rank24]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default0]:[rank24]: model = self._init_model_instance() [default0]:[rank24]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default0]:[rank24]: model = self._init_model( [default0]:[rank24]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default0]:[rank24]: model = build_model( [default0]:[rank24]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default0]:[rank24]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default0]:[rank24]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default0]:[rank24]: self.pp_block = self.module_builder(**self.module_kwargs) [default0]:[rank24]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default0]:[rank24]: self.attn = CausalSelfAttention( [default0]:[rank24]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default0]:[rank24]: config.num_attention_heads % tp_pg.size() == 0 [default0]:[rank24]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default6]:[rank30]: Traceback (most recent call last): [default6]:[rank30]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default6]:[rank30]: trainer = DistributedTrainer(config_file) [default6]:[rank30]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default6]:[rank30]: self.model = self.init_model() # Defines self.model [default6]:[rank30]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default6]:[rank30]: model = self._init_model_instance() [default6]:[rank30]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default6]:[rank30]: model = self._init_model( [default6]:[rank30]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default6]:[rank30]: model = build_model( [default6]:[rank30]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default6]:[rank30]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default6]:[rank30]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default6]:[rank30]: self.pp_block = self.module_builder(**self.module_kwargs) [default6]:[rank30]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default6]:[rank30]: self.attn = CausalSelfAttention( [default6]:[rank30]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default6]:[rank30]: config.num_attention_heads % tp_pg.size() == 0 [default6]:[rank30]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default3]:[rank27]: Traceback (most recent call last): [default3]:[rank27]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default3]:[rank27]: trainer = DistributedTrainer(config_file) [default3]:[rank27]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default3]:[rank27]: self.model = self.init_model() # Defines self.model [default3]:[rank27]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default3]:[rank27]: model = self._init_model_instance() [default3]:[rank27]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default3]:[rank27]: model = self._init_model( [default3]:[rank27]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default3]:[rank27]: model = build_model( [default3]:[rank27]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default3]:[rank27]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default3]:[rank27]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default3]:[rank27]: self.pp_block = self.module_builder(**self.module_kwargs) [default3]:[rank27]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default3]:[rank27]: self.attn = CausalSelfAttention( [default3]:[rank27]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default3]:[rank27]: config.num_attention_heads % tp_pg.size() == 0 [default3]:[rank27]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default7]:[rank31]: Traceback (most recent call last): [default7]:[rank31]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default7]:[rank31]: trainer = DistributedTrainer(config_file) [default7]:[rank31]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default7]:[rank31]: self.model = self.init_model() # Defines self.model [default7]:[rank31]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default7]:[rank31]: model = self._init_model_instance() [default7]:[rank31]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default7]:[rank31]: model = self._init_model( [default7]:[rank31]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default7]:[rank31]: model = build_model( [default7]:[rank31]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default7]:[rank31]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default7]:[rank31]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default7]:[rank31]: self.pp_block = self.module_builder(**self.module_kwargs) [default7]:[rank31]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default7]:[rank31]: self.attn = CausalSelfAttention( [default7]:[rank31]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default7]:[rank31]: config.num_attention_heads % tp_pg.size() == 0 [default7]:[rank31]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default2]:[rank34]: Traceback (most recent call last): [default2]:[rank34]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default2]:[rank34]: trainer = DistributedTrainer(config_file) [default2]:[rank34]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default2]:[rank34]: self.model = self.init_model() # Defines self.model [default2]:[rank34]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default2]:[rank34]: model = self._init_model_instance() [default2]:[rank34]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default2]:[rank34]: model = self._init_model( [default2]:[rank34]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default3]:[rank51]: Traceback (most recent call last): [default3]:[rank51]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default3]:[rank51]: trainer = DistributedTrainer(config_file) [default3]:[rank51]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default3]:[rank51]: self.model = self.init_model() # Defines self.model [default3]:[rank51]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default3]:[rank51]: model = self._init_model_instance() [default3]:[rank51]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default3]:[rank51]: model = self._init_model( [default3]:[rank51]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default3]:[rank51]: model = build_model( [def[default2]:[rank34]: model = build_model( [default2]:[rank34]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default2]:[rank34]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default2]:[rank34]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default2]:[rank34]: self.pp_block = self.module_builder(**self.module_kwargs) [default2]:[rank34]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default2]:[rank34]: self.attn = CausalSelfAttention( [default2]:[rank34]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default2]:[rank34]: config.num_attention_heads % tp_pg.size() == 0 [default2]:[rank34]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). ault3]:[rank51]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default3]:[rank51]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default3]:[rank51]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default3]:[rank51]: self.pp_block = self.module_builder(**self.module_kwargs) [default3]:[rank51]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default3]:[rank51]: self.attn = CausalSelfAttention( [default3]:[rank51]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default2]:[rank58]: Traceback (most recent call last): [default2]:[rank58]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default2]:[rank58]: trainer = DistributedTrainer(config_file) [default2]:[rank58]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default2]:[rank58]: self.model = self.init_model() # Defines self.model [default2]:[rank58]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default2]:[rank58]: model = self._init_model_instance() [default2]:[rank58]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default2]:[rank58]: model = self._init_model( [default2]:[rank58]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default2]:[rank58]: model = build_model( [def[default3]:[rank51]: config.num_attention_heads % tp_pg.size() == 0 [default3]:[rank51]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). ault2]:[rank58]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default2]:[rank58]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default2]:[rank58]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default2]:[rank58]: self.pp_block = self.module_builder(**self.module_kwargs) [default2]:[rank58]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default2]:[rank58]: self.attn = CausalSelfAttention( [default2]:[rank58]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default2]:[rank58]: config.num_attention_heads % tp_pg.size() == 0 [default2]:[rank58]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default2]:[rank18]: Traceback (most recent call last): [default2]:[rank18]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default4]:[rank20]: Traceback (most recent call last): [default4]:[rank20]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default2]:[rank18]: trainer = DistributedTrainer(config_file) [default2]:[rank18]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default2]:[rank18]: self.model = self.init_model() # Defines self.model [default2]:[rank18]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default2]:[rank18]: model = self._init_model_instance() [default2]:[rank18]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default2]:[rank18]: model = self._init_model( [default2]:[rank18]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default2]:[rank18]: model = build_model( [default2]:[rank18]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default2]:[rank18]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default4]:[rank20]: trainer = DistributedTrainer(config_file) [default0]:[rank16]: Traceback (most recent call last): [default0]:[rank16]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default4]:[rank20]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default2]:[rank18]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default0]:[rank16]: trainer = DistributedTrainer(config_file) [default4]:[rank20]: self.model = self.init_model() # Defines self.model [default0]:[rank16]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default0]:[rank16]: self.model = self.init_model() # Defines self.model [default2]:[rank18]: self.pp_block = self.module_builder(**self.module_kwargs) [default4]:[rank20]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default0]:[rank16]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default2]:[rank18]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default4]:[rank20]: model = self._init_model_instance() [default2]:[rank18]: self.attn = CausalSelfAttention( [default0]:[rank16]: model = self._init_model_instance() [default4]:[rank20]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default0]:[rank16]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default4]:[rank20]: model = self._init_model( [default4]:[rank20]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default0]:[rank16]: model = self._init_model( [default0]:[rank16]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default2]:[rank18]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default2]:[rank18]: config.num_attention_heads % tp_pg.size() == 0 [default0]:[rank16]: model = build_model( [default0]:[rank16]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default0]:[rank16]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default0]:[rank16]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default4]:[rank20]: model = build_model( [default2]:[rank10]: Traceback (most recent call last): [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default2]:[rank10]: trainer = DistributedTrainer(config_file) [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default2]:[rank10]: self.model = self.init_model() # Defines self.model [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default2]:[rank10]: model = self._init_model_instance() [default2]:[rank18]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default4]:[rank20]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default0]:[rank16]: self.pp_block = self.module_builder(**self.module_kwargs) [default0]:[rank16]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default4]:[rank20]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default2]:[rank10]: model = self._init_model( [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default2]:[rank10]: model = build_model( [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default2]:[rank10]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default2]:[rank10]: self.pp_block = self.module_builder(**self.module_kwargs) [default0]:[rank16]: self.attn = CausalSelfAttention( [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default2]:[rank10]: self.attn = CausalSelfAttention( [default2]:[rank10]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default2]:[rank10]: config.num_attention_heads % tp_pg.size() == 0 [default2]:[rank10]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default0]:[rank16]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default0]:[rank16]: config.num_attention_heads % tp_pg.size() == 0 [default0]:[rank16]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default4]:[rank20]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default4]:[rank20]: self.pp_block = self.module_builder(**self.module_kwargs) [default4]:[rank20]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default4]:[rank20]: self.attn = CausalSelfAttention( [default4]:[rank20]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default4]:[rank20]: config.num_attention_heads % tp_pg.size() == 0 [default4]:[rank20]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default5]:[rank45]: Traceback (most recent call last): [default5]:[rank45]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default5]:[rank45]: trainer = DistributedTrainer(config_file) [default5]:[rank45]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default5]:[rank45]: self.model = self.init_model() # Defines self.model [default5]:[rank45]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default5]:[rank45]: model = self._init_model_instance() [default5]:[rank45]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default5]:[rank45]: model = self._init_model( [default5]:[rank45]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default5]:[rank45]: model = build_model( [default5]:[rank45]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default5]:[rank45]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default5]:[rank45]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default5]:[rank45]: self.pp_block = self.module_builder(**self.module_kwargs) [default5]:[rank45]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default5]:[rank45]: self.attn = CausalSelfAttention( [default5]:[rank45]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default5]:[rank45]: config.num_attention_heads % tp_pg.size() == 0 [default5]:[rank45]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default2]:[rank42]: Traceback (most recent call last): [default2]:[rank42]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default3]:[rank43]: Traceback (most recent call last): [default0]:[rank40]: Traceback (most recent call last): [default3]:[rank11]: Traceback (most recent call last): [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default2]:[rank42]: trainer = DistributedTrainer(config_file) [default1]:[rank41]: Traceback (most recent call last): [default1]:[rank41]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default0]:[rank40]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default2]:[rank42]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default7]:[rank47]: Traceback (most recent call last): [default0]:[rank40]: trainer = DistributedTrainer(config_file) [default3]:[rank11]: trainer = DistributedTrainer(config_file) [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default0]:[rank40]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default3]:[rank11]: self.model = self.init_model() # Defines self.model [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default3]:[rank11]: model = self._init_model_instance() [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default3]:[rank11]: model = self._init_model( [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default3]:[rank11]: model = build_model( [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default0]:[rank40]: self.model = self.init_model() # Defines self.model [default7]:[rank47]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default3]:[rank11]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default3]:[rank11]: self.pp_block = self.module_builder(**self.module_kwargs) [default2]:[rank42]: self.model = self.init_model() # Defines self.model [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default3]:[rank11]: self.attn = CausalSelfAttention( [default3]:[rank43]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default1]:[rank41]: trainer = DistributedTrainer(config_file) [default3]:[rank11]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default3]:[rank11]: config.num_attention_heads % tp_pg.size() == 0 [default3]:[rank11]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default2]:[rank42]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default2]:[rank50]: Traceback (most recent call last): [default2]:[rank50]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default0]:[rank40]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default7]:[rank47]: trainer = DistributedTrainer(config_file) [default3]:[rank43]: trainer = DistributedTrainer(config_file) [default3]:[rank43]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default1]:[rank41]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default7]:[rank15]: Traceback (most recent call last): [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default7]:[rank15]: trainer = DistributedTrainer(config_file) [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default7]:[rank15]: self.model = self.init_model() # Defines self.model [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default7]:[rank15]: model = self._init_model_instance() [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default7]:[rank15]: model = self._init_model( [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default7]:[rank15]: model = build_model( [def[default2]:[rank50]: trainer = DistributedTrainer(config_file) [default2]:[rank42]: model = self._init_model_instance() [default2]:[rank42]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance ault7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default2]:[rank50]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default2]:[rank50]: self.model = self.init_model() # Defines self.model [default2]:[rank50]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default2]:[rank50]: model = self._init_model_instance() [default2]:[rank50]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default2]:[rank50]: model = self._init_model( [default2]:[rank50]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default2]:[rank50]: model = build_model( [default0]:[rank40]: model = self._init_model_instance() [default1]:[rank41]: self.model = self.init_model() # Defines self.model [default7]:[rank15]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default7]:[rank15]: self.pp_block = self.module_builder(**self.module_kwargs) [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default7]:[rank15]: self.attn = CausalSelfAttention( [default7]:[rank15]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default7]:[rank15]: config.num_attention_heads % tp_pg.size() == 0 [default7]:[rank15]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default2]:[rank50]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default2]:[rank50]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default2]:[rank42]: model = self._init_model( [default2]:[rank50]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default2]:[rank50]: self.pp_block = self.module_builder(**self.module_kwargs) [default7]:[rank47]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default2]:[rank50]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default2]:[rank50]: self.attn = CausalSelfAttention( [default1]:[rank41]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default2]:[rank50]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default2]:[rank50]: config.num_attention_heads % tp_pg.size() == 0 [default2]:[rank50]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default3]:[rank43]: self.model = self.init_model() # Defines self.model [default2]:[rank42]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default7]:[rank47]: self.model = self.init_model() # Defines self.model [default0]:[rank40]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default3]:[rank43]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default1]:[rank41]: model = self._init_model_instance() [default0]:[rank40]: model = self._init_model( [default7]:[rank47]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default2]:[rank42]: model = build_model( [default2]:[rank42]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default7]:[rank47]: model = self._init_model_instance() [default1]:[rank41]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default3]:[rank43]: model = self._init_model_instance() [default7]:[rank47]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default0]:[rank40]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default2]:[rank42]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default3]:[rank43]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default1]:[rank41]: model = self._init_model( [default0]:[rank40]: model = build_model( [default7]:[rank47]: model = self._init_model( [default1]:[rank41]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default3]:[rank43]: model = self._init_model( [default2]:[rank42]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default1]:[rank41]: model = build_model( [default7]:[rank47]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default3]:[rank43]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default0]:[rank40]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default7]:[rank63]: Traceback (most recent call last): [default4]:[rank60]: Traceback (most recent call last): [default7]:[rank47]: model = build_model( [default3]:[rank43]: model = build_model( [default0]:[rank40]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default7]:[rank47]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default1]:[rank41]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default2]:[rank42]: self.pp_block = self.module_builder(**self.module_kwargs) [default2]:[rank42]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default7]:[rank47]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default7]:[rank63]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default7]:[rank63]: trainer = DistributedTrainer(config_file) [default1]:[rank41]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default1]:[rank41]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default3]:[rank43]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default7]:[rank63]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default7]:[rank63]: self.model = self.init_model() # Defines self.model [default7]:[rank63]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default7]:[rank63]: model = self._init_model_instance() [default7]:[rank63]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default7]:[rank63]: model = self._init_model( [default7]:[rank47]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default7]:[rank63]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default7]:[rank63]: model = build_model( [default7]:[rank63]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default7]:[rank63]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default7]:[rank63]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default7]:[rank63]: self.pp_block = self.module_builder(**self.module_kwargs) [default2]:[rank42]: self.attn = CausalSelfAttention( [default7]:[rank47]: self.pp_block = self.module_builder(**self.module_kwargs) [default1]:[rank41]: self.pp_block = self.module_builder(**self.module_kwargs) [default0]:[rank40]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default3]:[rank43]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default1]:[rank41]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default0]:[rank40]: self.pp_block = self.module_builder(**self.module_kwargs) [default3]:[rank43]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default7]:[rank47]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default2]:[rank42]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default3]:[rank43]: self.pp_block = self.module_builder(**self.module_kwargs) [default3]:[rank43]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default7]:[rank63]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default4]:[rank60]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default1]:[rank41]: self.attn = CausalSelfAttention( [default7]:[rank47]: self.attn = CausalSelfAttention( [default6]:[rank22]: Traceback (most recent call last): [default0]:[rank40]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default5]:[rank53]: Traceback (most recent call last): [default5]:[rank53]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default5]:[rank53]: trainer = DistributedTrainer(config_file) [default5]:[rank53]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default5]:[rank53]: self.model = self.init_model() # Defines self.model [default5]:[rank53]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default5]:[rank53]: model = self._init_model_instance() [default5]:[rank53]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default5]:[rank53]: model = self._init_model( [default5]:[rank53]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default5]:[rank53]: model = build_model( [def[default4]:[rank60]: trainer = DistributedTrainer(config_file) [default2]:[rank42]: config.num_attention_heads % tp_pg.size() == 0 [default3]:[rank43]: self.attn = CausalSelfAttention( [default1]:[rank41]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default0]:[rank40]: self.attn = CausalSelfAttention( [default3]:[rank19]: Traceback (most recent call last): [default3]:[rank19]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default3]:[rank19]: trainer = DistributedTrainer(config_file) [default6]:[rank22]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default6]:[rank22]: trainer = DistributedTrainer(config_file) [default6]:[rank22]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default7]:[rank47]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default6]:[rank22]: self.model = self.init_model() # Defines self.model [default1]:[rank41]: config.num_attention_heads % tp_pg.size() == 0 [default6]:[rank22]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default0]:[rank40]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default7]:[rank47]: config.num_attention_heads % tp_pg.size() == 0 ault5]:[rank53]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default5]:[rank53]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default5]:[rank53]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default5]:[rank53]: self.pp_block = self.module_builder(**self.module_kwargs) [default5]:[rank53]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default5]:[rank53]: self.attn = CausalSelfAttention( [default1]:[rank41]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default5]:[rank53]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default5]:[rank53]: config.num_attention_heads % tp_pg.size() == 0 [default5]:[rank53]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default3]:[rank43]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default6]:[rank22]: model = self._init_model_instance() [default6]:[rank22]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default6]:[rank22]: model = self._init_model( [default6]:[rank22]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default6]:[rank22]: model = build_model( [default3]:[rank43]: config.num_attention_heads % tp_pg.size() == 0 [default6]:[rank22]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default7]:[rank47]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default0]:[rank40]: config.num_attention_heads % tp_pg.size() == 0 [default0]:[rank56]: Traceback (most recent call last): [default0]:[rank56]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default6]:[rank22]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default3]:[rank19]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default1]:[rank17]: Traceback (most recent call last): [default6]:[rank22]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default3]:[rank19]: self.model = self.init_model() # Defines self.model [default4]:[rank60]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default7]:[rank63]: self.attn = CausalSelfAttention( [default1]:[rank17]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default5]:[rank61]: Traceback (most recent call last): [default1]:[rank17]: trainer = DistributedTrainer(config_file) [default5]:[rank61]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default4]:[rank60]: self.model = self.init_model() # Defines self.model [default1]:[rank57]: Traceback (most recent call last): [default1]:[rank57]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default6]:[rank62]: Traceback (most recent call last): [default6]:[rank62]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default4]:[rank60]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default6]:[rank22]: self.pp_block = self.module_builder(**self.module_kwargs) [default5]:[rank61]: trainer = DistributedTrainer(config_file) [default5]:[rank61]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default3]:[rank19]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default6]:[rank22]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default3]:[rank19]: model = self._init_model_instance() [default6]:[rank22]: self.attn = CausalSelfAttention( [default3]:[rank19]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default0]:[rank56]: trainer = DistributedTrainer(config_file) [default1]:[rank57]: trainer = DistributedTrainer(config_file) [default1]:[rank17]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default6]:[rank62]: trainer = DistributedTrainer(config_file) [default0]:[rank56]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default4]:[rank60]: model = self._init_model_instance() [default6]:[rank54]: Traceback (most recent call last): [default6]:[rank62]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default6]:[rank62]: self.model = self.init_model() # Defines self.model [default6]:[rank62]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default1]:[rank17]: self.model = self.init_model() # Defines self.model [default1]:[rank17]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default6]:[rank22]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default6]:[rank22]: config.num_attention_heads % tp_pg.size() == 0 [default1]:[rank17]: model = self._init_model_instance() [default1]:[rank17]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default1]:[rank17]: model = self._init_model( [default7]:[rank23]: Traceback (most recent call last): [default3]:[rank19]: model = self._init_model( [default6]:[rank22]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default6]:[rank54]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default6]:[rank54]: trainer = DistributedTrainer(config_file) [default6]:[rank54]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default6]:[rank54]: self.model = self.init_model() # Defines self.model [default6]:[rank54]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default4]:[rank60]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default1]:[rank57]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default1]:[rank57]: self.model = self.init_model() # Defines self.model [default6]:[rank54]: model = self._init_model_instance() [default6]:[rank54]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default1]:[rank57]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default6]:[rank62]: model = self._init_model_instance() [default6]:[rank54]: model = self._init_model( [default6]:[rank54]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default6]:[rank54]: model = build_model( [default3]:[rank19]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default6]:[rank54]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default6]:[rank54]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default7]:[rank23]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default1]:[rank17]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default6]:[rank54]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default7]:[rank23]: trainer = DistributedTrainer(config_file) [default6]:[rank54]: self.pp_block = self.module_builder(**self.module_kwargs) [default3]:[rank19]: model = build_model( [default3]:[rank19]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default6]:[rank54]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default6]:[rank54]: self.attn = CausalSelfAttention( [default6]:[rank62]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default5]:[rank61]: self.model = self.init_model() # Defines self.model [default1]:[rank57]: model = self._init_model_instance() [default6]:[rank54]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default6]:[rank54]: config.num_attention_heads % tp_pg.size() == 0 [default6]:[rank54]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default1]:[rank57]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default4]:[rank60]: model = self._init_model( [default7]:[rank63]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default1]:[rank57]: model = self._init_model( [default5]:[rank61]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default7]:[rank63]: config.num_attention_heads % tp_pg.size() == 0 [default4]:[rank60]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default7]:[rank23]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default7]:[rank23]: self.model = self.init_model() # Defines self.model [default1]:[rank17]: model = build_model( [default7]:[rank23]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default7]:[rank23]: model = self._init_model_instance() [default7]:[rank23]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default3]:[rank19]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default5]:[rank61]: model = self._init_model_instance() [default5]:[rank61]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default1]:[rank57]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default0]:[rank56]: self.model = self.init_model() # Defines self.model [default0]:[rank56]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default0]:[rank56]: model = self._init_model_instance() [default3]:[rank19]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default7]:[rank23]: model = self._init_model( [default7]:[rank23]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default3]:[rank19]: self.pp_block = self.module_builder(**self.module_kwargs) [default1]:[rank17]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default6]:[rank62]: model = self._init_model( [default7]:[rank63]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default1]:[rank17]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default7]:[rank23]: model = build_model( [default3]:[rank19]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default3]:[rank19]: self.attn = CausalSelfAttention( [default3]:[rank59]: Traceback (most recent call last): [default4]:[rank60]: model = build_model( [default4]:[rank60]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default5]:[rank61]: model = self._init_model( [default0]:[rank56]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default3]:[rank59]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default3]:[rank59]: trainer = DistributedTrainer(config_file) [default5]:[rank61]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default1]:[rank17]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default1]:[rank17]: self.pp_block = self.module_builder(**self.module_kwargs) [default5]:[rank61]: model = build_model( [default7]:[rank23]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default4]:[rank60]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default4]:[rank60]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default1]:[rank17]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default1]:[rank57]: model = build_model( [default7]:[rank23]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default7]:[rank23]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default1]:[rank17]: self.attn = CausalSelfAttention( [default3]:[rank19]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default3]:[rank19]: config.num_attention_heads % tp_pg.size() == 0 [default7]:[rank23]: self.pp_block = self.module_builder(**self.module_kwargs) [default7]:[rank23]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default1]:[rank17]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default1]:[rank17]: config.num_attention_heads % tp_pg.size() == 0 [default6]:[rank62]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default3]:[rank59]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default5]:[rank61]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default3]:[rank19]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default5]:[rank61]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default6]:[rank62]: model = build_model( [default6]:[rank62]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default3]:[rank59]: self.model = self.init_model() # Defines self.model [default4]:[rank60]: self.pp_block = self.module_builder(**self.module_kwargs) [default4]:[rank60]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default5]:[rank61]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default6]:[rank62]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default1]:[rank17]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default7]:[rank23]: self.attn = CausalSelfAttention( [default7]:[rank23]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default7]:[rank23]: config.num_attention_heads % tp_pg.size() == 0 [default7]:[rank23]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default6]:[rank62]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default1]:[rank57]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default1]:[rank57]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default0]:[rank56]: model = self._init_model( [default4]:[rank60]: self.attn = CausalSelfAttention( [default0]:[rank56]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default0]:[rank56]: model = build_model( [default1]:[rank57]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default1]:[rank57]: self.pp_block = self.module_builder(**self.module_kwargs) [default4]:[rank60]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default0]:[rank56]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default0]:[rank56]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default1]:[rank57]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default1]:[rank57]: self.attn = CausalSelfAttention( [default5]:[rank61]: self.pp_block = self.module_builder(**self.module_kwargs) [default5]:[rank21]: Traceback (most recent call last): [default5]:[rank21]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default5]:[rank21]: trainer = DistributedTrainer(config_file) [default5]:[rank21]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default5]:[rank21]: self.model = self.init_model() # Defines self.model [default5]:[rank21]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default5]:[rank21]: model = self._init_model_instance() [default5]:[rank21]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default5]:[rank21]: model = self._init_model( [default5]:[rank21]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default5]:[rank21]: model = build_model( [def[default0]:[rank56]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default1]:[rank57]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default5]:[rank61]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default3]:[rank59]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default3]:[rank59]: model = self._init_model_instance() [default5]:[rank61]: self.attn = CausalSelfAttention( [default1]:[rank57]: config.num_attention_heads % tp_pg.size() == 0 [default3]:[rank59]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance ault5]:[rank21]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default5]:[rank21]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default5]:[rank21]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default5]:[rank21]: self.pp_block = self.module_builder(**self.module_kwargs) [default5]:[rank21]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default5]:[rank21]: self.attn = CausalSelfAttention( [default5]:[rank21]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default5]:[rank21]: config.num_attention_heads % tp_pg.size() == 0 [default5]:[rank21]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default3]:[rank59]: model = self._init_model( [default5]:[rank61]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default5]:[rank61]: config.num_attention_heads % tp_pg.size() == 0 [default4]:[rank60]: config.num_attention_heads % tp_pg.size() == 0 [default6]:[rank62]: self.pp_block = self.module_builder(**self.module_kwargs) [default6]:[rank62]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default3]:[rank59]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default1]:[rank57]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default6]:[rank62]: self.attn = CausalSelfAttention( [default6]:[rank62]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default0]:[rank56]: self.pp_block = self.module_builder(**self.module_kwargs) [default4]:[rank60]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default5]:[rank61]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default0]:[rank56]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default0]:[rank56]: self.attn = CausalSelfAttention( [default0]:[rank56]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default0]:[rank56]: config.num_attention_heads % tp_pg.size() == 0 [default0]:[rank56]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default6]:[rank62]: config.num_attention_heads % tp_pg.size() == 0 [default6]:[rank62]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default3]:[rank59]: model = build_model( [default3]:[rank59]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default3]:[rank59]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default3]:[rank59]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default3]:[rank59]: self.pp_block = self.module_builder(**self.module_kwargs) [default3]:[rank59]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default3]:[rank59]: self.attn = CausalSelfAttention( [default3]:[rank59]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default3]:[rank59]: config.num_attention_heads % tp_pg.size() == 0 [default3]:[rank59]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default3]:[rank43]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default2]:[rank42]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default0]:[rank40]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default4]:[rank44]: Traceback (most recent call last): [default4]:[rank44]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default4]:[rank44]: trainer = DistributedTrainer(config_file) [default4]:[rank44]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default4]:[rank44]: self.model = self.init_model() # Defines self.model [default4]:[rank44]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default4]:[rank44]: model = self._init_model_instance() [default4]:[rank44]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default4]:[rank44]: model = self._init_model( [default4]:[rank44]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default4]:[rank44]: model = build_model( [default4]:[rank44]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default4]:[rank44]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default4]:[rank44]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default4]:[rank44]: self.pp_block = self.module_builder(**self.module_kwargs) [default4]:[rank44]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default4]:[rank44]: self.attn = CausalSelfAttention( [default4]:[rank44]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default4]:[rank44]: config.num_attention_heads % tp_pg.size() == 0 [default4]:[rank44]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). [default6]:[rank46]: Traceback (most recent call last): [default6]:[rank46]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py", line 233, in [default6]:[rank46]: trainer = DistributedTrainer(config_file) [default6]:[rank46]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 172, in __init__ [default6]:[rank46]: self.model = self.init_model() # Defines self.model [default6]:[rank46]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 672, in init_model [default6]:[rank46]: model = self._init_model_instance() [default6]:[rank46]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 682, in _init_model_instance [default6]:[rank46]: model = self._init_model( [default6]:[rank46]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/trainer.py", line 751, in _init_model [default6]:[rank46]: model = build_model( [default6]:[rank46]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/base.py", line 230, in build_model [default6]:[rank46]: block.build_and_set_rank(target_pp_ranks[target_pp_rank_idx]) [default6]:[rank46]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/parallel/pipeline_parallel/block.py", line 52, in build_and_set_rank [default6]:[rank46]: self.pp_block = self.module_builder(**self.module_kwargs) [default6]:[rank46]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 613, in __init__ [default6]:[rank46]: self.attn = CausalSelfAttention( [default6]:[rank46]: File "/fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/src/nanotron/models/llama.py", line 271, in __init__ [default6]:[rank46]: config.num_attention_heads % tp_pg.size() == 0 [default6]:[rank46]: AssertionError: Number of attention heads (32) must be divisible by TP size (64). E0702 23:40:05.577000 139649365083968 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 1122470) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 E0702 23:40:05.577000 140306096875328 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 1757459) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 E0702 23:40:05.578000 140698504841024 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 1380073) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 E0702 23:40:05.579000 139987077363520 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 3745268) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 E0702 23:40:05.579000 140094763059008 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 862934) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 E0702 23:40:05.579000 140156427769664 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 3873708) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 E0702 23:40:05.580000 139788966299456 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 1401178) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 E0702 23:40:05.579000 139814066579264 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 849628) of binary: /fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/python3.10 Traceback (most recent call last): File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in sys.exit(main()) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper Traceback (most recent call last): File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in Traceback (most recent call last): File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in return f(*args, **kwargs) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main sys.exit(main()) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper return f(*args, **kwargs) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main sys.exit(main()) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper run(args) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run Traceback (most recent call last): File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in return f(*args, **kwargs) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main run(args) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run elastic_launch( File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__ sys.exit(main()) return launch_agent(self._config, self._entrypoint, list(args)) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent run(args) elastic_launch( File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__ File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run Traceback (most recent call last): File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED ------------------------------------------------------------ Failures: [1]: time : 2024-07-02_23:40:05 host : ip-26-0-162-233.ec2.internal rank : 33 (local_rank: 1) exitcode : 1 (pid: 1380074) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [2]: time : 2024-07-02_23:40:05 host : ip-26-0-162-233.ec2.internal rank : 34 (local_rank: 2) exitcode : 1 (pid: 1380075) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [3]: time : 2024-07-02_23:40:05 host : ip-26-0-162-233.ec2.internal rank : 35 (local_rank: 3) exitcode : 1 (pid: 1380076) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [4]: time : 2024-07-02_23:40:05 host : ip-26-0-162-233.ec2.internal rank : 36 (local_rank: 4) exitcode : 1 (pid: 1380077) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [5]: time : 2024-07-02_23:40:05 host : ip-26-0-162-233.ec2.internal rank : 37 (local_rank: 5) exitcode : 1 (pid: 1380078) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [6]: time : 2024-07-02_23:40:05 host : ip-26-0-162-233.ec2.internal rank : 38 (local_rank: 6) exitcode : 1 (pid: 1380079) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [7]: time : 2024-07-02_23:40:05 host : ip-26-0-162-233.ec2.internal rank : 39 (local_rank: 7) exitcode : 1 (pid: 1380080) error_file: traceback : To enable traceback see: https://pytorch.org/docs/st return f(*args, **kwargs) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main able/elastic/errors.html ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-07-02_23:40:05 host : ip-26-0-162-233.ec2.internal rank : 32 (local_rank: 0) exitcode : 1 (pid: 1380073) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================ return launch_agent(self._config, self._entrypoint, list(args)) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent elastic_launch( sys.exit(main()) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__ raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED ------------------------------------------------------------ Failures: [1]: time : 2024-07-02_23:40:05 host : ip-26-0-161-153.ec2.internal rank : 17 (local_rank: 1) exitcode : 1 (pid: 1401179) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [2]: time : 2024-07-02_23:40:05 host : ip-26-0-161-153.ec2.internal rank : 18 (local_rank: 2) exitcode : 1 (pid: 1401180) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [3]: time : 2024-07-02_23:40:05 host : ip-26-0-161-153.ec2.internal rank : 19 (local_rank: 3) exitcode : 1 (pid: 1401181) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [4]: time : 2024-07-02_23:40:05 host : ip-26-0-161-153.ec2.internal rank : 20 (local_rank: 4) exitcode : 1 (pid: 1401182) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [5]: time : 2024-07-02_23:40:05 host : ip-26-0-161-153.ec2.internal rank : 21 (local_rank: 5) exitcode : 1 (pid: 1401183) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [6]: time : 2024-07-02_23:40:05 host : ip-26-0-161-153.ec2.internal rank : 22 (local_rank: 6) exitcode : 1 (pid: 1401184) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [7]: time : 2024-07-02_23:40:05 host : ip-26-0-161-153.ec2.internal rank : 23 (local_rank: 7) exitcode : 1 (pid: 1401185) error_file: traceback : To enable traceback see: https://pytorch.org/docs/st return launch_agent(self._config, self._entrypoint, list(args)) run(args) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run able/elastic/errors.html ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-07-02_23:40:05 host : ip-26-0-161-153.ec2.internal rank : 16 (local_rank: 0) exitcode : 1 (pid: 1401178) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================ return f(*args, **kwargs) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED ------------------------------------------------------------ Failures: [1]: time : 2024-07-02_23:40:05 host : ip-26-0-171-102.ec2.internal rank : 41 (local_rank: 1) exitcode : 1 (pid: 3745269) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [2]: time : 2024-07-02_23:40:05 host : ip-26-0-171-102.ec2.internal rank : 42 (local_rank: 2) exitcode : 1 (pid: 3745270) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [3]: time : 2024-07-02_23:40:05 host : ip-26-0-171-102.ec2.internal rank : 43 (local_rank: 3) exitcode : 1 (pid: 3745271) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [4]: time : 2024-07-02_23:40:05 host : ip-26-0-171-102.ec2.internal rank : 44 (local_rank: 4) exitcode : 1 (pid: 3745272) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [5]: time : 2024-07-02_23:40:05 host : ip-26-0-171-102.ec2.internal rank : 45 (local_rank: 5) exitcode : 1 (pid: 3745273) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [6]: time : 2024-07-02_23:40:05 host : ip-26-0-171-102.ec2.internal rank : 46 (local_rank: 6) exitcode : 1 (pid: 3745274) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [7]: time : 2024-07-02_23:40:05 host : ip-26-0-171-102.ec2.internal rank : 47 (local_rank: 7) exitcode : 1 (pid: 3745275) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-07-02_23:40:05 host : ip-26-0-171-102.ec2.internal rank : 40 (local_rank: 0) exitcode : 1 (pid: 3745268) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================ elastic_launch( File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__ run(args) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run return launch_agent(self._config, self._entrypoint, list(args)) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent elastic_launch( File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__ raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED ------------------------------------------------------------ Failures: [1]: time : 2024-07-02_23:40:05 host : ip-26-0-161-103.ec2.internal rank : 9 (local_rank: 1) exitcode : 1 (pid: 849629) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [2]: time : 2024-07-02_23:40:05 host : ip-26-0-161-103.ec2.internal rank : 10 (local_rank: 2) exitcode : 1 (pid: 849630) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [3]: time : 2024-07-02_23:40:05 host : ip-26-0-161-103.ec2.internal rank : 11 (local_rank: 3) exitcode : 1 (pid: 849631) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [4]: time : 2024-07-02_23:40:05 host : ip-26-0-161-103.ec2.internal rank : 12 (local_rank: 4) exitcode : 1 (pid: 849632) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [5]: time : 2024-07-02_23:40:05 host : ip-26-0-161-103.ec2.internal rank : 13 (local_rank: 5) exitcode : 1 (pid: 849633) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [6]: time : 2024-07-02_23:40:05 host : ip-26-0-161-103.ec2.internal rank : 14 (local_rank: 6) exitcode : 1 (pid: 849634) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [7]: time : 2024-07-02_23:40:05 host : ip-26-0-161-103.ec2.internal rank : 15 (local_rank: 7) exitcode : 1 (pid: 849635) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-07-02_23:40:05 host : ip-26-0-161-103.ec2.internal rank : 8 (local_rank: 0) exitcode : 1 (pid: 849628) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================ return launch_agent(self._config, self._entrypoint, list(args)) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED ------------------------------------------------------------ Failures: [1]: time : 2024-07-02_23:40:05 host : ip-26-0-161-78.ec2.internal rank : 25 (local_rank: 1) exitcode : 1 (pid: 1122471) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [2]: time : 2024-07-02_23:40:05 host : ip-26-0-161-78.ec2.internal rank : 26 (local_rank: 2) exitcode : 1 (pid: 1122472) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [3]: time : 2024-07-02_23:40:05 host : ip-26-0-161-78.ec2.internal rank : 27 (local_rank: 3) exitcode : 1 (pid: 1122473) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [4]: time : 2024-07-02_23:40:05 host : ip-26-0-161-78.ec2.internal rank : 28 (local_rank: 4) exitcode : 1 (pid: 1122474) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [5]: time : 2024-07-02_23:40:05 host : ip-26-0-161-78.ec2.internal rank : 29 (local_rank: 5) exitcode : 1 (pid: 1122475) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [6]: time : 2024-07-02_23:40:05 host : ip-26-0-161-78.ec2.internal rank : 30 (local_rank: 6) exitcode : 1 (pid: 1122476) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [7]: time : 2024-07-02_23:40:05 host : ip-26-0-161-78.ec2.internal rank : 31 (local_rank: 7) exitcode : 1 (pid: 1122477) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-07-02_23:40:05 host : ip-26-0-161-78.ec2.internal rank : 24 (local_rank: 0) exitcode : 1 (pid: 1122470) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================ Traceback (most recent call last): File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in sys.exit(main()) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper return f(*args, **kwargs) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main Traceback (most recent call last): File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in run(args) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run sys.exit(main()) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper elastic_launch( File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__ return f(*args, **kwargs) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main return launch_agent(self._config, self._entrypoint, list(args)) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent run(args) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED ------------------------------------------------------------ Failures: [1]: time : 2024-07-02_23:40:05 host : ip-26-0-160-225.ec2.internal rank : 1 (local_rank: 1) exitcode : 1 (pid: 1757460) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [2]: time : 2024-07-02_23:40:05 host : ip-26-0-160-225.ec2.internal rank : 2 (local_rank: 2) exitcode : 1 (pid: 1757461) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [3]: time : 2024-07-02_23:40:05 host : ip-26-0-160-225.ec2.internal rank : 3 (local_rank: 3) exitcode : 1 (pid: 1757462) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [4]: time : 2024-07-02_23:40:05 host : ip-26-0-160-225.ec2.internal rank : 4 (local_rank: 4) exitcode : 1 (pid: 1757463) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [5]: time : 2024-07-02_23:40:05 host : ip-26-0-160-225.ec2.internal rank : 5 (local_rank: 5) exitcode : 1 (pid: 1757464) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [6]: time : 2024-07-02_23:40:05 host : ip-26-0-160-225.ec2.internal rank : 6 (local_rank: 6) exitcode : 1 (pid: 1757465) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [7]: time : 2024-07-02_23:40:05 host : ip-26-0-160-225.ec2.internal rank : 7 (local_rank: 7) exitcode : 1 (pid: 1757466) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-07-02_23:40:05 host : ip-26-0-160-225.ec2.internal rank : 0 (local_rank: 0) exitcode : 1 (pid: 1757459) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================ elastic_launch( File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED ------------------------------------------------------------ Failures: [1]: time : 2024-07-02_23:40:05 host : ip-26-0-171-62.ec2.internal rank : 49 (local_rank: 1) exitcode : 1 (pid: 3873709) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [2]: time : 2024-07-02_23:40:05 host : ip-26-0-171-62.ec2.internal rank : 50 (local_rank: 2) exitcode : 1 (pid: 3873710) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [3]: time : 2024-07-02_23:40:05 host : ip-26-0-171-62.ec2.internal rank : 51 (local_rank: 3) exitcode : 1 (pid: 3873711) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [4]: time : 2024-07-02_23:40:05 host : ip-26-0-171-62.ec2.internal rank : 52 (local_rank: 4) exitcode : 1 (pid: 3873712) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [5]: time : 2024-07-02_23:40:05 host : ip-26-0-171-62.ec2.internal rank : 53 (local_rank: 5) exitcode : 1 (pid: 3873713) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [6]: time : 2024-07-02_23:40:05 host : ip-26-0-171-62.ec2.internal rank : 54 (local_rank: 6) exitcode : 1 (pid: 3873714) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [7]: time : 2024-07-02_23:40:05 host : ip-26-0-171-62.ec2.internal rank : 55 (local_rank: 7) exitcode : 1 (pid: 3873715) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-07-02_23:40:05 host : ip-26-0-171-62.ec2.internal rank : 48 (local_rank: 0) exitcode : 1 (pid: 3873708) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================ Traceback (most recent call last): File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/bin/torchrun", line 8, in sys.exit(main()) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper return f(*args, **kwargs) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main run(args) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run elastic_launch( File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ /fsx/ferdinandmom/ferdinand-hf/bench_cluster/nanotron/run_train.py FAILED ------------------------------------------------------------ Failures: [1]: time : 2024-07-02_23:40:05 host : ip-26-0-171-88.ec2.internal rank : 57 (local_rank: 1) exitcode : 1 (pid: 862935) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [2]: time : 2024-07-02_23:40:05 host : ip-26-0-171-88.ec2.internal rank : 58 (local_rank: 2) exitcode : 1 (pid: 862936) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [3]: time : 2024-07-02_23:40:05 host : ip-26-0-171-88.ec2.internal rank : 59 (local_rank: 3) exitcode : 1 (pid: 862937) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [4]: time : 2024-07-02_23:40:05 host : ip-26-0-171-88.ec2.internal rank : 60 (local_rank: 4) exitcode : 1 (pid: 862938) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [5]: time : 2024-07-02_23:40:05 host : ip-26-0-171-88.ec2.internal rank : 61 (local_rank: 5) exitcode : 1 (pid: 862939) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [6]: time : 2024-07-02_23:40:05 host : ip-26-0-171-88.ec2.internal rank : 62 (local_rank: 6) exitcode : 1 (pid: 862940) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [7]: time : 2024-07-02_23:40:05 host : ip-26-0-171-88.ec2.internal rank : 63 (local_rank: 7) exitcode : 1 (pid: 862941) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-07-02_23:40:05 host : ip-26-0-171-88.ec2.internal rank : 56 (local_rank: 0) exitcode : 1 (pid: 862934) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================ srun: error: ip-26-0-160-225: task 0: Exited with exit code 1 srun: error: ip-26-0-161-78: task 1: Exited with exit code 1 srun: error: ip-26-0-162-233: task 4: Exited with exit code 1 srun: error: ip-26-0-161-153: task 3: Exited with exit code 1 srun: error: ip-26-0-171-62: task 5: Exited with exit code 1 srun: error: ip-26-0-171-88: task 6: Exited with exit code 1 srun: error: ip-26-0-171-102: task 7: Exited with exit code 1 srun: error: ip-26-0-161-103: task 2: Exited with exit code 1 Consider using `hf_transfer` for faster uploads. This solution comes with some limitations. See https://huggingface.co/docs/huggingface_hub/hf_transfer for more details.