llama-1B/8_GPUS/dp-2_tp-1_pp-4_mbz-2/log.out

========================
START TIME: Wed Jul  3 23:48:12 UTC 2024
python3 version = Python 3.10.14
========================
The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well.
Token is valid (permission: write).
Your token has been saved to /admin/home/ferdinand_mom/.cache/huggingface/token
Login successful
Already on 'bench_cluster'
M	examples/config_tiny_llama.py
M	examples/config_tiny_llama.yaml
M	examples/train_tiny_llama.sh
M	src/nanotron/models/llama.py
M	src/nanotron/trainer.py
Your branch is up to date with 'origin/bench_cluster'.
Job status: RUNNING
W0703 23:48:15.519000 140292654851904 torch/distributed/run.py:757] 
W0703 23:48:15.519000 140292654851904 torch/distributed/run.py:757] *****************************************
W0703 23:48:15.519000 140292654851904 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0703 23:48:15.519000 140292654851904 torch/distributed/run.py:757] *****************************************
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]: Config:
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]: Config(general=GeneralArgs(project='bench_cluster',
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                            run='%date_%jobid',
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                            seed=42,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                            step=None,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                            consumed_train_samples=None,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                            benchmark_csv_path=None,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                            ignore_sanity_checks=True),
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:        parallelism=ParallelismArgs(dp=2,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                    pp=4,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                    tp=1,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                    pp_engine=<nanotron.parallel.pipeline_parallel.engine.OneForwardOneBackwardPipelineEngine object at 0x7f0ea8be8880>,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                    tp_mode=<TensorParallelLinearMode.REDUCE_SCATTER: 2>,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                    tp_linear_async_communication=False,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                    expert_parallel_size=1),
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:        model=ModelArgs(model_config=LlamaConfig(bos_token_id=1,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                 eos_token_id=2,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                 hidden_act='silu',
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                 hidden_size=2048,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                 initializer_range=0.02,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                 intermediate_size=4096,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                 is_llama_config=True,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                 max_position_embeddings=4096,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                 num_attention_heads=32,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                 num_hidden_layers=24,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                 num_key_value_heads=32,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                 pad_token_id=None,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                 pretraining_tp=1,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                 rms_norm_eps=1e-05,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                 rope_scaling=None,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                 rope_theta=10000.0,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                 tie_word_embeddings=True,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                 use_cache=True,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                 vocab_size=50257),
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                        init_method=RandomInit(std=0.025),
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                        dtype=torch.bfloat16,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                        make_vocab_size_divisible_by=1,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                        ddp_bucket_cap_mb=25),
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:        tokenizer=TokenizerArgs(tokenizer_name_or_path='openai-community/gpt2',
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                tokenizer_revision=None,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                tokenizer_max_length=None),
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:        checkpoints=CheckpointsArgs(checkpoints_path=Path('/dev/null'),
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                    checkpoint_interval=100000,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                    save_initial_state=False,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                    resume_checkpoint_path=None,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                    checkpoints_path_is_shared_file_system=False),
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:        logging=LoggingArgs(log_level='info',
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                            log_level_replica='info',
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                            iteration_step_info_interval=1),
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:        tokens=TokensArgs(sequence_length=4096,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                          train_steps=20,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                          micro_batch_size=2,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                          batch_accumulation_per_replica=256,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                          val_check_interval=-1,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                          limit_val_batches=0,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                          limit_test_batches=0),
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:        optimizer=OptimizerArgs(optimizer_factory=AdamWOptimizerArgs(adam_eps=1e-08,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                                     adam_beta1=0.9,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                                     adam_beta2=0.95,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                                     torch_adam_is_fused=True,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                                     name='adamW'),
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                zero_stage=1,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                weight_decay=0.01,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                clip_grad=1.0,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                accumulate_grad_in_fp32=True,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                learning_rate_scheduler=LRSchedulerArgs(learning_rate=0.0001,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                                        lr_warmup_steps=1,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                                        lr_warmup_style='linear',
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                                        lr_decay_style='linear',
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                                        lr_decay_steps=19,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                                        lr_decay_starting_step=None,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                                        min_decay_lr=1e-05)),
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:        data_stages=[DatasetStageArgs(name='Training Stage',
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                      start_training_step=1,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                      data=DataArgs(dataset=PretrainDatasetsArgs(hf_dataset_or_datasets='roneneldan/TinyStories',
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                                                 hf_dataset_splits='train',
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                                                 hf_dataset_config_name=None,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                                                 dataset_processing_num_proc_per_process=64,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                                                 dataset_overwrite_cache=False,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                                                 text_column_name='text'),
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                    seed=42,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:                                                    num_loading_workers=0))],
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:        profiler=ProfilerArgs(profiler_export_path=Path('/fsx/ferdinandmom/ferdinand-hf/bench_cluster/results/llama-1B/8_GPUS/dp-2_tp-1_pp-4_mbz-2')),
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:        lighteval=None)
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]: Model Config:
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]: LlamaConfig(bos_token_id=1,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:             eos_token_id=2,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:             hidden_act='silu',
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:             hidden_size=2048,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:             initializer_range=0.02,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:             intermediate_size=4096,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:             is_llama_config=True,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:             max_position_embeddings=4096,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:             num_attention_heads=32,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:             num_hidden_layers=24,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:             num_key_value_heads=32,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:             pad_token_id=None,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:             pretraining_tp=1,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:             rms_norm_eps=1e-05,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:             rope_scaling=None,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:             rope_theta=10000.0,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:             tie_word_embeddings=True,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:             use_cache=True,
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:             vocab_size=50257)
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]: Building model..
[default0]:07/03/2024 23:48:32 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]: Setting PP block ranks...
[default0]:07/03/2024 23:48:44 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]: Total number of parameters: 1.21G (2312.82MiB)
[default0]:07/03/2024 23:48:44 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]: Local number of parameters: 397M (756.37MiB)
[default0]:07/03/2024 23:48:44 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]: [After model building] Memory usage: 763.38MiB. Peak allocated: 765.41MiB Peak reserved: 792.00MiB
[default0]:07/03/2024 23:48:44 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]: No checkpoint path provided.
[default0]:07/03/2024 23:48:44 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]: Parametrizing model parameters using StandardParametrizator
[default2]:07/03/2024 23:48:44 [INFO|DP=0|PP=1|TP=0|ip-26-0-163-220]: Local number of parameters: 294M (560.05MiB)
[default2]:07/03/2024 23:48:44 [INFO|DP=0|PP=1|TP=0|ip-26-0-163-220]: [After model building] Memory usage: 567.07MiB. Peak allocated: 569.10MiB Peak reserved: 594.00MiB
[default2]:07/03/2024 23:48:44 [INFO|DP=0|PP=1|TP=0|ip-26-0-163-220]: No checkpoint path provided.
[default4]:07/03/2024 23:48:44 [INFO|DP=0|PP=2|TP=0|ip-26-0-163-220]: Local number of parameters: 252M (480.05MiB)
[default4]:07/03/2024 23:48:44 [INFO|DP=0|PP=2|TP=0|ip-26-0-163-220]: [After model building] Memory usage: 486.06MiB. Peak allocated: 488.09MiB Peak reserved: 502.00MiB
[default4]:07/03/2024 23:48:44 [INFO|DP=0|PP=2|TP=0|ip-26-0-163-220]: No checkpoint path provided.
[default6]:07/03/2024 23:48:44 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-220]: Local number of parameters: 271M (516.35MiB)
[default6]:07/03/2024 23:48:44 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-220]: [After model building] Memory usage: 520.36MiB. Peak allocated: 522.39MiB Peak reserved: 534.00MiB
[default6]:07/03/2024 23:48:44 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-220]: No checkpoint path provided.
[default7]:07/03/2024 23:48:44 [INFO|DP=1|PP=3|TP=0|ip-26-0-163-220]: No checkpoint path provided.
[default1]:07/03/2024 23:48:44 [INFO|DP=1|PP=0|TP=0|ip-26-0-163-220]: No checkpoint path provided.
[default5]:07/03/2024 23:48:44 [INFO|DP=1|PP=2|TP=0|ip-26-0-163-220]: No checkpoint path provided.
[default3]:07/03/2024 23:48:44 [INFO|DP=1|PP=1|TP=0|ip-26-0-163-220]: No checkpoint path provided.
[default0]:07/03/2024 23:48:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]: [Optimizer Building] Using LearningRateForSP as learning rate
[default0]:07/03/2024 23:48:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]: [ZeRO sharding] Size of optimizer params per rank:
[default0]:07/03/2024 23:48:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]: [ZeRO sharding] DP Rank 0 has 198M out of 397M (50.00%) params' optimizer states
[default0]:07/03/2024 23:48:47 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]: [ZeRO sharding] DP Rank 1 has 198M out of 397M (50.00%) params' optimizer states
[default0]:07/03/2024 23:48:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]: [Training Plan] Stage Training Stage has 19 remaining training steps and has consumed 0 samples
[default0]:07/03/2024 23:48:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]: Using `datasets` library
[default0]:07/03/2024 23:48:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]: Loading tokenizer from openai-community/gpt2 and transformers/hf_hub versions ('4.41.2', '0.23.4')
[default0]:Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/03/2024 23:48:48 [WARNING|DP=0|PP=0|TP=0|ip-26-0-163-220]: Repo card metadata block was not found. Setting CardData to empty.
[default0]:07/03/2024 23:48:49 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]: [Training Plan] There are 1 training stages 
[default0]:07/03/2024 23:48:49 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]: [Stage Training Stage] start from step 1 
[default0]:07/03/2024 23:48:49 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]: 
[default0]:07/03/2024 23:48:49 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]: [Start training] datetime: 2024-07-03 23:48:49.221044 | mbs: 2 | grad_accum: 256 | global_batch_size: 1024 | sequence_length: 4096 | train_steps: 20 | start_iteration_step: 0 | consumed_train_samples: 0
[default0]:07/03/2024 23:48:49 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]: Resuming training from stage Training Stage, it has trained for 0 samples and has 19 remaining train steps
[default0]:07/03/2024 23:48:49 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:  Memory usage: 3032.50MiB. Peak allocated 3032.50MiB. Peak reserved: 3064.00MiB
[default5]:07/03/2024 23:48:49 [WARNING|DP=1|PP=2|TP=0|ip-26-0-163-220]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:Repo card metadata block was not found. Setting CardData to empty.
[default4]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:07/03/2024 23:48:49 [WARNING|DP=1|PP=3|TP=0|ip-26-0-163-220]: Repo card metadata block was not found. Setting CardData to empty.
[default1]:07/03/2024 23:48:49 [WARNING|DP=1|PP=0|TP=0|ip-26-0-163-220]: Repo card metadata block was not found. Setting CardData to empty.
[default4]:07/03/2024 23:48:49 [WARNING|DP=0|PP=2|TP=0|ip-26-0-163-220]: Repo card metadata block was not found. Setting CardData to empty.
[default2]:07/03/2024 23:48:49 [WARNING|DP=0|PP=1|TP=0|ip-26-0-163-220]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:07/03/2024 23:48:49 [WARNING|DP=1|PP=1|TP=0|ip-26-0-163-220]: Repo card metadata block was not found. Setting CardData to empty.
[default6]:07/03/2024 23:48:49 [WARNING|DP=0|PP=3|TP=0|ip-26-0-163-220]: Repo card metadata block was not found. Setting CardData to empty.
[default3]:Repo card metadata block was not found. Setting CardData to empty.
[default1]:Repo card metadata block was not found. Setting CardData to empty.
[default2]:Repo card metadata block was not found. Setting CardData to empty.
[default5]:Repo card metadata block was not found. Setting CardData to empty.
[default7]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
[default7]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[default5]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
[default5]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[default6]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
[default6]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[default3]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
[default3]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[default4]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
[default4]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[default1]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
[default1]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[default2]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
[default2]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[default0]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: Attempting to run cuBLAS, but there was no current CUDA context! Attempting to set the primary context... (Triggered internally at ../aten/src/ATen/cuda/CublasHandlePool.cpp:135.)
[default0]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[default0]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ../torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)
[default0]:  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[default7]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
[default7]:  warnings.warn(
[default1]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
[default1]:  warnings.warn(
[default6]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
[default6]:  warnings.warn(
[default0]:/fsx/ferdinandmom/miniforge3/envs/env-bench-cluster/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2261: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
[default0]:  warnings.warn(
[default0]:07/03/2024 23:49:29 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:  Memory usage: 3100.07MiB. Peak allocated 20330.05MiB. Peak reserved: 20644.00MiB
[default0]:07/03/2024 23:49:29 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:  Memory usage: 4612.83MiB. Peak allocated 6503.76MiB. Peak reserved: 22740.00MiB
[default6]:07/03/2024 23:49:29 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-220]: iteration: 1 / 20 | consumed_tokens: 4.19M | elapsed_time_per_iteration_ms: 33.4K | tokens_per_sec: 126K | tokens_per_sec_per_gpu: 15.7K | global_batch_size: 1.02K | lm_loss: 11.1 | lr: 0.0001 | model_tflops_per_gpu: 142 | hardware_tflops_per_gpu: 142 | grad_norm: 25.1 | cuda_memory_allocated: 3.32G | cuda_max_memory_reserved: 9.02G | hd_total_memory_tb: 312G | hd_used_memory_tb: 65.6G | hd_free_memory_tb: 247G
[default0]:07/03/2024 23:49:52 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:  Memory usage: 4612.83MiB. Peak allocated 21842.82MiB. Peak reserved: 22742.00MiB
[default6]:07/03/2024 23:49:52 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-220]: iteration: 2 / 20 | consumed_tokens: 8.39M | elapsed_time_per_iteration_ms: 22.5K | tokens_per_sec: 187K | tokens_per_sec_per_gpu: 23.4K | global_batch_size: 1.02K | lm_loss: 11.1 | lr: 9.53e-05 | model_tflops_per_gpu: 212 | hardware_tflops_per_gpu: 212 | grad_norm: 25.2 | cuda_memory_allocated: 3.32G | cuda_max_memory_reserved: 10.7G | hd_total_memory_tb: 312G | hd_used_memory_tb: 65.6G | hd_free_memory_tb: 247G
[default0]:07/03/2024 23:49:52 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:  Memory usage: 4612.83MiB. Peak allocated 6503.76MiB. Peak reserved: 22742.00MiB
[default0]:07/03/2024 23:50:14 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:  Memory usage: 4612.83MiB. Peak allocated 21842.82MiB. Peak reserved: 22742.00MiB
[default0]:STAGE:2024-07-03 23:50:14 436335:436335 ActivityProfilerController.cpp:314] Completed Stage: Warm Up
[default6]:07/03/2024 23:50:14 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-220]: iteration: 3 / 20 | consumed_tokens: 12.6M | elapsed_time_per_iteration_ms: 22.5K | tokens_per_sec: 187K | tokens_per_sec_per_gpu: 23.3K | global_batch_size: 1.02K | lm_loss: 11.4 | lr: 9.05e-05 | model_tflops_per_gpu: 212 | hardware_tflops_per_gpu: 212 | grad_norm: 217 | cuda_memory_allocated: 3.32G | cuda_max_memory_reserved: 10.7G | hd_total_memory_tb: 312G | hd_used_memory_tb: 65.6G | hd_free_memory_tb: 247G
[default0]:07/03/2024 23:50:14 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:  Memory usage: 4612.83MiB. Peak allocated 6503.76MiB. Peak reserved: 22742.00MiB
[default6]:07/03/2024 23:50:37 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-220]: iteration: 4 / 20 | consumed_tokens: 16.8M | elapsed_time_per_iteration_ms: 22.5K | tokens_per_sec: 187K | tokens_per_sec_per_gpu: 23.3K | global_batch_size: 1.02K | lm_loss: 13.8 | lr: 8.58e-05 | model_tflops_per_gpu: 212 | hardware_tflops_per_gpu: 212 | grad_norm: 22.5 | cuda_memory_allocated: 3.32G | cuda_max_memory_reserved: 10.7G | hd_total_memory_tb: 312G | hd_used_memory_tb: 65.6G | hd_free_memory_tb: 247G
[default0]:07/03/2024 23:50:37 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:  Memory usage: 4612.83MiB. Peak allocated 21842.82MiB. Peak reserved: 22742.00MiB
[default0]:07/03/2024 23:50:37 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:  Memory usage: 4612.83MiB. Peak allocated 6503.76MiB. Peak reserved: 22742.00MiB
[default6]:07/03/2024 23:50:59 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-220]: iteration: 5 / 20 | consumed_tokens: 21M | elapsed_time_per_iteration_ms: 22.5K | tokens_per_sec: 186K | tokens_per_sec_per_gpu: 23.3K | global_batch_size: 1.02K | lm_loss: 9.98 | lr: 8.11e-05 | model_tflops_per_gpu: 211 | hardware_tflops_per_gpu: 211 | grad_norm: 16.4
[default0]:07/03/2024 23:50:59 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:  Memory usage: 4612.83MiB. Peak allocated 21842.82MiB. Peak reserved: 22742.00MiB
[default6]:07/03/2024 23:51:22 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-220]: iteration: 6 / 20 | consumed_tokens: 25.2M | elapsed_time_per_iteration_ms: 22.5K | tokens_per_sec: 187K | tokens_per_sec_per_gpu: 23.3K | global_batch_size: 1.02K | lm_loss: 10.9 | lr: 7.63e-05 | model_tflops_per_gpu: 212 | hardware_tflops_per_gpu: 212 | grad_norm: 93.9
[default0]:STAGE:2024-07-03 23:52:04 436335:436335 ActivityProfilerController.cpp:320] Completed Stage: Collection
[default0]:STAGE:2024-07-03 23:52:07 436335:436335 ActivityProfilerController.cpp:324] Completed Stage: Post Processing
[default0]:07/03/2024 23:57:04 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:  Memory usage: 4612.83MiB. Peak allocated 21842.82MiB. Peak reserved: 22742.00MiB
[default6]:07/03/2024 23:57:26 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-220]: iteration: 7 / 20 | consumed_tokens: 29.4M | elapsed_time_per_iteration_ms: 365K | tokens_per_sec: 11.5K | tokens_per_sec_per_gpu: 1.44K | global_batch_size: 1.02K | lm_loss: 9.16 | lr: 7.16e-05 | model_tflops_per_gpu: 13 | hardware_tflops_per_gpu: 13 | grad_norm: 19.8
[default0]:07/03/2024 23:57:26 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:  Memory usage: 4612.83MiB. Peak allocated 21842.82MiB. Peak reserved: 22742.00MiB
[default6]:07/03/2024 23:57:49 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-220]: iteration: 8 / 20 | consumed_tokens: 33.6M | elapsed_time_per_iteration_ms: 22.4K | tokens_per_sec: 188K | tokens_per_sec_per_gpu: 23.5K | global_batch_size: 1.02K | lm_loss: 8.83 | lr: 6.68e-05 | model_tflops_per_gpu: 213 | hardware_tflops_per_gpu: 213 | grad_norm: 6.08
[default0]:07/03/2024 23:57:49 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:  Memory usage: 4612.83MiB. Peak allocated 21842.82MiB. Peak reserved: 22742.00MiB
[default6]:07/03/2024 23:58:11 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-220]: iteration: 9 / 20 | consumed_tokens: 37.7M | elapsed_time_per_iteration_ms: 22.4K | tokens_per_sec: 188K | tokens_per_sec_per_gpu: 23.4K | global_batch_size: 1.02K | lm_loss: 8.47 | lr: 6.21e-05 | model_tflops_per_gpu: 213 | hardware_tflops_per_gpu: 213 | grad_norm: 5.23
[default0]:07/03/2024 23:58:11 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:  Memory usage: 4612.83MiB. Peak allocated 21842.82MiB. Peak reserved: 22742.00MiB
[default0]:07/03/2024 23:58:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:  Memory usage: 4612.83MiB. Peak allocated 21842.82MiB. Peak reserved: 22742.00MiB
[default6]:07/03/2024 23:58:33 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-220]: iteration: 10 / 20 | consumed_tokens: 41.9M | elapsed_time_per_iteration_ms: 22.4K | tokens_per_sec: 187K | tokens_per_sec_per_gpu: 23.4K | global_batch_size: 1.02K | lm_loss: 8.17 | lr: 5.74e-05 | model_tflops_per_gpu: 212 | hardware_tflops_per_gpu: 212 | grad_norm: 7.71
[default6]:07/03/2024 23:58:56 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-220]: iteration: 11 / 20 | consumed_tokens: 46.1M | elapsed_time_per_iteration_ms: 22.5K | tokens_per_sec: 187K | tokens_per_sec_per_gpu: 23.3K | global_batch_size: 1.02K | lm_loss: 7.93 | lr: 5.26e-05 | model_tflops_per_gpu: 212 | hardware_tflops_per_gpu: 212 | grad_norm: 5.54
[default0]:07/03/2024 23:58:56 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:  Memory usage: 4612.83MiB. Peak allocated 21842.82MiB. Peak reserved: 22742.00MiB
[default6]:07/03/2024 23:59:18 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-220]: iteration: 12 / 20 | consumed_tokens: 50.3M | elapsed_time_per_iteration_ms: 22.4K | tokens_per_sec: 187K | tokens_per_sec_per_gpu: 23.4K | global_batch_size: 1.02K | lm_loss: 7.75 | lr: 4.79e-05 | model_tflops_per_gpu: 212 | hardware_tflops_per_gpu: 212 | grad_norm: 4.64
[default0]:07/03/2024 23:59:18 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:  Memory usage: 4612.83MiB. Peak allocated 21842.82MiB. Peak reserved: 22742.00MiB
[default6]:07/03/2024 23:59:41 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-220]: iteration: 13 / 20 | consumed_tokens: 54.5M | elapsed_time_per_iteration_ms: 22.4K | tokens_per_sec: 187K | tokens_per_sec_per_gpu: 23.4K | global_batch_size: 1.02K | lm_loss: 7.58 | lr: 4.32e-05 | model_tflops_per_gpu: 212 | hardware_tflops_per_gpu: 212 | grad_norm: 2.9
[default0]:07/03/2024 23:59:41 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:  Memory usage: 4612.83MiB. Peak allocated 21842.82MiB. Peak reserved: 22742.00MiB
[default6]:07/04/2024 00:00:03 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-220]: iteration: 14 / 20 | consumed_tokens: 58.7M | elapsed_time_per_iteration_ms: 22.5K | tokens_per_sec: 187K | tokens_per_sec_per_gpu: 23.4K | global_batch_size: 1.02K | lm_loss: 7.5 | lr: 3.84e-05 | model_tflops_per_gpu: 212 | hardware_tflops_per_gpu: 212 | grad_norm: 4.18
[default0]:07/04/2024 00:00:03 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:  Memory usage: 4612.83MiB. Peak allocated 21842.82MiB. Peak reserved: 22742.00MiB
[default6]:07/04/2024 00:00:26 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-220]: iteration: 15 / 20 | consumed_tokens: 62.9M | elapsed_time_per_iteration_ms: 22.6K | tokens_per_sec: 186K | tokens_per_sec_per_gpu: 23.2K | global_batch_size: 1.02K | lm_loss: 7.4 | lr: 3.37e-05 | model_tflops_per_gpu: 211 | hardware_tflops_per_gpu: 211 | grad_norm: 3.86
[default0]:07/04/2024 00:00:26 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:  Memory usage: 4612.83MiB. Peak allocated 21842.82MiB. Peak reserved: 22742.00MiB
[default6]:07/04/2024 00:00:48 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-220]: iteration: 16 / 20 | consumed_tokens: 67.1M | elapsed_time_per_iteration_ms: 22.4K | tokens_per_sec: 187K | tokens_per_sec_per_gpu: 23.4K | global_batch_size: 1.02K | lm_loss: 7.29 | lr: 2.89e-05 | model_tflops_per_gpu: 212 | hardware_tflops_per_gpu: 212 | grad_norm: 3.07
[default0]:07/04/2024 00:00:48 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:  Memory usage: 4612.83MiB. Peak allocated 21842.82MiB. Peak reserved: 22742.00MiB
[default6]:07/04/2024 00:01:10 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-220]: iteration: 17 / 20 | consumed_tokens: 71.3M | elapsed_time_per_iteration_ms: 22.4K | tokens_per_sec: 188K | tokens_per_sec_per_gpu: 23.4K | global_batch_size: 1.02K | lm_loss: 7.19 | lr: 2.42e-05 | model_tflops_per_gpu: 213 | hardware_tflops_per_gpu: 213 | grad_norm: 2.39
[default0]:07/04/2024 00:01:10 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:  Memory usage: 4612.83MiB. Peak allocated 21842.82MiB. Peak reserved: 22742.00MiB
[default0]:07/04/2024 00:01:33 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:  Memory usage: 4612.83MiB. Peak allocated 21842.82MiB. Peak reserved: 22742.00MiB
[default6]:07/04/2024 00:01:33 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-220]: iteration: 18 / 20 | consumed_tokens: 75.5M | elapsed_time_per_iteration_ms: 22.4K | tokens_per_sec: 187K | tokens_per_sec_per_gpu: 23.4K | global_batch_size: 1.02K | lm_loss: 7.13 | lr: 1.95e-05 | model_tflops_per_gpu: 212 | hardware_tflops_per_gpu: 212 | grad_norm: 2.2
[default0]:07/04/2024 00:01:55 [INFO|DP=0|PP=0|TP=0|ip-26-0-163-220]:  Memory usage: 4612.83MiB. Peak allocated 21842.82MiB. Peak reserved: 22742.00MiB
[default6]:07/04/2024 00:01:55 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-220]: iteration: 19 / 20 | consumed_tokens: 79.7M | elapsed_time_per_iteration_ms: 22.4K | tokens_per_sec: 187K | tokens_per_sec_per_gpu: 23.4K | global_batch_size: 1.02K | lm_loss: 7.08 | lr: 1.47e-05 | model_tflops_per_gpu: 212 | hardware_tflops_per_gpu: 212 | grad_norm: 2.64
[default6]:07/04/2024 00:02:18 [INFO|DP=0|PP=3|TP=0|ip-26-0-163-220]: iteration: 20 / 20 | consumed_tokens: 83.9M | elapsed_time_per_iteration_ms: 22.6K | tokens_per_sec: 186K | tokens_per_sec_per_gpu: 23.2K | global_batch_size: 1.02K | lm_loss: 7.03 | lr: 1e-05 | model_tflops_per_gpu: 211 | hardware_tflops_per_gpu: 211 | grad_norm: 2.3
Saved 1 csv files over 1 completed logs
Processing file: /fsx/ferdinandmom/ferdinand-hf/bench_cluster/results/llama-1B/8_GPUS/dp-2_tp-1_pp-4_mbz-2/profiler/ip-26-0-163-220_436335.1720050960368134686.pt.trace.json
Results written to /fsx/ferdinandmom/ferdinand-hf/bench_cluster/results/llama-1B/8_GPUS/dp-2_tp-1_pp-4_mbz-2/profiler.csv
Consider using `hf_transfer` for faster uploads. This solution comes with some limitations. See https://huggingface.co/docs/huggingface_hub/hf_transfer for more details.

ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   0%|          | 0.00/10.3G [00:00<?, ?B/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   0%|          | 16.0M/10.3G [00:00<03:26, 49.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   0%|          | 32.0M/10.3G [00:00<03:09, 54.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   0%|          | 48.0M/10.3G [00:01<04:32, 37.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   1%|          | 64.0M/10.3G [00:01<03:46, 45.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   1%|          | 80.0M/10.3G [00:01<03:28, 49.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   1%|          | 96.0M/10.3G [00:01<03:20, 50.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   1%|          | 112M/10.3G [00:02<03:36, 46.9MB/s] 
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   1%|          | 128M/10.3G [00:03<06:34, 25.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   1%|▏         | 144M/10.3G [00:03<05:31, 30.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   2%|▏         | 160M/10.3G [00:04<04:28, 37.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   2%|▏         | 176M/10.3G [00:04<03:55, 43.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   2%|▏         | 192M/10.3G [00:04<03:27, 48.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   2%|▏         | 208M/10.3G [00:04<02:59, 56.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   2%|▏         | 224M/10.3G [00:05<02:57, 56.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   2%|▏         | 240M/10.3G [00:05<02:39, 62.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   2%|▏         | 256M/10.3G [00:05<02:57, 56.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   3%|▎         | 272M/10.3G [00:05<03:06, 53.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   3%|▎         | 288M/10.3G [00:06<03:29, 47.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   3%|▎         | 304M/10.3G [00:06<03:28, 47.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   3%|▎         | 320M/10.3G [00:06<03:05, 53.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   3%|▎         | 336M/10.3G [00:07<03:21, 49.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   3%|▎         | 352M/10.3G [00:07<03:07, 53.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   4%|▎         | 368M/10.3G [00:07<03:01, 54.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   4%|▎         | 384M/10.3G [00:08<02:55, 56.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   4%|▍         | 400M/10.3G [00:08<02:33, 64.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   4%|▍         | 416M/10.3G [00:08<02:40, 61.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   4%|▍         | 432M/10.3G [00:08<02:34, 63.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   4%|▍         | 448M/10.3G [00:08<02:29, 65.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   5%|▍         | 464M/10.3G [00:09<03:09, 51.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   5%|▍         | 480M/10.3G [00:09<02:54, 56.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   5%|▍         | 496M/10.3G [00:09<02:47, 58.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   5%|▍         | 512M/10.3G [00:10<02:44, 59.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   5%|▌         | 528M/10.3G [00:10<02:47, 58.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   5%|▌         | 544M/10.3G [00:10<02:29, 64.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   5%|▌         | 560M/10.3G [00:10<02:38, 61.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   6%|▌         | 576M/10.3G [00:11<02:27, 65.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   6%|▌         | 592M/10.3G [00:11<02:18, 70.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   6%|▌         | 608M/10.3G [00:11<02:29, 64.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   6%|▌         | 624M/10.3G [00:11<02:33, 62.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   6%|▌         | 640M/10.3G [00:12<02:41, 59.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   6%|▋         | 656M/10.3G [00:12<02:44, 58.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   7%|▋         | 672M/10.3G [00:12<02:49, 56.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   7%|▋         | 688M/10.3G [00:13<02:55, 54.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   7%|▋         | 704M/10.3G [00:13<02:48, 56.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   7%|▋         | 720M/10.3G [00:13<02:53, 55.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   7%|▋         | 736M/10.3G [00:13<02:59, 53.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   7%|▋         | 752M/10.3G [00:14<03:17, 48.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   7%|▋         | 768M/10.3G [00:14<03:12, 49.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   8%|▊         | 784M/10.3G [00:15<03:20, 47.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   8%|▊         | 797M/10.3G [00:15<02:58, 53.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   8%|▊         | 803M/10.3G [00:15<04:24, 35.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   8%|▊         | 816M/10.3G [00:16<04:14, 37.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   8%|▊         | 832M/10.3G [00:16<03:42, 42.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   8%|▊         | 848M/10.3G [00:16<03:12, 49.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   8%|▊         | 864M/10.3G [00:16<03:21, 46.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   9%|▊         | 880M/10.3G [00:17<03:03, 51.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   9%|▊         | 896M/10.3G [00:17<03:11, 49.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   9%|▉         | 912M/10.3G [00:18<04:12, 37.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   9%|▉         | 928M/10.3G [00:18<03:37, 42.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   9%|▉         | 944M/10.3G [00:18<03:13, 48.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   9%|▉         | 960M/10.3G [00:19<03:36, 43.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:   9%|▉         | 976M/10.3G [00:19<03:14, 47.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  10%|▉         | 992M/10.3G [00:19<03:13, 48.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  10%|▉         | 1.01G/10.3G [00:20<03:09, 48.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  10%|▉         | 1.02G/10.3G [00:20<03:03, 50.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  10%|█         | 1.04G/10.3G [00:20<03:04, 49.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  10%|█         | 1.06G/10.3G [00:21<03:22, 45.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  10%|█         | 1.07G/10.3G [00:21<03:01, 50.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  11%|█         | 1.09G/10.3G [00:21<03:11, 48.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  11%|█         | 1.10G/10.3G [00:21<02:53, 52.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  11%|█         | 1.12G/10.3G [00:22<03:14, 47.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  11%|█         | 1.13G/10.3G [00:22<02:52, 53.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  11%|█         | 1.14G/10.3G [00:22<03:36, 42.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  11%|█         | 1.15G/10.3G [00:23<03:15, 46.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  11%|█▏        | 1.17G/10.3G [00:23<03:16, 46.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  12%|█▏        | 1.18G/10.3G [00:23<03:08, 48.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  12%|█▏        | 1.20G/10.3G [00:23<02:58, 50.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  12%|█▏        | 1.22G/10.3G [00:24<03:21, 45.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  12%|█▏        | 1.22G/10.3G [00:24<03:16, 46.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  12%|█▏        | 1.23G/10.3G [00:24<03:35, 42.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  12%|█▏        | 1.23G/10.3G [00:24<04:09, 36.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  12%|█▏        | 1.24G/10.3G [00:25<07:51, 19.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  12%|█▏        | 1.25G/10.3G [00:26<08:38, 17.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  12%|█▏        | 1.26G/10.3G [00:26<06:41, 22.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  12%|█▏        | 1.26G/10.3G [00:26<05:41, 26.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  12%|█▏        | 1.27G/10.3G [00:26<06:50, 22.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  12%|█▏        | 1.28G/10.3G [00:27<06:01, 24.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  13%|█▎        | 1.30G/10.3G [00:27<03:53, 38.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  13%|█▎        | 1.30G/10.3G [00:27<05:26, 27.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  13%|█▎        | 1.31G/10.3G [00:28<04:53, 30.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  13%|█▎        | 1.31G/10.3G [00:28<05:53, 25.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  13%|█▎        | 1.33G/10.3G [00:28<04:16, 34.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  13%|█▎        | 1.34G/10.3G [00:28<03:14, 45.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  13%|█▎        | 1.36G/10.3G [00:29<03:08, 47.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  13%|█▎        | 1.38G/10.3G [00:29<03:12, 46.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  14%|█▎        | 1.39G/10.3G [00:29<03:13, 46.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  14%|█▎        | 1.41G/10.3G [00:30<02:56, 50.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  14%|█▍        | 1.42G/10.3G [00:30<02:50, 51.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  14%|█▍        | 1.44G/10.3G [00:30<03:07, 47.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  14%|█▍        | 1.46G/10.3G [00:31<02:59, 49.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  14%|█▍        | 1.47G/10.3G [00:31<03:00, 48.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  14%|█▍        | 1.49G/10.3G [00:31<02:42, 54.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  15%|█▍        | 1.50G/10.3G [00:32<03:11, 45.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  15%|█▍        | 1.52G/10.3G [00:32<02:58, 49.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  15%|█▍        | 1.54G/10.3G [00:32<03:07, 46.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  15%|█▌        | 1.55G/10.3G [00:33<03:08, 46.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  15%|█▌        | 1.57G/10.3G [00:33<03:11, 45.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  15%|█▌        | 1.58G/10.3G [00:33<02:53, 50.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  16%|█▌        | 1.60G/10.3G [00:34<02:40, 54.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  16%|█▌        | 1.62G/10.3G [00:34<02:41, 53.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  16%|█▌        | 1.63G/10.3G [00:34<02:31, 57.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  16%|█▌        | 1.65G/10.3G [00:34<02:38, 54.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  16%|█▌        | 1.66G/10.3G [00:35<02:36, 55.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  16%|█▋        | 1.68G/10.3G [00:35<02:39, 53.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  17%|█▋        | 1.70G/10.3G [00:35<02:31, 56.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  17%|█▋        | 1.71G/10.3G [00:36<02:29, 57.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  17%|█▋        | 1.73G/10.3G [00:36<02:27, 57.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  17%|█▋        | 1.74G/10.3G [00:36<02:22, 59.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  17%|█▋        | 1.76G/10.3G [00:37<03:11, 44.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  17%|█▋        | 1.78G/10.3G [00:37<03:12, 44.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  17%|█▋        | 1.79G/10.3G [00:37<03:05, 45.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  18%|█▊        | 1.81G/10.3G [00:38<03:00, 47.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  18%|█▊        | 1.82G/10.3G [00:38<03:01, 46.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  18%|█▊        | 1.84G/10.3G [00:38<02:54, 48.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  18%|█▊        | 1.86G/10.3G [00:39<02:37, 53.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  18%|█▊        | 1.87G/10.3G [00:39<02:41, 52.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  18%|█▊        | 1.89G/10.3G [00:39<02:30, 55.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  19%|█▊        | 1.90G/10.3G [00:39<02:24, 58.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  19%|█▊        | 1.92G/10.3G [00:40<02:38, 52.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  19%|█▉        | 1.94G/10.3G [00:43<10:45, 12.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  19%|█▉        | 1.95G/10.3G [00:43<08:09, 17.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  19%|█▉        | 1.97G/10.3G [00:44<06:39, 20.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  19%|█▉        | 1.98G/10.3G [00:44<05:21, 25.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  19%|█▉        | 2.00G/10.3G [00:44<04:36, 29.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  20%|█▉        | 2.02G/10.3G [00:45<04:00, 34.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  20%|█▉        | 2.03G/10.3G [00:45<03:44, 36.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  20%|█▉        | 2.05G/10.3G [00:45<03:27, 39.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  20%|██        | 2.06G/10.3G [00:46<03:02, 45.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  20%|██        | 2.08G/10.3G [00:46<02:55, 46.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  20%|██        | 2.10G/10.3G [00:46<02:54, 47.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  21%|██        | 2.11G/10.3G [00:47<02:44, 49.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  21%|██        | 2.13G/10.3G [00:47<02:41, 50.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  21%|██        | 2.14G/10.3G [00:47<02:31, 53.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  21%|██        | 2.16G/10.3G [00:47<02:35, 52.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  21%|██        | 2.18G/10.3G [00:48<02:17, 59.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  21%|██▏       | 2.19G/10.3G [00:48<02:29, 54.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  21%|██▏       | 2.21G/10.3G [00:48<02:35, 51.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  22%|██▏       | 2.22G/10.3G [00:49<02:28, 54.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  22%|██▏       | 2.24G/10.3G [00:49<02:26, 54.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  22%|██▏       | 2.26G/10.3G [00:49<02:39, 50.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  22%|██▏       | 2.27G/10.3G [00:50<02:42, 49.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  22%|██▏       | 2.29G/10.3G [00:50<02:41, 49.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  22%|██▏       | 2.30G/10.3G [00:50<02:29, 53.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  23%|██▎       | 2.32G/10.3G [00:50<02:28, 53.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  23%|██▎       | 2.34G/10.3G [00:51<02:37, 50.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  23%|██▎       | 2.35G/10.3G [00:51<02:25, 54.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  23%|██▎       | 2.37G/10.3G [00:51<02:21, 56.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  23%|██▎       | 2.38G/10.3G [00:52<03:18, 39.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  23%|██▎       | 2.40G/10.3G [00:52<02:48, 46.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  24%|██▎       | 2.42G/10.3G [00:53<02:53, 45.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  24%|██▎       | 2.43G/10.3G [00:53<02:39, 49.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  24%|██▍       | 2.45G/10.3G [00:53<02:39, 49.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  24%|██▍       | 2.46G/10.3G [00:53<02:19, 55.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  24%|██▍       | 2.48G/10.3G [00:54<02:07, 61.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  24%|██▍       | 2.50G/10.3G [00:54<02:14, 57.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  24%|██▍       | 2.51G/10.3G [00:54<02:24, 53.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  25%|██▍       | 2.53G/10.3G [00:54<02:18, 55.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  25%|██▍       | 2.54G/10.3G [00:55<02:09, 59.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  25%|██▍       | 2.56G/10.3G [00:55<02:11, 58.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  25%|██▌       | 2.58G/10.3G [00:55<02:28, 51.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  25%|██▌       | 2.59G/10.3G [00:56<02:17, 55.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  25%|██▌       | 2.61G/10.3G [00:56<02:05, 61.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  26%|██▌       | 2.62G/10.3G [00:56<02:05, 61.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  26%|██▌       | 2.64G/10.3G [00:56<02:17, 55.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  26%|██▌       | 2.66G/10.3G [00:57<02:16, 55.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  26%|██▌       | 2.67G/10.3G [00:57<02:16, 55.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  26%|██▌       | 2.69G/10.3G [00:57<02:23, 52.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  26%|██▋       | 2.70G/10.3G [00:58<02:18, 54.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  26%|██▋       | 2.72G/10.3G [00:58<02:08, 58.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  27%|██▋       | 2.74G/10.3G [00:58<02:04, 60.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  27%|██▋       | 2.75G/10.3G [00:58<02:02, 61.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  27%|██▋       | 2.77G/10.3G [00:59<02:05, 59.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  27%|██▋       | 2.78G/10.3G [00:59<02:02, 61.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  27%|██▋       | 2.80G/10.3G [00:59<01:53, 66.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  27%|██▋       | 2.82G/10.3G [00:59<01:48, 68.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  28%|██▊       | 2.83G/10.3G [01:00<02:01, 61.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  28%|██▊       | 2.85G/10.3G [01:00<01:57, 63.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  28%|██▊       | 2.86G/10.3G [01:00<02:04, 59.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  28%|██▊       | 2.88G/10.3G [01:00<02:10, 56.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  28%|██▊       | 2.90G/10.3G [01:01<02:00, 61.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  28%|██▊       | 2.91G/10.3G [01:01<01:59, 61.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  28%|██▊       | 2.93G/10.3G [01:01<02:05, 58.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  29%|██▊       | 2.94G/10.3G [01:02<02:16, 53.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  29%|██▉       | 2.96G/10.3G [01:02<02:06, 57.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  29%|██▉       | 2.98G/10.3G [01:02<02:03, 59.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  29%|██▉       | 2.99G/10.3G [01:02<02:08, 56.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  29%|██▉       | 3.01G/10.3G [01:03<02:11, 55.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  29%|██▉       | 3.02G/10.3G [01:03<02:08, 56.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  30%|██▉       | 3.04G/10.3G [01:03<02:05, 57.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  30%|██▉       | 3.06G/10.3G [01:03<02:02, 59.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  30%|██▉       | 3.07G/10.3G [01:04<01:49, 65.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  30%|███       | 3.09G/10.3G [01:04<02:12, 54.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  30%|███       | 3.10G/10.3G [01:04<02:28, 48.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  30%|███       | 3.12G/10.3G [01:05<02:34, 46.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  31%|███       | 3.14G/10.3G [01:05<02:22, 50.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  31%|███       | 3.15G/10.3G [01:05<02:21, 50.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  31%|███       | 3.17G/10.3G [01:06<02:06, 56.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  31%|███       | 3.18G/10.3G [01:06<01:59, 59.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  31%|███       | 3.20G/10.3G [01:06<01:53, 62.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  31%|███▏      | 3.22G/10.3G [01:06<01:58, 59.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  31%|███▏      | 3.23G/10.3G [01:07<01:59, 58.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  32%|███▏      | 3.25G/10.3G [01:07<01:53, 62.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  32%|███▏      | 3.26G/10.3G [01:07<02:00, 58.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  32%|███▏      | 3.28G/10.3G [01:07<02:08, 54.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  32%|███▏      | 3.30G/10.3G [01:08<02:00, 57.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  32%|███▏      | 3.31G/10.3G [01:08<02:01, 57.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  32%|███▏      | 3.33G/10.3G [01:08<02:01, 57.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  33%|███▎      | 3.34G/10.3G [01:09<02:12, 52.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  33%|███▎      | 3.36G/10.3G [01:09<02:08, 53.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  33%|███▎      | 3.38G/10.3G [01:09<02:21, 48.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  33%|███▎      | 3.39G/10.3G [01:10<02:20, 49.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  33%|███▎      | 3.41G/10.3G [01:10<02:18, 49.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  33%|███▎      | 3.42G/10.3G [01:10<02:19, 49.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  33%|███▎      | 3.44G/10.3G [01:11<02:25, 47.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  34%|███▎      | 3.46G/10.3G [01:11<02:25, 46.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  34%|███▍      | 3.47G/10.3G [01:11<02:19, 49.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  34%|███▍      | 3.49G/10.3G [01:12<02:23, 47.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  34%|███▍      | 3.50G/10.3G [01:12<02:11, 51.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  34%|███▍      | 3.52G/10.3G [01:12<02:17, 49.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  34%|███▍      | 3.53G/10.3G [01:12<01:54, 58.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  34%|███▍      | 3.54G/10.3G [01:13<02:25, 46.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  35%|███▍      | 3.55G/10.3G [01:13<02:36, 43.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  35%|███▍      | 3.57G/10.3G [01:13<02:16, 49.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  35%|███▍      | 3.58G/10.3G [01:14<02:11, 50.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  35%|███▌      | 3.60G/10.3G [01:14<02:14, 49.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  35%|███▌      | 3.62G/10.3G [01:14<02:14, 49.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  35%|███▌      | 3.63G/10.3G [01:15<02:13, 49.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  35%|███▌      | 3.65G/10.3G [01:15<02:04, 53.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  36%|███▌      | 3.66G/10.3G [01:15<02:01, 54.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  36%|███▌      | 3.68G/10.3G [01:15<01:59, 55.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  36%|███▌      | 3.70G/10.3G [01:16<01:59, 55.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  36%|███▌      | 3.71G/10.3G [01:16<01:50, 59.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  36%|███▋      | 3.73G/10.3G [01:16<01:42, 63.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  36%|███▋      | 3.74G/10.3G [01:16<01:48, 60.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  37%|███▋      | 3.76G/10.3G [01:17<01:49, 59.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  37%|███▋      | 3.78G/10.3G [01:17<01:58, 54.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  37%|███▋      | 3.79G/10.3G [01:18<02:30, 43.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  37%|███▋      | 3.81G/10.3G [01:18<02:27, 43.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  37%|███▋      | 3.82G/10.3G [01:18<02:14, 48.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  37%|███▋      | 3.84G/10.3G [01:18<02:07, 50.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  38%|███▊      | 3.86G/10.3G [01:19<01:57, 54.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  38%|███▊      | 3.87G/10.3G [01:19<02:00, 53.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  38%|███▊      | 3.89G/10.3G [01:19<02:04, 51.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  38%|███▊      | 3.90G/10.3G [01:20<01:54, 55.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  38%|███▊      | 3.92G/10.3G [01:20<01:55, 55.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  38%|███▊      | 3.94G/10.3G [01:20<01:51, 56.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  38%|███▊      | 3.95G/10.3G [01:20<01:57, 53.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  39%|███▊      | 3.97G/10.3G [01:21<02:01, 51.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  39%|███▉      | 3.98G/10.3G [01:21<01:56, 54.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  39%|███▉      | 4.00G/10.3G [01:21<01:53, 55.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  39%|███▉      | 4.02G/10.3G [01:22<01:42, 60.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  39%|███▉      | 4.03G/10.3G [01:22<01:44, 59.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  39%|███▉      | 4.05G/10.3G [01:22<01:33, 66.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  40%|███▉      | 4.06G/10.3G [01:22<02:01, 51.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  40%|███▉      | 4.08G/10.3G [01:23<01:57, 52.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  40%|███▉      | 4.10G/10.3G [01:23<01:52, 55.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  40%|████      | 4.11G/10.3G [01:23<01:57, 52.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  40%|████      | 4.13G/10.3G [01:24<01:57, 52.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  40%|████      | 4.14G/10.3G [01:24<01:57, 52.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  40%|████      | 4.16G/10.3G [01:24<01:49, 55.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  41%|████      | 4.18G/10.3G [01:25<01:56, 52.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  41%|████      | 4.19G/10.3G [01:25<01:49, 55.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  41%|████      | 4.21G/10.3G [01:26<03:32, 28.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  41%|████      | 4.22G/10.3G [01:26<03:00, 33.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  41%|████▏     | 4.24G/10.3G [01:27<02:39, 37.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  41%|████▏     | 4.26G/10.3G [01:27<02:18, 43.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  42%|████▏     | 4.27G/10.3G [01:27<02:14, 44.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  42%|████▏     | 4.29G/10.3G [01:27<02:04, 48.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  42%|████▏     | 4.30G/10.3G [01:28<01:54, 52.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  42%|████▏     | 4.32G/10.3G [01:28<01:59, 50.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  42%|████▏     | 4.34G/10.3G [01:28<01:48, 54.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  42%|████▏     | 4.35G/10.3G [01:29<01:43, 57.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  42%|████▏     | 4.37G/10.3G [01:29<01:57, 50.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  43%|████▎     | 4.38G/10.3G [01:29<01:46, 55.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  43%|████▎     | 4.40G/10.3G [01:29<01:36, 60.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  43%|████▎     | 4.42G/10.3G [01:30<01:34, 62.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  43%|████▎     | 4.43G/10.3G [01:30<01:32, 63.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  43%|████▎     | 4.45G/10.3G [01:30<01:25, 68.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  43%|████▎     | 4.46G/10.3G [01:30<01:30, 64.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  44%|████▎     | 4.48G/10.3G [01:30<01:23, 69.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  44%|████▎     | 4.50G/10.3G [01:31<01:30, 64.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  44%|████▍     | 4.51G/10.3G [01:31<01:30, 63.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  44%|████▍     | 4.53G/10.3G [01:31<01:33, 61.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  44%|████▍     | 4.54G/10.3G [01:32<01:30, 63.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  44%|████▍     | 4.56G/10.3G [01:32<01:26, 65.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  45%|████▍     | 4.58G/10.3G [01:32<01:41, 56.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  45%|████▍     | 4.59G/10.3G [01:32<01:40, 56.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  45%|████▍     | 4.61G/10.3G [01:33<01:38, 57.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  45%|████▍     | 4.62G/10.3G [01:33<01:36, 58.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  45%|████▌     | 4.64G/10.3G [01:33<01:37, 57.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  45%|████▌     | 4.66G/10.3G [01:34<01:49, 51.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  45%|████▌     | 4.67G/10.3G [01:34<01:46, 52.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  46%|████▌     | 4.69G/10.3G [01:34<01:42, 54.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  46%|████▌     | 4.70G/10.3G [01:34<01:43, 54.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  46%|████▌     | 4.72G/10.3G [01:35<01:38, 56.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  46%|████▌     | 4.74G/10.3G [01:35<01:37, 56.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  46%|████▌     | 4.75G/10.3G [01:35<01:36, 57.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  46%|████▋     | 4.77G/10.3G [01:36<01:36, 57.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  47%|████▋     | 4.78G/10.3G [01:36<02:15, 40.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  47%|████▋     | 4.80G/10.3G [01:37<02:06, 43.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  47%|████▋     | 4.82G/10.3G [01:37<01:50, 49.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  47%|████▋     | 4.83G/10.3G [01:37<01:41, 53.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  47%|████▋     | 4.85G/10.3G [01:37<01:40, 54.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  47%|████▋     | 4.86G/10.3G [01:38<01:38, 55.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  47%|████▋     | 4.88G/10.3G [01:38<01:44, 51.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  48%|████▊     | 4.90G/10.3G [01:38<01:39, 54.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  48%|████▊     | 4.91G/10.3G [01:38<01:33, 57.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  48%|████▊     | 4.93G/10.3G [01:39<01:39, 53.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  48%|████▊     | 4.94G/10.3G [01:39<02:06, 42.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  48%|████▊     | 4.96G/10.3G [01:40<01:58, 44.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  48%|████▊     | 4.98G/10.3G [01:40<01:55, 45.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  49%|████▊     | 4.99G/10.3G [01:40<01:44, 50.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  49%|████▊     | 5.01G/10.3G [01:41<02:09, 40.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  49%|████▉     | 5.02G/10.3G [01:41<01:54, 45.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  49%|████▉     | 5.04G/10.3G [01:41<01:42, 51.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  49%|████▉     | 5.06G/10.3G [01:41<01:32, 56.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  49%|████▉     | 5.07G/10.3G [01:42<01:33, 55.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  50%|████▉     | 5.09G/10.3G [01:42<01:34, 54.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  50%|████▉     | 5.10G/10.3G [01:42<01:33, 55.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  50%|████▉     | 5.12G/10.3G [01:43<01:27, 59.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  50%|████▉     | 5.14G/10.3G [01:43<01:35, 53.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  50%|█████     | 5.15G/10.3G [01:43<01:49, 46.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  50%|█████     | 5.17G/10.3G [01:44<02:06, 40.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  50%|█████     | 5.18G/10.3G [01:44<01:52, 45.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  51%|█████     | 5.20G/10.3G [01:44<01:42, 49.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  51%|█████     | 5.22G/10.3G [01:45<01:38, 51.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  51%|█████     | 5.23G/10.3G [01:45<01:36, 52.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  51%|█████     | 5.25G/10.3G [01:45<01:35, 52.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  51%|█████     | 5.26G/10.3G [01:46<01:29, 56.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  51%|█████▏    | 5.28G/10.3G [01:46<01:27, 57.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  52%|█████▏    | 5.30G/10.3G [01:46<01:26, 57.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  52%|█████▏    | 5.31G/10.3G [01:46<01:33, 52.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  52%|█████▏    | 5.33G/10.3G [01:47<01:31, 54.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  52%|█████▏    | 5.34G/10.3G [01:47<01:27, 56.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  52%|█████▏    | 5.36G/10.3G [01:47<01:21, 60.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  52%|█████▏    | 5.38G/10.3G [01:47<01:18, 62.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  52%|█████▏    | 5.39G/10.3G [01:48<01:19, 61.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  53%|█████▎    | 5.41G/10.3G [01:48<01:26, 56.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  53%|█████▎    | 5.42G/10.3G [01:48<01:21, 59.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  53%|█████▎    | 5.44G/10.3G [01:49<01:28, 55.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  53%|█████▎    | 5.46G/10.3G [01:49<01:22, 58.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  53%|█████▎    | 5.47G/10.3G [01:49<01:14, 64.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  53%|█████▎    | 5.49G/10.3G [01:50<01:51, 43.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  54%|█████▎    | 5.50G/10.3G [01:50<01:47, 44.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  54%|█████▎    | 5.52G/10.3G [01:50<01:41, 46.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  54%|█████▍    | 5.54G/10.3G [01:51<01:26, 54.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  54%|█████▍    | 5.55G/10.3G [01:51<01:23, 56.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  54%|█████▍    | 5.57G/10.3G [01:51<01:16, 61.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  54%|█████▍    | 5.58G/10.3G [01:51<01:14, 63.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  54%|█████▍    | 5.60G/10.3G [01:51<01:11, 65.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  55%|█████▍    | 5.62G/10.3G [01:52<01:13, 63.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  55%|█████▍    | 5.63G/10.3G [01:52<01:18, 59.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  55%|█████▍    | 5.65G/10.3G [01:52<01:21, 56.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  55%|█████▌    | 5.66G/10.3G [01:53<01:21, 56.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  55%|█████▌    | 5.68G/10.3G [01:53<01:28, 52.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  55%|█████▌    | 5.70G/10.3G [01:53<01:23, 55.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  56%|█████▌    | 5.71G/10.3G [01:53<01:19, 57.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  56%|█████▌    | 5.73G/10.3G [01:54<01:14, 60.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  56%|█████▌    | 5.74G/10.3G [01:54<01:14, 60.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  56%|█████▌    | 5.76G/10.3G [01:54<01:16, 59.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  56%|█████▌    | 5.78G/10.3G [01:54<01:08, 65.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  56%|█████▋    | 5.79G/10.3G [01:55<01:12, 61.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  57%|█████▋    | 5.81G/10.3G [01:55<01:09, 64.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  57%|█████▋    | 5.82G/10.3G [01:56<01:34, 47.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  57%|█████▋    | 5.84G/10.3G [01:56<01:31, 48.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  57%|█████▋    | 5.86G/10.3G [01:56<01:28, 49.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  57%|█████▋    | 5.87G/10.3G [01:56<01:29, 49.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  57%|█████▋    | 5.89G/10.3G [01:57<01:20, 54.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  57%|█████▋    | 5.90G/10.3G [01:57<01:18, 55.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  58%|█████▊    | 5.92G/10.3G [01:57<01:21, 53.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  58%|█████▊    | 5.94G/10.3G [01:58<01:18, 55.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  58%|█████▊    | 5.95G/10.3G [01:58<01:12, 59.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  58%|█████▊    | 5.97G/10.3G [01:58<01:13, 58.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  58%|█████▊    | 5.98G/10.3G [01:58<01:10, 60.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  58%|█████▊    | 6.00G/10.3G [01:59<01:08, 62.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  59%|█████▊    | 6.02G/10.3G [01:59<01:13, 58.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  59%|█████▊    | 6.03G/10.3G [01:59<01:16, 55.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  59%|█████▉    | 6.05G/10.3G [01:59<01:12, 58.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  59%|█████▉    | 6.06G/10.3G [02:00<01:11, 59.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  59%|█████▉    | 6.08G/10.3G [02:00<01:12, 57.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  59%|█████▉    | 6.10G/10.3G [02:00<01:11, 58.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  59%|█████▉    | 6.11G/10.3G [02:01<01:31, 45.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  60%|█████▉    | 6.13G/10.3G [02:01<01:37, 42.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  60%|█████▉    | 6.14G/10.3G [02:02<01:30, 45.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  60%|█████▉    | 6.16G/10.3G [02:02<01:27, 47.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  60%|██████    | 6.18G/10.3G [02:02<01:18, 52.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  60%|██████    | 6.19G/10.3G [02:02<01:15, 54.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  60%|██████    | 6.21G/10.3G [02:03<01:16, 53.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  61%|██████    | 6.22G/10.3G [02:03<01:32, 43.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  61%|██████    | 6.24G/10.3G [02:03<01:18, 51.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  61%|██████    | 6.26G/10.3G [02:04<01:14, 54.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  61%|██████    | 6.27G/10.3G [02:04<01:08, 58.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  61%|██████    | 6.29G/10.3G [02:04<01:17, 51.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  61%|██████▏   | 6.30G/10.3G [02:05<02:14, 29.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  61%|██████▏   | 6.32G/10.3G [02:05<01:48, 36.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  62%|██████▏   | 6.34G/10.3G [02:06<01:31, 43.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  62%|██████▏   | 6.35G/10.3G [02:06<01:23, 47.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  62%|██████▏   | 6.37G/10.3G [02:06<01:20, 48.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  62%|██████▏   | 6.38G/10.3G [02:07<01:13, 52.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  62%|██████▏   | 6.40G/10.3G [02:07<01:14, 51.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  62%|██████▏   | 6.42G/10.3G [02:07<01:08, 56.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  63%|██████▎   | 6.43G/10.3G [02:07<01:12, 53.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  63%|██████▎   | 6.45G/10.3G [02:08<01:07, 56.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  63%|██████▎   | 6.46G/10.3G [02:08<01:04, 58.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  63%|██████▎   | 6.48G/10.3G [02:08<01:05, 58.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  63%|██████▎   | 6.50G/10.3G [02:08<01:05, 58.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  63%|██████▎   | 6.51G/10.3G [02:09<01:07, 55.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  64%|██████▎   | 6.53G/10.3G [02:09<01:06, 56.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  64%|██████▎   | 6.54G/10.3G [02:09<01:14, 50.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  64%|██████▍   | 6.56G/10.3G [02:10<01:07, 54.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  64%|██████▍   | 6.58G/10.3G [02:10<01:06, 55.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  64%|██████▍   | 6.59G/10.3G [02:10<01:05, 56.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  64%|██████▍   | 6.61G/10.3G [02:10<01:02, 58.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  64%|██████▍   | 6.62G/10.3G [02:11<01:02, 58.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  65%|██████▍   | 6.64G/10.3G [02:11<01:02, 58.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  65%|██████▍   | 6.66G/10.3G [02:11<00:53, 67.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  65%|██████▍   | 6.67G/10.3G [02:11<00:53, 67.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  65%|██████▌   | 6.69G/10.3G [02:12<00:55, 64.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  65%|██████▌   | 6.70G/10.3G [02:12<00:56, 62.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  65%|██████▌   | 6.72G/10.3G [02:12<01:05, 54.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  66%|██████▌   | 6.74G/10.3G [02:13<01:01, 57.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  66%|██████▌   | 6.75G/10.3G [02:13<01:01, 57.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  66%|██████▌   | 6.77G/10.3G [02:13<01:02, 56.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  66%|██████▌   | 6.78G/10.3G [02:14<01:30, 38.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  66%|██████▌   | 6.80G/10.3G [02:14<01:22, 42.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  66%|██████▋   | 6.82G/10.3G [02:14<01:18, 44.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  66%|██████▋   | 6.83G/10.3G [02:15<01:09, 49.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  67%|██████▋   | 6.85G/10.3G [02:15<01:05, 52.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  67%|██████▋   | 6.86G/10.3G [02:15<01:01, 55.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  67%|██████▋   | 6.88G/10.3G [02:15<00:57, 59.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  67%|██████▋   | 6.90G/10.3G [02:16<00:56, 59.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  67%|██████▋   | 6.91G/10.3G [02:16<00:56, 59.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  67%|██████▋   | 6.93G/10.3G [02:16<00:59, 56.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  68%|██████▊   | 6.94G/10.3G [02:17<00:55, 59.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  68%|██████▊   | 6.96G/10.3G [02:17<01:05, 50.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  68%|██████▊   | 6.98G/10.3G [02:17<01:04, 51.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  68%|██████▊   | 6.99G/10.3G [02:18<01:00, 54.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  68%|██████▊   | 7.01G/10.3G [02:18<01:02, 52.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  68%|██████▊   | 7.02G/10.3G [02:18<00:58, 55.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  68%|██████▊   | 7.04G/10.3G [02:18<01:02, 52.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  69%|██████▊   | 7.06G/10.3G [02:19<00:57, 56.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  69%|██████▉   | 7.07G/10.3G [02:19<00:54, 58.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  69%|██████▉   | 7.09G/10.3G [02:19<00:53, 60.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  69%|██████▉   | 7.10G/10.3G [02:20<00:56, 56.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  69%|██████▉   | 7.12G/10.3G [02:20<01:03, 49.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  69%|██████▉   | 7.14G/10.3G [02:20<01:00, 51.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  70%|██████▉   | 7.15G/10.3G [02:21<00:59, 52.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  70%|██████▉   | 7.17G/10.3G [02:21<01:03, 48.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  70%|██████▉   | 7.18G/10.3G [02:21<01:01, 50.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  70%|███████   | 7.20G/10.3G [02:22<01:03, 48.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  70%|███████   | 7.22G/10.3G [02:22<00:59, 51.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  70%|███████   | 7.23G/10.3G [02:22<00:56, 53.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  71%|███████   | 7.25G/10.3G [02:22<01:02, 48.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  71%|███████   | 7.26G/10.3G [02:23<01:04, 46.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  71%|███████   | 7.28G/10.3G [02:23<01:03, 47.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  71%|███████   | 7.30G/10.3G [02:23<00:58, 50.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  71%|███████   | 7.31G/10.3G [02:24<00:59, 49.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  71%|███████▏  | 7.33G/10.3G [02:24<01:01, 48.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  71%|███████▏  | 7.34G/10.3G [02:24<01:02, 47.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  72%|███████▏  | 7.36G/10.3G [02:25<01:10, 41.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  72%|███████▏  | 7.38G/10.3G [02:25<01:02, 46.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  72%|███████▏  | 7.39G/10.3G [02:25<00:56, 50.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  72%|███████▏  | 7.41G/10.3G [02:26<00:56, 50.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  72%|███████▏  | 7.42G/10.3G [02:26<00:52, 54.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  72%|███████▏  | 7.44G/10.3G [02:26<00:45, 63.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  73%|███████▎  | 7.46G/10.3G [02:26<00:42, 66.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  73%|███████▎  | 7.47G/10.3G [02:27<01:11, 39.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  73%|███████▎  | 7.49G/10.3G [02:28<01:07, 41.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  73%|███████▎  | 7.50G/10.3G [02:28<00:57, 48.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  73%|███████▎  | 7.52G/10.3G [02:28<00:51, 53.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  73%|███████▎  | 7.54G/10.3G [02:28<00:48, 56.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  73%|███████▎  | 7.55G/10.3G [02:28<00:46, 58.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  74%|███████▎  | 7.57G/10.3G [02:29<00:48, 55.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  74%|███████▍  | 7.58G/10.3G [02:29<00:43, 62.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  74%|███████▍  | 7.60G/10.3G [02:29<00:45, 58.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  74%|███████▍  | 7.62G/10.3G [02:30<00:55, 48.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  74%|███████▍  | 7.63G/10.3G [02:30<00:51, 51.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  74%|███████▍  | 7.65G/10.3G [02:30<00:48, 54.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  75%|███████▍  | 7.66G/10.3G [02:31<00:49, 52.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  75%|███████▍  | 7.68G/10.3G [02:31<00:46, 55.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  75%|███████▍  | 7.70G/10.3G [02:31<00:47, 54.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  75%|███████▌  | 7.71G/10.3G [02:31<00:43, 58.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  75%|███████▌  | 7.73G/10.3G [02:32<00:49, 52.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  75%|███████▌  | 7.74G/10.3G [02:33<01:37, 25.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  76%|███████▌  | 7.76G/10.3G [02:34<01:28, 28.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  76%|███████▌  | 7.78G/10.3G [02:34<01:15, 33.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  76%|███████▌  | 7.79G/10.3G [02:34<01:07, 37.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  76%|███████▌  | 7.81G/10.3G [02:37<02:45, 14.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  76%|███████▌  | 7.82G/10.3G [02:37<02:08, 19.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  76%|███████▋  | 7.84G/10.3G [02:37<01:46, 22.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  76%|███████▋  | 7.86G/10.3G [02:38<01:22, 29.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  77%|███████▋  | 7.87G/10.3G [02:38<01:13, 32.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  77%|███████▋  | 7.89G/10.3G [02:39<01:39, 24.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  77%|███████▋  | 7.90G/10.3G [02:39<01:18, 30.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  77%|███████▋  | 7.92G/10.3G [02:39<01:05, 36.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  77%|███████▋  | 7.94G/10.3G [02:40<00:55, 41.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  77%|███████▋  | 7.95G/10.3G [02:40<00:48, 48.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  78%|███████▊  | 7.97G/10.3G [02:40<00:45, 50.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  78%|███████▊  | 7.98G/10.3G [02:40<00:41, 55.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  78%|███████▊  | 8.00G/10.3G [02:41<00:38, 59.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  78%|███████▊  | 8.02G/10.3G [02:41<00:38, 58.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  78%|███████▊  | 8.03G/10.3G [02:41<00:40, 54.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  78%|███████▊  | 8.05G/10.3G [02:42<00:40, 54.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  78%|███████▊  | 8.06G/10.3G [02:42<00:39, 56.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  79%|███████▊  | 8.08G/10.3G [02:42<00:37, 58.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  79%|███████▉  | 8.10G/10.3G [02:42<00:35, 61.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  79%|███████▉  | 8.11G/10.3G [02:43<00:37, 57.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  79%|███████▉  | 8.13G/10.3G [02:43<00:38, 55.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  79%|███████▉  | 8.14G/10.3G [02:43<00:41, 51.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  79%|███████▉  | 8.16G/10.3G [02:44<00:39, 53.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  80%|███████▉  | 8.18G/10.3G [02:44<00:37, 56.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  80%|███████▉  | 8.19G/10.3G [02:44<00:36, 57.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  80%|███████▉  | 8.21G/10.3G [02:44<00:36, 56.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  80%|████████  | 8.22G/10.3G [02:45<00:35, 57.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  80%|████████  | 8.24G/10.3G [02:45<00:43, 46.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  80%|████████  | 8.26G/10.3G [02:45<00:43, 46.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  80%|████████  | 8.27G/10.3G [02:46<00:42, 47.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  81%|████████  | 8.29G/10.3G [02:46<00:40, 49.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  81%|████████  | 8.30G/10.3G [02:46<00:36, 54.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  81%|████████  | 8.32G/10.3G [02:47<00:39, 49.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  81%|████████  | 8.34G/10.3G [02:48<01:01, 31.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  81%|████████▏ | 8.35G/10.3G [02:48<00:54, 35.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  81%|████████▏ | 8.37G/10.3G [02:48<00:48, 39.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  82%|████████▏ | 8.38G/10.3G [02:49<00:43, 43.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  82%|████████▏ | 8.40G/10.3G [02:49<00:37, 49.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  82%|████████▏ | 8.42G/10.3G [02:49<00:33, 55.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  82%|████████▏ | 8.43G/10.3G [02:49<00:32, 56.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  82%|████████▏ | 8.45G/10.3G [02:50<00:31, 58.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  82%|████████▏ | 8.46G/10.3G [02:50<00:29, 62.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  83%|████████▎ | 8.48G/10.3G [02:50<00:28, 62.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  83%|████████▎ | 8.50G/10.3G [02:50<00:29, 59.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  83%|████████▎ | 8.51G/10.3G [02:50<00:27, 63.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  83%|████████▎ | 8.53G/10.3G [02:51<00:29, 60.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  83%|████████▎ | 8.54G/10.3G [02:52<00:51, 33.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  83%|████████▎ | 8.56G/10.3G [02:52<00:44, 39.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  83%|████████▎ | 8.58G/10.3G [02:52<00:38, 44.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  84%|████████▎ | 8.59G/10.3G [02:53<00:34, 48.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  84%|████████▍ | 8.61G/10.3G [02:53<00:33, 49.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  84%|████████▍ | 8.62G/10.3G [02:53<00:31, 53.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  84%|████████▍ | 8.64G/10.3G [02:53<00:29, 55.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  84%|████████▍ | 8.66G/10.3G [02:53<00:25, 64.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  84%|████████▍ | 8.67G/10.3G [02:54<00:28, 57.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  85%|████████▍ | 8.69G/10.3G [02:54<00:25, 61.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  85%|████████▍ | 8.70G/10.3G [02:54<00:24, 64.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  85%|████████▍ | 8.72G/10.3G [02:55<00:34, 45.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  85%|████████▍ | 8.74G/10.3G [02:55<00:31, 49.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  85%|████████▌ | 8.75G/10.3G [02:55<00:29, 52.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  85%|████████▌ | 8.77G/10.3G [02:56<00:27, 55.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  85%|████████▌ | 8.78G/10.3G [02:56<00:27, 55.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  86%|████████▌ | 8.80G/10.3G [02:56<00:26, 56.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  86%|████████▌ | 8.82G/10.3G [02:56<00:24, 59.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  86%|████████▌ | 8.83G/10.3G [02:57<00:25, 57.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  86%|████████▌ | 8.85G/10.3G [02:57<00:27, 51.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  86%|████████▌ | 8.86G/10.3G [02:57<00:28, 49.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  86%|████████▋ | 8.88G/10.3G [02:58<00:25, 55.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  87%|████████▋ | 8.90G/10.3G [02:59<00:47, 29.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  87%|████████▋ | 8.91G/10.3G [02:59<00:37, 36.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  87%|████████▋ | 8.93G/10.3G [02:59<00:30, 44.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  87%|████████▋ | 8.94G/10.3G [03:00<00:30, 43.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  87%|████████▋ | 8.96G/10.3G [03:00<00:28, 45.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  87%|████████▋ | 8.98G/10.3G [03:00<00:31, 41.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  87%|████████▋ | 8.99G/10.3G [03:01<00:27, 46.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  88%|████████▊ | 9.01G/10.3G [03:01<00:25, 49.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  88%|████████▊ | 9.02G/10.3G [03:01<00:23, 53.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  88%|████████▊ | 9.04G/10.3G [03:01<00:22, 54.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  88%|████████▊ | 9.06G/10.3G [03:02<00:20, 58.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  88%|████████▊ | 9.07G/10.3G [03:02<00:21, 55.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  88%|████████▊ | 9.09G/10.3G [03:02<00:20, 59.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  89%|████████▊ | 9.10G/10.3G [03:02<00:20, 58.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  89%|████████▊ | 9.12G/10.3G [03:03<00:18, 62.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  89%|████████▉ | 9.14G/10.3G [03:03<00:18, 63.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  89%|████████▉ | 9.15G/10.3G [03:03<00:23, 47.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  89%|████████▉ | 9.17G/10.3G [03:04<00:23, 46.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  89%|████████▉ | 9.18G/10.3G [03:04<00:21, 51.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  90%|████████▉ | 9.20G/10.3G [03:04<00:21, 51.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  90%|████████▉ | 9.22G/10.3G [03:05<00:21, 50.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  90%|████████▉ | 9.23G/10.3G [03:05<00:23, 44.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  90%|████████▉ | 9.25G/10.3G [03:05<00:21, 48.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  90%|█████████ | 9.26G/10.3G [03:06<00:19, 51.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  90%|█████████ | 9.28G/10.3G [03:06<00:18, 54.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  90%|█████████ | 9.30G/10.3G [03:06<00:16, 57.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  91%|█████████ | 9.31G/10.3G [03:06<00:16, 57.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  91%|█████████ | 9.33G/10.3G [03:07<00:16, 57.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  91%|█████████ | 9.34G/10.3G [03:07<00:15, 60.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  91%|█████████ | 9.36G/10.3G [03:07<00:16, 55.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  91%|█████████ | 9.38G/10.3G [03:08<00:15, 57.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  91%|█████████▏| 9.39G/10.3G [03:08<00:22, 38.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  92%|█████████▏| 9.41G/10.3G [03:09<00:20, 42.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  92%|█████████▏| 9.42G/10.3G [03:09<00:20, 40.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  92%|█████████▏| 9.44G/10.3G [03:09<00:19, 43.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  92%|█████████▏| 9.46G/10.3G [03:10<00:17, 46.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  92%|█████████▏| 9.47G/10.3G [03:10<00:14, 53.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  92%|█████████▏| 9.49G/10.3G [03:10<00:13, 56.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  92%|█████████▏| 9.50G/10.3G [03:10<00:12, 61.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  93%|█████████▎| 9.52G/10.3G [03:11<00:12, 62.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  93%|█████████▎| 9.54G/10.3G [03:11<00:12, 61.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  93%|█████████▎| 9.55G/10.3G [03:11<00:11, 60.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  93%|█████████▎| 9.57G/10.3G [03:11<00:11, 61.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  93%|█████████▎| 9.58G/10.3G [03:12<00:14, 49.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  93%|█████████▎| 9.60G/10.3G [03:12<00:14, 47.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  94%|█████████▎| 9.62G/10.3G [03:12<00:13, 50.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  94%|█████████▎| 9.63G/10.3G [03:13<00:12, 53.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  94%|█████████▍| 9.65G/10.3G [03:13<00:11, 57.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  94%|█████████▍| 9.66G/10.3G [03:13<00:10, 60.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  94%|█████████▍| 9.68G/10.3G [03:13<00:10, 55.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  94%|█████████▍| 9.70G/10.3G [03:14<00:11, 49.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  94%|█████████▍| 9.71G/10.3G [03:14<00:10, 52.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  95%|█████████▍| 9.73G/10.3G [03:14<00:09, 56.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  95%|█████████▍| 9.74G/10.3G [03:15<00:09, 57.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  95%|█████████▍| 9.76G/10.3G [03:15<00:08, 60.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  95%|█████████▌| 9.78G/10.3G [03:15<00:08, 61.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  95%|█████████▌| 9.79G/10.3G [03:15<00:08, 58.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  95%|█████████▌| 9.81G/10.3G [03:16<00:07, 63.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  96%|█████████▌| 9.82G/10.3G [03:16<00:07, 59.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  96%|█████████▌| 9.84G/10.3G [03:16<00:07, 58.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  96%|█████████▌| 9.86G/10.3G [03:17<00:07, 53.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  96%|█████████▌| 9.87G/10.3G [03:17<00:07, 56.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  96%|█████████▌| 9.89G/10.3G [03:17<00:06, 56.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  96%|█████████▋| 9.90G/10.3G [03:17<00:07, 53.2MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  97%|█████████▋| 9.92G/10.3G [03:18<00:07, 50.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  97%|█████████▋| 9.94G/10.3G [03:18<00:05, 57.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  97%|█████████▋| 9.95G/10.3G [03:18<00:05, 57.6MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  97%|█████████▋| 9.97G/10.3G [03:19<00:05, 56.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  97%|█████████▋| 9.98G/10.3G [03:19<00:04, 60.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  97%|█████████▋| 10.0G/10.3G [03:19<00:05, 51.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  97%|█████████▋| 10.0G/10.3G [03:19<00:05, 52.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  98%|█████████▊| 10.0G/10.3G [03:20<00:05, 49.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  98%|█████████▊| 10.0G/10.3G [03:20<00:04, 49.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  98%|█████████▊| 10.1G/10.3G [03:20<00:04, 51.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  98%|█████████▊| 10.1G/10.3G [03:21<00:03, 51.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  98%|█████████▊| 10.1G/10.3G [03:21<00:03, 52.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  98%|█████████▊| 10.1G/10.3G [03:21<00:02, 58.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  99%|█████████▊| 10.1G/10.3G [03:22<00:02, 60.5MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  99%|█████████▊| 10.1G/10.3G [03:22<00:02, 60.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  99%|█████████▉| 10.2G/10.3G [03:22<00:01, 62.4MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  99%|█████████▉| 10.2G/10.3G [03:22<00:01, 62.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  99%|█████████▉| 10.2G/10.3G [03:22<00:01, 65.8MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  99%|█████████▉| 10.2G/10.3G [03:23<00:01, 65.1MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json:  99%|█████████▉| 10.2G/10.3G [03:23<00:00, 59.9MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json: 100%|█████████▉| 10.2G/10.3G [03:23<00:00, 55.3MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json: 100%|█████████▉| 10.3G/10.3G [03:25<00:00, 28.0MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json: 100%|█████████▉| 10.3G/10.3G [03:25<00:00, 32.7MB/s]
ip-26-0-163-220_436335.1720050960368134686.pt.trace.json: 100%|██████████| 10.3G/10.3G [03:25<00:00, 50.0MB/s]