WARNING:torch.distributed.run: ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. ***************************************** WARNING:torch.distributed.run: ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. ***************************************** WARNING:torch.distributed.run: ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. ***************************************** [2024-01-29 09:54:52,371] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 09:54:52,371] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 09:54:52,371] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 09:54:52,378] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 09:54:52,379] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 09:54:52,383] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 09:54:52,393] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 09:54:52,393] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 09:54:58,476] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 09:54:58,476] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 09:54:58,479] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 09:54:58,484] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 09:54:58,487] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 09:54:58,490] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 09:54:58,493] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 09:54:58,495] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 09:55:00,463] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 09:55:00,463] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 09:55:00,463] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 09:55:00,463] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 09:55:00,463] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 09:55:00,463] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 09:55:00,463] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 09:55:00,463] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 09:55:20,105] [INFO] [comm.py:637:init_distributed] cdb=None [2024-01-29 09:55:20,133] [INFO] [comm.py:637:init_distributed] cdb=None [2024-01-29 09:55:20,139] [INFO] [comm.py:637:init_distributed] cdb=None [2024-01-29 09:55:20,171] [INFO] [comm.py:637:init_distributed] cdb=None [2024-01-29 09:55:20,204] [INFO] [comm.py:637:init_distributed] cdb=None [2024-01-29 09:55:20,207] [INFO] [comm.py:637:init_distributed] cdb=None [2024-01-29 09:55:20,210] [INFO] [comm.py:637:init_distributed] cdb=None [2024-01-29 09:55:20,211] [INFO] [comm.py:637:init_distributed] cdb=None [2024-01-29 09:55:15,206] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl 01/29/2024 09:55:24 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False 01/29/2024 09:55:24 - INFO - __main__ - Training/evaluation parameters TrainingArguments( _n_gpu=1, adafactor=False, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, auto_find_batch_size=False, bf16=True, bf16_full_eval=True, data_seed=None, dataloader_drop_last=False, dataloader_num_workers=8, dataloader_pin_memory=True, ddp_bucket_cap_mb=None, ddp_find_unused_parameters=None, ddp_timeout=72000, debug=[], deepspeed=/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/train/deepspeed_config_bf16.json, disable_tqdm=False, do_eval=False, do_predict=False, do_train=True, eval_accumulation_steps=None, eval_delay=0, eval_steps=None, evaluation_strategy=no, fp16=False, fp16_backend=auto, fp16_full_eval=False, fp16_opt_level=O1, fsdp=[], fsdp_config={'fsdp_min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False}, fsdp_min_num_params=0, fsdp_transformer_layer_cls_to_wrap=None, full_determinism=False, gradient_accumulation_steps=8, gradient_checkpointing=True, greater_is_better=None, group_by_length=False, ha01/29/2024 09:55:24 - WARNING - __main__ - Process rank: 7, device: cuda:7, n_gpu: 1distributed training: True, 16-bits training: False /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( 01/29/2024 09:55:24 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False 01/29/2024 09:55:24 - INFO - __main__ - Training/evaluation parameters TrainingArguments( _n_gpu=1, adafactor=False, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, auto_find_batch_size=False, bf16=True, bf16_full_eval=True, data_seed=None, dataloader_drop_last=False, dataloader_num_workers=8, dataloader_pin_memory=True, ddp_bucket_cap_mb=None, ddp_find_unused_parameters=None, ddp_timeout=72000, debug=[], deepspeed=/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/train/deepspeed_config_bf16.json, disable_tqdm=False, do_eval=False, do_predict=False, do_train=True, eval_accumulation_steps=None, eval_delay=0, eval_steps=None, evaluation_strategy=no, fp16=False, fp16_backend=auto, fp16_full_eval=False, fp16_opt_level=O1, fsdp=[], fsdp_config={'fsdp_min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False}, fsdp_min_num_params=0, fsdp_transformer_layer_cls_to_wrap=None, full_determinism=False, gradient_accumulation_steps=8, gradient_checkpointing=True, greater_is_better=None, group_by_length=False, half_precision_backend=auto, hub_model_id=None, hub_private_repo=False, hub_strategy=every_save, hub_token=, ignore_data_skip=False, include_inputs_for_metrics=False, jit_mode_eval=False, label_names=None, label_smoothing_factor=0.0, learning_rate=2e-05, length_column_name=length, load_best_model_at_end=False, local_rank=0, log_level=passive01/29/2024 09:55:24 - WARNING - __main__ - Process rank: 6, device: cuda:6, n_gpu: 1distributed training: True, 16-bits training: False 01/29/2024 09:55:24 - WARNING - __main__ - Process rank: 7, device: cuda:7, n_gpu: 1distributed training: True, 16-bits training: False 01/29/2024 09:55:24 - WARNING - __main__ - Process rank: 3, device: cuda:3, n_gpu: 1distributed training: True, 16-bits training: False /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( 01/29/2024 09:55:24 - WARNING - __main__ - Process rank: 4, device: cuda:4, n_gpu: 1distributed training: True, 16-bits training: False 01/29/2024 09:55:24 - WARNING - __main__ - Process rank: 5, device: cuda:5, n_gpu: 1distributed training: True, 16-bits training: False 01/29/2024 09:55:24 - WARNING - __main__ - Process rank: 1, device: cuda:1, n_gpu: 1distributed training: True, 16-bits training: False 01/29/2024 09:55:24 - WARNING - __main__ - Process rank: 2, device: cuda:2, n_gpu: 1distributed training: True, 16-bits training: False /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( No config specified, defaulting to the single config: red_pajama-data-1_t-sample/plain_text 01/29/2024 09:55:25 - INFO - datasets.builder - No config specified, defaulting to the single config: red_pajama-data-1_t-sample/plain_text Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/modules/datasets_modules/datasets/RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 09:55:25 - INFO - datasets.info - Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/modules/datasets_modules/datasets/RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 Overwrite dataset info from restored data version if exists. 01/29/2024 09:55:25 - INFO - datasets.builder - Overwrite dataset info from restored data version if exists. Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 09:55:25 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 Found cached dataset red_pajama-data-1_t-sample (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039) 01/29/2024 09:55:25 - INFO - datasets.builder - Found cached dataset red_pajama-data-1_t-sample (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039) Loading Dataset info from /apdcephfs/share_733425Overwrite dataset info from restored data version if exists. 01/29/2024 09:55:26 - INFO - datasets.builder - Overwrite dataset info from restored data version if exists. Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 09:55:26 - INFO -No config specified, defaulting to the single config: red_pajama-data-1_t-sample/plain_text 01/29/2024 09:55:26 - INFO - datasets.builder - No config specified, defaulting to the single config: red_pajama-data-1_t-sample/plain_text Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/modules/datasets_modules/datasets/RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 09:55:26 - INFO - datasets.info - Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/modules/datasets_modules/datasets/RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 Overwrite dataset info from restored data version if exists. 01/29/2024 09:55:27 - INFO - datasets.builder - Overwrite dataset info from restored data version if exists. Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 09:55:27 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e90Overwrite dataset info from restored data version if exists. 01/29/2024 09:55:27 - INFO - datasets.builder - Overwrite dataset info from restored data version if exists. Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 09:55:27 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e90Overwrite dataset info from restored data version if exists. 01/29/2024 09:55:28 - INFO - datasets.builder - Overwrite dataset info from restored data version if exists. Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 09:55:28 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 Found cached dataset red_pajama-data-1_t-sample (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039) 01/29/2024 09:55:28 - INFO - datasets.builder - Found cached dataset red_pajama-data-1_t-sample (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039) Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 09:55:28 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 No config specified, defaulting to the single config: red_pajama-data-1_t-sample/plain_text 01/29/2024 09:55:28 - INFO - datasets.builder - No config specified, defaulting to the single config: red_pajama-data-1_t-sample/plain_text Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/modules/datasets_modules/datasets/RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 09:55:28 - INFO - datasets.info - Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/modules/datasets_modules/datasets/RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 Overwrite dataset info from restored data version if exists. 01/29/2024 09:55:28 - INFO - datasets.builder - Overwrite dataset info from restored data version if exists. Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 09:55:28 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 Found cached dataset red_pajama-data-1_t-sample (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039) 01/29/2024 09:55:28 - INFO - datasets.builder - Found cached dataset red_pajama-data-1_t-sample (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039) Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 09:55:28 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 [INFO|configuration_utils.py:666] 2024-01-29 09:55:28,611 >> loading configuration file /apdcephfs/share_733425/vinnylywang/jianhuipang/opensourcellms/llama2/Llama-2-13b-hf/config.json [INFO|configuration_utils.py:720] 2024-01-29 09:55:28,612 >> Model config LlamaConfig { "_name_or_path": "/apdcephfs/share_733425/vinnylywang/jianhuipang/opensourcellms/llama2/Llama-2-13b-hf", "architectures": [ "LlamaForCausalLM" ], "bos_token_id": 1, "eos_token_id": 2, "hidden_act": "silu", "hidden_size": 5120, "initializer_range": 0.02, "intermediate_size": 13824, "max_position_embeddings": 4096, "model_type": "llama", "num_attention_heads": 40, "num_hidden_layers": 40, "num_key_value_heads": 40, "pad_token_id": 0, "pretraining_tp": 1, "rms_norm_eps": 1e-05, "rope_scaling": null, "tie_word_embeddings": false, "torch_dtype": "float16", "transformers_version": "4.28.0.dev0", "use_cache": true, "vocab_size"ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48748:48748 [0] NCCL INFO Bootstrap : Using eth1:11.219.11.45<0> ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48748:48748 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48748:48748 [0] NCCL INFO cudaDriverVersion 11070 NCCL version 2.14.3+cuda11.7 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48753:48753 [5] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48753:48753 [5] NCCL INFO Bootstrap : Using eth1:11.219.11.45 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118721:118721 [4] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118721:118721 [4] NCCL INFO Bootstrap : Using eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118724:118724 [7] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118721:118721 [4] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118724:119607 [7] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118724:119607 [7] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118721:119608 [4] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118721:119608 [4] NCCL INFO Using nets-cbba87ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118717:118717 [0] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118723:118723 [6] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118717:118717 [0] NCCL INFO Bootstrap : Using eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118723:118723 [6] NCCL INFO Bootstrap : Using eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118718:118718 [1] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118718:118718 [1] NCCL INFO Bootstrap : Using eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118717:118717 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118723:118723 [6] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118718:118718 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118717:119611 [0] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118717:119611 [0] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118723:119612 [6] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118723:119612 [6] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118718:119613 [1] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118718:119613 [1] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118722:118722 [5] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118722:118722 [5] NCCL INFO Bootstrap : Using eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118722:118722 [5] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118722:119617 [5] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118722:119617 [5] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118720:118720 [3] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118720:118720 [3] NCCL INFO Bootstrap : Using eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118720:118720 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118720:119624 [3] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-workts-cbba87c5e7504a249f5127103d9ce40f-launcher:48751:49695 [3] NCCL INFO Setting affinity for GPU 3 to ffff,ffffffff,00000000,0000ffff,ffffffff ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48748:49689 [0] NCCL INFO Setting affinity for GPU 0 to ffff,ffffffff,00000000,0000ffff,ffffffff ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48755:49699 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffff0000,00000000,ffffffff,ffff0000,00000000 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48749:49693 [1] NCCL INFO Setting affinity for GPU 1 to ffff,ffffffff,00000000,0000ffff,ffffffff ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48752:49701 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffff0000,00000000,ffffffff,ffff0000,00000000 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48753:49690 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffff0000,00000000,ffffffff,ffff0000,00000000 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48750:49702 [2] NCCL INFO Setting affinity for GPU 2 to ffff,ffffffff,00000000,0000ffff,ffffffff ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48754:49697 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffff0000,000ts-cbba87c5e7504a249f5127103d9ce4ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48748:49689 [0] NCCL INFO Channel 00/02 : 0 3 10 15 14 13 12 9 8 11 18 23 22 21 20 17 16 19 26 31 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48748:49689 [0] NCCL INFO Channel 01/02 : 0 3 10 15 14 13 12 9 8 11 18 23 22 21 20 17 16 19 26 31 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48748:49689 [0] NCCL INFO Trees [0] 1/-1/-1->0->7 [1] 1/-1/-1->0->7 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48755:49699 [7] NCCL INFO Trees [0] 0/-1/-1->7->6 [1] 0/-1/-1->7->6 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48749:49693 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48750:49702 [2] NCCL INFO Trees [0] 3/18/-1->2->-1 [1] 3/-1/-1->2->10 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48753:49690 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48754:49697 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48752:49701 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48751:49695 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48748:49689 [0] NCCL INFO Channel 00/0 : 0[e000] -> 3[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48752:49701 [4] NCCL INFO Channel 00/0 : 4[93000] -> 1[13000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48750:49702 [2] NCCL INFO Channel 00/0 : 2[4b000] -> 7[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48748:49689 [0] NCCL INFO Channel 01/0 : 0[e000] -> 3[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48752:49701 [4] NCCL INFO Channel 01/0 : 4[93000] -> 1[13000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48750:49702 [2] NCCL INFO Channel 01/0 : 2[4b000] -> 7[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48753:49690 [5] NCCL INFO Channel 00/0 : 5[99000] -> 4[93000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48754:49697 [6] NCCL INFO Channel 00/0 : 6[cb000] -> 5[99000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48753:49690 [5] NCCL INFO Channel 01/0 : 5[99000] -> 4[93000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48754:49697 [6] NCCL INFO Channel 01/0 : 6[cb000] -> 5[99000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48749:49693 [1] NCCL INFO Channel 00/0 : 1[13000] -> 0[e000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48751:49695 [3] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48755:49699 [7] NCCL INFO Channel 00/0 : 7[d0000] -> 6[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48749:49693 [1] NCCL INFO Channel 01/0 : 1[13000] -> 0[e000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48755:49699 [7] NCCL INFO Channel 01/0 : 7[d0000] -> 6[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48752:49701 [4] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48755:49699 [7] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48749:49693 [1] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48748:49689 [0] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48748:49689 [0] NCCL INFO Channel 00/0 : 0[e000] -> 1[13000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48748:49689 [0] NCCL INFO Channel 01/0 : 0[e000] -> 1[13000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48753:49690 [5] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48754:49697 [6] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48752:49701 [4] NCCL INFO Channel 00/0 : 4[93000] -> 5[99000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48752:49701 [4] NCCL INFO Channel 01/0 : 4[93000] -> 5[99000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48753:49690 [5] NCCL INFO Channel 00/0 : 5[99000] -> 6[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48749:49693 [1] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48749:49693 [1] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48749:49693 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48754:49697 [6] NCCL INFO Channel 00/0 : 6[cb000] -> 7[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48748:49689 [0] NCCL INFO Channel 00/0 : 0[e000] -> 7[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48753:49690 [5] NCCL INFO Channel 01/0 : 5[99000] -> 6[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48754:49697 [6] NCCL INFO Channel 01/0 : 6[cb000] -> 7[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48748:49689 [0] NCCL INFts-cbba87c5e7504a249f5127103d9ce40f-worker-2:69938:708ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48753:49690 [5] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48753:49690 [5] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48753:49690 [5] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48754:49697 [6] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48754:49697 [6] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48754:49697 [6] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48755:49699 [7] NCCL INFO Channel 00/0 : 7[d0000] -> 0[e000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48755:49699 [7] NCCL INFO Channel 01/0 : 7[d0000] -> 0[e000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48748:49689 [0] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48748:49689 [0] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48748:49689 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48755:49699 [7] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48755:49699 [7] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48755:49699 [7] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48750:49702 [2] NCCL INFO Channel 00/0 : 27[51000] -> 2[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48751:49695 [3] NCCL INFO Channel 00/0 : 3[51000] -> 10[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48749:49693 [1] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48754:49697 [6] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48753:49690 [5] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48755:49699 [7] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48748:49689 [0] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48751:49695 [3] NCCL INFO Channel 01/0 : 3[51000] -> 10[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48750:49702 [2] NCCL INFO Channel 01/0 : 27[51000] -> 2[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48751:49710 [3] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48750:49716 [2] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48750:49716 [2] NCCL INFO NCCL_IB_SL set by environment to 3. ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48751:49695 [3] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48751:49695 [3] NCCL INFO Channel 00/0 : 3[51000] -> 4[93000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48751:49695 [3] NCCL INFO Channel 01/0 : 3[51000] -> 4[93000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48750:49702 [2] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48750:49702 [2] NCCL INFO Channel 00/0 : 2[4b000] -> 3[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48750:49702 [2] NCCL INFO Channel 01/0 : 2[4b000] -> 3[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48752:49701 [4] NCCL INFO Channel 00/0 : 4[93000] -> 3[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48750:49702 [2] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48752:49701 [4] NCCL INFO Channel 01/0 : 4[93000] -> 3[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48751:49695 [3] NCCL INFO Channel 00/0 : 3[51000] -> 2[4b000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48751:49695 [3] NCCL INFO Channel 01/0 : 3[51000] -> 2[4b000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48752:49701 [4] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48752:49701 [4] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48752:49701 [4] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48750:49702 [2] NCCL INFO Channel 01/0 : 2[4b000] -> 10[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48752:49701 [4] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48750:49702 [2] NCCL INFO Channel 00/0 : 18[4b000] -> 2[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48750:49702 [2] NCCL INFO Channel 00/0 : 2[4b000] -> 18[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48750:49702 [2] NCCL INFO Channel 01/0 : 10[4b000] -> 2[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48750:49702 [2] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48750:49702 [2] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48750:49702 [2] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48751:49695 [3] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48751:49695 [3] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48751:49695 [3] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48754:49697 [6] NCCL INFO comm 0x44a515c0 rank 6 nranks 32 cudaDev 6 busId cb000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48753:49690 [5] NCCL INFO comm 0x43e95220 rank 5 nranks 32 cudaDev 5 busId 99000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48755:49699 [7] NCCL INFO comm 0x43fcd730 rank 7 nranks 32 cudaDev 7 busId d0000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48752:49701 [4] NCCL INFO comm 0x42f18fe0 rank 4 nranks 32 cudaDev 4 busId 93000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48749:49693 [1] NCCL INFO comm 0x44d5c800 rank 1 nranks 32 cudaDev 1 busId 13000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48751:49695 [3] NCCL INFO comm 0x42b49b70 rank 3 nranks 32 cudaDev 3 busId 51000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48748:49689 [0] NCCL INFO comm 0x42d72100 rank 0 nranks 32 cudaDev 0 busId e000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48750:49702 [2] NCCL INFO comm 0x44dcd090 rank 2 nranks 32 cudaDev 2 busId 4b000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-launcher:48751:49718 [3] NCCL INFO NCCL_IB_SL set by environment to 3. [2024-01-29 09:55:36,614] [INFO] [partition_parameters.py:347:__exit__] finished initializing model - num_params = 363, num_elems = 13.02B Loading checkpoint shards: 0%| | 0/3 [00:00 11[51000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118719:119626 [2] NCCL INFO Channel 01/0 : 2[4b000] -> 10[4b000ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95674:96580 [2] NCCL INFO Channel 00/0 : 2[4b000] -> 18[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95676:96593 [3] NCCL INFO NCCL_IB_SL set by environment to 3. ts-cbba87c5e7504a249f5127103d9ce40f-workets-cbba87c5e7504a249f5127103d9ce40f-worker-0:118719:119626 [2] NCCL INFO Channel 01/0 : 26[4b0ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118719:119626 [2] NCCL INFO Channel 01/0 : 10[4b000] -> 26[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118719:119626 [2] NCCL INFO Channel 00/0 : 19ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95676:ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118719:119626 [2] NCCL INFO Channel 01/0 : 10[4b000] -> 2[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118720:119631 [3] NCCL INFO NCCL_IB_SL set by environment to 3. ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118720:119624 [3] NCCL INFO Channel 00/0 : 11[51000] -> 10[4b000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118720:119624 [3] NCCL INFO Channel 01/0 : 11[51000] -> 10[4b000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118721:119608 [4] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118721:119608 [4] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118721:119608 [4] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118719:119626 [2] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118719:119626 [2] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118719:119626 [2] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118720:119624 [3] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118720:119624 [3] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118720:119624 [3ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95684:96582 [7] NCCL INFO comm 0x42e71dd0 rank 23 nranks 32 cudaDev 7 busId d0000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-workerts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118717:119611 [0] NCCL INFO comm 0x4666f760 rank 8 nranks 32 cudaDev 0 busId e000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118723:119612 [6] NCCL INFO comm 0x428c2130 rank 14 nranks 32 cudaDev 6 busId cb000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118719:119626 [2] NCCL INFO comm 0x442af620 rank 10 nranks 32 cudaDev 2 busId 4b000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118721:119608 [4] NCCL INFO comm 0x43ffeda0 rank 12 nranks 32 cudaDev 4 busId 93000 - Ints-cbba87c5e7504a249f5127103d9ce40f-worker-2:69941:70853 [3] NCCL INFO NCCL_IB_SL set by environment to 3. nranks 32 cudaDev 1 busId 13000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118720:119624 [3] NCCL INFO comm 0x468cbc90 rank 11 nranks 32 cudaDev 3 busId 51000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118724:119607 [7] NCCL INFO comm 0x4627cac0 rank 15 nranks 32 cudaDev 7 busId d0000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118722:119617 [5] NCCL INFO comm 0x44fc7970 rank 13 nranks 32 cudaDev 5 busId 99000 - Init COMPLETE Loading checkpoint shards: 0%| | 0/3 [00:00> Using pad_token, but it is not set yet. Loading checkpoint shards: 100%|██████████| 3/3 [01:00<00:00, 19.18s/it] Loading checkpoint shards: 100%|██████████| 3/3 [01:00<00:00, 20.06s/it] Loading checkpoint shards: 100%|██████████| 3/3 [01:00<00:00, 19.18s/it] Loading checkpoint shards: 100%|██████████| 3/3 [01:00<00:00, 20.06s/it] Loading checkpoint shards: 100%|██████████| 3/3 [01:00<00:00, 19.19s/it] Loading checkpoint shards: 100%|██████████| 3/3 [01:00<00:00, 20.06s/it] Loading checkpoint shards: 100%|██████████| 3/3 [01:00<00:00, 19.19s/it] Loading checkpoint shards: 100%|██████████| 3/3 [01:00<00:00, 20.06s/it] [INFO|modeling_utils.py:3029] 2024-01-29 09:56:36,879 >> All model checkpoint weights were used when initializing LlamaForCausalLM. [INFO|modeling_utils.py:3037] 2024-01-29 09:56:36,879 >> All the weights of LlamaForCausalLM were initialized from the model checkpoint at /apdcephfs/share_733425/vinnylywang/jianhuipang/opensourcellms/llama2/Llama-2-13b-hf. If your task is similar to the task the model of the checkpoint was tr Loading checkpoint shards: 100%|██████████| 3/3 [01:00<00:00, 19.20s/it] Loading checkpoint shards: 100%|██████████| 3/3 [01:00<00:00, 20.08s/it] Loading checkpoint shards: 100%|██████████| 3/3 [01:00<00:00, 19.21s/it] Loading checkpoint shards: 100%|██████████| 3/3 [01:00<00:00, 20.08s/it] [ERROR|tokenization_utils_base.py:1042] 2024-01-29 09:56:36,955 >> Using pad_token, but it is not set yet. [ERROR|tokenization_utils_base.py:1042] 2024-01-29 09:56:36,962 >> Using pad_token, but it is not set yet. Loading checkpoint shards: 100%|██████████| 3/3 [01:00<00:00, 19.19s/it] Loading checkpoint shards: 100%|██████████| 3/3 [01:00<00:00, 20.11s/it] [INFO|modeling_utils.py:3029] 2024-01-29 09:56:37,015 >> All model checkpoint weights were used when initializing LlamaForCausalLM. [INFO|modeling_utils.py:3037] 2024-01-29 09:56:37,015 >> All the weights of LlamaForCausalLM were initialized from the model checkpoint at /apdcephfs/share_733425/vinnylywang/jianhuipang/opensourcellms/llama2/Llama-2-13b-hf. If your task is similar to the task the model of the checkpoint was trained on, you can already use LlamaForCausalLM for predictions without further training. [INFO|configuration_utils.py:535] 2024-01-29 09:56:37,024 >> loading configuration file /apdcephfs/share_733425/vinnylywang/jianhuipang/opensourcellms/llama2/Llama-2-13b-hf/generation_config.json [INFO|configuration_utils.py:575] 2024-01-29 09:56:37,025 >> Generate config GenerationConfig { "bos_token_id": 1, "do_sample": true, "eos_token_id": 2, "max_length": 4096, "pad_token_id": 0, "temperature": 0.6, "top_p": 0.9, "transformers_version": "4.28.0.dev0" } [ERROR|tokenization_utils_base.py:1042] 2024-01-29 09:56:37,025 >> Using pad_token, but it is not set yet. [INFO|tokenization_utils_base.py:907] 2024-01-29 09:56:37,025 >> Assigning [PAD] to the pad_token key of the tokenizer [INFO|tokenization_utils.py:426] 2024-01-29 09:56:37,025 >> Adding [PAD] to the vocabulary [INFO|tokenization_utils_base.py:907] 2024-01-29 09:56:40,969 >> Assigning to the eos_token key of the tokenizer [INFO|tokenization_utils_base.py:907] 2024-01-29 09:56:40,969 >> Assigning to the bos_token key of the tokenizer [INFO|tokenization_utils_base.py:907] 2024-01-29 09:56:40,969 >> Assigning to the unk_token key of the tokenizer [INFO|tokenization_utils.py:426] 2024-01-29 09:56:41,040 >> Adding to the vocabulary 01/29/2024 09:56:41 - INFO - __main__ - We have added new 1 token as an anchor Process #0 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00000_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #0 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00000_of_00032.arrow Process #1 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00001_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #1 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00001_of_00032.arrow Process #2 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00002_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #2 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00002_of_00032.arrow Process #3 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00003_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #3 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00003_of_00032.arrow Process #4 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00004_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #4 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00004_of_00032.arrow Process #5 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00005_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #5 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00005_of_00032.arrow Process #6 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00006_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #6 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00006_of_00032.arrow Process #7 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00007_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #7 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00007_of_00032.arrow Process #8 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00008_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #8 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00008_of_00032.arrow Process #9 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00009_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #9 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00009_of_00032.arrow Process #10 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00010_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #10 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00010_of_00032.arrow Process #11 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00011_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #11 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00011_of_00032.arrow Process #12 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00012_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #12 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00012_of_00032.arrow Process #13 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00013_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #13 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00013_of_00032.arrow Process #14 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00014_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #14 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00014_of_00032.arrow Process #15 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00015_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #15 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00015_of_00032.arrow Process #16 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00016_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #16 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00016_of_00032.arrow Process #17 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00017_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #17 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00017_of_00032.arrow Process #18 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00018_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #18 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00018_of_00032.arrow Process #19 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00019_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #19 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00019_of_00032.arrow Process #20 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00020_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #20 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00020_of_00032.arrow Process #21 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00021_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #21 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00021_of_00032.arrow Process #22 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00022_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #22 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00022_of_00032.arrow Process #23 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00023_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #23 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00023_of_00032.arrow Process #24 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00024_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #24 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00024_of_00032.arrow Process #25 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00025_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #25 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00025_of_00032.arrow Process #26 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00026_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #26 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00026_of_00032.arrow Process #27 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00027_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #27 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00027_of_00032.arrow Process #28 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00028_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #28 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00028_of_00032.arrow Process #29 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00029_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #29 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00029_of_00032.arrow Process #30 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00030_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #30 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00030_of_00032.arrow Process #31 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00031_of_00032.arrow 01/29/2024 09:56:43 - INFO - datasets.arrow_dataset - Process #31 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-66ee3d7dd3d5e980_00031_of_00032.arrow Spawning 32 processes 01/29/2024 09:56:44 - INFO - datasets.arrow_dataset - Spawning 32 processes Map (num_proc=32): 0%| | 0/930514 [00:00> Using cuda_amp half precision backend [2024-01-29 10:09:21,178] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.11.1, git-hash=unknown, git-branch=unknown Map (num_proc=32): 0%| | 0/930514 [00:00> Using cuda_amp half precision backend Map (num_proc=32): 0%| | 0/930514 [00:00 [2024-01-29 10:28:55,829] [INFO] [logging.py:96:log_dist] [Rank 0] Creating fp16 ZeRO stage 3 optimizer, MiCS is enabled False, Hierarchical params gather False [2024-01-29 10:28:55,829] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 3 optimizer [2024-01-29 10:28:56,015] [INFO] [utils.py:802:see_memory_usage] Stage 3 initialize beginning [2024-01-29 10:28:56,016] [INFO] [utils.py:803:see_memory_usage] MA 0.65 GB Max_MA 1.3 GB CA 1.61 GB Max_CA 2 GB [2024-01-29 10:28:56,016] [INFO] [utils.py:810:see_memory_usage] CPU Virtual Memory: used = 60.61 GB, percent = 6.0% [2024-01-29 10:28:56,020] [INFO] [stage3.py:126:__init__] Reduce bucket size 26214400 [2024-01-29 10:28:56,020] [INFO] [stage3.py:127:__init__] Prefetch bucket size 23592960 [2024-01-29 10:28:56,199] [INFO] [utils.py:802:see_memory_usage] DeepSpeedZeRoOffload initialize [begin] [2024-01-29 10:28:56,199] [INFO] [utils.py:803:see_memory_usage] MA 0.65 GB Max_MA 0.65 GB CA 1.61 GB Max_CA 2 GB [2024-01-29 10:28:56,200] [INFO] [utils.py:810:see_memory_usage] CPU Virtual Memory: used = 60.55 GB, percent = 6.0% Parameter Offload: Total persistent parameters: 414720 in 81 params [2024-01-29 10:28:56,874] [INFO] [utils.py:802:see_memory_usage] DeepSpeedZeRoOffload initialize [end] [2024-01-29 10:28:56,875] [INFO] [utils.py:803:see_memory_usage] MA 0.04 GB Max_MA 0.65 GB CA 1.61 GB Max_CA 2 GB [2024-01-29 10:28:56,876] [INFO] [utils.py:810:see_memory_usage] CPU Virtual Memory: used = 60.55 GB, percent = 6.0% [2024-01-29 10:28:57,057] [INFO] [utils.py:802:see_memory_usage] Before creating fp16 partitions [2024-01-29 10:28:57,058] [INFO] [utils.py:803:see_memory_usage] MA 0.04 GB Max_MA 0.04 GB CA 1.61 GB Max_CA 2 GB [2024-01-29 10:28:57,058] [INFO] [utils.py:810:see_memory_usage] CPU Virtual Memory: used = 60.55 GB, percent = 6.0% [2024-01-29 10:28:58,092] [INFO] [utils.py:802:see_memory_usage] After creating fp16 partitions: 1 [2024-01-29 10:28:58,093] [INFO] [utils.py:803:see_memory_usage] MA 0.04 GB Max_MA 0.04 GB CA 1.61 GB Max_CA 2 GB [2024-01-29 10:28:58,093] [INFO] [utils.py:810:see_memory_usage] CPU Virtual Memory: used = 74.34 GB, percent = 7.4% [2024-01-29 10:28:58,303] [INFO] [utils.py:802:see_memory_usage] Before creating fp32 partitions [2024-01-29 10:28:58,304] [INFO] [utils.py:803:see_memory_usage] MA 0.04 GB Max_MA 0.04 GB CA 1.61 GB Max_CA 2 GB [2024-01-29 10:28:58,304] [INFO] [utils.py:810:see_memory_usage] CPU Virtual Memory: used = 78.98 GB, percent = 7.8% [2024-01-29 10:28:59,196] [INFO] [utils.py:802:see_memory_usage] After creating fp32 partitions [2024-01-29 10:28:59,197] [INFO] [utils.py:803:see_memory_usage] MA 0.04 GB Max_MA 0.04 GB CA 1.61 GB Max_CA 2 GB [2024-01-29 10:28:59,197] [INFO] [utils.py:810:see_memory_usage] CPU Virtual Memory: used = 80.84 GB, percent = 8.0% 86, 1736, 2651, 2036, 29912, 12443, 10578, 29906, 29900, 29900, 29945, 5062, 11235, 29892, 13260, 333, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 13, 29871, 696, 7586, 29906, 29900, 29896, 29946, 7610, 345, 2941, 271, 24541, 29892, 697, 309, 29906, 29900, 29896, 29953, 311, 5365, 550, 332, 6950, 29913, 505, 2931, 1891, 278, 13, 479, 5275, 310, 12362, 29914, 6585, 7562, 18540, 313, 5800, 1799, 29897, 773, 320, 7278, 29912, 6734, 26946, 952, 1118, 13, 4716, 3867, 1880, 29899, 29567, 6089, 541, 526, 9078, 297, 2159, 313, 29906, 29899, 29945, 5940, 29968, 18777, 29897, 13, 392, 508, 367, 4768, 1463, 491, 5221, 424, 23460, 29889, 32001, 512, 29871, 29906, 29900, 29900, 29947, 2261, 801, 2681, 634, 394, 8770, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29913, 18043, 263, 3031, 979, 13, 16961, 29899, 7052, 313, 1454, 278, 931, 29897, 6559, 373, 18322, 1799, 320, 7278, 29912, 479, 5275, 773, 1375, 292, 7047, 13, 29871, 28914, 313, 4345, 29934, 29897, 13698, 1836, 32001, 2688, 29537, 287, 278, 3978, 310, 29871, 29896, 5940, 29924, 17737, 29560, 13, 4746, 278, 7562, 2831, 479, 1404, 2566, 322, 611, 6504, 1051, 3190, 3145, 975, 278, 13, 29896, 29929, 29929, 29929, 489, 29906, 29900, 29900, 29945, 3785, 29892, 773, 408, 18470, 2472, 2788, 304, 1749, 29879, 29901, 4876, 21904, 13, 392, 17998, 1283, 7224, 29889, 32001, 450, 12399, 3785, 313, 29955, 2440, 29897, 297, 2651, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29913, 338, 20511, 1135, 29871, 13, 5816, 338, 12399, 297, 278, 2198, 5650, 313, 29945, 29900, 2440, 29897, 322, 278, 848, 8974, 526, 29871, 13, 27489, 873, 1422, 29936, 411, 393, 297, 3458, 29892, 1749, 2582, 1510, 263, 10029, 7200, 14978, 310, 29871, 13, 15654, 273, 325, 13217, 29940, 2072, 3082, 20706, 29889, 32001, 7280, 29190, 936, 664, 515, 29871, 29906, 29900, 29896, 29900, 491, 14619, 29882, 371, 4099, 29894, 322, 13, 29950, 309, 1372, 2651, 2036, 29912, 29873, 19426, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29913, 4023, 29894, 2868, 1583, 29899, 7099, 433, 1127, 1737, 12122, 13, 2029, 800, 310, 25492, 15303, 8304, 3598, 1494, 1009, 12368, 29892, 13, 15914, 292, 2472, 363, 779, 14850, 4535, 29892, 29955, 29900, 5940, 29968, 25492, 4160, 29889, 32001, 319, 1407, 7786, 13, 1287, 2651, 2036, 29912, 21686, 29879, 29906, 29900, 29906, 29896, 2209, 479, 5275, 29913, 491, 399, 496, 29879, 634, 394, 13217, 5349, 1737, 324, 542, 630, 4203, 263, 7284, 13, 28712, 16046, 4160, 29892, 2534, 26869, 472, 3203, 29871, 29896, 29900, 29900, 25741, 1269, 29892, 322, 1058, 13, 1311, 29899, 7099, 8663, 14354, 373, 1009, 25492, 28723, 29889, 32001, 5806, 278, 6559, 338, 13, 3149, 29899, 262, 29899, 2230, 408, 310, 29871, 29906, 29900, 29906, 29896, 29892, 278, 15717, 7252, 1009, 1284, 886, 13, 351, 475, 303, 2651, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29892, 1850, 29882, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29913, 304, 13, 18609, 675, 278, 14675, 310, 18322, 1799, 1737, 5275, 975, 278, 931, 15101, 845, 1862, 4586, 491, 13, 1552, 2211, 11898, 29889, 32001, 3831, 1965, 411, 3517, 29190, 936, 1736, 29892, 1749, 6559, 338, 1568, 7200, 6287, 5634, 29882, 5555, 13, 7054, 12339, 287, 29871, 29946, 29941, 7284, 29905, 13007, 15717, 310, 29871, 29906, 29889, 29906, 24464, 29905, 13007, 25741, 515, 29871, 29896, 29953, 29900, 7284, 29905, 13007, 13, 16418, 5634, 5426, 11267, 979, 975, 29871, 29945, 29900, 2440, 310, 970, 775, 20706, 3265, 1135, 13, 3149, 297, 931, 29892, 322, 884, 901, 2691, 29899, 3874, 1312, 313, 2541, 1629, 29899, 1609, 29899, 6360, 3803, 1070, 537, 975, 13, 1552, 8900, 3785, 467, 32001, 8108, 1189, 1711, 29892, 1749, 6559, 337, 3687, 373, 10079, 11264, 13, 3924, 313, 8257, 29903, 29897, 9063, 848, 3265, 1135, 7481, 29899, 7099, 433, 1127, 4423, 2472, 29889, 32001, 5901, 1736, 5634, 262, 3153, 278, 664, 491, 8432, 2651, 2036, 29912, 18386, 709, 29906, 29900, 29896, 29941, 2209, 29881, 24974, 29913, 13, 392, 29892, 901, 10325, 29892, 390, 579, 468, 29875, 634, 394, 8770, 2036, 29912, 29878, 579, 468, 29875, 29906, 29900, 29896, 29953, 479, 711, 3173, 29892, 13, 29871, 364, 579, 468, 29875, 29906, 29900, 29896, 29947, 479, 711, 3173, 29892, 544, 1648, 29906, 29900, 29906, 29896, 479, 468, 1581, 29881, 24974, 29913, 5634, 17532, 12399, 1737, 12122, 13, 29905, 7278, 29912, 29881, 24974, 322, 24003, 1118, 474, 29889, 29872, 1696, 278, 15834, 304, 607, 278, 3978, 310, 18322, 1799, 13, 17426, 6602, 1009, 11465, 1230, 14137, 14188, 29889, 32001, 512, 445, 664, 591, 2931, 1891, 1737, 12122, 6894, 537, 297, 970, 775, 363, 278, 937, 13, 2230, 472, 445, 6287, 29892, 1716, 297, 4958, 310, 17737, 29560, 322, 15500, 3785, 29889, 32001, 1334, 437, 13, 1333, 22002, 280, 278, 24003, 10696, 29892, 541, 3867, 29190, 936, 848, 322, 1284, 886, 393, 508, 367, 13, 280, 369, 4063, 304, 393, 1095, 408, 5434, 664, 29889, 32001, 320, 7278, 29912, 12756, 7047, 21639, 29913, 2651, 2036, 29912, 2276, 5824, 19982, 29906, 29900, 29900, 29955, 23705, 1338, 29893, 996, 29913, 338, 278, 13, 1491, 29899, 2671, 310, 7047, 21639, 393, 756, 29537, 287, 278, 18066, 267, 310, 21640, 13, 6734, 24771, 13149, 635, 29892, 3704, 278, 2702, 5932, 310, 920, 304, 5376, 13, 2541, 1737, 12122, 6894, 537, 2651, 2036, 29912, 13376, 303, 456, 29906, 29900, 29900, 29953, 10945, 3359, 29892, 1424, 29440, 29906, 29900, 29896, 29946, 23027, 5933, 1836, 32001, 3826, 3076, 2678, 278, 2198, 6559, 8128, 10757, 393, 508, 367, 1304, 29892, 297, 278, 13, 14940, 1206, 310, 970, 775, 322, 472, 263, 1407, 2919, 6287, 29892, 304, 11539, 607, 13, 14032, 4637, 310, 5534, 7047, 21639, 505, 9820, 484, 15774, 29889, 32001, 320, 2042, 29912, 4062, 3002, 29913, 13, 29905, 1643, 29912, 3471, 29901, 5696, 29913, 13, 13, 13, 29905, 1482, 361, 29905, 361, 29887, 798, 386, 1003, 29871, 320, 29887, 798, 386, 1003, 3009, 13, 29905, 361, 29887, 798, 386, 1003, 13, 29905, 463, 29912, 4532, 29913, 13, 29871, 320, 7313, 29961, 2103, 2013, 4914, 2103, 3199, 6360, 368, 29899, 2055, 1169, 29913, 13, 29871, 320, 6671, 29912, 12883, 368, 970, 25741, 975, 931, 313, 1188, 6287, 467, 13, 29913, 32001, 320, 1643, 29912, 1003, 29901, 29887, 798, 386, 29913, 13, 29905, 355, 29912, 4532, 29913, 13, 29905, 7241, 13, 13, 29905, 26956, 29912, 16390, 24541, 29913, 13, 13, 4806, 27387, 515, 320, 23066, 29950, 29914, 2651, 2036, 29912, 29879, 1332, 29899, 1516, 29878, 29906, 29900, 29896, 29929, 29899, 24713, 29913, 599, 25741, 3190, 2347, 2745, 320, 25832, 1964, 579, 1523, 2415, 2539, 6294, 32001, 2688, 5253, 304, 320, 14573, 5261, 1169, 22131, 29914, 25741, 29892, 5412, 491, 317, 15715, 29896, 15882, 29892, 4023, 29894, 2868, 515, 320, 25832, 1299, 7288, 5261, 1169, 797, 7068, 29914, 970, 9279, 6421, 515, 4655, 5849, 363, 2710, 313, 28712, 16046, 29892, 11786, 28632, 29892, 2992, 1846, 32001, 322, 3577, 28914, 313, 10251, 713, 29892, 10772, 2227, 29892, 405, 13427, 29892, 2992, 6250, 32001, 1876, 1169, 297, 278, 8783, 526, 491, 320, 25832, 6344, 2806, 943, 22131, 29914, 15717, 29892, 5412, 491, 779, 6990, 29938, 978, 29892, 4876, 4535, 5854, 29938, 11000, 29889, 32001, 450, 8783, 2996, 408, 1023, 1104, 1288, 6131, 29892, 697, 363, 25741, 322, 697, 363, 15717, 29892, 411, 278, 4642, 29371, 278, 7480, 3025, 263, 9117, 1820, 29889, 32001, 320, 361, 5426, 13, 9760, 1948, 297, 278, 9063, 1591, 3743, 278, 1494, 4235, 29901, 9063, 317, 15715, 29896, 15882, 29892, 4148, 322, 844, 5171, 5335, 342, 15092, 29892, 4148, 322, 844, 5171, 2893, 14903, 313, 20275, 16750, 278, 4148, 1591, 467, 32001, 450, 21578, 1546, 9063, 15717, 322, 9063, 2153, 2041, 515, 11786, 29892, 607, 6511, 304, 9063, 263, 1735, 4148, 287, 491, 4856, 1683, 29889, 32001, 1152, 445, 6559, 591, 21309, 373, 15717, 322, 17262, 9063, 2153, 29892, 408, 278, 4328, 1546, 278, 1023, 338, 451, 8018, 363, 1749, 5925, 5155, 322, 278, 5253, 310, 25741, 411, 263, 844, 5171, 916, 1135, 967, 4148, 338, 3480, 3473, 1821, 29889, 32001, 320, 7241, 13, 2831, 1269, 6251, 297, 278, 4148, 1591, 591, 505, 4148, 2989, 1024, 322, 4876, 408, 1023, 5004, 6031, 310, 10650, 6262, 29889, 32001, 1334, 6206, 2411, 15273, 1821, 470, 18325, 519, 2983, 393, 29901, 526, 451, 1602, 397, 519, 408, 18351, 29899, 29947, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12283, 7099, 397, 519, 29914, 4148, 2983, 6206, 511, 526, 4876, 14157, 2012, 310, 2983, 3441, 25832, 6344, 2806, 943, 29934, 29885, 9823, 29914, 4954, 7039, 4907, 511, 5718, 310, 871, 9654, 4890, 3441, 25832, 6344, 2806, 943, 29934, 29885, 10358, 804, 23201, 1712, 901, 1135, 29871, 29896, 29900, 8958, 1661, 29899, 1026, 2153, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12283, 15670, 23201, 526, 5520, 1135, 29871, 29896, 29900, 29900, 4890, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12229, 549, 12495, 32001, 2860, 21166, 29892, 1048, 320, 25832, 6344, 2806, 943, 29925, 15273, 1821, 2052, 307, 29916, 29914, 15717, 3441, 25832, 6344, 2806, 943, 29925, 15273, 1821, 29925, 312, 29914, 310, 278, 2847, 8783, 29897, 9488, 363, 4340, 7418, 29889, 32001, 3940, 393, 278, 5253, 310, 970, 775, 25741, 313, 392, 15717, 29897, 11122, 297, 278, 13, 11228, 8783, 25088, 18709, 9247, 975, 13, 2230, 2651, 2036, 29912, 29879, 1332, 29899, 771, 854, 749, 29899, 331, 344, 1012, 361, 29887, 798, 386, 1003, 29892, 408, 4318, 363, 25741, 297, 13, 29905, 29907, 999, 29912, 1003, 29901, 29887, 798, 386, 1012, 2870, 29901, 515, 395, 29896, 29900, 29985, 29946, 29938, 25741, 297, 29871, 29896, 29929, 29955, 29896, 29892, 304, 395, 29896, 29900, 29985, 29953, 29938, 297, 29871, 29896, 29929, 29929, 29947, 29892, 304, 13, 284, 3242, 395, 29896, 29900, 29985, 29929, 29938, 297, 29871, 29906, 29900, 29906, 29900, 29905, 7241, 29889, 32001, 1094, 263, 17004, 278, 8900, 534, 1975, 10331, 304, 367, 901, 13, 13844, 297, 7786, 1602, 3076, 1135, 297, 29871, 29946, 29900, 29974, 1629, 29899, 1025, 6743, 29892, 2861, 304, 13964, 4586, 373, 13, 735, 1112, 9247, 7200, 23093, 29889, 32001, 320, 26956, 29912, 7999, 324, 10610, 29913, 13, 13, 29905, 463, 29912, 4532, 29913, 13, 29871, 320, 9525, 13, 29871, 320, 7313, 29961, 24049, 29892, 15450, 29922, 29953, 4912, 29871, 29953, 4912, 29871, 29900, 29871, 29900, 29892, 2103, 2013, 16292, 3199, 1491, 1727, 1080, 29899, 2470, 29913, 13, 29871, 320, 6671, 29912, 1576, 320, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 1304, 408, 1737, 324, 10610, 22525, 5003, 32001, 320, 1643, 29912, 1003, 29901, 11526, 1958, 29913, 13, 29905, 355, 29912, 4532, 29913, 13, 13, 2887, 1737, 324, 10610, 22525, 591, 671, 11758, 3186, 12786, 10723, 515, 278, 3303, 18269, 1737, 359, 305, 2004, 2651, 2036, 29912, 348, 29896, 29929, 29929, 29929, 479, 359, 305, 2004, 1836, 32001, 1763, 4772, 2432, 3381, 491, 2919, 10916, 313, 29872, 29889, 29887, 1696, 7551, 470, 12710, 29897, 2629, 11758, 12786, 29892, 591, 19412, 322, 6219, 777, 12786, 2729, 373, 1737, 12122, 23203, 537, 322, 278, 19383, 310, 758, 331, 8946, 16375, 29769, 5680, 29892, 1316, 408, 19182, 4086, 29889, 32001, 320, 29907, 999, 29912, 1003, 29901, 11526, 1958, 29913, 3697, 278, 2186, 1051, 310, 320, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 1304, 408, 1737, 324, 10610, 22525, 297, 445, 6559, 29889, 32001, 1879, 324, 10610, 310, 9063, 15717, 304, 3186, 12786, 3913, 278, 1023, 19595, 653, 13698, 9129, 297, 2651, 2036, 29912, 293, 344, 29899, 344, 275, 29899, 29906, 29900, 29906, 29906, 29899, 26098, 1118, 23359, 337, 13998, 2400, 29889, 32001, 450, 937, 697, 337, 3687, 373, 278, 4234, 775, 2246, 29899, 5563, 5354, 313, 617, 29911, 10249, 29897, 310, 4876, 14157, 23892, 515, 9063, 15562, 29892, 321, 29889, 29887, 1696, 320, 16266, 26139, 1341, 1118, 320, 16266, 26139, 582, 1118, 320, 16266, 26139, 18038, 1118, 2992, 29889, 32001, 1334, 4687, 515, 278, 306, 2190, 29909, 1051, 310, 13548, 2931, 21759, 29911, 10249, 29879, 2651, 2036, 29912, 6011, 29899, 29883, 312, 430, 29913, 322, 7522, 20545, 1269, 6590, 20123, 304, 263, 3646, 3186, 5120, 29889, 32001, 450, 1473, 1737, 324, 10610, 11043, 3913, 278, 17998, 9210, 310, 9063, 5335, 342, 15092, 313, 29872, 29889, 29887, 1696, 17998, 29899, 29900, 29945, 29901, 29900, 29900, 29897, 322, 4148, 2983, 304, 8161, 278, 1556, 5517, 3186, 5120, 310, 278, 9063, 4148, 29889, 32001, 1152, 1269, 17998, 9210, 591, 8161, 263, 1051, 310, 15878, 7600, 313, 13509, 29892, 2106, 29892, 470, 14278, 20123, 29897, 297, 278, 3186, 393, 29892, 472, 278, 931, 310, 393, 9063, 29892, 750, 393, 17998, 9210, 29936, 9063, 931, 338, 1820, 1244, 29892, 408, 4234, 17998, 1283, 7224, 13100, 975, 931, 2861, 304, 29431, 3620, 29889, 32001, 1763, 1207, 445, 3683, 3381, 591, 671, 278, 306, 2190, 29909, 931, 10640, 2566, 2651, 2036, 29912, 17559, 1272, 1836, 32001, 1987, 591, 3566, 304, 1269, 2058, 263, 8158, 393, 4332, 1973, 278, 4188, 22342, 393, 263, 2183, 4148, 1024, 338, 17443, 310, 372, 29889, 32001, 1763, 445, 1095, 591, 671, 278, 28297, 29890, 15451, 8783, 310, 278, 29511, 310, 278, 1556, 3619, 937, 322, 3942, 2983, 607, 29892, 439, 11427, 515, 2651, 2036, 29912, 1079, 29890, 799, 29899, 7039, 6177, 2802, 277, 12181, 4954, 16123, 2247, 278, 26368, 5528, 5084, 310, 363, 264, 1280, 322, 269, 595, 1280, 7371, 515, 263, 2566, 310, 320, 1949, 29912, 29946, 29871, 29900, 29946, 29946, 29871, 29945, 29946, 29953, 29871, 29929, 29941, 29947, 29913, 2305, 313, 29945, 29945, 29889, 29945, 8958, 310, 8471, 2305, 297, 29871, 29906, 29900, 29896, 29946, 467, 32001, 1094, 310, 3839, 29871, 29906, 29900, 29896, 29929, 372, 18469, 320, 1949, 29912, 29906, 29955, 29871, 29953, 29953, 29906, 29871, 29947, 29900, 29896, 29913, 363, 264, 1280, 322, 320, 1949, 29912, 29906, 29955, 29871, 29906, 29900, 29953, 29871, 29947, 29906, 29896, 29913, 269, 595, 1280, 297, 29871, 29906, 29941, 29953, 24894, 8977, 1080, 29889, 4907, 29913, 32001, 1094, 297, 1749, 8783, 15717, 526, 2989, 1024, 6031, 313, 29878, 1624, 1135, 6219, 491, 937, 29914, 11922, 1024, 511, 591, 937, 5993, 675, 2983, 313, 1609, 1999, 1331, 322, 1206, 3620, 29897, 322, 769, 16280, 5375, 18897, 297, 1716, 937, 322, 3942, 2983, 10868, 8857, 29889, 32001, 1152, 1269, 1543, 1476, 297, 1024, 8857, 591, 22932, 278, 2058, 4665, 29905, 6661, 1333, 24212, 8875, 491, 278, 1024, 10868, 304, 4017, 263, 5645, 393, 338, 29839, 304, 278, 1353, 310, 12407, 24638, 393, 1024, 313, 6979, 29897, 297, 278, 2702, 2058, 29889, 32001, 320, 6661, 1333, 300, 1062, 29912, 1762, 4017, 4665, 2025, 1338, 5634, 294, 278, 17837, 310, 4954, 6689, 4907, 338, 25745, 23724, 29901, 2989, 10916, 325, 13217, 29879, 29399, 310, 2919, 10916, 805, 9450, 2999, 931, 29920, 2873, 5634, 705, 671, 263, 29544, 310, 7601, 8974, 313, 29872, 29889, 29887, 1696, 5874, 28007, 511, 322, 1661, 29899, 16072, 6743, 313, 29872, 29889, 29887, 1696, 14109, 7456, 467, 29913, 32001, 1334, 2533, 445, 4377, 363, 599, 3161, 304, 4017, 263, 2058, 8158, 29892, 17140, 701, 411, 263, 1051, 310, 779, 6990, 29938, 6689, 29892, 8158, 4535, 5854, 29938, 11000, 29889, 32001, 1334, 769, 8877, 445, 1051, 491, 278, 3186, 5120, 393, 263, 2058, 14393, 304, 322, 2533, 278, 8158, 363, 599, 278, 7600, 297, 1269, 5120, 304, 4017, 385, 12463, 8158, 29892, 6590, 304, 278, 4188, 22342, 393, 278, 9063, 14393, 304, 263, 2183, 3186, 5120, 29889, 32001, 1334, 3566, 278, 6257, 9063, 408, 6421, 515, 278, 3186, 5120, 411, 278, 9939, 8158, 29889, 32001, 450, 4876, 29899, 6707, 11043, 9378, 414, 515, 278, 9078, 322, 443, 5521, 8362, 671, 310, 21759, 29911, 10249, 29879, 29901, 1556, 18777, 671, 10035, 323, 10249, 29879, 1316, 408, 320, 16266, 26139, 510, 1118, 320, 16266, 26139, 990, 1118, 470, 320, 16266, 26139, 1212, 1836, 32001, 12808, 445, 947, 451, 3799, 26018, 4822, 20542, 29901, 3148, 29899, 6707, 18777, 29892, 363, 1342, 29892, 671, 278, 320, 16266, 26139, 375, 29913, 21759, 29911, 10249, 1568, 901, 269, 2495, 290, 368, 1135, 1009, 7824, 6795, 20895, 29889, 32001, 1551, 278, 916, 1361, 278, 9210, 29914, 978, 29899, 6707, 11043, 337, 3687, 373, 278, 17998, 9210, 310, 278, 9063, 5335, 342, 15092, 29889, 32001, 16809, 304, 5780, 22920, 373, 13897, 731, 14340, 29892, 263, 2919, 1353, 310, 25741, 297, 278, 8783, 756, 385, 17998, 9210, 5186, 304, 5225, 29889, 32001, 910, 6602, 29879, 3109, 7786, 25741, 3441, 14573, 5261, 1169, 29911, 29999, 29999, 13985, 1349, 681, 392, 27418, 6478, 29914, 310, 29871, 29906, 29900, 29906, 29900, 29879, 25741, 505, 263, 5225, 9210, 29897, 1135, 9642, 6743, 3441, 14573, 5261, 1169, 29911, 29999, 29999, 13985, 1349, 681, 392, 29914, 297, 29871, 29906, 29900, 29900, 29900, 467, 32001, 1094, 263, 1121, 278, 9210, 29914, 978, 29899, 6707, 11043, 1033, 1095, 701, 6459, 292, 263, 2919, 6232, 310, 9642, 25741, 408, 4148, 287, 491, 11715, 18777, 29892, 322, 304, 263, 3109, 261, 15834, 4092, 550, 29889, 32001, 1763, 6795, 1438, 5626, 591, 14405, 278, 1023, 1737, 324, 10610, 13698, 4208, 491, 15399, 278, 9210, 29914, 978, 29899, 6707, 13698, 304, 599, 25741, 411, 263, 1661, 29899, 9171, 17998, 9210, 29892, 322, 278, 4876, 29899, 6707, 373, 304, 599, 916, 25741, 29889, 32001, 320, 2042, 29912, 12191, 322, 8565, 1558, 291, 29913, 13, 29905, 1643, 29912, 3471, 29901, 9902, 29913, 13, 13, 29905, 463, 29912, 4532, 4044, 13, 29871, 320, 9525, 13, 29871, 320, 7313, 29961, 2103, 2013, 16292, 3199, 1429, 287, 29889, 5140, 29913, 13, 29871, 320, 6671, 29912, 29934, 20819, 310, 25741, 313, 27215, 29897, 322, 6136, 15717, 313, 22503, 29897, 491, 3186, 10640, 975, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29900, 3785, 5003, 32001, 320, 9868, 29961, 14732, 3199, 7264, 287, 2594, 8727, 6445, 278, 3186, 10640, 364, 2219, 359, 363, 25741, 322, 15717, 975, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29900, 3785, 5003, 32001, 320, 1643, 29912, 1003, 29901, 9902, 29913, 13, 29905, 355, 29912, 4532, 4044, 13, 13, 13, 29871, 13, 1762, 1234, 320, 25029, 29912, 29878, 29939, 29901, 479, 397, 24974, 29913, 591, 22229, 278, 1353, 310, 25741, 322, 8359, 15717, 639, 1629, 322, 639, 3186, 10640, 29889, 32001, 1334, 2198, 278, 7625, 2582, 297, 320, 29907, 999, 29912, 1003, 29901, 9902, 29913, 408, 1023, 5096, 287, 2594, 24469, 29892, 6445, 1629, 368, 2867, 3204, 29879, 363, 25741, 322, 15717, 8307, 29889, 32001, 7569, 2594, 11524, 263, 1629, 322, 338, 8877, 287, 297, 269, 29399, 6445, 278, 9063, 29914, 8921, 11959, 363, 1269, 310, 278, 3186, 12786, 310, 320, 29907, 999, 29912, 1003, 29901, 11526, 1958, 29913, 297, 393, 1629, 29889, 32001, 1763, 4772, 714, 27801, 2861, 304, 805, 272, 26538, 17737, 29560, 29892, 297, 278, 4148, 8727, 591, 871, 2050, 15717, 2534, 26869, 472, 3203, 29871, 29945, 25741, 297, 263, 2183, 1629, 29889, 32001, 5806, 5366, 1747, 534, 1975, 297, 278, 24469, 6456, 393, 278, 3001, 3694, 310, 25741, 322, 15717, 6548, 18709, 9247, 975, 931, 29889, 32001, 10133, 363, 278, 937, 2440, 297, 278, 24469, 29892, 278, 1353, 310, 848, 3291, 297, 777, 3186, 12786, 508, 367, 14154, 2319, 29892, 411, 8178, 27721, 373, 278, 25806, 310, 534, 1975, 29889, 32001, 320, 26956, 29912, 7999, 12122, 6894, 537, 975, 931, 29913, 13, 13, 3563, 497, 29892, 278, 2498, 534, 355, 5692, 304, 367, 393, 278, 320, 4534, 29912, 479, 12122, 6894, 537, 297, 970, 775, 338, 10231, 6177, 4644, 6813, 322, 4092, 5136, 630, 1009, 4954, 24130, 749, 4907, 2745, 278, 7256, 310, 278, 29871, 29929, 29900], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [1, 320, 2042, 29912, 25898, 29913, 13, 29905, 1643, 29912, 3471, 29901, 23333, 29913, 13, 13, 29905, 7278, 29912, 29954, 1581, 6894, 537, 1118, 470, 901, 4049, 967, 10225, 727, 974, 29892, 4249, 27138, 304, 13, 20415, 5849, 14188, 756, 1063, 26606, 12399, 297, 7786, 2440, 29889, 32001, 512, 13, 1595, 16311, 29892, 278, 10122, 310, 29892, 9545, 310, 29892, 322, 6795, 1004, 25414, 363, 320, 7278, 29912, 26098, 13, 29871, 24003, 29913, 297, 12362, 29914, 6585, 7562, 18540, 313, 5800, 1799, 29897, 505, 4520, 263, 3287, 310, 8570, 13, 957, 278, 4940, 316, 6332, 2651, 2036, 29912, 29881, 16093, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 3855, 5871, 29906, 29900, 29896, 29900, 29895, 311, 29893, 2770, 29892, 13, 29871, 1055, 29888, 375, 29906, 29900, 29896, 29906, 5041, 267, 29892, 413, 434, 305, 1358, 29906, 29900, 29896, 29906, 26098, 29888, 2209, 29892, 19723, 5475, 4979, 29906, 29900, 29896, 29946, 26098, 29892, 13, 29871, 697, 309, 29906, 29900, 29896, 29953, 311, 5365, 550, 332, 6950, 29892, 696, 7586, 29906, 29900, 29896, 29953, 29893, 2770, 7192, 2209, 29892, 1935, 15044, 29906, 29900, 29896, 29955, 26098, 29892, 13, 29871, 503, 562, 305, 3350, 492, 29906, 29900, 29906, 29896, 26098, 1836, 32001, 320, 7278, 29912, 7999, 12122, 6894, 537, 29913, 338, 373, 278, 916, 1361, 278, 13, 14380, 310, 6894, 537, 393, 380, 1567, 515, 27138, 297, 777, 5534, 6354, 6421, 13, 3166, 1422, 3186, 12786, 322, 4185, 1973, 29889, 32001, 26304, 6894, 537, 297, 18322, 1799, 756, 4520, 13774, 2217, 8570, 297, 21344, 368, 13, 13129, 29889, 32001, 512, 3153, 29892, 1550, 3031, 979, 18994, 29899, 6707, 322, 13, 3149, 29899, 262, 29899, 2230, 18350, 29899, 7052, 11898, 310, 278, 1737, 12122, 1677, 1144, 310, 18322, 1799, 13, 21570, 29560, 1863, 2651, 2036, 29912, 12443, 10578, 29906, 29900, 29900, 29945, 5062, 11235, 29892, 13260, 333, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 13, 29871, 2594, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29892, 1850, 29882, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29892, 696, 7586, 29906, 29900, 29896, 29946, 7610, 345, 2941, 271, 24541, 29892, 13, 29871, 281, 496, 29879, 29906, 29900, 29906, 29896, 2209, 479, 5275, 1118, 2919, 29899, 7052, 25579, 979, 11898, 310, 278, 1737, 12122, 13, 12574, 310, 18322, 1799, 17737, 29560, 526, 1603, 10225, 292, 29889, 32001, 10506, 263, 4323, 23378, 13, 18609, 2133, 723, 367, 5407, 304, 1871, 1602, 12112, 4475, 304, 5534, 13, 25431, 10907, 2651, 2036, 29912, 2276, 5824, 19982, 29906, 29900, 29900, 29955, 23705, 1338, 29893, 996, 29913, 322, 298, 8491, 16650, 583, 297, 278, 13, 19678, 15483, 313, 1806, 29897, 9999, 29892, 408, 1532, 408, 29126, 2114, 950, 2472, 13, 517, 278, 2553, 1078, 373, 278, 17407, 10879, 322, 5374, 3002, 310, 18322, 1799, 2820, 278, 3186, 29889, 32001, 320, 26956, 29912, 1323, 3224, 29879, 29913, 13, 13, 3047, 445, 664, 591, 29126, 304, 3802, 445, 17261, 491, 7512, 292, 320, 4534, 29912, 1552, 937, 13, 29871, 25579, 979, 6559, 310, 278, 1737, 12122, 3978, 310, 17737, 29560, 304, 970, 775, 13, 29871, 975, 29871, 29945, 29900, 2440, 5003, 32001, 26321, 29892, 591, 3867, 263, 758, 2576, 3821, 1234, 304, 278, 13, 23031, 292, 5925, 1139, 29901, 13, 29905, 463, 29912, 690, 2842, 12470, 29913, 13, 29871, 3645, 607, 3186, 12786, 437, 15717, 310, 970, 368, 3625, 25741, 2041, 515, 13, 29871, 322, 920, 756, 372, 3939, 975, 278, 4940, 29871, 29945, 29900, 2440, 29973, 32001, 320, 1643, 29912, 29878, 29939, 29901, 479, 397, 24974, 29913, 13, 29905, 355, 29912, 690, 2842, 12470, 29913, 13, 4806, 671, 408, 8783, 278, 320, 23066, 29950, 29914, 18871, 2651, 2036, 29912, 29879, 1332, 29875, 4569, 29906, 29900, 29896, 29955, 29913, 322, 27599, 515, 372, 13, 29906, 29889, 29906, 24464, 29905, 13007, 25741, 3190, 2347, 515, 29871, 29896, 29953, 29900, 7284, 29905, 13007, 9279, 322, 4148, 287, 491, 13, 29946, 29941, 7284, 29905, 13007, 15717, 2645, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29896, 931, 3785, 29889, 32001, 1334, 1737, 324, 542, 403, 18777, 304, 13, 29905, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 29892, 773, 408, 18470, 4876, 4234, 775, 2246, 29899, 5563, 21904, 313, 617, 29911, 10249, 29879, 29897, 322, 29871, 13, 8921, 313, 4102, 29914, 4230, 29897, 2983, 9401, 411, 1024, 18822, 2820, 278, 3186, 29892, 322, 17998, 1283, 7224, 29871, 13, 1195, 287, 515, 9063, 15562, 29889, 32001, 1334, 1284, 10757, 310, 278, 4688, 8022, 749, 310, 4644, 6813, 297, 1722, 2752, 13, 20415, 29892, 2678, 8772, 491, 4092, 29889, 32001, 2860, 393, 3785, 29892, 278, 1737, 12122, 6894, 537, 29871, 13, 262, 970, 775, 756, 1063, 21003, 10231, 29889, 32001, 1334, 884, 12439, 8018, 15839, 528, 17741, 13, 12817, 304, 278, 1095, 310, 278, 8291, 6415, 29129, 322, 278, 7910, 310, 14137, 4631, 4135, 297, 13, 23369, 1705, 322, 4275, 14325, 29892, 408, 1532, 408, 310, 2545, 1664, 17292, 28342, 763, 25539, 1608, 322, 13, 25719, 10298, 4822, 10916, 313, 6727, 16783, 29914, 331, 16783, 467, 32001, 320, 26956, 29912, 1469, 20847, 3097, 5003, 32001, 319, 1634, 1414, 3577, 363, 445, 5650, 338, 3625, 515, 796, 264, 8144, 472, 13, 29905, 2271, 29912, 991, 597, 1867, 29875, 29889, 990, 29914, 29896, 29900, 29889, 29945, 29906, 29947, 29896, 29914, 2256, 8144, 29889, 29953, 29941, 29929, 29900, 29941, 29945, 29945, 29913, 2651, 2036, 29912, 3445, 1414, 29899, 5113, 1836, 32001, 320, 2042, 29912, 9662, 630, 5244, 29913, 13, 29905, 1643, 29912, 3471, 29901, 12817, 29913, 13, 13, 29933, 720, 4688, 322, 7786, 1736, 2651, 2036, 29912, 12443, 10578, 29906, 29900, 29900, 29945, 5062, 11235, 29892, 13260, 333, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 13, 29871, 696, 7586, 29906, 29900, 29896, 29946, 7610, 345, 2941, 271, 24541, 29892, 697, 309, 29906, 29900, 29896, 29953, 311, 5365, 550, 332, 6950, 29913, 505, 2931, 1891, 278, 13, 479, 5275, 310, 12362, 29914, 6585, 7562, 18540, 313, 5800, 1799, 29897, 773, 320, 7278, 29912, 6734, 26946, 952, 1118, 13, 4716, 3867, 1880, 29899, 29567, 6089, 541, 526, 9078, 297, 2159, 313, 29906, 29899, 29945, 5940, 29968, 18777, 29897, 13, 392, 508, 367, 4768, 1463, 491, 5221, 424, 23460, 29889, 32001, 512, 29871, 29906, 29900, 29900, 29947, 2261, 801, 2681, 634, 394, 8770, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29913, 18043, 263, 3031, 979, 13, 16961, 29899, 7052, 313, 1454, 278, 931, 29897, 6559, 373, 18322, 1799, 320, 7278, 29912, 479, 5275, 773, 1375, 292, 7047, 13, 29871, 28914, 313, 4345, 29934, 29897, 13698, 1836, 32001, 2688, 29537, 287, 278, 3978, 310, 29871, 29896, 5940, 29924, 17737, 29560, 13, 4746, 278, 7562, 2831, 479, 1404, 2566, 322, 611, 6504, 1051, 3190, 3145, 975, 278, 13, 29896, 29929, 29929, 29929, 489, 29906, 29900, 29900, 29945, 3785, 29892, 773, 408, 18470, 2472, 2788, 304, 1749, 29879, 29901, 4876, 21904, 13, 392, 17998, 1283, 7224, 29889, 32001, 450, 12399, 3785, 313, 29955, 2440, 29897, 297, 2651, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29913, 338, 20511, 1135, 29871, 13, 5816, 338, 12399, 297, 278, 2198, 5650, 313, 29945, 29900, 2440, 29897, 322, 278, 848, 8974, 526, 29871, 13, 27489, 873, 1422, 29936, 411, 393, 297, 3458, 29892, 1749, 2582, 1510, 263, 10029, 7200, 14978, 310, 29871, 13, 15654, 273, 325, 13217, 29940, 2072, 3082, 20706, 29889, 32001, 7280, 29190, 936, 664, 515, 29871, 29906, 29900, 29896, 29900, 491, 14619, 29882, 371, 4099, 29894, 322, 13, 29950, 309, 1372, 2651, 2036, 29912, 29873, 19426, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29913, 4023, 29894, 2868, 1583, 29899, 7099, 433, 1127, 1737, 12122, 13, 2029, 800, 310, 25492, 15303, 8304, 3598, 1494, 1009, 12368, 29892, 13, 15914, 292, 2472, 363, 779, 14850, 4535, 29892, 29955, 29900, 5940, 29968, 25492, 4160, 29889, 32001, 319, 1407, 7786, 13, 1287, 2651, 2036, 29912, 21686, 29879, 29906, 29900, 29906, 29896, 2209, 479, 5275, 29913, 491, 399, 496, 29879, 634, 394, 13217, 5349, 1737, 324, 542, 630, 4203, 263, 7284, 13, 28712, 16046, 4160, 29892, 2534, 26869, 472, 3203, 29871, 29896, 29900, 29900, 25741, 1269, 29892, 322, 1058, 13, 1311, 29899, 7099, 8663, 14354, 373, 1009, 25492, 28723, 29889, 32001, 5806, 278, 6559, 338, 13, 3149, 29899, 262, 29899, 2230, 408, 310, 29871, 29906, 29900, 29906, 29896, 29892, 278, 15717, 7252, 1009, 1284, 886, 13, 351, 475, 303, 2651, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29892, 1850, 29882, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29913, 304, 13, 18609, 675, 278, 14675, 310, 18322, 1799, 1737, 5275, 975, 278, 931, 15101, 845, 1862, 4586, 491, 13, 1552, 2211, 11898, 29889, 32001, 3831, 1965, 411, 3517, 29190, 936, 1736, 29892, 1749, 6559, 338, 1568, 7200, 6287, 5634, 29882, 5555, 13, 7054, 12339, 287, 29871, 29946, 29941, 7284, 29905, 13007, 15717, 310, 29871, 29906, 29889, 29906, 24464, 29905, 13007, 25741, 515, 29871, 29896, 29953, 29900, 7284, 29905, 13007, 13, 16418, 5634, 5426, 11267, 979, 975, 29871, 29945, 29900, 2440, 310, 970, 775, 20706, 3265, 1135, 13, 3149, 297, 931, 29892, 322, 884, 901, 2691, 29899, 3874, 1312, 313, 2541, 1629, 29899, 1609, 29899, 6360, 3803, 1070, 537, 975, 13, 1552, 8900, 3785, 467, 32001, 8108, 1189, 1711, 29892, 1749, 6559, 337, 3687, 373, 10079, 11264, 13, 3924, 313, 8257, 29903, 29897, 9063, 848, 3265, 1135, 7481, 29899, 7099, 433, 1127, 4423, 2472, 29889, 32001, 5901, 1736, 5634, 262, 3153, 278, 664, 491, 8432, 2651, 2036, 29912, 18386, 709, 29906, 29900, 29896, 29941, 2209, 29881, 24974, 29913, 13, 392, 29892, 901, 10325, 29892, 390, 579, 468, 29875, 634, 394, 8770, 2036, 29912, 29878, 579, 468, 29875, 29906, 29900, 29896, 29953, 479, 711, 3173, 29892, 13, 29871, 364, 579, 468, 29875, 29906, 29900, 29896, 29947, 479, 711, 3173, 29892, 544, 1648, 29906, 29900, 29906, 29896, 479, 468, 1581, 29881, 24974, 29913, 5634, 17532, 12399, 1737, 12122, 13, 29905, 7278, 29912, 29881, 24974, 322, 24003, 1118, 474, 29889, 29872, 1696, 278, 15834, 304, 607, 278, 3978, 310, 18322, 1799, 13, 17426, 6602, 1009, 11465, 1230, 14137, 14188, 29889, 32001, 512, 445, 664, 591, 2931, 1891, 1737, 12122, 6894, 537, 297, 970, 775, 363, 278, 937, 13, 2230, 472, 445, 6287, 29892, 1716, 297, 4958, 310, 17737, 29560, 322, 15500, 3785, 29889, 32001, 1334, 437, 13, 1333, 22002, 280, 278, 24003, 10696, 29892, 541, 3867, 29190, 936, 848, 322, 1284, 886, 393, 508, 367, 13, 280, 369, 4063, 304, 393, 1095, 408, 5434, 664, 29889, 32001, 320, 7278, 29912, 12756, 7047, 21639, 29913, 2651, 2036, 29912, 2276, 5824, 19982, 29906, 29900, 29900, 29955, 23705, 1338, 29893, 996, 29913, 338, 278, 13, 1491, 29899, 2671, 310, 7047, 21639, 393, 756, 29537, 287, 278, 18066, 267, 310, 21640, 13, 6734, 24771, 13149, 635, 29892, 3704, 278, 2702, 5932, 310, 920, 304, 5376, 13, 2541, 1737, 12122, 6894, 537, 2651, 2036, 29912, 13376, 303, 456, 29906, 29900, 29900, 29953, 10945, 3359, 29892, 1424, 29440, 29906, 29900, 29896, 29946, 23027, 5933, 1836, 32001, 3826, 3076, 2678, 278, 2198, 6559, 8128, 10757, 393, 508, 367, 1304, 29892, 297, 278, 13, 14940, 1206, 310, 970, 775, 322, 472, 263, 1407, 2919, 6287, 29892, 304, 11539, 607, 13, 14032, 4637, 310, 5534, 7047, 21639, 505, 9820, 484, 15774, 29889, 32001, 320, 2042, 29912, 4062, 3002, 29913, 13, 29905, 1643, 29912, 3471, 29901, 5696, 29913, 13, 13, 13, 29905, 1482, 361, 29905, 361, 29887, 798, 386, 1003, 29871, 320, 29887, 798, 386, 1003, 3009, 13, 29905, 361, 29887, 798, 386, 1003, 13, 29905, 463, 29912, 4532, 29913, 13, 29871, 320, 7313, 29961, 2103, 2013, 4914, 2103, 3199, 6360, 368, 29899, 2055, 1169, 29913, 13, 29871, 320, 6671, 29912, 12883, 368, 970, 25741, 975, 931, 313, 1188, 6287, 467, 13, 29913, 32001, 320, 1643, 29912, 1003, 29901, 29887, 798, 386, 29913, 13, 29905, 355, 29912, 4532, 29913, 13, 29905, 7241, 13, 13, 29905, 26956, 29912, 16390, 24541, 29913, 13, 13, 4806, 27387, 515, 320, 23066, 29950, 29914, 2651, 2036, 29912, 29879, 1332, 29899, 1516, 29878, 29906, 29900, 29896, 29929, 29899, 24713, 29913, 599, 25741, 3190, 2347, 2745, 320, 25832, 1964, 579, 1523, 2415, 2539, 6294, 32001, 2688, 5253, 304, 320, 14573, 5261, 1169, 22131, 29914, 25741, 29892, 5412, 491, 317, 15715, 29896, 15882, 29892, 4023, 29894, 2868, 515, 320, 25832, 1299, 7288, 5261, 1169, 797, 7068, 29914, 970, 9279, 6421, 515, 4655, 5849, 363, 2710, 313, 28712, 16046, 29892, 11786, 28632, 29892, 2992, 1846, 32001, 322, 3577, 28914, 313, 10251, 713, 29892, 10772, 2227, 29892, 405, 13427, 29892, 2992, 6250, 32001, 1876, 1169, 297, 278, 8783, 526, 491, 320, 25832, 6344, 2806, 943, 22131, 29914, 15717, 29892, 5412, 491, 779, 6990, 29938, 978, 29892, 4876, 4535, 5854, 29938, 11000, 29889, 32001, 450, 8783, 2996, 408, 1023, 1104, 1288, 6131, 29892, 697, 363, 25741, 322, 697, 363, 15717, 29892, 411, 278, 4642, 29371, 278, 7480, 3025, 263, 9117, 1820, 29889, 32001, 320, 361, 5426, 13, 9760, 1948, 297, 278, 9063, 1591, 3743, 278, 1494, 4235, 29901, 9063, 317, 15715, 29896, 15882, 29892, 4148, 322, 844, 5171, 5335, 342, 15092, 29892, 4148, 322, 844, 5171, 2893, 14903, 313, 20275, 16750, 278, 4148, 1591, 467, 32001, 450, 21578, 1546, 9063, 15717, 322, 9063, 2153, 2041, 515, 11786, 29892, 607, 6511, 304, 9063, 263, 1735, 4148, 287, 491, 4856, 1683, 29889, 32001, 1152, 445, 6559, 591, 21309, 373, 15717, 322, 17262, 9063, 2153, 29892, 408, 278, 4328, 1546, 278, 1023, 338, 451, 8018, 363, 1749, 5925, 5155, 322, 278, 5253, 310, 25741, 411, 263, 844, 5171, 916, 1135, 967, 4148, 338, 3480, 3473, 1821, 29889, 32001, 320, 7241, 13, 2831, 1269, 6251, 297, 278, 4148, 1591, 591, 505, 4148, 2989, 1024, 322, 4876, 408, 1023, 5004, 6031, 310, 10650, 6262, 29889, 32001, 1334, 6206, 2411, 15273, 1821, 470, 18325, 519, 2983, 393, 29901, 526, 451, 1602, 397, 519, 408, 18351, 29899, 29947, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12283, 7099, 397, 519, 29914, 4148, 2983, 6206, 511, 526, 4876, 14157, 2012, 310, 2983, 3441, 25832, 6344, 2806, 943, 29934, 29885, 9823, 29914, 4954, 7039, 4907, 511, 5718, 310, 871, 9654, 4890, 3441, 25832, 6344, 2806, 943, 29934, 29885, 10358, 804, 23201, 1712, 901, 1135, 29871, 29896, 29900, 8958, 1661, 29899, 1026, 2153, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12283, 15670, 23201, 526, 5520, 1135, 29871, 29896, 29900, 29900, 4890, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12229, 549, 12495, 32001, 2860, 21166, 29892, 1048, 320, 25832, 6344, 2806, 943, 29925, 15273, 1821, 2052, 307, 29916, 29914, 15717, 3441, 25832, 6344, 2806, 943, 29925, 15273, 1821, 29925, 312, 29914, 310, 278, 2847, 8783, 29897, 9488, 363, 4340, 7418, 29889, 32001, 3940, 393, 278, 5253, 310, 970, 775, 25741, 313, 392, 15717, 29897, 11122, 297, 278, 13, 11228, 8783, 25088, 18709, 9247, 975, 13, 2230, 2651, 2036, 29912, 29879, 1332, 29899, 771, 854, 749, 29899, 331, 344, 1012, 361, 29887, 798, 386, 1003, 29892, 408, 4318, 363, 25741, 297, 13, 29905, 29907, 999, 29912, 1003, 29901, 29887, 798, 386, 1012, 2870, 29901, 515, 395, 29896, 29900, 29985, 29946, 29938, 25741, 297, 29871, 29896, 29929, 29955, 29896, 29892, 304, 395, 29896, 29900, 29985, 29953, 29938, 297, 29871, 29896, 29929, 29929, 29947, 29892, 304, 13, 284, 3242, 395, 29896, 29900, 29985, 29929, 29938, 297, 29871, 29906, 29900, 29906, 29900, 29905, 7241, 29889, 32001, 1094, 263, 17004, 278, 8900, 534, 1975, 10331, 304, 367, 901, 13, 13844, 297, 7786, 1602, 3076, 1135, 297, 29871, 29946, 29900, 29974, 1629, 29899, 1025, 6743, 29892, 2861, 304, 13964, 4586, 373, 13, 735, 1112, 9247, 7200, 23093, 29889, 32001, 320, 26956, 29912, 7999, 324, 10610, 29913, 13, 13, 29905, 463, 29912, 4532, 29913, 13, 29871, 320, 9525, 13, 29871, 320, 7313, 29961, 24049, 29892, 15450, 29922, 29953, 4912, 29871, 29953, 4912, 29871, 29900, 29871, 29900, 29892, 2103, 2013, 16292, 3199, 1491, 1727, 1080, 29899, 2470, 29913, 13, 29871, 320, 6671, 29912, 1576, 320, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 1304, 408, 1737, 324, 10610, 22525, 5003, 32001, 320, 1643, 29912, 1003, 29901, 11526, 1958, 29913, 13, 29905, 355, 29912, 4532, 29913, 13, 13, 2887, 1737, 324, 10610, 22525, 591, 671, 11758, 3186, 12786, 10723, 515, 278, 3303, 18269, 1737, 359, 305, 2004, 2651, 2036, 29912, 348, 29896, 29929, 29929, 29929, 479, 359, 305, 2004, 1836, 32001, 1763, 4772, 2432, 3381, 491, 2919, 10916, 313, 29872, 29889, 29887, 1696, 7551, 470, 12710, 29897, 2629, 11758, 12786, 29892, 591, 19412, 322, 6219, 777, 12786, 2729, 373, 1737, 12122, 23203, 537, 322, 278, 19383, 310, 758, 331, 8946, 16375, 29769, 5680, 29892, 1316, 408, 19182, 4086, 29889, 32001, 320, 29907, 999, 29912, 1003, 29901, 11526, 1958, 29913, 3697, 278, 2186, 1051, 310, 320, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 1304, 408, 1737, 324, 10610, 22525, 297, 445, 6559, 29889, 32001, 1879, 324, 10610, 310, 9063, 15717, 304, 3186, 12786, 3913, 278, 1023, 19595, 653, 13698, 9129, 297, 2651, 2036, 29912, 293, 344, 29899, 344, 275, 29899, 29906, 29900, 29906, 29906, 29899, 26098, 1118, 23359, 337, 13998, 2400, 29889, 32001, 450, 937, 697, 337, 3687, 373, 278, 4234, 775, 2246, 29899, 5563, 5354, 313, 617, 29911, 10249, 29897, 310, 4876, 14157, 23892, 515, 9063, 15562, 29892, 321, 29889, 29887, 1696, 320, 16266, 26139, 1341, 1118, 320, 16266, 26139, 582, 1118, 320, 16266, 26139, 18038, 1118, 2992, 29889, 32001, 1334, 4687, 515, 278, 306, 2190, 29909, 1051, 310, 13548, 2931, 21759, 29911, 10249, 29879, 2651, 2036, 29912, 6011, 29899, 29883, 312, 430, 29913, 322, 7522, 20545, 1269, 6590, 20123, 304, 263, 3646, 3186, 5120, 29889, 32001, 450, 1473, 1737, 324, 10610, 11043, 3913, 278, 17998, 9210, 310, 9063, 5335, 342, 15092, 313, 29872, 29889, 29887, 1696, 17998, 29899, 29900, 29945, 29901, 29900, 29900, 29897, 322, 4148, 2983, 304, 8161, 278, 1556, 5517, 3186, 5120, 310, 278, 9063, 4148, 29889, 32001, 1152, 1269, 17998, 9210, 591, 8161, 263, 1051, 310, 15878, 7600, 313, 13509, 29892, 2106, 29892, 470, 14278, 20123, 29897, 297, 278, 3186, 393, 29892, 472, 278, 931, 310, 393, 9063, 29892, 750, 393, 17998, 9210, 29936, 9063, 931, 338, 1820, 1244, 29892, 408, 4234, 17998, 1283, 7224, 13100, 975, 931, 2861, 304, 29431, 3620, 29889, 32001, 1763, 1207, 445, 3683, 3381, 591, 671, 278, 306, 2190, 29909, 931, 10640, 2566, 2651, 2036, 29912, 17559, 1272, 1836, 32001, 1987, 591, 3566, 304, 1269, 2058, 263, 8158, 393, 4332, 1973, 278, 4188, 22342, 393, 263, 2183, 4148, 1024, 338, 17443, 310, 372, 29889, 32001, 1763, 445, 1095, 591, 671, 278, 28297, 29890, 15451, 8783, 310, 278, 29511, 310, 278, 1556, 3619, 937, 322, 3942, 2983, 607, 29892, 439, 11427, 515, 2651, 2036, 29912, 1079, 29890, 799, 29899, 7039, 6177, 2802, 277, 12181, 4954, 16123, 2247, 278, 26368, 5528, 5084, 310, 363, 264, 1280, 322, 269, 595, 1280, 7371, 515, 263, 2566, 310, 320, 1949, 29912, 29946, 29871, 29900, 29946, 29946, 29871, 29945, 29946, 29953, 29871, 29929, 29941, 29947, 29913, 2305, 313, 29945, 29945, 29889, 29945, 8958, 310, 8471, 2305, 297, 29871, 29906, 29900, 29896, 29946, 467, 32001, 1094, 310, 3839, 29871, 29906, 29900, 29896, 29929, 372, 18469, 320, 1949, 29912, 29906, 29955, 29871, 29953, 29953, 29906, 29871, 29947, 29900, 29896, 29913, 363, 264, 1280, 322, 320, 1949, 29912, 29906, 29955, 29871, 29906, 29900, 29953, 29871, 29947, 29906, 29896, 29913, 269, 595, 1280, 297, 29871, 29906, 29941, 29953, 24894, 8977, 1080, 29889, 4907, 29913, 32001, 1094, 297, 1749, 8783, 15717, 526, 2989, 1024, 6031, 313, 29878, 1624, 1135, 6219, 491, 937, 29914, 11922, 1024, 511, 591, 937, 5993, 675, 2983, 313, 1609, 1999, 1331, 322, 1206, 3620, 29897, 322, 769, 16280, 5375, 18897, 297, 1716, 937, 322, 3942, 2983, 10868, 8857, 29889, 32001, 1152, 1269, 1543, 1476, 297, 1024, 8857, 591, 22932, 278, 2058, 4665, 29905, 6661, 1333, 24212, 8875, 491, 278, 1024, 10868, 304, 4017, 263, 5645, 393, 338, 29839, 304, 278, 1353, 310, 12407, 24638, 393, 1024, 313, 6979, 29897, 297, 278, 2702, 2058, 29889, 32001, 320, 6661, 1333, 300, 1062, 29912, 1762, 4017, 4665, 2025, 1338, 5634, 294, 278, 17837, 310, 4954, 6689, 4907, 338, 25745, 23724, 29901, 2989, 10916, 325, 13217, 29879, 29399, 310, 2919, 10916, 805, 9450, 2999, 931, 29920, 2873, 5634, 705, 671, 263, 29544, 310, 7601, 8974, 313, 29872, 29889, 29887, 1696, 5874, 28007, 511, 322, 1661, 29899, 16072, 6743, 313, 29872, 29889, 29887, 1696, 14109, 7456, 467, 29913, 32001, 1334, 2533, 445, 4377, 363, 599, 3161, 304, 4017, 263, 2058, 8158, 29892, 17140, 701, 411, 263, 1051, 310, 779, 6990, 29938, 6689, 29892, 8158, 4535, 5854, 29938, 11000, 29889, 32001, 1334, 769, 8877, 445, 1051, 491, 278, 3186, 5120, 393, 263, 2058, 14393, 304, 322, 2533, 278, 8158, 363, 599, 278, 7600, 297, 1269, 5120, 304, 4017, 385, 12463, 8158, 29892, 6590, 304, 278, 4188, 22342, 393, 278, 9063, 14393, 304, 263, 2183, 3186, 5120, 29889, 32001, 1334, 3566, 278, 6257, 9063, 408, 6421, 515, 278, 3186, 5120, 411, 278, 9939, 8158, 29889, 32001, 450, 4876, 29899, 6707, 11043, 9378, 414, 515, 278, 9078, 322, 443, 5521, 8362, 671, 310, 21759, 29911, 10249, 29879, 29901, 1556, 18777, 671, 10035, 323, 10249, 29879, 1316, 408, 320, 16266, 26139, 510, 1118, 320, 16266, 26139, 990, 1118, 470, 320, 16266, 26139, 1212, 1836, 32001, 12808, 445, 947, 451, 3799, 26018, 4822, 20542, 29901, 3148, 29899, 6707, 18777, 29892, 363, 1342, 29892, 671, 278, 320, 16266, 26139, 375, 29913, 21759, 29911, 10249, 1568, 901, 269, 2495, 290, 368, 1135, 1009, 7824, 6795, 20895, 29889, 32001, 1551, 278, 916, 1361, 278, 9210, 29914, 978, 29899, 6707, 11043, 337, 3687, 373, 278, 17998, 9210, 310, 278, 9063, 5335, 342, 15092, 29889, 32001, 16809, 304, 5780, 22920, 373, 13897, 731, 14340, 29892, 263, 2919, 1353, 310, 25741, 297, 278, 8783, 756, 385, 17998, 9210, 5186, 304, 5225, 29889, 32001, 910, 6602, 29879, 3109, 7786, 25741, 3441, 14573, 5261, 1169, 29911, 29999, 29999, 13985, 1349, 681, 392, 27418, 6478, 29914, 310, 29871, 29906, 29900, 29906, 29900, 29879, 25741, 505, 263, 5225, 9210, 29897, 1135, 9642, 6743, 3441, 14573, 5261, 1169, 29911, 29999, 29999, 13985, 1349, 681, 392, 29914, 297, 29871, 29906, 29900, 29900, 29900, 467, 32001, 1094, 263, 1121, 278, 9210, 29914, 978, 29899, 6707, 11043, 1033, 1095, 701, 6459, 292, 263, 2919, 6232, 310, 9642, 25741, 408, 4148, 287, 491, 11715, 18777, 29892, 322, 304, 263, 3109, 261, 15834, 4092, 550, 29889, 32001, 1763, 6795, 1438, 5626, 591, 14405, 278, 1023, 1737, 324, 10610, 13698, 4208, 491, 15399, 278, 9210, 29914, 978, 29899, 6707, 13698, 304, 599, 25741, 411, 263, 1661, 29899, 9171, 17998, 9210, 29892, 322, 278, 4876, 29899, 6707, 373, 304, 599, 916, 25741, 29889, 32001, 320, 2042, 29912, 12191, 322, 8565, 1558, 291, 29913, 13, 29905, 1643, 29912, 3471, 29901, 9902, 29913, 13, 13, 29905, 463, 29912, 4532, 4044, 13, 29871, 320, 9525, 13, 29871, 320, 7313, 29961, 2103, 2013, 16292, 3199, 1429, 287, 29889, 5140, 29913, 13, 29871, 320, 6671, 29912, 29934, 20819, 310, 25741, 313, 27215, 29897, 322, 6136, 15717, 313, 22503, 29897, 491, 3186, 10640, 975, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29900, 3785, 5003, 32001, 320, 9868, 29961, 14732, 3199, 7264, 287, 2594, 8727, 6445, 278, 3186, 10640, 364, 2219, 359, 363, 25741, 322, 15717, 975, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29900, 3785, 5003, 32001, 320, 1643, 29912, 1003, 29901, 9902, 29913, 13, 29905, 355, 29912, 4532, 4044, 13, 13, 13, 29871, 13, 1762, 1234, 320, 25029, 29912, 29878, 29939, 29901, 479, 397, 24974, 29913, 591, 22229, 278, 1353, 310, 25741, 322, 8359, 15717, 639, 1629, 322, 639, 3186, 10640, 29889, 32001, 1334, 2198, 278, 7625, 2582, 297, 320, 29907, 999, 29912, 1003, 29901, 9902, 29913, 408, 1023, 5096, 287, 2594, 24469, 29892, 6445, 1629, 368, 2867, 3204, 29879, 363, 25741, 322, 15717, 8307, 29889, 32001, 7569, 2594, 11524, 263, 1629, 322, 338, 8877, 287, 297, 269, 29399, 6445, 278, 9063, 29914, 8921, 11959, 363, 1269, 310, 278, 3186, 12786, 310, 320, 29907, 999, 29912, 1003, 29901, 11526, 1958, 29913, 297, 393, 1629, 29889, 32001, 1763, 4772, 714, 27801, 2861, 304, 805, 272, 26538, 17737, 29560, 29892, 297, 278, 4148, 8727, 591, 871, 2050, 15717, 2534, 26869, 472, 3203, 29871, 29945, 25741, 297, 263, 2183, 1629, 29889, 32001, 5806, 5366, 1747, 534, 1975, 297, 278, 24469, 6456, 393, 278, 3001, 3694, 310, 25741, 322, 15717, 6548, 18709, 9247, 975, 931, 29889, 32001, 10133, 363, 278, 937, 2440, 297, 278, 24469, 29892, 278, 1353, 310, 848, 3291, 297, 777, 3186, 12786, 508, 367, 14154, 2319, 29892, 411, 8178, 27721, 373, 278, 25806, 310, 534, 1975, 29889, 32001, 320, 26956, 29912, 7999, 12122, 6894, 537, 975, 931, 29913, 13, 13, 3563, 497, 29892, 278, 2498, 534, 355, 5692, 304, 367, 393, 278, 320, 4534, 29912, 479, 12122, 6894, 537, 297, 970, 775, 338, 10231, 6177, 4644, 6813, 322, 4092, 5136, 630, 1009, 4954, 24130, 749, 4907, 2745, 278, 7256, 310, 278, 29871, 29929, 29900]} {'input_ids': [1, 320, 2042, 29912, 25898, 29913, 13, 29905, 1643, 29912, 3471, 29901, 23333, 29913, 13, 13, 29905, 7278, 29912, 29954, 1581, 6894, 537, 1118, 470, 901, 4049, 967, 10225, 727, 974, 29892, 4249, 27138, 304, 13, 20415, 5849, 14188, 756, 1063, 26606, 12399, 297, 7786, 2440, 29889, 32001, 512, 13, 1595, 16311, 29892, 278, 10122, 310, 29892, 9545, 310, 29892, 322, 6795, 1004, 25414, 363, 320, 7278, 29912, 26098, 13, 29871, 24003, 29913, 297, 12362, 29914, 6585, 7562, 18540, 313, 5800, 1799, 29897, 505, 4520, 263, 3287, 310, 8570, 13, 957, 278, 4940, 316, 6332, 2651, 2036, 29912, 29881, 16093, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 3855, 5871, 29906, 29900, 29896, 29900, 29895, 311, 29893, 2770, 29892, 13, 29871, 1055, 29888, 375, 29906, 29900, 29896, 29906, 5041, 267, 29892, 413, 434, 305, 1358, 29906, 29900, 29896, 29906, 26098, 29888, 2209, 29892, 19723, 5475, 4979, 29906, 29900, 29896, 29946, 26098, 29892, 13, 29871, 697, 309, 29906, 29900, 29896, 29953, 311, 5365, 550, 332, 6950, 29892, 696, 7586, 29906, 29900, 29896, 29953, 29893, 2770, 7192, 2209, 29892, 1935, 15044, 29906, 29900, 29896, 29955, 26098, 29892, 13, 29871, 503, 562, 305, 3350, 492, 29906, 29900, 29906, 29896, 26098, 1836, 32001, 320, 7278, 29912, 7999, 12122, 6894, 537, 29913, 338, 373, 278, 916, 1361, 278, 13, 14380, 310, 6894, 537, 393, 380, 1567, 515, 27138, 297, 777, 5534, 6354, 6421, 13, 3166, 1422, 3186, 12786, 322, 4185, 1973, 29889, 32001, 26304, 6894, 537, 297, 18322, 1799, 756, 4520, 13774, 2217, 8570, 297, 21344, 368, 13, 13129, 29889, 32001, 512, 3153, 29892, 1550, 3031, 979, 18994, 29899, 6707, 322, 13, 3149, 29899, 262, 29899, 2230, 18350, 29899, 7052, 11898, 310, 278, 1737, 12122, 1677, 1144, 310, 18322, 1799, 13, 21570, 29560, 1863, 2651, 2036, 29912, 12443, 10578, 29906, 29900, 29900, 29945, 5062, 11235, 29892, 13260, 333, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 13, 29871, 2594, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29892, 1850, 29882, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29892, 696, 7586, 29906, 29900, 29896, 29946, 7610, 345, 2941, 271, 24541, 29892, 13, 29871, 281, 496, 29879, 29906, 29900, 29906, 29896, 2209, 479, 5275, 1118, 2919, 29899, 7052, 25579, 979, 11898, 310, 278, 1737, 12122, 13, 12574, 310, 18322, 1799, 17737, 29560, 526, 1603, 10225, 292, 29889, 32001, 10506, 263, 4323, 23378, 13, 18609, 2133, 723, 367, 5407, 304, 1871, 1602, 12112, 4475, 304, 5534, 13, 25431, 10907, 2651, 2036, 29912, 2276, 5824, 19982, 29906, 29900, 29900, 29955, 23705, 1338, 29893, 996, 29913, 322, 298, 8491, 16650, 583, 297, 278, 13, 19678, 15483, 313, 1806, 29897, 9999, 29892, 408, 1532, 408, 29126, 2114, 950, 2472, 13, 517, 278, 2553, 1078, 373, 278, 17407, 10879, 322, 5374, 3002, 310, 18322, 1799, 2820, 278, 3186, 29889, 32001, 320, 26956, 29912, 1323, 3224, 29879, 29913, 13, 13, 3047, 445, 664, 591, 29126, 304, 3802, 445, 17261, 491, 7512, 292, 320, 4534, 29912, 1552, 937, 13, 29871, 25579, 979, 6559, 310, 278, 1737, 12122, 3978, 310, 17737, 29560, 304, 970, 775, 13, 29871, 975, 29871, 29945, 29900, 2440, 5003, 32001, 26321, 29892, 591, 3867, 263, 758, 2576, 3821, 1234, 304, 278, 13, 23031, 292, 5925, 1139, 29901, 13, 29905, 463, 29912, 690, 2842, 12470, 29913, 13, 29871, 3645, 607, 3186, 12786, 437, 15717, 310, 970, 368, 3625, 25741, 2041, 515, 13, 29871, 322, 920, 756, 372, 3939, 975, 278, 4940, 29871, 29945, 29900, 2440, 29973, 32001, 320, 1643, 29912, 29878, 29939, 29901, 479, 397, 24974, 29913, 13, 29905, 355, 29912, 690, 2842, 12470, 29913, 13, 4806, 671, 408, 8783, 278, 320, 23066, 29950, 29914, 18871, 2651, 2036, 29912, 29879, 1332, 29875, 4569, 29906, 29900, 29896, 29955, 29913, 322, 27599, 515, 372, 13, 29906, 29889, 29906, 24464, 29905, 13007, 25741, 3190, 2347, 515, 29871, 29896, 29953, 29900, 7284, 29905, 13007, 9279, 322, 4148, 287, 491, 13, 29946, 29941, 7284, 29905, 13007, 15717, 2645, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29896, 931, 3785, 29889, 32001, 1334, 1737, 324, 542, 403, 18777, 304, 13, 29905, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 29892, 773, 408, 18470, 4876, 4234, 775, 2246, 29899, 5563, 21904, 313, 617, 29911, 10249, 29879, 29897, 322, 29871, 13, 8921, 313, 4102, 29914, 4230, 29897, 2983, 9401, 411, 1024, 18822, 2820, 278, 3186, 29892, 322, 17998, 1283, 7224, 29871, 13, 1195, 287, 515, 9063, 15562, 29889, 32001, 1334, 1284, 10757, 310, 278, 4688, 8022, 749, 310, 4644, 6813, 297, 1722, 2752, 13, 20415, 29892, 2678, 8772, 491, 4092, 29889, 32001, 2860, 393, 3785, 29892, 278, 1737, 12122, 6894, 537, 29871, 13, 262, 970, 775, 756, 1063, 21003, 10231, 29889, 32001, 1334, 884, 12439, 8018, 15839, 528, 17741, 13, 12817, 304, 278, 1095, 310, 278, 8291, 6415, 29129, 322, 278, 7910, 310, 14137, 4631, 4135, 297, 13, 23369, 1705, 322, 4275, 14325, 29892, 408, 1532, 408, 310, 2545, 1664, 17292, 28342, 763, 25539, 1608, 322, 13, 25719, 10298, 4822, 10916, 313, 6727, 16783, 29914, 331, 16783, 467, 32001, 320, 26956, 29912, 1469, 20847, 3097, 5003, 32001, 319, 1634, 1414, 3577, 363, 445, 5650, 338, 3625, 515, 796, 264, 8144, 472, 13, 29905, 2271, 29912, 991, 597, 1867, 29875, 29889, 990, 29914, 29896, 29900, 29889, 29945, 29906, 29947, 29896, 29914, 2256, 8144, 29889, 29953, 29941, 29929, 29900, 29941, 29945, 29945, 29913, 2651, 2036, 29912, 3445, 1414, 29899, 5113, 1836, 32001, 320, 2042, 29912, 9662, 630, 5244, 29913, 13, 29905, 1643, 29912, 3471, 29901, 12817, 29913, 13, 13, 29933, 720, 4688, 322, 7786, 1736, 2651, 2036, 29912, 12443, 10578, 29906, 29900, 29900, 29945, 5062, 11235, 29892, 13260, 333, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 13, 29871, 696, 7586, 29906, 29900, 29896, 29946, 7610, 345, 2941, 271, 24541, 29892, 697, 309, 29906, 29900, 29896, 29953, 311, 5365, 550, 332, 6950, 29913, 505, 2931, 1891, 278, 13, 479, 5275, 310, 12362, 29914, 6585, 7562, 18540, 313, 5800, 1799, 29897, 773, 320, 7278, 29912, 6734, 26946, 952, 1118, 13, 4716, 3867, 1880, 29899, 29567, 6089, 541, 526, 9078, 297, 2159, 313, 29906, 29899, 29945, 5940, 29968, 18777, 29897, 13, 392, 508, 367, 4768, 1463, 491, 5221, 424, 23460, 29889, 32001, 512, 29871, 29906, 29900, 29900, 29947, 2261, 801, 2681, 634, 394, 8770, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29913, 18043, 263, 3031, 979, 13, 16961, 29899, 7052, 313, 1454, 278, 931, 29897, 6559, 373, 18322, 1799, 320, 7278, 29912, 479, 5275, 773, 1375, 292, 7047, 13, 29871, 28914, 313, 4345, 29934, 29897, 13698, 1836, 32001, 2688, 29537, 287, 278, 3978, 310, 29871, 29896, 5940, 29924, 17737, 29560, 13, 4746, 278, 7562, 2831, 479, 1404, 2566, 322, 611, 6504, 1051, 3190, 3145, 975, 278, 13, 29896, 29929, 29929, 29929, 489, 29906, 29900, 29900, 29945, 3785, 29892, 773, 408, 18470, 2472, 2788, 304, 1749, 29879, 29901, 4876, 21904, 13, 392, 17998, 1283, 7224, 29889, 32001, 450, 12399, 3785, 313, 29955, 2440, 29897, 297, 2651, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29913, 338, 20511, 1135, 29871, 13, 5816, 338, 12399, 297, 278, 2198, 5650, 313, 29945, 29900, 2440, 29897, 322, 278, 848, 8974, 526, 29871, 13, 27489, 873, 1422, 29936, 411, 393, 297, 3458, 29892, 1749, 2582, 1510, 263, 10029, 7200, 14978, 310, 29871, 13, 15654, 273, 325, 13217, 29940, 2072, 3082, 20706, 29889, 32001, 7280, 29190, 936, 664, 515, 29871, 29906, 29900, 29896, 29900, 491, 14619, 29882, 371, 4099, 29894, 322, 13, 29950, 309, 1372, 2651, 2036, 29912, 29873, 19426, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29913, 4023, 29894, 2868, 1583, 29899, 7099, 433, 1127, 1737, 12122, 13, 2029, 800, 310, 25492, 15303, 8304, 3598, 1494, 1009, 12368, 29892, 13, 15914, 292, 2472, 363, 779, 14850, 4535, 29892, 29955, 29900, 5940, 29968, 25492, 4160, 29889, 32001, 319, 1407, 7786, 13, 1287, 2651, 2036, 29912, 21686, 29879, 29906, 29900, 29906, 29896, 2209, 479, 5275, 29913, 491, 399, 496, 29879, 634, 394, 13217, 5349, 1737, 324, 542, 630, 4203, 263, 7284, 13, 28712, 16046, 4160, 29892, 2534, 26869, 472, 3203, 29871, 29896, 29900, 29900, 25741, 1269, 29892, 322, 1058, 13, 1311, 29899, 7099, 8663, 14354, 373, 1009, 25492, 28723, 29889, 32001, 5806, 278, 6559, 338, 13, 3149, 29899, 262, 29899, 2230, 408, 310, 29871, 29906, 29900, 29906, 29896, 29892, 278, 15717, 7252, 1009, 1284, 886, 13, 351, 475, 303, 2651, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29892, 1850, 29882, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29913, 304, 13, 18609, 675, 278, 14675, 310, 18322, 1799, 1737, 5275, 975, 278, 931, 15101, 845, 1862, 4586, 491, 13, 1552, 2211, 11898, 29889, 32001, 3831, 1965, 411, 3517, 29190, 936, 1736, 29892, 1749, 6559, 338, 1568, 7200, 6287, 5634, 29882, 5555, 13, 7054, 12339, 287, 29871, 29946, 29941, 7284, 29905, 13007, 15717, 310, 29871, 29906, 29889, 29906, 24464, 29905, 13007, 25741, 515, 29871, 29896, 29953, 29900, 7284, 29905, 13007, 13, 16418, 5634, 5426, 11267, 979, 975, 29871, 29945, 29900, 2440, 310, 970, 775, 20706, 3265, 1135, 13, 3149, 297, 931, 29892, 322, 884, 901, 2691, 29899, 3874, 1312, 313, 2541, 1629, 29899, 1609, 29899, 6360, 3803, 1070, 537, 975, 13, 1552, 8900, 3785, 467, 32001, 8108, 1189, 1711, 29892, 1749, 6559, 337, 3687, 373, 10079, 11264, 13, 3924, 313, 8257, 29903, 29897, 9063, 848, 3265, 1135, 7481, 29899, 7099, 433, 1127, 4423, 2472, 29889, 32001, 5901, 1736, 5634, 262, 3153, 278, 664, 491, 8432, 2651, 2036, 29912, 18386, 709, 29906, 29900, 29896, 29941, 2209, 29881, 24974, 29913, 13, 392, 29892, 901, 10325, 29892, 390, 579, 468, 29875, 634, 394, 8770, 2036, 29912, 29878, 579, 468, 29875, 29906, 29900, 29896, 29953, 479, 711, 3173, 29892, 13, 29871, 364, 579, 468, 29875, 29906, 29900, 29896, 29947, 479, 711, 3173, 29892, 544, 1648, 29906, 29900, 29906, 29896, 479, 468, 1581, 29881, 24974, 29913, 5634, 17532, 12399, 1737, 12122, 13, 29905, 7278, 29912, 29881, 24974, 322, 24003, 1118, 474, 29889, 29872, 1696, 278, 15834, 304, 607, 278, 3978, 310, 18322, 1799, 13, 17426, 6602, 1009, 11465, 1230, 14137, 14188, 29889, 32001, 512, 445, 664, 591, 2931, 1891, 1737, 12122, 6894, 537, 297, 970, 775, 363, 278, 937, 13, 2230, 472, 445, 6287, 29892, 1716, 297, 4958, 310, 17737, 29560, 322, 15500, 3785, 29889, 32001, 1334, 437, 13, 1333, 22002, 280, 278, 24003, 10696, 29892, 541, 3867, 29190, 936, 848, 322, 1284, 886, 393, 508, 367, 13, 280, 369, 4063, 304, 393, 1095, 408, 5434, 664, 29889, 32001, 320, 7278, 29912, 12756, 7047, 21639, 29913, 2651, 2036, 29912, 2276, 5824, 19982, 29906, 29900, 29900, 29955, 23705, 1338, 29893, 996, 29913, 338, 278, 13, 1491, 29899, 2671, 310, 7047, 21639, 393, 756, 29537, 287, 278, 18066, 267, 310, 21640, 13, 6734, 24771, 13149, 635, 29892, 3704, 278, 2702, 5932, 310, 920, 304, 5376, 13, 2541, 1737, 12122, 6894, 537, 2651, 2036, 29912, 13376, 303, 456, 29906, 29900, 29900, 29953, 10945, 3359, 29892, 1424, 29440, 29906, 29900, 29896, 29946, 23027, 5933, 1836, 32001, 3826, 3076, 2678, 278, 2198, 6559, 8128, 10757, 393, 508, 367, 1304, 29892, 297, 278, 13, 14940, 1206, 310, 970, 775, 322, 472, 263, 1407, 2919, 6287, 29892, 304, 11539, 607, 13, 14032, 4637, 310, 5534, 7047, 21639, 505, 9820, 484, 15774, 29889, 32001, 320, 2042, 29912, 4062, 3002, 29913, 13, 29905, 1643, 29912, 3471, 29901, 5696, 29913, 13, 13, 13, 29905, 1482, 361, 29905, 361, 29887, 798, 386, 1003, 29871, 320, 29887, 798, 386, 1003, 3009, 13, 29905, 361, 29887, 798, 386, 1003, 13, 29905, 463, 29912, 4532, 29913, 13, 29871, 320, 7313, 29961, 2103, 2013, 4914, 2103, 3199, 6360, 368, 29899, 2055, 1169, 29913, 13, 29871, 320, 6671, 29912, 12883, 368, 970, 25741, 975, 931, 313, 1188, 6287, 467, 13, 29913, 32001, 320, 1643, 29912, 1003, 29901, 29887, 798, 386, 29913, 13, 29905, 355, 29912, 4532, 29913, 13, 29905, 7241, 13, 13, 29905, 26956, 29912, 16390, 24541, 29913, 13, 13, 4806, 27387, 515, 320, 23066, 29950, 29914, 2651, 2036, 29912, 29879, 1332, 29899, 1516, 29878, 29906, 29900, 29896, 29929, 29899, 24713, 29913, 599, 25741, 3190, 2347, 2745, 320, 25832, 1964, 579, 1523, 2415, 2539, 6294, 32001, 2688, 5253, 304, 320, 14573, 5261, 1169, 22131, 29914, 25741, 29892, 5412, 491, 317, 15715, 29896, 15882, 29892, 4023, 29894, 2868, 515, 320, 25832, 1299, 7288, 5261, 1169, 797, 7068, 29914, 970, 9279, 6421, 515, 4655, 5849, 363, 2710, 313, 28712, 16046, 29892, 11786, 28632, 29892, 2992, 1846, 32001, 322, 3577, 28914, 313, 10251, 713, 29892, 10772, 2227, 29892, 405, 13427, 29892, 2992, 6250, 32001, 1876, 1169, 297, 278, 8783, 526, 491, 320, 25832, 6344, 2806, 943, 22131, 29914, 15717, 29892, 5412, 491, 779, 6990, 29938, 978, 29892, 4876, 4535, 5854, 29938, 11000, 29889, 32001, 450, 8783, 2996, 408, 1023, 1104, 1288, 6131, 29892, 697, 363, 25741, 322, 697, 363, 15717, 29892, 411, 278, 4642, 29371, 278, 7480, 3025, 263, 9117, 1820, 29889, 32001, 320, 361, 5426, 13, 9760, 1948, 297, 278, 9063, 1591, 3743, 278, 1494, 4235, 29901, 9063, 317, 15715, 29896, 15882, 29892, 4148, 322, 844, 5171, 5335, 342, 15092, 29892, 4148, 322, 844, 5171, 2893, 14903, 313, 20275, 16750, 278, 4148, 1591, 467, 32001, 450, 21578, 1546, 9063, 15717, 322, 9063, 2153, 2041, 515, 11786, 29892, 607, 6511, 304, 9063, 263, 1735, 4148, 287, 491, 4856, 1683, 29889, 32001, 1152, 445, 6559, 591, 21309, 373, 15717, 322, 17262, 9063, 2153, 29892, 408, 278, 4328, 1546, 278, 1023, 338, 451, 8018, 363, 1749, 5925, 5155, 322, 278, 5253, 310, 25741, 411, 263, 844, 5171, 916, 1135, 967, 4148, 338, 3480, 3473, 1821, 29889, 32001, 320, 7241, 13, 2831, 1269, 6251, 297, 278, 4148, 1591, 591, 505, 4148, 2989, 1024, 322, 4876, 408, 1023, 5004, 6031, 310, 10650, 6262, 29889, 32001, 1334, 6206, 2411, 15273, 1821, 470, 18325, 519, 2983, 393, 29901, 526, 451, 1602, 397, 519, 408, 18351, 29899, 29947, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12283, 7099, 397, 519, 29914, 4148, 2983, 6206, 511, 526, 4876, 14157, 2012, 310, 2983, 3441, 25832, 6344, 2806, 943, 29934, 29885, 9823, 29914, 4954, 7039, 4907, 511, 5718, 310, 871, 9654, 4890, 3441, 25832, 6344, 2806, 943, 29934, 29885, 10358, 804, 23201, 1712, 901, 1135, 29871, 29896, 29900, 8958, 1661, 29899, 1026, 2153, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12283, 15670, 23201, 526, 5520, 1135, 29871, 29896, 29900, 29900, 4890, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12229, 549, 12495, 32001, 2860, 21166, 29892, 1048, 320, 25832, 6344, 2806, 943, 29925, 15273, 1821, 2052, 307, 29916, 29914, 15717, 3441, 25832, 6344, 2806, 943, 29925, 15273, 1821, 29925, 312, 29914, 310, 278, 2847, 8783, 29897, 9488, 363, 4340, 7418, 29889, 32001, 3940, 393, 278, 5253, 310, 970, 775, 25741, 313, 392, 15717, 29897, 11122, 297, 278, 13, 11228, 8783, 25088, 18709, 9247, 975, 13, 2230, 2651, 2036, 29912, 29879, 1332, 29899, 771, 854, 749, 29899, 331, 344, 1012, 361, 29887, 798, 386, 1003, 29892, 408, 4318, 363, 25741, 297, 13, 29905, 29907, 999, 29912, 1003, 29901, 29887, 798, 386, 1012, 2870, 29901, 515, 395, 29896, 29900, 29985, 29946, 29938, 25741, 297, 29871, 29896, 29929, 29955, 29896, 29892, 304, 395, 29896, 29900, 29985, 29953, 29938, 297, 29871, 29896, 29929, 29929, 29947, 29892, 304, 13, 284, 3242, 395, 29896, 29900, 29985, 29929, 29938, 297, 29871, 29906, 29900, 29906, 29900, 29905, 7241, 29889, 32001, 1094, 263, 17004, 278, 8900, 534, 1975, 10331, 304, 367, 901, 13, 13844, 297, 7786, 1602, 3076, 1135, 297, 29871, 29946, 29900, 29974, 1629, 29899, 1025, 6743, 29892, 2861, 304, 13964, 4586, 373, 13, 735, 1112, 9247, 7200, 23093, 29889, 32001, 320, 26956, 29912, 7999, 324, 10610, 29913, 13, 13, 29905, 463, 29912, 4532, 29913, 13, 29871, 320, 9525, 13, 29871, 320, 7313, 29961, 24049, 29892, 15450, 29922, 29953, 4912, 29871, 29953, 4912, 29871, 29900, 29871, 29900, 29892, 2103, 2013, 16292, 3199, 1491, 1727, 1080, 29899, 2470, 29913, 13, 29871, 320, 6671, 29912, 1576, 320, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 1304, 408, 1737, 324, 10610, 22525, 5003, 32001, 320, 1643, 29912, 1003, 29901, 11526, 1958, 29913, 13, 29905, 355, 29912, 4532, 29913, 13, 13, 2887, 1737, 324, 10610, 22525, 591, 671, 11758, 3186, 12786, 10723, 515, 278, 3303, 18269, 1737, 359, 305, 2004, 2651, 2036, 29912, 348, 29896, 29929, 29929, 29929, 479, 359, 305, 2004, 1836, 32001, 1763, 4772, 2432, 3381, 491, 2919, 10916, 313, 29872, 29889, 29887, 1696, 7551, 470, 12710, 29897, 2629, 11758, 12786, 29892, 591, 19412, 322, 6219, 777, 12786, 2729, 373, 1737, 12122, 23203, 537, 322, 278, 19383, 310, 758, 331, 8946, 16375, 29769, 5680, 29892, 1316, 408, 19182, 4086, 29889, 32001, 320, 29907, 999, 29912, 1003, 29901, 11526, 1958, 29913, 3697, 278, 2186, 1051, 310, 320, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 1304, 408, 1737, 324, 10610, 22525, 297, 445, 6559, 29889, 32001, 1879, 324, 10610, 310, 9063, 15717, 304, 3186, 12786, 3913, 278, 1023, 19595, 653, 13698, 9129, 297, 2651, 2036, 29912, 293, 344, 29899, 344, 275, 29899, 29906, 29900, 29906, 29906, 29899, 26098, 1118, 23359, 337, 13998, 2400, 29889, 32001, 450, 937, 697, 337, 3687, 373, 278, 4234, 775, 2246, 29899, 5563, 5354, 313, 617, 29911, 10249, 29897, 310, 4876, 14157, 23892, 515, 9063, 15562, 29892, 321, 29889, 29887, 1696, 320, 16266, 26139, 1341, 1118, 320, 16266, 26139, 582, 1118, 320, 16266, 26139, 18038, 1118, 2992, 29889, 32001, 1334, 4687, 515, 278, 306, 2190, 29909, 1051, 310, 13548, 2931, 21759, 29911, 10249, 29879, 2651, 2036, 29912, 6011, 29899, 29883, 312, 430, 29913, 322, 7522, 20545, 1269, 6590, 20123, 304, 263, 3646, 3186, 5120, 29889, 32001, 450, 1473, 1737, 324, 10610, 11043, 3913, 278, 17998, 9210, 310, 9063, 5335, 342, 15092, 313, 29872, 29889, 29887, 1696, 17998, 29899, 29900, 29945, 29901, 29900, 29900, 29897, 322, 4148, 2983, 304, 8161, 278, 1556, 5517, 3186, 5120, 310, 278, 9063, 4148, 29889, 32001, 1152, 1269, 17998, 9210, 591, 8161, 263, 1051, 310, 15878, 7600, 313, 13509, 29892, 2106, 29892, 470, 14278, 20123, 29897, 297, 278, 3186, 393, 29892, 472, 278, 931, 310, 393, 9063, 29892, 750, 393, 17998, 9210, 29936, 9063, 931, 338, 1820, 1244, 29892, 408, 4234, 17998, 1283, 7224, 13100, 975, 931, 2861, 304, 29431, 3620, 29889, 32001, 1763, 1207, 445, 3683, 3381, 591, 671, 278, 306, 2190, 29909, 931, 10640, 2566, 2651, 2036, 29912, 17559, 1272, 1836, 32001, 1987, 591, 3566, 304, 1269, 2058, 263, 8158, 393, 4332, 1973, 278, 4188, 22342, 393, 263, 2183, 4148, 1024, 338, 17443, 310, 372, 29889, 32001, 1763, 445, 1095, 591, 671, 278, 28297, 29890, 15451, 8783, 310, 278, 29511, 310, 278, 1556, 3619, 937, 322, 3942, 2983, 607, 29892, 439, 11427, 515, 2651, 2036, 29912, 1079, 29890, 799, 29899, 7039, 6177, 2802, 277, 12181, 4954, 16123, 2247, 278, 26368, 5528, 5084, 310, 363, 264, 1280, 322, 269, 595, 1280, 7371, 515, 263, 2566, 310, 320, 1949, 29912, 29946, 29871, 29900, 29946, 29946, 29871, 29945, 29946, 29953, 29871, 29929, 29941, 29947, 29913, 2305, 313, 29945, 29945, 29889, 29945, 8958, 310, 8471, 2305, 297, 29871, 29906, 29900, 29896, 29946, 467, 32001, 1094, 310, 3839, 29871, 29906, 29900, 29896, 29929, 372, 18469, 320, 1949, 29912, 29906, 29955, 29871, 29953, 29953, 29906, 29871, 29947, 29900, 29896, 29913, 363, 264, 1280, 322, 320, 1949, 29912, 29906, 29955, 29871, 29906, 29900, 29953, 29871, 29947, 29906, 29896, 29913, 269, 595, 1280, 297, 29871, 29906, 29941, 29953, 24894, 8977, 1080, 29889, 4907, 29913, 32001, 1094, 297, 1749, 8783, 15717, 526, 2989, 1024, 6031, 313, 29878, 1624, 1135, 6219, 491, 937, 29914, 11922, 1024, 511, 591, 937, 5993, 675, 2983, 313, 1609, 1999, 1331, 322, 1206, 3620, 29897, 322, 769, 16280, 5375, 18897, 297, 1716, 937, 322, 3942, 2983, 10868, 8857, 29889, 32001, 1152, 1269, 1543, 1476, 297, 1024, 8857, 591, 22932, 278, 2058, 4665, 29905, 6661, 1333, 24212, 8875, 491, 278, 1024, 10868, 304, 4017, 263, 5645, 393, 338, 29839, 304, 278, 1353, 310, 12407, 24638, 393, 1024, 313, 6979, 29897, 297, 278, 2702, 2058, 29889, 32001, 320, 6661, 1333, 300, 1062, 29912, 1762, 4017, 4665, 2025, 1338, 5634, 294, 278, 17837, 310, 4954, 6689, 4907, 338, 25745, 23724, 29901, 2989, 10916, 325, 13217, 29879, 29399, 310, 2919, 10916, 805, 9450, 2999, 931, 29920, 2873, 5634, 705, 671, 263, 29544, 310, 7601, 8974, 313, 29872, 29889, 29887, 1696, 5874, 28007, 511, 322, 1661, 29899, 16072, 6743, 313, 29872, 29889, 29887, 1696, 14109, 7456, 467, 29913, 32001, 1334, 2533, 445, 4377, 363, 599, 3161, 304, 4017, 263, 2058, 8158, 29892, 17140, 701, 411, 263, 1051, 310, 779, 6990, 29938, 6689, 29892, 8158, 4535, 5854, 29938, 11000, 29889, 32001, 1334, 769, 8877, 445, 1051, 491, 278, 3186, 5120, 393, 263, 2058, 14393, 304, 322, 2533, 278, 8158, 363, 599, 278, 7600, 297, 1269, 5120, 304, 4017, 385, 12463, 8158, 29892, 6590, 304, 278, 4188, 22342, 393, 278, 9063, 14393, 304, 263, 2183, 3186, 5120, 29889, 32001, 1334, 3566, 278, 6257, 9063, 408, 6421, 515, 278, 3186, 5120, 411, 278, 9939, 8158, 29889, 32001, 450, 4876, 29899, 6707, 11043, 9378, 414, 515, 278, 9078, 322, 443, 5521, 8362, 671, 310, 21759, 29911, 10249, 29879, 29901, 1556, 18777, 671, 10035, 323, 10249, 29879, 1316, 408, 320, 16266, 26139, 510, 1118, 320, 16266, 26139, 990, 1118, 470, 320, 16266, 26139, 1212, 1836, 32001, 12808, 445, 947, 451, 3799, 26018, 4822, 20542, 29901, 3148, 29899, 6707, 18777, 29892, 363, 1342, 29892, 671, 278, 320, 16266, 26139, 375, 29913, 21759, 29911, 10249, 1568, 901, 269, 2495, 290, 368, 1135, 1009, 7824, 6795, 20895, 29889, 32001, 1551, 278, 916, 1361, 278, 9210, 29914, 978, 29899, 6707, 11043, 337, 3687, 373, 278, 17998, 9210, 310, 278, 9063, 5335, 342, 15092, 29889, 32001, 16809, 304, 5780, 22920, 373, 13897, 731, 14340, 29892, 263, 2919, 1353, 310, 25741, 297, 278, 8783, 756, 385, 17998, 9210, 5186, 304, 5225, 29889, 32001, 910, 6602, 29879, 3109, 7786, 25741, 3441, 14573, 5261, 1169, 29911, 29999, 29999, 13985, 1349, 681, 392, 27418, 6478, 29914, 310, 29871, 29906, 29900, 29906, 29900, 29879, 25741, 505, 263, 5225, 9210, 29897, 1135, 9642, 6743, 3441, 14573, 5261, 1169, 29911, 29999, 29999, 13985, 1349, 681, 392, 29914, 297, 29871, 29906, 29900, 29900, 29900, 467, 32001, 1094, 263, 1121, 278, 9210, 29914, 978, 29899, 6707, 11043, 1033, 1095, 701, 6459, 292, 263, 2919, 6232, 310, 9642, 25741, 408, 4148, 287, 491, 11715, 18777, 29892, 322, 304, 263, 3109, 261, 15834, 4092, 550, 29889, 32001, 1763, 6795, 1438, 5626, 591, 14405, 278, 1023, 1737, 324, 10610, 13698, 4208, 491, 15399, 278, 9210, 29914, 978, 29899, 6707, 13698, 304, 599, 25741, 411, 263, 1661, 29899, 9171, 17998, 9210, 29892, 322, 278, 4876, 29899, 6707, 373, 304, 599, 916, 25741, 29889, 32001, 320, 2042, 29912, 12191, 322, 8565, 1558, 291, 29913, 13, 29905, 1643, 29912, 3471, 29901, 9902, 29913, 13, 13, 29905, 463, 29912, 4532, 4044, 13, 29871, 320, 9525, 13, 29871, 320, 7313, 29961, 2103, 2013, 16292, 3199, 1429, 287, 29889, 5140, 29913, 13, 29871, 320, 6671, 29912, 29934, 20819, 310, 25741, 313, 27215, 29897, 322, 6136, 15717, 313, 22503, 29897, 491, 3186, 10640, 975, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29900, 3785, 5003, 32001, 320, 9868, 29961, 14732, 3199, 7264, 287, 2594, 8727, 6445, 278, 3186, 10640, 364, 2219, 359, 363, 25741, 322, 15717, 975, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29900, 3785, 5003, 32001, 320, 1643, 29912, 1003, 29901, 9902, 29913, 13, 29905, 355, 29912, 4532, 4044, 13, 13, 13, 29871, 13, 1762, 1234, 320, 25029, 29912, 29878, 29939, 29901, 479, 397, 24974, 29913, 591, 22229, 278, 1353, 310, 25741, 322, 8359, 15717, 639, 1629, 322, 639, 3186, 10640, 29889, 32001, 1334, 2198, 278, 7625, 2582, 297, 320, 29907, 999, 29912, 1003, 29901, 9902, 29913, 408, 1023, 5096, 287, 2594, 24469, 29892, 6445, 1629, 368, 2867, 3204, 29879, 363, 25741, 322, 15717, 8307, 29889, 32001, 7569, 2594, 11524, 263, 1629, 322, 338, 8877, 287, 297, 269, 29399, 6445, 278, 9063, 29914, 8921, 11959, 363, 1269, 310, 278, 3186, 12786, 310, 320, 29907, 999, 29912, 1003, 29901, 11526, 1958, 29913, 297, 393, 1629, 29889, 32001, 1763, 4772, 714, 27801, 2861, 304, 805, 272, 26538, 17737, 29560, 29892, 297, 278, 4148, 8727, 591, 871, 2050, 15717, 2534, 26869, 472, 3203, 29871, 29945, 25741, 297, 263, 2183, 1629, 29889, 32001, 5806, 5366, 1747, 534, 1975, 297, 278, 24469, 6456, 393, 278, 3001, 3694, 310, 25741, 322, 15717, 6548, 18709, 9247, 975, 931, 29889, 32001, 10133, 363, 278, 937, 2440, 297, 278, 24469, 29892, 278, 1353, 310, 848, 3291, 297, 777, 3186, 12786, 508, 367, 14154, 2319, 29892, 411, 8178, 27721, 373, 278, 25806, 310, 534, 1975, 29889, 32001, 320, 26956, 29912, 7999, 12122, 6894, 537, 975, 931, 29913, 13, 13, 3563, 497, 29892, 278, 2498, 534, 355, 5692, 304, 367, 393, 278, 320, 4534, 29912, 479, 12122, 6894, 537, 297, 970, 775, 338, 10231, 6177, 4644, 6813, 322, 4092, 5136, 630, 1009, 4954, 24130, 749, 4907, 2745, 278, 7256, 310, 278, 29871, 29929, 29900], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [1, 320, 2042, 29912, 25898, 29913, 13, 29905, 1643, 29912, 3471, 29901, 23333, 29913, 13, 13, 29905, 7278, 29912, 29954, 1581, 6894, 537, 1118, 470, 901, 4049, 967, 10225, 727, 974, 29892, 4249, 27138, 304, 13, 20415, 5849, 14188, 756, 1063, 26606, 12399, 297, 7786, 2440, 29889, 32001, 512, 13, 1595, 16311, 29892, 278, 10122, 310, 29892, 9545, 310, 29892, 322, 6795, 1004, 25414, 363, 320, 7278, 29912, 26098, 13, 29871, 24003, 29913, 297, 12362, 29914, 6585, 7562, 18540, 313, 5800, 1799, 29897, 505, 4520, 263, 3287, 310, 8570, 13, 957, 278, 4940, 316, 6332, 2651, 2036, 29912, 29881, 16093, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 3855, 5871, 29906, 29900, 29896, 29900, 29895, 311, 29893, 2770, 29892, 13, 29871, 1055, 29888, 375, 29906, 29900, 29896, 29906, 5041, 267, 29892, 413, 434, 305, 1358, 29906, 29900, 29896, 29906, 26098, 29888, 2209, 29892, 19723, 5475, 4979, 29906, 29900, 29896, 29946, 26098, 29892, 13, 29871, 697, 309, 29906, 29900, 29896, 29953, 311, 5365, 550, 332, 6950, 29892, 696, 7586, 29906, 29900, 29896, 29953, 29893, 2770, 7192, 2209, 29892, 1935, 15044, 29906, 29900, 29896, 29955, 26098, 29892, 13, 29871, 503, 562, 305, 3350, 492, 29906, 29900, 29906, 29896, 26098, 1836, 32001, 320, 7278, 29912, 7999, 12122, 6894, 537, 29913, 338, 373, 278, 916, 1361, 278, 13, 14380, 310, 6894, 537, 393, 380, 1567, 515, 27138, 297, 777, 5534, 6354, 6421, 13, 3166, 1422, 3186, 12786, 322, 4185, 1973, 29889, 32001, 26304, 6894, 537, 297, 18322, 1799, 756, 4520, 13774, 2217, 8570, 297, 21344, 368, 13, 13129, 29889, 32001, 512, 3153, 29892, 1550, 3031, 979, 18994, 29899, 6707, 322, 13, 3149, 29899, 262, 29899, 2230, 18350, 29899, 7052, 11898, 310, 278, 1737, 12122, 1677, 1144, 310, 18322, 1799, 13, 21570, 29560, 1863, 2651, 2036, 29912, 12443, 10578, 29906, 29900, 29900, 29945, 5062, 11235, 29892, 13260, 333, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 13, 29871, 2594, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29892, 1850, 29882, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29892, 696, 7586, 29906, 29900, 29896, 29946, 7610, 345, 2941, 271, 24541, 29892, 13, 29871, 281, 496, 29879, 29906, 29900, 29906, 29896, 2209, 479, 5275, 1118, 2919, 29899, 7052, 25579, 979, 11898, 310, 278, 1737, 12122, 13, 12574, 310, 18322, 1799, 17737, 29560, 526, 1603, 10225, 292, 29889, 32001, 10506, 263, 4323, 23378, 13, 18609, 2133, 723, 367, 5407, 304, 1871, 1602, 12112, 4475, 304, 5534, 13, 25431, 10907, 2651, 2036, 29912, 2276, 5824, 19982, 29906, 29900, 29900, 29955, 23705, 1338, 29893, 996, 29913, 322, 298, 8491, 16650, 583, 297, 278, 13, 19678, 15483, 313, 1806, 29897, 9999, 29892, 408, 1532, 408, 29126, 2114, 950, 2472, 13, 517, 278, 2553, 1078, 373, 278, 17407, 10879, 322, 5374, 3002, 310, 18322, 1799, 2820, 278, 3186, 29889, 32001, 320, 26956, 29912, 1323, 3224, 29879, 29913, 13, 13, 3047, 445, 664, 591, 29126, 304, 3802, 445, 17261, 491, 7512, 292, 320, 4534, 29912, 1552, 937, 13, 29871, 25579, 979, 6559, 310, 278, 1737, 12122, 3978, 310, 17737, 29560, 304, 970, 775, 13, 29871, 975, 29871, 29945, 29900, 2440, 5003, 32001, 26321, 29892, 591, 3867, 263, 758, 2576, 3821, 1234, 304, 278, 13, 23031, 292, 5925, 1139, 29901, 13, 29905, 463, 29912, 690, 2842, 12470, 29913, 13, 29871, 3645, 607, 3186, 12786, 437, 15717, 310, 970, 368, 3625, 25741, 2041, 515, 13, 29871, 322, 920, 756, 372, 3939, 975, 278, 4940, 29871, 29945, 29900, 2440, 29973, 32001, 320, 1643, 29912, 29878, 29939, 29901, 479, 397, 24974, 29913, 13, 29905, 355, 29912, 690, 2842, 12470, 29913, 13, 4806, 671, 408, 8783, 278, 320, 23066, 29950, 29914, 18871, 2651, 2036, 29912, 29879, 1332, 29875, 4569, 29906, 29900, 29896, 29955, 29913, 322, 27599, 515, 372, 13, 29906, 29889, 29906, 24464, 29905, 13007, 25741, 3190, 2347, 515, 29871, 29896, 29953, 29900, 7284, 29905, 13007, 9279, 322, 4148, 287, 491, 13, 29946, 29941, 7284, 29905, 13007, 15717, 2645, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29896, 931, 3785, 29889, 32001, 1334, 1737, 324, 542, 403, 18777, 304, 13, 29905, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 29892, 773, 408, 18470, 4876, 4234, 775, 2246, 29899, 5563, 21904, 313, 617, 29911, 10249, 29879, 29897, 322, 29871, 13, 8921, 313, 4102, 29914, 4230, 29897, 2983, 9401, 411, 1024, 18822, 2820, 278, 3186, 29892, 322, 17998, 1283, 7224, 29871, 13, 1195, 287, 515, 9063, 15562, 29889, 32001, 1334, 1284, 10757, 310, 278, 4688, 8022, 749, 310, 4644, 6813, 297, 1722, 2752, 13, 20415, 29892, 2678, 8772, 491, 4092, 29889, 32001, 2860, 393, 3785, 29892, 278, 1737, 12122, 6894, 537, 29871, 13, 262, 970, 775, 756, 1063, 21003, 10231, 29889, 32001, 1334, 884, 12439, 8018, 15839, 528, 17741, 13, 12817, 304, 278, 1095, 310, 278, 8291, 6415, 29129, 322, 278, 7910, 310, 14137, 4631, 4135, 297, 13, 23369, 1705, 322, 4275, 14325, 29892, 408, 1532, 408, 310, 2545, 1664, 17292, 28342, 763, 25539, 1608, 322, 13, 25719, 10298, 4822, 10916, 313, 6727, 16783, 29914, 331, 16783, 467, 32001, 320, 26956, 29912, 1469, 20847, 3097, 5003, 32001, 319, 1634, 1414, 3577, 363, 445, 5650, 338, 3625, 515, 796, 264, 8144, 472, 13, 29905, 2271, 29912, 991, 597, 1867, 29875, 29889, 990, 29914, 29896, 29900, 29889, 29945, 29906, 29947, 29896, 29914, 2256, 8144, 29889, 29953, 29941, 29929, 29900, 29941, 29945, 29945, 29913, 2651, 2036, 29912, 3445, 1414, 29899, 5113, 1836, 32001, 320, 2042, 29912, 9662, 630, 5244, 29913, 13, 29905, 1643, 29912, 3471, 29901, 12817, 29913, 13, 13, 29933, 720, 4688, 322, 7786, 1736, 2651, 2036, 29912, 12443, 10578, 29906, 29900, 29900, 29945, 5062, 11235, 29892, 13260, 333, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 13, 29871, 696, 7586, 29906, 29900, 29896, 29946, 7610, 345, 2941, 271, 24541, 29892, 697, 309, 29906, 29900, 29896, 29953, 311, 5365, 550, 332, 6950, 29913, 505, 2931, 1891, 278, 13, 479, 5275, 310, 12362, 29914, 6585, 7562, 18540, 313, 5800, 1799, 29897, 773, 320, 7278, 29912, 6734, 26946, 952, 1118, 13, 4716, 3867, 1880, 29899, 29567, 6089, 541, 526, 9078, 297, 2159, 313, 29906, 29899, 29945, 5940, 29968, 18777, 29897, 13, 392, 508, 367, 4768, 1463, 491, 5221, 424, 23460, 29889, 32001, 512, 29871, 29906, 29900, 29900, 29947, 2261, 801, 2681, 634, 394, 8770, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29913, 18043, 263, 3031, 979, 13, 16961, 29899, 7052, 313, 1454, 278, 931, 29897, 6559, 373, 18322, 1799, 320, 7278, 29912, 479, 5275, 773, 1375, 292, 7047, 13, 29871, 28914, 313, 4345, 29934, 29897, 13698, 1836, 32001, 2688, 29537, 287, 278, 3978, 310, 29871, 29896, 5940, 29924, 17737, 29560, 13, 4746, 278, 7562, 2831, 479, 1404, 2566, 322, 611, 6504, 1051, 3190, 3145, 975, 278, 13, 29896, 29929, 29929, 29929, 489, 29906, 29900, 29900, 29945, 3785, 29892, 773, 408, 18470, 2472, 2788, 304, 1749, 29879, 29901, 4876, 21904, 13, 392, 17998, 1283, 7224, 29889, 32001, 450, 12399, 3785, 313, 29955, 2440, 29897, 297, 2651, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29913, 338, 20511, 1135, 29871, 13, 5816, 338, 12399, 297, 278, 2198, 5650, 313, 29945, 29900, 2440, 29897, 322, 278, 848, 8974, 526, 29871, 13, 27489, 873, 1422, 29936, 411, 393, 297, 3458, 29892, 1749, 2582, 1510, 263, 10029, 7200, 14978, 310, 29871, 13, 15654, 273, 325, 13217, 29940, 2072, 3082, 20706, 29889, 32001, 7280, 29190, 936, 664, 515, 29871, 29906, 29900, 29896, 29900, 491, 14619, 29882, 371, 4099, 29894, 322, 13, 29950, 309, 1372, 2651, 2036, 29912, 29873, 19426, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29913, 4023, 29894, 2868, 1583, 29899, 7099, 433, 1127, 1737, 12122, 13, 2029, 800, 310, 25492, 15303, 8304, 3598, 1494, 1009, 12368, 29892, 13, 15914, 292, 2472, 363, 779, 14850, 4535, 29892, 29955, 29900, 5940, 29968, 25492, 4160, 29889, 32001, 319, 1407, 7786, 13, 1287, 2651, 2036, 29912, 21686, 29879, 29906, 29900, 29906, 29896, 2209, 479, 5275, 29913, 491, 399, 496, 29879, 634, 394, 13217, 5349, 1737, 324, 542, 630, 4203, 263, 7284, 13, 28712, 16046, 4160, 29892, 2534, 26869, 472, 3203, 29871, 29896, 29900, 29900, 25741, 1269, 29892, 322, 1058, 13, 1311, 29899, 7099, 8663, 14354, 373, 1009, 25492, 28723, 29889, 32001, 5806, 278, 6559, 338, 13, 3149, 29899, 262, 29899, 2230, 408, 310, 29871, 29906, 29900, 29906, 29896, 29892, 278, 15717, 7252, 1009, 1284, 886, 13, 351, 475, 303, 2651, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29892, 1850, 29882, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29913, 304, 13, 18609, 675, 278, 14675, 310, 18322, 1799, 1737, 5275, 975, 278, 931, 15101, 845, 1862, 4586, 491, 13, 1552, 2211, 11898, 29889, 32001, 3831, 1965, 411, 3517, 29190, 936, 1736, 29892, 1749, 6559, 338, 1568, 7200, 6287, 5634, 29882, 5555, 13, 7054, 12339, 287, 29871, 29946, 29941, 7284, 29905, 13007, 15717, 310, 29871, 29906, 29889, 29906, 24464, 29905, 13007, 25741, 515, 29871, 29896, 29953, 29900, 7284, 29905, 13007, 13, 16418, 5634, 5426, 11267, 979, 975, 29871, 29945, 29900, 2440, 310, 970, 775, 20706, 3265, 1135, 13, 3149, 297, 931, 29892, 322, 884, 901, 2691, 29899, 3874, 1312, 313, 2541, 1629, 29899, 1609, 29899, 6360, 3803, 1070, 537, 975, 13, 1552, 8900, 3785, 467, 32001, 8108, 1189, 1711, 29892, 1749, 6559, 337, 3687, 373, 10079, 11264, 13, 3924, 313, 8257, 29903, 29897, 9063, 848, 3265, 1135, 7481, 29899, 7099, 433, 1127, 4423, 2472, 29889, 32001, 5901, 1736, 5634, 262, 3153, 278, 664, 491, 8432, 2651, 2036, 29912, 18386, 709, 29906, 29900, 29896, 29941, 2209, 29881, 24974, 29913, 13, 392, 29892, 901, 10325, 29892, 390, 579, 468, 29875, 634, 394, 8770, 2036, 29912, 29878, 579, 468, 29875, 29906, 29900, 29896, 29953, 479, 711, 3173, 29892, 13, 29871, 364, 579, 468, 29875, 29906, 29900, 29896, 29947, 479, 711, 3173, 29892, 544, 1648, 29906, 29900, 29906, 29896, 479, 468, 1581, 29881, 24974, 29913, 5634, 17532, 12399, 1737, 12122, 13, 29905, 7278, 29912, 29881, 24974, 322, 24003, 1118, 474, 29889, 29872, 1696, 278, 15834, 304, 607, 278, 3978, 310, 18322, 1799, 13, 17426, 6602, 1009, 11465, 1230, 14137, 14188, 29889, 32001, 512, 445, 664, 591, 2931, 1891, 1737, 12122, 6894, 537, 297, 970, 775, 363, 278, 937, 13, 2230, 472, 445, 6287, 29892, 1716, 297, 4958, 310, 17737, 29560, 322, 15500, 3785, 29889, 32001, 1334, 437, 13, 1333, 22002, 280, 278, 24003, 10696, 29892, 541, 3867, 29190, 936, 848, 322, 1284, 886, 393, 508, 367, 13, 280, 369, 4063, 304, 393, 1095, 408, 5434, 664, 29889, 32001, 320, 7278, 29912, 12756, 7047, 21639, 29913, 2651, 2036, 29912, 2276, 5824, 19982, 29906, 29900, 29900, 29955, 23705, 1338, 29893, 996, 29913, 338, 278, 13, 1491, 29899, 2671, 310, 7047, 21639, 393, 756, 29537, 287, 278, 18066, 267, 310, 21640, 13, 6734, 24771, 13149, 635, 29892, 3704, 278, 2702, 5932, 310, 920, 304, 5376, 13, 2541, 1737, 12122, 6894, 537, 2651, 2036, 29912, 13376, 303, 456, 29906, 29900, 29900, 29953, 10945, 3359, 29892, 1424, 29440, 29906, 29900, 29896, 29946, 23027, 5933, 1836, 32001, 3826, 3076, 2678, 278, 2198, 6559, 8128, 10757, 393, 508, 367, 1304, 29892, 297, 278, 13, 14940, 1206, 310, 970, 775, 322, 472, 263, 1407, 2919, 6287, 29892, 304, 11539, 607, 13, 14032, 4637, 310, 5534, 7047, 21639, 505, 9820, 484, 15774, 29889, 32001, 320, 2042, 29912, 4062, 3002, 29913, 13, 29905, 1643, 29912, 3471, 29901, 5696, 29913, 13, 13, 13, 29905, 1482, 361, 29905, 361, 29887, 798, 386, 1003, 29871, 320, 29887, 798, 386, 1003, 3009, 13, 29905, 361, 29887, 798, 386, 1003, 13, 29905, 463, 29912, 4532, 29913, 13, 29871, 320, 7313, 29961, 2103, 2013, 4914, 2103, 3199, 6360, 368, 29899, 2055, 1169, 29913, 13, 29871, 320, 6671, 29912, 12883, 368, 970, 25741, 975, 931, 313, 1188, 6287, 467, 13, 29913, 32001, 320, 1643, 29912, 1003, 29901, 29887, 798, 386, 29913, 13, 29905, 355, 29912, 4532, 29913, 13, 29905, 7241, 13, 13, 29905, 26956, 29912, 16390, 24541, 29913, 13, 13, 4806, 27387, 515, 320, 23066, 29950, 29914, 2651, 2036, 29912, 29879, 1332, 29899, 1516, 29878, 29906, 29900, 29896, 29929, 29899, 24713, 29913, 599, 25741, 3190, 2347, 2745, 320, 25832, 1964, 579, 1523, 2415, 2539, 6294, 32001, 2688, 5253, 304, 320, 14573, 5261, 1169, 22131, 29914, 25741, 29892, 5412, 491, 317, 15715, 29896, 15882, 29892, 4023, 29894, 2868, 515, 320, 25832, 1299, 7288, 5261, 1169, 797, 7068, 29914, 970, 9279, 6421, 515, 4655, 5849, 363, 2710, 313, 28712, 16046, 29892, 11786, 28632, 29892, 2992, 1846, 32001, 322, 3577, 28914, 313, 10251, 713, 29892, 10772, 2227, 29892, 405, 13427, 29892, 2992, 6250, 32001, 1876, 1169, 297, 278, 8783, 526, 491, 320, 25832, 6344, 2806, 943, 22131, 29914, 15717, 29892, 5412, 491, 779, 6990, 29938, 978, 29892, 4876, 4535, 5854, 29938, 11000, 29889, 32001, 450, 8783, 2996, 408, 1023, 1104, 1288, 6131, 29892, 697, 363, 25741, 322, 697, 363, 15717, 29892, 411, 278, 4642, 29371, 278, 7480, 3025, 263, 9117, 1820, 29889, 32001, 320, 361, 5426, 13, 9760, 1948, 297, 278, 9063, 1591, 3743, 278, 1494, 4235, 29901, 9063, 317, 15715, 29896, 15882, 29892, 4148, 322, 844, 5171, 5335, 342, 15092, 29892, 4148, 322, 844, 5171, 2893, 14903, 313, 20275, 16750, 278, 4148, 1591, 467, 32001, 450, 21578, 1546, 9063, 15717, 322, 9063, 2153, 2041, 515, 11786, 29892, 607, 6511, 304, 9063, 263, 1735, 4148, 287, 491, 4856, 1683, 29889, 32001, 1152, 445, 6559, 591, 21309, 373, 15717, 322, 17262, 9063, 2153, 29892, 408, 278, 4328, 1546, 278, 1023, 338, 451, 8018, 363, 1749, 5925, 5155, 322, 278, 5253, 310, 25741, 411, 263, 844, 5171, 916, 1135, 967, 4148, 338, 3480, 3473, 1821, 29889, 32001, 320, 7241, 13, 2831, 1269, 6251, 297, 278, 4148, 1591, 591, 505, 4148, 2989, 1024, 322, 4876, 408, 1023, 5004, 6031, 310, 10650, 6262, 29889, 32001, 1334, 6206, 2411, 15273, 1821, 470, 18325, 519, 2983, 393, 29901, 526, 451, 1602, 397, 519, 408, 18351, 29899, 29947, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12283, 7099, 397, 519, 29914, 4148, 2983, 6206, 511, 526, 4876, 14157, 2012, 310, 2983, 3441, 25832, 6344, 2806, 943, 29934, 29885, 9823, 29914, 4954, 7039, 4907, 511, 5718, 310, 871, 9654, 4890, 3441, 25832, 6344, 2806, 943, 29934, 29885, 10358, 804, 23201, 1712, 901, 1135, 29871, 29896, 29900, 8958, 1661, 29899, 1026, 2153, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12283, 15670, 23201, 526, 5520, 1135, 29871, 29896, 29900, 29900, 4890, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12229, 549, 12495, 32001, 2860, 21166, 29892, 1048, 320, 25832, 6344, 2806, 943, 29925, 15273, 1821, 2052, 307, 29916, 29914, 15717, 3441, 25832, 6344, 2806, 943, 29925, 15273, 1821, 29925, 312, 29914, 310, 278, 2847, 8783, 29897, 9488, 363, 4340, 7418, 29889, 32001, 3940, 393, 278, 5253, 310, 970, 775, 25741, 313, 392, 15717, 29897, 11122, 297, 278, 13, 11228, 8783, 25088, 18709, 9247, 975, 13, 2230, 2651, 2036, 29912, 29879, 1332, 29899, 771, 854, 749, 29899, 331, 344, 1012, 361, 29887, 798, 386, 1003, 29892, 408, 4318, 363, 25741, 297, 13, 29905, 29907, 999, 29912, 1003, 29901, 29887, 798, 386, 1012, 2870, 29901, 515, 395, 29896, 29900, 29985, 29946, 29938, 25741, 297, 29871, 29896, 29929, 29955, 29896, 29892, 304, 395, 29896, 29900, 29985, 29953, 29938, 297, 29871, 29896, 29929, 29929, 29947, 29892, 304, 13, 284, 3242, 395, 29896, 29900, 29985, 29929, 29938, 297, 29871, 29906, 29900, 29906, 29900, 29905, 7241, 29889, 32001, 1094, 263, 17004, 278, 8900, 534, 1975, 10331, 304, 367, 901, 13, 13844, 297, 7786, 1602, 3076, 1135, 297, 29871, 29946, 29900, 29974, 1629, 29899, 1025, 6743, 29892, 2861, 304, 13964, 4586, 373, 13, 735, 1112, 9247, 7200, 23093, 29889, 32001, 320, 26956, 29912, 7999, 324, 10610, 29913, 13, 13, 29905, 463, 29912, 4532, 29913, 13, 29871, 320, 9525, 13, 29871, 320, 7313, 29961, 24049, 29892, 15450, 29922, 29953, 4912, 29871, 29953, 4912, 29871, 29900, 29871, 29900, 29892, 2103, 2013, 16292, 3199, 1491, 1727, 1080, 29899, 2470, 29913, 13, 29871, 320, 6671, 29912, 1576, 320, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 1304, 408, 1737, 324, 10610, 22525, 5003, 32001, 320, 1643, 29912, 1003, 29901, 11526, 1958, 29913, 13, 29905, 355, 29912, 4532, 29913, 13, 13, 2887, 1737, 324, 10610, 22525, 591, 671, 11758, 3186, 12786, 10723, 515, 278, 3303, 18269, 1737, 359, 305, 2004, 2651, 2036, 29912, 348, 29896, 29929, 29929, 29929, 479, 359, 305, 2004, 1836, 32001, 1763, 4772, 2432, 3381, 491, 2919, 10916, 313, 29872, 29889, 29887, 1696, 7551, 470, 12710, 29897, 2629, 11758, 12786, 29892, 591, 19412, 322, 6219, 777, 12786, 2729, 373, 1737, 12122, 23203, 537, 322, 278, 19383, 310, 758, 331, 8946, 16375, 29769, 5680, 29892, 1316, 408, 19182, 4086, 29889, 32001, 320, 29907, 999, 29912, 1003, 29901, 11526, 1958, 29913, 3697, 278, 2186, 1051, 310, 320, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 1304, 408, 1737, 324, 10610, 22525, 297, 445, 6559, 29889, 32001, 1879, 324, 10610, 310, 9063, 15717, 304, 3186, 12786, 3913, 278, 1023, 19595, 653, 13698, 9129, 297, 2651, 2036, 29912, 293, 344, 29899, 344, 275, 29899, 29906, 29900, 29906, 29906, 29899, 26098, 1118, 23359, 337, 13998, 2400, 29889, 32001, 450, 937, 697, 337, 3687, 373, 278, 4234, 775, 2246, 29899, 5563, 5354, 313, 617, 29911, 10249, 29897, 310, 4876, 14157, 23892, 515, 9063, 15562, 29892, 321, 29889, 29887, 1696, 320, 16266, 26139, 1341, 1118, 320, 16266, 26139, 582, 1118, 320, 16266, 26139, 18038, 1118, 2992, 29889, 32001, 1334, 4687, 515, 278, 306, 2190, 29909, 1051, 310, 13548, 2931, 21759, 29911, 10249, 29879, 2651, 2036, 29912, 6011, 29899, 29883, 312, 430, 29913, 322, 7522, 20545, 1269, 6590, 20123, 304, 263, 3646, 3186, 5120, 29889, 32001, 450, 1473, 1737, 324, 10610, 11043, 3913, 278, 17998, 9210, 310, 9063, 5335, 342, 15092, 313, 29872, 29889, 29887, 1696, 17998, 29899, 29900, 29945, 29901, 29900, 29900, 29897, 322, 4148, 2983, 304, 8161, 278, 1556, 5517, 3186, 5120, 310, 278, 9063, 4148, 29889, 32001, 1152, 1269, 17998, 9210, 591, 8161, 263, 1051, 310, 15878, 7600, 313, 13509, 29892, 2106, 29892, 470, 14278, 20123, 29897, 297, 278, 3186, 393, 29892, 472, 278, 931, 310, 393, 9063, 29892, 750, 393, 17998, 9210, 29936, 9063, 931, 338, 1820, 1244, 29892, 408, 4234, 17998, 1283, 7224, 13100, 975, 931, 2861, 304, 29431, 3620, 29889, 32001, 1763, 1207, 445, 3683, 3381, 591, 671, 278, 306, 2190, 29909, 931, 10640, 2566, 2651, 2036, 29912, 17559, 1272, 1836, 32001, 1987, 591, 3566, 304, 1269, 2058, 263, 8158, 393, 4332, 1973, 278, 4188, 22342, 393, 263, 2183, 4148, 1024, 338, 17443, 310, 372, 29889, 32001, 1763, 445, 1095, 591, 671, 278, 28297, 29890, 15451, 8783, 310, 278, 29511, 310, 278, 1556, 3619, 937, 322, 3942, 2983, 607, 29892, 439, 11427, 515, 2651, 2036, 29912, 1079, 29890, 799, 29899, 7039, 6177, 2802, 277, 12181, 4954, 16123, 2247, 278, 26368, 5528, 5084, 310, 363, 264, 1280, 322, 269, 595, 1280, 7371, 515, 263, 2566, 310, 320, 1949, 29912, 29946, 29871, 29900, 29946, 29946, 29871, 29945, 29946, 29953, 29871, 29929, 29941, 29947, 29913, 2305, 313, 29945, 29945, 29889, 29945, 8958, 310, 8471, 2305, 297, 29871, 29906, 29900, 29896, 29946, 467, 32001, 1094, 310, 3839, 29871, 29906, 29900, 29896, 29929, 372, 18469, 320, 1949, 29912, 29906, 29955, 29871, 29953, 29953, 29906, 29871, 29947, 29900, 29896, 29913, 363, 264, 1280, 322, 320, 1949, 29912, 29906, 29955, 29871, 29906, 29900, 29953, 29871, 29947, 29906, 29896, 29913, 269, 595, 1280, 297, 29871, 29906, 29941, 29953, 24894, 8977, 1080, 29889, 4907, 29913, 32001, 1094, 297, 1749, 8783, 15717, 526, 2989, 1024, 6031, 313, 29878, 1624, 1135, 6219, 491, 937, 29914, 11922, 1024, 511, 591, 937, 5993, 675, 2983, 313, 1609, 1999, 1331, 322, 1206, 3620, 29897, 322, 769, 16280, 5375, 18897, 297, 1716, 937, 322, 3942, 2983, 10868, 8857, 29889, 32001, 1152, 1269, 1543, 1476, 297, 1024, 8857, 591, 22932, 278, 2058, 4665, 29905, 6661, 1333, 24212, 8875, 491, 278, 1024, 10868, 304, 4017, 263, 5645, 393, 338, 29839, 304, 278, 1353, 310, 12407, 24638, 393, 1024, 313, 6979, 29897, 297, 278, 2702, 2058, 29889, 32001, 320, 6661, 1333, 300, 1062, 29912, 1762, 4017, 4665, 2025, 1338, 5634, 294, 278, 17837, 310, 4954, 6689, 4907, 338, 25745, 23724, 29901, 2989, 10916, 325, 13217, 29879, 29399, 310, 2919, 10916, 805, 9450, 2999, 931, 29920, 2873, 5634, 705, 671, 263, 29544, 310, 7601, 8974, 313, 29872, 29889, 29887, 1696, 5874, 28007, 511, 322, 1661, 29899, 16072, 6743, 313, 29872, 29889, 29887, 1696, 14109, 7456, 467, 29913, 32001, 1334, 2533, 445, 4377, 363, 599, 3161, 304, 4017, 263, 2058, 8158, 29892, 17140, 701, 411, 263, 1051, 310, 779, 6990, 29938, 6689, 29892, 8158, 4535, 5854, 29938, 11000, 29889, 32001, 1334, 769, 8877, 445, 1051, 491, 278, 3186, 5120, 393, 263, 2058, 14393, 304, 322, 2533, 278, 8158, 363, 599, 278, 7600, 297, 1269, 5120, 304, 4017, 385, 12463, 8158, 29892, 6590, 304, 278, 4188, 22342, 393, 278, 9063, 14393, 304, 263, 2183, 3186, 5120, 29889, 32001, 1334, 3566, 278, 6257, 9063, 408, 6421, 515, 278, 3186, 5120, 411, 278, 9939, 8158, 29889, 32001, 450, 4876, 29899, 6707, 11043, 9378, 414, 515, 278, 9078, 322, 443, 5521, 8362, 671, 310, 21759, 29911, 10249, 29879, 29901, 1556, 18777, 671, 10035, 323, 10249, 29879, 1316, 408, 320, 16266, 26139, 510, 1118, 320, 16266, 26139, 990, 1118, 470, 320, 16266, 26139, 1212, 1836, 32001, 12808, 445, 947, 451, 3799, 26018, 4822, 20542, 29901, 3148, 29899, 6707, 18777, 29892, 363, 1342, 29892, 671, 278, 320, 16266, 26139, 375, 29913, 21759, 29911, 10249, 1568, 901, 269, 2495, 290, 368, 1135, 1009, 7824, 6795, 20895, 29889, 32001, 1551, 278, 916, 1361, 278, 9210, 29914, 978, 29899, 6707, 11043, 337, 3687, 373, 278, 17998, 9210, 310, 278, 9063, 5335, 342, 15092, 29889, 32001, 16809, 304, 5780, 22920, 373, 13897, 731, 14340, 29892, 263, 2919, 1353, 310, 25741, 297, 278, 8783, 756, 385, 17998, 9210, 5186, 304, 5225, 29889, 32001, 910, 6602, 29879, 3109, 7786, 25741, 3441, 14573, 5261, 1169, 29911, 29999, 29999, 13985, 1349, 681, 392, 27418, 6478, 29914, 310, 29871, 29906, 29900, 29906, 29900, 29879, 25741, 505, 263, 5225, 9210, 29897, 1135, 9642, 6743, 3441, 14573, 5261, 1169, 29911, 29999, 29999, 13985, 1349, 681, 392, 29914, 297, 29871, 29906, 29900, 29900, 29900, 467, 32001, 1094, 263, 1121, 278, 9210, 29914, 978, 29899, 6707, 11043, 1033, 1095, 701, 6459, 292, 263, 2919, 6232, 310, 9642, 25741, 408, 4148, 287, 491, 11715, 18777, 29892, 322, 304, 263, 3109, 261, 15834, 4092, 550, 29889, 32001, 1763, 6795, 1438, 5626, 591, 14405, 278, 1023, 1737, 324, 10610, 13698, 4208, 491, 15399, 278, 9210, 29914, 978, 29899, 6707, 13698, 304, 599, 25741, 411, 263, 1661, 29899, 9171, 17998, 9210, 29892, 322, 278, 4876, 29899, 6707, 373, 304, 599, 916, 25741, 29889, 32001, 320, 2042, 29912, 12191, 322, 8565, 1558, 291, 29913, 13, 29905, 1643, 29912, 3471, 29901, 9902, 29913, 13, 13, 29905, 463, 29912, 4532, 4044, 13, 29871, 320, 9525, 13, 29871, 320, 7313, 29961, 2103, 2013, 16292, 3199, 1429, 287, 29889, 5140, 29913, 13, 29871, 320, 6671, 29912, 29934, 20819, 310, 25741, 313, 27215, 29897, 322, 6136, 15717, 313, 22503, 29897, 491, 3186, 10640, 975, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29900, 3785, 5003, 32001, 320, 9868, 29961, 14732, 3199, 7264, 287, 2594, 8727, 6445, 278, 3186, 10640, 364, 2219, 359, 363, 25741, 322, 15717, 975, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29900, 3785, 5003, 32001, 320, 1643, 29912, 1003, 29901, 9902, 29913, 13, 29905, 355, 29912, 4532, 4044, 13, 13, 13, 29871, 13, 1762, 1234, 320, 25029, 29912, 29878, 29939, 29901, 479, 397, 24974, 29913, 591, 22229, 278, 1353, 310, 25741, 322, 8359, 15717, 639, 1629, 322, 639, 3186, 10640, 29889, 32001, 1334, 2198, 278, 7625, 2582, 297, 320, 29907, 999, 29912, 1003, 29901, 9902, 29913, 408, 1023, 5096, 287, 2594, 24469, 29892, 6445, 1629, 368, 2867, 3204, 29879, 363, 25741, 322, 15717, 8307, 29889, 32001, 7569, 2594, 11524, 263, 1629, 322, 338, 8877, 287, 297, 269, 29399, 6445, 278, 9063, 29914, 8921, 11959, 363, 1269, 310, 278, 3186, 12786, 310, 320, 29907, 999, 29912, 1003, 29901, 11526, 1958, 29913, 297, 393, 1629, 29889, 32001, 1763, 4772, 714, 27801, 2861, 304, 805, 272, 26538, 17737, 29560, 29892, 297, 278, 4148, 8727, 591, 871, 2050, 15717, 2534, 26869, 472, 3203, 29871, 29945, 25741, 297, 263, 2183, 1629, 29889, 32001, 5806, 5366, 1747, 534, 1975, 297, 278, 24469, 6456, 393, 278, 3001, 3694, 310, 25741, 322, 15717, 6548, 18709, 9247, 975, 931, 29889, 32001, 10133, 363, 278, 937, 2440, 297, 278, 24469, 29892, 278, 1353, 310, 848, 3291, 297, 777, 3186, 12786, 508, 367, 14154, 2319, 29892, 411, 8178, 27721, 373, 278, 25806, 310, 534, 1975, 29889, 32001, 320, 26956, 29912, 7999, 12122, 6894, 537, 975, 931, 29913, 13, 13, 3563, 497, 29892, 278, 2498, 534, 355, 5692, 304, 367, 393, 278, 320, 4534, 29912, 479, 12122, 6894, 537, 297, 970, 775, 338, 10231, 6177, 4644, 6813, 322, 4092, 5136, 630, 1009, 4954, 24130, 749, 4907, 2745, 278, 7256, 310, 278, 29871, 29929, 29900]} {'input_ids': [1, 320, 2042, 29912, 25898, 29913, 13, 29905, 1643, 29912, 3471, 29901, 23333, 29913, 13, 13, 29905, 7278, 29912, 29954, 1581, 6894, 537, 1118, 470, 901, 4049, 967, 10225, 727, 974, 29892, 4249, 27138, 304, 13, 20415, 5849, 14188, 756, 1063, 26606, 12399, 297, 7786, 2440, 29889, 32001, 512, 13, 1595, 16311, 29892, 278, 10122, 310, 29892, 9545, 310, 29892, 322, 6795, 1004, 25414, 363, 320, 7278, 29912, 26098, 13, 29871, 24003, 29913, 297, 12362, 29914, 6585, 7562, 18540, 313, 5800, 1799, 29897, 505, 4520, 263, 3287, 310, 8570, 13, 957, 278, 4940, 316, 6332, 2651, 2036, 29912, 29881, 16093, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 3855, 5871, 29906, 29900, 29896, 29900, 29895, 311, 29893, 2770, 29892, 13, 29871, 1055, 29888, 375, 29906, 29900, 29896, 29906, 5041, 267, 29892, 413, 434, 305, 1358, 29906, 29900, 29896, 29906, 26098, 29888, 2209, 29892, 19723, 5475, 4979, 29906, 29900, 29896, 29946, 26098, 29892, 13, 29871, 697, 309, 29906, 29900, 29896, 29953, 311, 5365, 550, 332, 6950, 29892, 696, 7586, 29906, 29900, 29896, 29953, 29893, 2770, 7192, 2209, 29892, 1935, 15044, 29906, 29900, 29896, 29955, 26098, 29892, 13, 29871, 503, 562, 305, 3350, 492, 29906, 29900, 29906, 29896, 26098, 1836, 32001, 320, 7278, 29912, 7999, 12122, 6894, 537, 29913, 338, 373, 278, 916, 1361, 278, 13, 14380, 310, 6894, 537, 393, 380, 1567, 515, 27138, 297, 777, 5534, 6354, 6421, 13, 3166, 1422, 3186, 12786, 322, 4185, 1973, 29889, 32001, 26304, 6894, 537, 297, 18322, 1799, 756, 4520, 13774, 2217, 8570, 297, 21344, 368, 13, 13129, 29889, 32001, 512, 3153, 29892, 1550, 3031, 979, 18994, 29899, 6707, 322, 13, 3149, 29899, 262, 29899, 2230, 18350, 29899, 7052, 11898, 310, 278, 1737, 12122, 1677, 1144, 310, 18322, 1799, 13, 21570, 29560, 1863, 2651, 2036, 29912, 12443, 10578, 29906, 29900, 29900, 29945, 5062, 11235, 29892, 13260, 333, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 13, 29871, 2594, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29892, 1850, 29882, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29892, 696, 7586, 29906, 29900, 29896, 29946, 7610, 345, 2941, 271, 24541, 29892, 13, 29871, 281, 496, 29879, 29906, 29900, 29906, 29896, 2209, 479, 5275, 1118, 2919, 29899, 7052, 25579, 979, 11898, 310, 278, 1737, 12122, 13, 12574, 310, 18322, 1799, 17737, 29560, 526, 1603, 10225, 292, 29889, 32001, 10506, 263, 4323, 23378, 13, 18609, 2133, 723, 367, 5407, 304, 1871, 1602, 12112, 4475, 304, 5534, 13, 25431, 10907, 2651, 2036, 29912, 2276, 5824, 19982, 29906, 29900, 29900, 29955, 23705, 1338, 29893, 996, 29913, 322, 298, 8491, 16650, 583, 297, 278, 13, 19678, 15483, 313, 1806, 29897, 9999, 29892, 408, 1532, 408, 29126, 2114, 950, 2472, 13, 517, 278, 2553, 1078, 373, 278, 17407, 10879, 322, 5374, 3002, 310, 18322, 1799, 2820, 278, 3186, 29889, 32001, 320, 26956, 29912, 1323, 3224, 29879, 29913, 13, 13, 3047, 445, 664, 591, 29126, 304, 3802, 445, 17261, 491, 7512, 292, 320, 4534, 29912, 1552, 937, 13, 29871, 25579, 979, 6559, 310, 278, 1737, 12122, 3978, 310, 17737, 29560, 304, 970, 775, 13, 29871, 975, 29871, 29945, 29900, 2440, 5003, 32001, 26321, 29892, 591, 3867, 263, 758, 2576, 3821, 1234, 304, 278, 13, 23031, 292, 5925, 1139, 29901, 13, 29905, 463, 29912, 690, 2842, 12470, 29913, 13, 29871, 3645, 607, 3186, 12786, 437, 15717, 310, 970, 368, 3625, 25741, 2041, 515, 13, 29871, 322, 920, 756, 372, 3939, 975, 278, 4940, 29871, 29945, 29900, 2440, 29973, 32001, 320, 1643, 29912, 29878, 29939, 29901, 479, 397, 24974, 29913, 13, 29905, 355, 29912, 690, 2842, 12470, 29913, 13, 4806, 671, 408, 8783, 278, 320, 23066, 29950, 29914, 18871, 2651, 2036, 29912, 29879, 1332, 29875, 4569, 29906, 29900, 29896, 29955, 29913, 322, 27599, 515, 372, 13, 29906, 29889, 29906, 24464, 29905, 13007, 25741, 3190, 2347, 515, 29871, 29896, 29953, 29900, 7284, 29905, 13007, 9279, 322, 4148, 287, 491, 13, 29946, 29941, 7284, 29905, 13007, 15717, 2645, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29896, 931, 3785, 29889, 32001, 1334, 1737, 324, 542, 403, 18777, 304, 13, 29905, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 29892, 773, 408, 18470, 4876, 4234, 775, 2246, 29899, 5563, 21904, 313, 617, 29911, 10249, 29879, 29897, 322, 29871, 13, 8921, 313, 4102, 29914, 4230, 29897, 2983, 9401, 411, 1024, 18822, 2820, 278, 3186, 29892, 322, 17998, 1283, 7224, 29871, 13, 1195, 287, 515, 9063, 15562, 29889, 32001, 1334, 1284, 10757, 310, 278, 4688, 8022, 749, 310, 4644, 6813, 297, 1722, 2752, 13, 20415, 29892, 2678, 8772, 491, 4092, 29889, 32001, 2860, 393, 3785, 29892, 278, 1737, 12122, 6894, 537, 29871, 13, 262, 970, 775, 756, 1063, 21003, 10231, 29889, 32001, 1334, 884, 12439, 8018, 15839, 528, 17741, 13, 12817, 304, 278, 1095, 310, 278, 8291, 6415, 29129, 322, 278, 7910, 310, 14137, 4631, 4135, 297, 13, 23369, 1705, 322, 4275, 14325, 29892, 408, 1532, 408, 310, 2545, 1664, 17292, 28342, 763, 25539, 1608, 322, 13, 25719, 10298, 4822, 10916, 313, 6727, 16783, 29914, 331, 16783, 467, 32001, 320, 26956, 29912, 1469, 20847, 3097, 5003, 32001, 319, 1634, 1414, 3577, 363, 445, 5650, 338, 3625, 515, 796, 264, 8144, 472, 13, 29905, 2271, 29912, 991, 597, 1867, 29875, 29889, 990, 29914, 29896, 29900, 29889, 29945, 29906, 29947, 29896, 29914, 2256, 8144, 29889, 29953, 29941, 29929, 29900, 29941, 29945, 29945, 29913, 2651, 2036, 29912, 3445, 1414, 29899, 5113, 1836, 32001, 320, 2042, 29912, 9662, 630, 5244, 29913, 13, 29905, 1643, 29912, 3471, 29901, 12817, 29913, 13, 13, 29933, 720, 4688, 322, 7786, 1736, 2651, 2036, 29912, 12443, 10578, 29906, 29900, 29900, 29945, 5062, 11235, 29892, 13260, 333, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 13, 29871, 696, 7586, 29906, 29900, 29896, 29946, 7610, 345, 2941, 271, 24541, 29892, 697, 309, 29906, 29900, 29896, 29953, 311, 5365, 550, 332, 6950, 29913, 505, 2931, 1891, 278, 13, 479, 5275, 310, 12362, 29914, 6585, 7562, 18540, 313, 5800, 1799, 29897, 773, 320, 7278, 29912, 6734, 26946, 952, 1118, 13, 4716, 3867, 1880, 29899, 29567, 6089, 541, 526, 9078, 297, 2159, 313, 29906, 29899, 29945, 5940, 29968, 18777, 29897, 13, 392, 508, 367, 4768, 1463, 491, 5221, 424, 23460, 29889, 32001, 512, 29871, 29906, 29900, 29900, 29947, 2261, 801, 2681, 634, 394, 8770, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29913, 18043, 263, 3031, 979, 13, 16961, 29899, 7052, 313, 1454, 278, 931, 29897, 6559, 373, 18322, 1799, 320, 7278, 29912, 479, 5275, 773, 1375, 292, 7047, 13, 29871, 28914, 313, 4345, 29934, 29897, 13698, 1836, 32001, 2688, 29537, 287, 278, 3978, 310, 29871, 29896, 5940, 29924, 17737, 29560, 13, 4746, 278, 7562, 2831, 479, 1404, 2566, 322, 611, 6504, 1051, 3190, 3145, 975, 278, 13, 29896, 29929, 29929, 29929, 489, 29906, 29900, 29900, 29945, 3785, 29892, 773, 408, 18470, 2472, 2788, 304, 1749, 29879, 29901, 4876, 21904, 13, 392, 17998, 1283, 7224, 29889, 32001, 450, 12399, 3785, 313, 29955, 2440, 29897, 297, 2651, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29913, 338, 20511, 1135, 29871, 13, 5816, 338, 12399, 297, 278, 2198, 5650, 313, 29945, 29900, 2440, 29897, 322, 278, 848, 8974, 526, 29871, 13, 27489, 873, 1422, 29936, 411, 393, 297, 3458, 29892, 1749, 2582, 1510, 263, 10029, 7200, 14978, 310, 29871, 13, 15654, 273, 325, 13217, 29940, 2072, 3082, 20706, 29889, 32001, 7280, 29190, 936, 664, 515, 29871, 29906, 29900, 29896, 29900, 491, 14619, 29882, 371, 4099, 29894, 322, 13, 29950, 309, 1372, 2651, 2036, 29912, 29873, 19426, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29913, 4023, 29894, 2868, 1583, 29899, 7099, 433, 1127, 1737, 12122, 13, 2029, 800, 310, 25492, 15303, 8304, 3598, 1494, 1009, 12368, 29892, 13, 15914, 292, 2472, 363, 779, 14850, 4535, 29892, 29955, 29900, 5940, 29968, 25492, 4160, 29889, 32001, 319, 1407, 7786, 13, 1287, 2651, 2036, 29912, 21686, 29879, 29906, 29900, 29906, 29896, 2209, 479, 5275, 29913, 491, 399, 496, 29879, 634, 394, 13217, 5349, 1737, 324, 542, 630, 4203, 263, 7284, 13, 28712, 16046, 4160, 29892, 2534, 26869, 472, 3203, 29871, 29896, 29900, 29900, 25741, 1269, 29892, 322, 1058, 13, 1311, 29899, 7099, 8663, 14354, 373, 1009, 25492, 28723, 29889, 32001, 5806, 278, 6559, 338, 13, 3149, 29899, 262, 29899, 2230, 408, 310, 29871, 29906, 29900, 29906, 29896, 29892, 278, 15717, 7252, 1009, 1284, 886, 13, 351, 475, 303, 2651, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29892, 1850, 29882, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29913, 304, 13, 18609, 675, 278, 14675, 310, 18322, 1799, 1737, 5275, 975, 278, 931, 15101, 845, 1862, 4586, 491, 13, 1552, 2211, 11898, 29889, 32001, 3831, 1965, 411, 3517, 29190, 936, 1736, 29892, 1749, 6559, 338, 1568, 7200, 6287, 5634, 29882, 5555, 13, 7054, 12339, 287, 29871, 29946, 29941, 7284, 29905, 13007, 15717, 310, 29871, 29906, 29889, 29906, 24464, 29905, 13007, 25741, 515, 29871, 29896, 29953, 29900, 7284, 29905, 13007, 13, 16418, 5634, 5426, 11267, 979, 975, 29871, 29945, 29900, 2440, 310, 970, 775, 20706, 3265, 1135, 13, 3149, 297, 931, 29892, 322, 884, 901, 2691, 29899, 3874, 1312, 313, 2541, 1629, 29899, 1609, 29899, 6360, 3803, 1070, 537, 975, 13, 1552, 8900, 3785, 467, 32001, 8108, 1189, 1711, 29892, 1749, 6559, 337, 3687, 373, 10079, 11264, 13, 3924, 313, 8257, 29903, 29897, 9063, 848, 3265, 1135, 7481, 29899, 7099, 433, 1127, 4423, 2472, 29889, 32001, 5901, 1736, 5634, 262, 3153, 278, 664, 491, 8432, 2651, 2036, 29912, 18386, 709, 29906, 29900, 29896, 29941, 2209, 29881, 24974, 29913, 13, 392, 29892, 901, 10325, 29892, 390, 579, 468, 29875, 634, 394, 8770, 2036, 29912, 29878, 579, 468, 29875, 29906, 29900, 29896, 29953, 479, 711, 3173, 29892, 13, 29871, 364, 579, 468, 29875, 29906, 29900, 29896, 29947, 479, 711, 3173, 29892, 544, 1648, 29906, 29900, 29906, 29896, 479, 468, 1581, 29881, 24974, 29913, 5634, 17532, 12399, 1737, 12122, 13, 29905, 7278, 29912, 29881, 24974, 322, 24003, 1118, 474, 29889, 29872, 1696, 278, 15834, 304, 607, 278, 3978, 310, 18322, 1799, 13, 17426, 6602, 1009, 11465, 1230, 14137, 14188, 29889, 32001, 512, 445, 664, 591, 2931, 1891, 1737, 12122, 6894, 537, 297, 970, 775, 363, 278, 937, 13, 2230, 472, 445, 6287, 29892, 1716, 297, 4958, 310, 17737, 29560, 322, 15500, 3785, 29889, 32001, 1334, 437, 13, 1333, 22002, 280, 278, 24003, 10696, 29892, 541, 3867, 29190, 936, 848, 322, 1284, 886, 393, 508, 367, 13, 280, 369, 4063, 304, 393, 1095, 408, 5434, 664, 29889, 32001, 320, 7278, 29912, 12756, 7047, 21639, 29913, 2651, 2036, 29912, 2276, 5824, 19982, 29906, 29900, 29900, 29955, 23705, 1338, 29893, 996, 29913, 338, 278, 13, 1491, 29899, 2671, 310, 7047, 21639, 393, 756, 29537, 287, 278, 18066, 267, 310, 21640, 13, 6734, 24771, 13149, 635, 29892, 3704, 278, 2702, 5932, 310, 920, 304, 5376, 13, 2541, 1737, 12122, 6894, 537, 2651, 2036, 29912, 13376, 303, 456, 29906, 29900, 29900, 29953, 10945, 3359, 29892, 1424, 29440, 29906, 29900, 29896, 29946, 23027, 5933, 1836, 32001, 3826, 3076, 2678, 278, 2198, 6559, 8128, 10757, 393, 508, 367, 1304, 29892, 297, 278, 13, 14940, 1206, 310, 970, 775, 322, 472, 263, 1407, 2919, 6287, 29892, 304, 11539, 607, 13, 14032, 4637, 310, 5534, 7047, 21639, 505, 9820, 484, 15774, 29889, 32001, 320, 2042, 29912, 4062, 3002, 29913, 13, 29905, 1643, 29912, 3471, 29901, 5696, 29913, 13, 13, 13, 29905, 1482, 361, 29905, 361, 29887, 798, 386, 1003, 29871, 320, 29887, 798, 386, 1003, 3009, 13, 29905, 361, 29887, 798, 386, 1003, 13, 29905, 463, 29912, 4532, 29913, 13, 29871, 320, 7313, 29961, 2103, 2013, 4914, 2103, 3199, 6360, 368, 29899, 2055, 1169, 29913, 13, 29871, 320, 6671, 29912, 12883, 368, 970, 25741, 975, 931, 313, 1188, 6287, 467, 13, 29913, 32001, 320, 1643, 29912, 1003, 29901, 29887, 798, 386, 29913, 13, 29905, 355, 29912, 4532, 29913, 13, 29905, 7241, 13, 13, 29905, 26956, 29912, 16390, 24541, 29913, 13, 13, 4806, 27387, 515, 320, 23066, 29950, 29914, 2651, 2036, 29912, 29879, 1332, 29899, 1516, 29878, 29906, 29900, 29896, 29929, 29899, 24713, 29913, 599, 25741, 3190, 2347, 2745, 320, 25832, 1964, 579, 1523, 2415, 2539, 6294, 32001, 2688, 5253, 304, 320, 14573, 5261, 1169, 22131, 29914, 25741, 29892, 5412, 491, 317, 15715, 29896, 15882, 29892, 4023, 29894, 2868, 515, 320, 25832, 1299, 7288, 5261, 1169, 797, 7068, 29914, 970, 9279, 6421, 515, 4655, 5849, 363, 2710, 313, 28712, 16046, 29892, 11786, 28632, 29892, 2992, 1846, 32001, 322, 3577, 28914, 313, 10251, 713, 29892, 10772, 2227, 29892, 405, 13427, 29892, 2992, 6250, 32001, 1876, 1169, 297, 278, 8783, 526, 491, 320, 25832, 6344, 2806, 943, 22131, 29914, 15717, 29892, 5412, 491, 779, 6990, 29938, 978, 29892, 4876, 4535, 5854, 29938, 11000, 29889, 32001, 450, 8783, 2996, 408, 1023, 1104, 1288, 6131, 29892, 697, 363, 25741, 322, 697, 363, 15717, 29892, 411, 278, 4642, 29371, 278, 7480, 3025, 263, 9117, 1820, 29889, 32001, 320, 361, 5426, 13, 9760, 1948, 297, 278, 9063, 1591, 3743, 278, 1494, 4235, 29901, 9063, 317, 15715, 29896, 15882, 29892, 4148, 322, 844, 5171, 5335, 342, 15092, 29892, 4148, 322, 844, 5171, 2893, 14903, 313, 20275, 16750, 278, 4148, 1591, 467, 32001, 450, 21578, 1546, 9063, 15717, 322, 9063, 2153, 2041, 515, 11786, 29892, 607, 6511, 304, 9063, 263, 1735, 4148, 287, 491, 4856, 1683, 29889, 32001, 1152, 445, 6559, 591, 21309, 373, 15717, 322, 17262, 9063, 2153, 29892, 408, 278, 4328, 1546, 278, 1023, 338, 451, 8018, 363, 1749, 5925, 5155, 322, 278, 5253, 310, 25741, 411, 263, 844, 5171, 916, 1135, 967, 4148, 338, 3480, 3473, 1821, 29889, 32001, 320, 7241, 13, 2831, 1269, 6251, 297, 278, 4148, 1591, 591, 505, 4148, 2989, 1024, 322, 4876, 408, 1023, 5004, 6031, 310, 10650, 6262, 29889, 32001, 1334, 6206, 2411, 15273, 1821, 470, 18325, 519, 2983, 393, 29901, 526, 451, 1602, 397, 519, 408, 18351, 29899, 29947, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12283, 7099, 397, 519, 29914, 4148, 2983, 6206, 511, 526, 4876, 14157, 2012, 310, 2983, 3441, 25832, 6344, 2806, 943, 29934, 29885, 9823, 29914, 4954, 7039, 4907, 511, 5718, 310, 871, 9654, 4890, 3441, 25832, 6344, 2806, 943, 29934, 29885, 10358, 804, 23201, 1712, 901, 1135, 29871, 29896, 29900, 8958, 1661, 29899, 1026, 2153, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12283, 15670, 23201, 526, 5520, 1135, 29871, 29896, 29900, 29900, 4890, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12229, 549, 12495, 32001, 2860, 21166, 29892, 1048, 320, 25832, 6344, 2806, 943, 29925, 15273, 1821, 2052, 307, 29916, 29914, 15717, 3441, 25832, 6344, 2806, 943, 29925, 15273, 1821, 29925, 312, 29914, 310, 278, 2847, 8783, 29897, 9488, 363, 4340, 7418, 29889, 32001, 3940, 393, 278, 5253, 310, 970, 775, 25741, 313, 392, 15717, 29897, 11122, 297, 278, 13, 11228, 8783, 25088, 18709, 9247, 975, 13, 2230, 2651, 2036, 29912, 29879, 1332, 29899, 771, 854, 749, 29899, 331, 344, 1012, 361, 29887, 798, 386, 1003, 29892, 408, 4318, 363, 25741, 297, 13, 29905, 29907, 999, 29912, 1003, 29901, 29887, 798, 386, 1012, 2870, 29901, 515, 395, 29896, 29900, 29985, 29946, 29938, 25741, 297, 29871, 29896, 29929, 29955, 29896, 29892, 304, 395, 29896, 29900, 29985, 29953, 29938, 297, 29871, 29896, 29929, 29929, 29947, 29892, 304, 13, 284, 3242, 395, 29896, 29900, 29985, 29929, 29938, 297, 29871, 29906, 29900, 29906, 29900, 29905, 7241, 29889, 32001, 1094, 263, 17004, 278, 8900, 534, 1975, 10331, 304, 367, 901, 13, 13844, 297, 7786, 1602, 3076, 1135, 297, 29871, 29946, 29900, 29974, 1629, 29899, 1025, 6743, 29892, 2861, 304, 13964, 4586, 373, 13, 735, 1112, 9247, 7200, 23093, 29889, 32001, 320, 26956, 29912, 7999, 324, 10610, 29913, 13, 13, 29905, 463, 29912, 4532, 29913, 13, 29871, 320, 9525, 13, 29871, 320, 7313, 29961, 24049, 29892, 15450, 29922, 29953, 4912, 29871, 29953, 4912, 29871, 29900, 29871, 29900, 29892, 2103, 2013, 16292, 3199, 1491, 1727, 1080, 29899, 2470, 29913, 13, 29871, 320, 6671, 29912, 1576, 320, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 1304, 408, 1737, 324, 10610, 22525, 5003, 32001, 320, 1643, 29912, 1003, 29901, 11526, 1958, 29913, 13, 29905, 355, 29912, 4532, 29913, 13, 13, 2887, 1737, 324, 10610, 22525, 591, 671, 11758, 3186, 12786, 10723, 515, 278, 3303, 18269, 1737, 359, 305, 2004, 2651, 2036, 29912, 348, 29896, 29929, 29929, 29929, 479, 359, 305, 2004, 1836, 32001, 1763, 4772, 2432, 3381, 491, 2919, 10916, 313, 29872, 29889, 29887, 1696, 7551, 470, 12710, 29897, 2629, 11758, 12786, 29892, 591, 19412, 322, 6219, 777, 12786, 2729, 373, 1737, 12122, 23203, 537, 322, 278, 19383, 310, 758, 331, 8946, 16375, 29769, 5680, 29892, 1316, 408, 19182, 4086, 29889, 32001, 320, 29907, 999, 29912, 1003, 29901, 11526, 1958, 29913, 3697, 278, 2186, 1051, 310, 320, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 1304, 408, 1737, 324, 10610, 22525, 297, 445, 6559, 29889, 32001, 1879, 324, 10610, 310, 9063, 15717, 304, 3186, 12786, 3913, 278, 1023, 19595, 653, 13698, 9129, 297, 2651, 2036, 29912, 293, 344, 29899, 344, 275, 29899, 29906, 29900, 29906, 29906, 29899, 26098, 1118, 23359, 337, 13998, 2400, 29889, 32001, 450, 937, 697, 337, 3687, 373, 278, 4234, 775, 2246, 29899, 5563, 5354, 313, 617, 29911, 10249, 29897, 310, 4876, 14157, 23892, 515, 9063, 15562, 29892, 321, 29889, 29887, 1696, 320, 16266, 26139, 1341, 1118, 320, 16266, 26139, 582, 1118, 320, 16266, 26139, 18038, 1118, 2992, 29889, 32001, 1334, 4687, 515, 278, 306, 2190, 29909, 1051, 310, 13548, 2931, 21759, 29911, 10249, 29879, 2651, 2036, 29912, 6011, 29899, 29883, 312, 430, 29913, 322, 7522, 20545, 1269, 6590, 20123, 304, 263, 3646, 3186, 5120, 29889, 32001, 450, 1473, 1737, 324, 10610, 11043, 3913, 278, 17998, 9210, 310, 9063, 5335, 342, 15092, 313, 29872, 29889, 29887, 1696, 17998, 29899, 29900, 29945, 29901, 29900, 29900, 29897, 322, 4148, 2983, 304, 8161, 278, 1556, 5517, 3186, 5120, 310, 278, 9063, 4148, 29889, 32001, 1152, 1269, 17998, 9210, 591, 8161, 263, 1051, 310, 15878, 7600, 313, 13509, 29892, 2106, 29892, 470, 14278, 20123, 29897, 297, 278, 3186, 393, 29892, 472, 278, 931, 310, 393, 9063, 29892, 750, 393, 17998, 9210, 29936, 9063, 931, 338, 1820, 1244, 29892, 408, 4234, 17998, 1283, 7224, 13100, 975, 931, 2861, 304, 29431, 3620, 29889, 32001, 1763, 1207, 445, 3683, 3381, 591, 671, 278, 306, 2190, 29909, 931, 10640, 2566, 2651, 2036, 29912, 17559, 1272, 1836, 32001, 1987, 591, 3566, 304, 1269, 2058, 263, 8158, 393, 4332, 1973, 278, 4188, 22342, 393, 263, 2183, 4148, 1024, 338, 17443, 310, 372, 29889, 32001, 1763, 445, 1095, 591, 671, 278, 28297, 29890, 15451, 8783, 310, 278, 29511, 310, 278, 1556, 3619, 937, 322, 3942, 2983, 607, 29892, 439, 11427, 515, 2651, 2036, 29912, 1079, 29890, 799, 29899, 7039, 6177, 2802, 277, 12181, 4954, 16123, 2247, 278, 26368, 5528, 5084, 310, 363, 264, 1280, 322, 269, 595, 1280, 7371, 515, 263, 2566, 310, 320, 1949, 29912, 29946, 29871, 29900, 29946, 29946, 29871, 29945, 29946, 29953, 29871, 29929, 29941, 29947, 29913, 2305, 313, 29945, 29945, 29889, 29945, 8958, 310, 8471, 2305, 297, 29871, 29906, 29900, 29896, 29946, 467, 32001, 1094, 310, 3839, 29871, 29906, 29900, 29896, 29929, 372, 18469, 320, 1949, 29912, 29906, 29955, 29871, 29953, 29953, 29906, 29871, 29947, 29900, 29896, 29913, 363, 264, 1280, 322, 320, 1949, 29912, 29906, 29955, 29871, 29906, 29900, 29953, 29871, 29947, 29906, 29896, 29913, 269, 595, 1280, 297, 29871, 29906, 29941, 29953, 24894, 8977, 1080, 29889, 4907, 29913, 32001, 1094, 297, 1749, 8783, 15717, 526, 2989, 1024, 6031, 313, 29878, 1624, 1135, 6219, 491, 937, 29914, 11922, 1024, 511, 591, 937, 5993, 675, 2983, 313, 1609, 1999, 1331, 322, 1206, 3620, 29897, 322, 769, 16280, 5375, 18897, 297, 1716, 937, 322, 3942, 2983, 10868, 8857, 29889, 32001, 1152, 1269, 1543, 1476, 297, 1024, 8857, 591, 22932, 278, 2058, 4665, 29905, 6661, 1333, 24212, 8875, 491, 278, 1024, 10868, 304, 4017, 263, 5645, 393, 338, 29839, 304, 278, 1353, 310, 12407, 24638, 393, 1024, 313, 6979, 29897, 297, 278, 2702, 2058, 29889, 32001, 320, 6661, 1333, 300, 1062, 29912, 1762, 4017, 4665, 2025, 1338, 5634, 294, 278, 17837, 310, 4954, 6689, 4907, 338, 25745, 23724, 29901, 2989, 10916, 325, 13217, 29879, 29399, 310, 2919, 10916, 805, 9450, 2999, 931, 29920, 2873, 5634, 705, 671, 263, 29544, 310, 7601, 8974, 313, 29872, 29889, 29887, 1696, 5874, 28007, 511, 322, 1661, 29899, 16072, 6743, 313, 29872, 29889, 29887, 1696, 14109, 7456, 467, 29913, 32001, 1334, 2533, 445, 4377, 363, 599, 3161, 304, 4017, 263, 2058, 8158, 29892, 17140, 701, 411, 263, 1051, 310, 779, 6990, 29938, 6689, 29892, 8158, 4535, 5854, 29938, 11000, 29889, 32001, 1334, 769, 8877, 445, 1051, 491, 278, 3186, 5120, 393, 263, 2058, 14393, 304, 322, 2533, 278, 8158, 363, 599, 278, 7600, 297, 1269, 5120, 304, 4017, 385, 12463, 8158, 29892, 6590, 304, 278, 4188, 22342, 393, 278, 9063, 14393, 304, 263, 2183, 3186, 5120, 29889, 32001, 1334, 3566, 278, 6257, 9063, 408, 6421, 515, 278, 3186, 5120, 411, 278, 9939, 8158, 29889, 32001, 450, 4876, 29899, 6707, 11043, 9378, 414, 515, 278, 9078, 322, 443, 5521, 8362, 671, 310, 21759, 29911, 10249, 29879, 29901, 1556, 18777, 671, 10035, 323, 10249, 29879, 1316, 408, 320, 16266, 26139, 510, 1118, 320, 16266, 26139, 990, 1118, 470, 320, 16266, 26139, 1212, 1836, 32001, 12808, 445, 947, 451, 3799, 26018, 4822, 20542, 29901, 3148, 29899, 6707, 18777, 29892, 363, 1342, 29892, 671, 278, 320, 16266, 26139, 375, 29913, 21759, 29911, 10249, 1568, 901, 269, 2495, 290, 368, 1135, 1009, 7824, 6795, 20895, 29889, 32001, 1551, 278, 916, 1361, 278, 9210, 29914, 978, 29899, 6707, 11043, 337, 3687, 373, 278, 17998, 9210, 310, 278, 9063, 5335, 342, 15092, 29889, 32001, 16809, 304, 5780, 22920, 373, 13897, 731, 14340, 29892, 263, 2919, 1353, 310, 25741, 297, 278, 8783, 756, 385, 17998, 9210, 5186, 304, 5225, 29889, 32001, 910, 6602, 29879, 3109, 7786, 25741, 3441, 14573, 5261, 1169, 29911, 29999, 29999, 13985, 1349, 681, 392, 27418, 6478, 29914, 310, 29871, 29906, 29900, 29906, 29900, 29879, 25741, 505, 263, 5225, 9210, 29897, 1135, 9642, 6743, 3441, 14573, 5261, 1169, 29911, 29999, 29999, 13985, 1349, 681, 392, 29914, 297, 29871, 29906, 29900, 29900, 29900, 467, 32001, 1094, 263, 1121, 278, 9210, 29914, 978, 29899, 6707, 11043, 1033, 1095, 701, 6459, 292, 263, 2919, 6232, 310, 9642, 25741, 408, 4148, 287, 491, 11715, 18777, 29892, 322, 304, 263, 3109, 261, 15834, 4092, 550, 29889, 32001, 1763, 6795, 1438, 5626, 591, 14405, 278, 1023, 1737, 324, 10610, 13698, 4208, 491, 15399, 278, 9210, 29914, 978, 29899, 6707, 13698, 304, 599, 25741, 411, 263, 1661, 29899, 9171, 17998, 9210, 29892, 322, 278, 4876, 29899, 6707, 373, 304, 599, 916, 25741, 29889, 32001, 320, 2042, 29912, 12191, 322, 8565, 1558, 291, 29913, 13, 29905, 1643, 29912, 3471, 29901, 9902, 29913, 13, 13, 29905, 463, 29912, 4532, 4044, 13, 29871, 320, 9525, 13, 29871, 320, 7313, 29961, 2103, 2013, 16292, 3199, 1429, 287, 29889, 5140, 29913, 13, 29871, 320, 6671, 29912, 29934, 20819, 310, 25741, 313, 27215, 29897, 322, 6136, 15717, 313, 22503, 29897, 491, 3186, 10640, 975, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29900, 3785, 5003, 32001, 320, 9868, 29961, 14732, 3199, 7264, 287, 2594, 8727, 6445, 278, 3186, 10640, 364, 2219, 359, 363, 25741, 322, 15717, 975, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29900, 3785, 5003, 32001, 320, 1643, 29912, 1003, 29901, 9902, 29913, 13, 29905, 355, 29912, 4532, 4044, 13, 13, 13, 29871, 13, 1762, 1234, 320, 25029, 29912, 29878, 29939, 29901, 479, 397, 24974, 29913, 591, 22229, 278, 1353, 310, 25741, 322, 8359, 15717, 639, 1629, 322, 639, 3186, 10640, 29889, 32001, 1334, 2198, 278, 7625, 2582, 297, 320, 29907, 999, 29912, 1003, 29901, 9902, 29913, 408, 1023, 5096, 287, 2594, 24469, 29892, 6445, 1629, 368, 2867, 3204, 29879, 363, 25741, 322, 15717, 8307, 29889, 32001, 7569, 2594, 11524, 263, 1629, 322, 338, 8877, 287, 297, 269, 29399, 6445, 278, 9063, 29914, 8921, 11959, 363, 1269, 310, 278, 3186, 12786, 310, 320, 29907, 999, 29912, 1003, 29901, 11526, 1958, 29913, 297, 393, 1629, 29889, 32001, 1763, 4772, 714, 27801, 2861, 304, 805, 272, 26538, 17737, 29560, 29892, 297, 278, 4148, 8727, 591, 871, 2050, 15717, 2534, 26869, 472, 3203, 29871, 29945, 25741, 297, 263, 2183, 1629, 29889, 32001, 5806, 5366, 1747, 534, 1975, 297, 278, 24469, 6456, 393, 278, 3001, 3694, 310, 25741, 322, 15717, 6548, 18709, 9247, 975, 931, 29889, 32001, 10133, 363, 278, 937, 2440, 297, 278, 24469, 29892, 278, 1353, 310, 848, 3291, 297, 777, 3186, 12786, 508, 367, 14154, 2319, 29892, 411, 8178, 27721, 373, 278, 25806, 310, 534, 1975, 29889, 32001, 320, 26956, 29912, 7999, 12122, 6894, 537, 975, 931, 29913, 13, 13, 3563, 497, 29892, 278, 2498, 534, 355, 5692, 304, 367, 393, 278, 320, 4534, 29912, 479, 12122, 6894, 537, 297, 970, 775, 338, 10231, 6177, 4644, 6813, 322, 4092, 5136, 630, 1009, 4954, 24130, 749, 4907, 2745, 278, 7256, 310, 278, 29871, 29929, 29900], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [1, 320, 2042, 29912, 25898, 29913, 13, 29905, 1643, 29912, 3471, 29901, 23333, 29913, 13, 13, 29905, 7278, 29912, 29954, 1581, 6894, 537, 1118, 470, 901, 4049, 967, 10225, 727, 974, 29892, 4249, 27138, 304, 13, 20415, 5849, 14188, 756, 1063, 26606, 12399, 297, 7786, 2440, 29889, 32001, 512, 13, 1595, 16311, 29892, 278, 10122, 310, 29892, 9545, 310, 29892, 322, 6795, 1004, 25414, 363, 320, 7278, 29912, 26098, 13, 29871, 24003, 29913, 297, 12362, 29914, 6585, 7562, 18540, 313, 5800, 1799, 29897, 505, 4520, 263, 3287, 310, 8570, 13, 957, 278, 4940, 316, 6332, 2651, 2036, 29912, 29881, 16093, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 3855, 5871, 29906, 29900, 29896, 29900, 29895, 311, 29893, 2770, 29892, 13, 29871, 1055, 29888, 375, 29906, 29900, 29896, 29906, 5041, 267, 29892, 413, 434, 305, 1358, 29906, 29900, 29896, 29906, 26098, 29888, 2209, 29892, 19723, 5475, 4979, 29906, 29900, 29896, 29946, 26098, 29892, 13, 29871, 697, 309, 29906, 29900, 29896, 29953, 311, 5365, 550, 332, 6950, 29892, 696, 7586, 29906, 29900, 29896, 29953, 29893, 2770, 7192, 2209, 29892, 1935, 15044, 29906, 29900, 29896, 29955, 26098, 29892, 13, 29871, 503, 562, 305, 3350, 492, 29906, 29900, 29906, 29896, 26098, 1836, 32001, 320, 7278, 29912, 7999, 12122, 6894, 537, 29913, 338, 373, 278, 916, 1361, 278, 13, 14380, 310, 6894, 537, 393, 380, 1567, 515, 27138, 297, 777, 5534, 6354, 6421, 13, 3166, 1422, 3186, 12786, 322, 4185, 1973, 29889, 32001, 26304, 6894, 537, 297, 18322, 1799, 756, 4520, 13774, 2217, 8570, 297, 21344, 368, 13, 13129, 29889, 32001, 512, 3153, 29892, 1550, 3031, 979, 18994, 29899, 6707, 322, 13, 3149, 29899, 262, 29899, 2230, 18350, 29899, 7052, 11898, 310, 278, 1737, 12122, 1677, 1144, 310, 18322, 1799, 13, 21570, 29560, 1863, 2651, 2036, 29912, 12443, 10578, 29906, 29900, 29900, 29945, 5062, 11235, 29892, 13260, 333, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 13, 29871, 2594, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29892, 1850, 29882, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29892, 696, 7586, 29906, 29900, 29896, 29946, 7610, 345, 2941, 271, 24541, 29892, 13, 29871, 281, 496, 29879, 29906, 29900, 29906, 29896, 2209, 479, 5275, 1118, 2919, 29899, 7052, 25579, 979, 11898, 310, 278, 1737, 12122, 13, 12574, 310, 18322, 1799, 17737, 29560, 526, 1603, 10225, 292, 29889, 32001, 10506, 263, 4323, 23378, 13, 18609, 2133, 723, 367, 5407, 304, 1871, 1602, 12112, 4475, 304, 5534, 13, 25431, 10907, 2651, 2036, 29912, 2276, 5824, 19982, 29906, 29900, 29900, 29955, 23705, 1338, 29893, 996, 29913, 322, 298, 8491, 16650, 583, 297, 278, 13, 19678, 15483, 313, 1806, 29897, 9999, 29892, 408, 1532, 408, 29126, 2114, 950, 2472, 13, 517, 278, 2553, 1078, 373, 278, 17407, 10879, 322, 5374, 3002, 310, 18322, 1799, 2820, 278, 3186, 29889, 32001, 320, 26956, 29912, 1323, 3224, 29879, 29913, 13, 13, 3047, 445, 664, 591, 29126, 304, 3802, 445, 17261, 491, 7512, 292, 320, 4534, 29912, 1552, 937, 13, 29871, 25579, 979, 6559, 310, 278, 1737, 12122, 3978, 310, 17737, 29560, 304, 970, 775, 13, 29871, 975, 29871, 29945, 29900, 2440, 5003, 32001, 26321, 29892, 591, 3867, 263, 758, 2576, 3821, 1234, 304, 278, 13, 23031, 292, 5925, 1139, 29901, 13, 29905, 463, 29912, 690, 2842, 12470, 29913, 13, 29871, 3645, 607, 3186, 12786, 437, 15717, 310, 970, 368, 3625, 25741, 2041, 515, 13, 29871, 322, 920, 756, 372, 3939, 975, 278, 4940, 29871, 29945, 29900, 2440, 29973, 32001, 320, 1643, 29912, 29878, 29939, 29901, 479, 397, 24974, 29913, 13, 29905, 355, 29912, 690, 2842, 12470, 29913, 13, 4806, 671, 408, 8783, 278, 320, 23066, 29950, 29914, 18871, 2651, 2036, 29912, 29879, 1332, 29875, 4569, 29906, 29900, 29896, 29955, 29913, 322, 27599, 515, 372, 13, 29906, 29889, 29906, 24464, 29905, 13007, 25741, 3190, 2347, 515, 29871, 29896, 29953, 29900, 7284, 29905, 13007, 9279, 322, 4148, 287, 491, 13, 29946, 29941, 7284, 29905, 13007, 15717, 2645, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29896, 931, 3785, 29889, 32001, 1334, 1737, 324, 542, 403, 18777, 304, 13, 29905, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 29892, 773, 408, 18470, 4876, 4234, 775, 2246, 29899, 5563, 21904, 313, 617, 29911, 10249, 29879, 29897, 322, 29871, 13, 8921, 313, 4102, 29914, 4230, 29897, 2983, 9401, 411, 1024, 18822, 2820, 278, 3186, 29892, 322, 17998, 1283, 7224, 29871, 13, 1195, 287, 515, 9063, 15562, 29889, 32001, 1334, 1284, 10757, 310, 278, 4688, 8022, 749, 310, 4644, 6813, 297, 1722, 2752, 13, 20415, 29892, 2678, 8772, 491, 4092, 29889, 32001, 2860, 393, 3785, 29892, 278, 1737, 12122, 6894, 537, 29871, 13, 262, 970, 775, 756, 1063, 21003, 10231, 29889, 32001, 1334, 884, 12439, 8018, 15839, 528, 17741, 13, 12817, 304, 278, 1095, 310, 278, 8291, 6415, 29129, 322, 278, 7910, 310, 14137, 4631, 4135, 297, 13, 23369, 1705, 322, 4275, 14325, 29892, 408, 1532, 408, 310, 2545, 1664, 17292, 28342, 763, 25539, 1608, 322, 13, 25719, 10298, 4822, 10916, 313, 6727, 16783, 29914, 331, 16783, 467, 32001, 320, 26956, 29912, 1469, 20847, 3097, 5003, 32001, 319, 1634, 1414, 3577, 363, 445, 5650, 338, 3625, 515, 796, 264, 8144, 472, 13, 29905, 2271, 29912, 991, 597, 1867, 29875, 29889, 990, 29914, 29896, 29900, 29889, 29945, 29906, 29947, 29896, 29914, 2256, 8144, 29889, 29953, 29941, 29929, 29900, 29941, 29945, 29945, 29913, 2651, 2036, 29912, 3445, 1414, 29899, 5113, 1836, 32001, 320, 2042, 29912, 9662, 630, 5244, 29913, 13, 29905, 1643, 29912, 3471, 29901, 12817, 29913, 13, 13, 29933, 720, 4688, 322, 7786, 1736, 2651, 2036, 29912, 12443, 10578, 29906, 29900, 29900, 29945, 5062, 11235, 29892, 13260, 333, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 13, 29871, 696, 7586, 29906, 29900, 29896, 29946, 7610, 345, 2941, 271, 24541, 29892, 697, 309, 29906, 29900, 29896, 29953, 311, 5365, 550, 332, 6950, 29913, 505, 2931, 1891, 278, 13, 479, 5275, 310, 12362, 29914, 6585, 7562, 18540, 313, 5800, 1799, 29897, 773, 320, 7278, 29912, 6734, 26946, 952, 1118, 13, 4716, 3867, 1880, 29899, 29567, 6089, 541, 526, 9078, 297, 2159, 313, 29906, 29899, 29945, 5940, 29968, 18777, 29897, 13, 392, 508, 367, 4768, 1463, 491, 5221, 424, 23460, 29889, 32001, 512, 29871, 29906, 29900, 29900, 29947, 2261, 801, 2681, 634, 394, 8770, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29913, 18043, 263, 3031, 979, 13, 16961, 29899, 7052, 313, 1454, 278, 931, 29897, 6559, 373, 18322, 1799, 320, 7278, 29912, 479, 5275, 773, 1375, 292, 7047, 13, 29871, 28914, 313, 4345, 29934, 29897, 13698, 1836, 32001, 2688, 29537, 287, 278, 3978, 310, 29871, 29896, 5940, 29924, 17737, 29560, 13, 4746, 278, 7562, 2831, 479, 1404, 2566, 322, 611, 6504, 1051, 3190, 3145, 975, 278, 13, 29896, 29929, 29929, 29929, 489, 29906, 29900, 29900, 29945, 3785, 29892, 773, 408, 18470, 2472, 2788, 304, 1749, 29879, 29901, 4876, 21904, 13, 392, 17998, 1283, 7224, 29889, 32001, 450, 12399, 3785, 313, 29955, 2440, 29897, 297, 2651, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29913, 338, 20511, 1135, 29871, 13, 5816, 338, 12399, 297, 278, 2198, 5650, 313, 29945, 29900, 2440, 29897, 322, 278, 848, 8974, 526, 29871, 13, 27489, 873, 1422, 29936, 411, 393, 297, 3458, 29892, 1749, 2582, 1510, 263, 10029, 7200, 14978, 310, 29871, 13, 15654, 273, 325, 13217, 29940, 2072, 3082, 20706, 29889, 32001, 7280, 29190, 936, 664, 515, 29871, 29906, 29900, 29896, 29900, 491, 14619, 29882, 371, 4099, 29894, 322, 13, 29950, 309, 1372, 2651, 2036, 29912, 29873, 19426, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29913, 4023, 29894, 2868, 1583, 29899, 7099, 433, 1127, 1737, 12122, 13, 2029, 800, 310, 25492, 15303, 8304, 3598, 1494, 1009, 12368, 29892, 13, 15914, 292, 2472, 363, 779, 14850, 4535, 29892, 29955, 29900, 5940, 29968, 25492, 4160, 29889, 32001, 319, 1407, 7786, 13, 1287, 2651, 2036, 29912, 21686, 29879, 29906, 29900, 29906, 29896, 2209, 479, 5275, 29913, 491, 399, 496, 29879, 634, 394, 13217, 5349, 1737, 324, 542, 630, 4203, 263, 7284, 13, 28712, 16046, 4160, 29892, 2534, 26869, 472, 3203, 29871, 29896, 29900, 29900, 25741, 1269, 29892, 322, 1058, 13, 1311, 29899, 7099, 8663, 14354, 373, 1009, 25492, 28723, 29889, 32001, 5806, 278, 6559, 338, 13, 3149, 29899, 262, 29899, 2230, 408, 310, 29871, 29906, 29900, 29906, 29896, 29892, 278, 15717, 7252, 1009, 1284, 886, 13, 351, 475, 303, 2651, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29892, 1850, 29882, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29913, 304, 13, 18609, 675, 278, 14675, 310, 18322, 1799, 1737, 5275, 975, 278, 931, 15101, 845, 1862, 4586, 491, 13, 1552, 2211, 11898, 29889, 32001, 3831, 1965, 411, 3517, 29190, 936, 1736, 29892, 1749, 6559, 338, 1568, 7200, 6287, 5634, 29882, 5555, 13, 7054, 12339, 287, 29871, 29946, 29941, 7284, 29905, 13007, 15717, 310, 29871, 29906, 29889, 29906, 24464, 29905, 13007, 25741, 515, 29871, 29896, 29953, 29900, 7284, 29905, 13007, 13, 16418, 5634, 5426, 11267, 979, 975, 29871, 29945, 29900, 2440, 310, 970, 775, 20706, 3265, 1135, 13, 3149, 297, 931, 29892, 322, 884, 901, 2691, 29899, 3874, 1312, 313, 2541, 1629, 29899, 1609, 29899, 6360, 3803, 1070, 537, 975, 13, 1552, 8900, 3785, 467, 32001, 8108, 1189, 1711, 29892, 1749, 6559, 337, 3687, 373, 10079, 11264, 13, 3924, 313, 8257, 29903, 29897, 9063, 848, 3265, 1135, 7481, 29899, 7099, 433, 1127, 4423, 2472, 29889, 32001, 5901, 1736, 5634, 262, 3153, 278, 664, 491, 8432, 2651, 2036, 29912, 18386, 709, 29906, 29900, 29896, 29941, 2209, 29881, 24974, 29913, 13, 392, 29892, 901, 10325, 29892, 390, 579, 468, 29875, 634, 394, 8770, 2036, 29912, 29878, 579, 468, 29875, 29906, 29900, 29896, 29953, 479, 711, 3173, 29892, 13, 29871, 364, 579, 468, 29875, 29906, 29900, 29896, 29947, 479, 711, 3173, 29892, 544, 1648, 29906, 29900, 29906, 29896, 479, 468, 1581, 29881, 24974, 29913, 5634, 17532, 12399, 1737, 12122, 13, 29905, 7278, 29912, 29881, 24974, 322, 24003, 1118, 474, 29889, 29872, 1696, 278, 15834, 304, 607, 278, 3978, 310, 18322, 1799, 13, 17426, 6602, 1009, 11465, 1230, 14137, 14188, 29889, 32001, 512, 445, 664, 591, 2931, 1891, 1737, 12122, 6894, 537, 297, 970, 775, 363, 278, 937, 13, 2230, 472, 445, 6287, 29892, 1716, 297, 4958, 310, 17737, 29560, 322, 15500, 3785, 29889, 32001, 1334, 437, 13, 1333, 22002, 280, 278, 24003, 10696, 29892, 541, 3867, 29190, 936, 848, 322, 1284, 886, 393, 508, 367, 13, 280, 369, 4063, 304, 393, 1095, 408, 5434, 664, 29889, 32001, 320, 7278, 29912, 12756, 7047, 21639, 29913, 2651, 2036, 29912, 2276, 5824, 19982, 29906, 29900, 29900, 29955, 23705, 1338, 29893, 996, 29913, 338, 278, 13, 1491, 29899, 2671, 310, 7047, 21639, 393, 756, 29537, 287, 278, 18066, 267, 310, 21640, 13, 6734, 24771, 13149, 635, 29892, 3704, 278, 2702, 5932, 310, 920, 304, 5376, 13, 2541, 1737, 12122, 6894, 537, 2651, 2036, 29912, 13376, 303, 456, 29906, 29900, 29900, 29953, 10945, 3359, 29892, 1424, 29440, 29906, 29900, 29896, 29946, 23027, 5933, 1836, 32001, 3826, 3076, 2678, 278, 2198, 6559, 8128, 10757, 393, 508, 367, 1304, 29892, 297, 278, 13, 14940, 1206, 310, 970, 775, 322, 472, 263, 1407, 2919, 6287, 29892, 304, 11539, 607, 13, 14032, 4637, 310, 5534, 7047, 21639, 505, 9820, 484, 15774, 29889, 32001, 320, 2042, 29912, 4062, 3002, 29913, 13, 29905, 1643, 29912, 3471, 29901, 5696, 29913, 13, 13, 13, 29905, 1482, 361, 29905, 361, 29887, 798, 386, 1003, 29871, 320, 29887, 798, 386, 1003, 3009, 13, 29905, 361, 29887, 798, 386, 1003, 13, 29905, 463, 29912, 4532, 29913, 13, 29871, 320, 7313, 29961, 2103, 2013, 4914, 2103, 3199, 6360, 368, 29899, 2055, 1169, 29913, 13, 29871, 320, 6671, 29912, 12883, 368, 970, 25741, 975, 931, 313, 1188, 6287, 467, 13, 29913, 32001, 320, 1643, 29912, 1003, 29901, 29887, 798, 386, 29913, 13, 29905, 355, 29912, 4532, 29913, 13, 29905, 7241, 13, 13, 29905, 26956, 29912, 16390, 24541, 29913, 13, 13, 4806, 27387, 515, 320, 23066, 29950, 29914, 2651, 2036, 29912, 29879, 1332, 29899, 1516, 29878, 29906, 29900, 29896, 29929, 29899, 24713, 29913, 599, 25741, 3190, 2347, 2745, 320, 25832, 1964, 579, 1523, 2415, 2539, 6294, 32001, 2688, 5253, 304, 320, 14573, 5261, 1169, 22131, 29914, 25741, 29892, 5412, 491, 317, 15715, 29896, 15882, 29892, 4023, 29894, 2868, 515, 320, 25832, 1299, 7288, 5261, 1169, 797, 7068, 29914, 970, 9279, 6421, 515, 4655, 5849, 363, 2710, 313, 28712, 16046, 29892, 11786, 28632, 29892, 2992, 1846, 32001, 322, 3577, 28914, 313, 10251, 713, 29892, 10772, 2227, 29892, 405, 13427, 29892, 2992, 6250, 32001, 1876, 1169, 297, 278, 8783, 526, 491, 320, 25832, 6344, 2806, 943, 22131, 29914, 15717, 29892, 5412, 491, 779, 6990, 29938, 978, 29892, 4876, 4535, 5854, 29938, 11000, 29889, 32001, 450, 8783, 2996, 408, 1023, 1104, 1288, 6131, 29892, 697, 363, 25741, 322, 697, 363, 15717, 29892, 411, 278, 4642, 29371, 278, 7480, 3025, 263, 9117, 1820, 29889, 32001, 320, 361, 5426, 13, 9760, 1948, 297, 278, 9063, 1591, 3743, 278, 1494, 4235, 29901, 9063, 317, 15715, 29896, 15882, 29892, 4148, 322, 844, 5171, 5335, 342, 15092, 29892, 4148, 322, 844, 5171, 2893, 14903, 313, 20275, 16750, 278, 4148, 1591, 467, 32001, 450, 21578, 1546, 9063, 15717, 322, 9063, 2153, 2041, 515, 11786, 29892, 607, 6511, 304, 9063, 263, 1735, 4148, 287, 491, 4856, 1683, 29889, 32001, 1152, 445, 6559, 591, 21309, 373, 15717, 322, 17262, 9063, 2153, 29892, 408, 278, 4328, 1546, 278, 1023, 338, 451, 8018, 363, 1749, 5925, 5155, 322, 278, 5253, 310, 25741, 411, 263, 844, 5171, 916, 1135, 967, 4148, 338, 3480, 3473, 1821, 29889, 32001, 320, 7241, 13, 2831, 1269, 6251, 297, 278, 4148, 1591, 591, 505, 4148, 2989, 1024, 322, 4876, 408, 1023, 5004, 6031, 310, 10650, 6262, 29889, 32001, 1334, 6206, 2411, 15273, 1821, 470, 18325, 519, 2983, 393, 29901, 526, 451, 1602, 397, 519, 408, 18351, 29899, 29947, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12283, 7099, 397, 519, 29914, 4148, 2983, 6206, 511, 526, 4876, 14157, 2012, 310, 2983, 3441, 25832, 6344, 2806, 943, 29934, 29885, 9823, 29914, 4954, 7039, 4907, 511, 5718, 310, 871, 9654, 4890, 3441, 25832, 6344, 2806, 943, 29934, 29885, 10358, 804, 23201, 1712, 901, 1135, 29871, 29896, 29900, 8958, 1661, 29899, 1026, 2153, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12283, 15670, 23201, 526, 5520, 1135, 29871, 29896, 29900, 29900, 4890, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12229, 549, 12495, 32001, 2860, 21166, 29892, 1048, 320, 25832, 6344, 2806, 943, 29925, 15273, 1821, 2052, 307, 29916, 29914, 15717, 3441, 25832, 6344, 2806, 943, 29925, 15273, 1821, 29925, 312, 29914, 310, 278, 2847, 8783, 29897, 9488, 363, 4340, 7418, 29889, 32001, 3940, 393, 278, 5253, 310, 970, 775, 25741, 313, 392, 15717, 29897, 11122, 297, 278, 13, 11228, 8783, 25088, 18709, 9247, 975, 13, 2230, 2651, 2036, 29912, 29879, 1332, 29899, 771, 854, 749, 29899, 331, 344, 1012, 361, 29887, 798, 386, 1003, 29892, 408, 4318, 363, 25741, 297, 13, 29905, 29907, 999, 29912, 1003, 29901, 29887, 798, 386, 1012, 2870, 29901, 515, 395, 29896, 29900, 29985, 29946, 29938, 25741, 297, 29871, 29896, 29929, 29955, 29896, 29892, 304, 395, 29896, 29900, 29985, 29953, 29938, 297, 29871, 29896, 29929, 29929, 29947, 29892, 304, 13, 284, 3242, 395, 29896, 29900, 29985, 29929, 29938, 297, 29871, 29906, 29900, 29906, 29900, 29905, 7241, 29889, 32001, 1094, 263, 17004, 278, 8900, 534, 1975, 10331, 304, 367, 901, 13, 13844, 297, 7786, 1602, 3076, 1135, 297, 29871, 29946, 29900, 29974, 1629, 29899, 1025, 6743, 29892, 2861, 304, 13964, 4586, 373, 13, 735, 1112, 9247, 7200, 23093, 29889, 32001, 320, 26956, 29912, 7999, 324, 10610, 29913, 13, 13, 29905, 463, 29912, 4532, 29913, 13, 29871, 320, 9525, 13, 29871, 320, 7313, 29961, 24049, 29892, 15450, 29922, 29953, 4912, 29871, 29953, 4912, 29871, 29900, 29871, 29900, 29892, 2103, 2013, 16292, 3199, 1491, 1727, 1080, 29899, 2470, 29913, 13, 29871, 320, 6671, 29912, 1576, 320, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 1304, 408, 1737, 324, 10610, 22525, 5003, 32001, 320, 1643, 29912, 1003, 29901, 11526, 1958, 29913, 13, 29905, 355, 29912, 4532, 29913, 13, 13, 2887, 1737, 324, 10610, 22525, 591, 671, 11758, 3186, 12786, 10723, 515, 278, 3303, 18269, 1737, 359, 305, 2004, 2651, 2036, 29912, 348, 29896, 29929, 29929, 29929, 479, 359, 305, 2004, 1836, 32001, 1763, 4772, 2432, 3381, 491, 2919, 10916, 313, 29872, 29889, 29887, 1696, 7551, 470, 12710, 29897, 2629, 11758, 12786, 29892, 591, 19412, 322, 6219, 777, 12786, 2729, 373, 1737, 12122, 23203, 537, 322, 278, 19383, 310, 758, 331, 8946, 16375, 29769, 5680, 29892, 1316, 408, 19182, 4086, 29889, 32001, 320, 29907, 999, 29912, 1003, 29901, 11526, 1958, 29913, 3697, 278, 2186, 1051, 310, 320, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 1304, 408, 1737, 324, 10610, 22525, 297, 445, 6559, 29889, 32001, 1879, 324, 10610, 310, 9063, 15717, 304, 3186, 12786, 3913, 278, 1023, 19595, 653, 13698, 9129, 297, 2651, 2036, 29912, 293, 344, 29899, 344, 275, 29899, 29906, 29900, 29906, 29906, 29899, 26098, 1118, 23359, 337, 13998, 2400, 29889, 32001, 450, 937, 697, 337, 3687, 373, 278, 4234, 775, 2246, 29899, 5563, 5354, 313, 617, 29911, 10249, 29897, 310, 4876, 14157, 23892, 515, 9063, 15562, 29892, 321, 29889, 29887, 1696, 320, 16266, 26139, 1341, 1118, 320, 16266, 26139, 582, 1118, 320, 16266, 26139, 18038, 1118, 2992, 29889, 32001, 1334, 4687, 515, 278, 306, 2190, 29909, 1051, 310, 13548, 2931, 21759, 29911, 10249, 29879, 2651, 2036, 29912, 6011, 29899, 29883, 312, 430, 29913, 322, 7522, 20545, 1269, 6590, 20123, 304, 263, 3646, 3186, 5120, 29889, 32001, 450, 1473, 1737, 324, 10610, 11043, 3913, 278, 17998, 9210, 310, 9063, 5335, 342, 15092, 313, 29872, 29889, 29887, 1696, 17998, 29899, 29900, 29945, 29901, 29900, 29900, 29897, 322, 4148, 2983, 304, 8161, 278, 1556, 5517, 3186, 5120, 310, 278, 9063, 4148, 29889, 32001, 1152, 1269, 17998, 9210, 591, 8161, 263, 1051, 310, 15878, 7600, 313, 13509, 29892, 2106, 29892, 470, 14278, 20123, 29897, 297, 278, 3186, 393, 29892, 472, 278, 931, 310, 393, 9063, 29892, 750, 393, 17998, 9210, 29936, 9063, 931, 338, 1820, 1244, 29892, 408, 4234, 17998, 1283, 7224, 13100, 975, 931, 2861, 304, 29431, 3620, 29889, 32001, 1763, 1207, 445, 3683, 3381, 591, 671, 278, 306, 2190, 29909, 931, 10640, 2566, 2651, 2036, 29912, 17559, 1272, 1836, 32001, 1987, 591, 3566, 304, 1269, 2058, 263, 8158, 393, 4332, 1973, 278, 4188, 22342, 393, 263, 2183, 4148, 1024, 338, 17443, 310, 372, 29889, 32001, 1763, 445, 1095, 591, 671, 278, 28297, 29890, 15451, 8783, 310, 278, 29511, 310, 278, 1556, 3619, 937, 322, 3942, 2983, 607, 29892, 439, 11427, 515, 2651, 2036, 29912, 1079, 29890, 799, 29899, 7039, 6177, 2802, 277, 12181, 4954, 16123, 2247, 278, 26368, 5528, 5084, 310, 363, 264, 1280, 322, 269, 595, 1280, 7371, 515, 263, 2566, 310, 320, 1949, 29912, 29946, 29871, 29900, 29946, 29946, 29871, 29945, 29946, 29953, 29871, 29929, 29941, 29947, 29913, 2305, 313, 29945, 29945, 29889, 29945, 8958, 310, 8471, 2305, 297, 29871, 29906, 29900, 29896, 29946, 467, 32001, 1094, 310, 3839, 29871, 29906, 29900, 29896, 29929, 372, 18469, 320, 1949, 29912, 29906, 29955, 29871, 29953, 29953, 29906, 29871, 29947, 29900, 29896, 29913, 363, 264, 1280, 322, 320, 1949, 29912, 29906, 29955, 29871, 29906, 29900, 29953, 29871, 29947, 29906, 29896, 29913, 269, 595, 1280, 297, 29871, 29906, 29941, 29953, 24894, 8977, 1080, 29889, 4907, 29913, 32001, 1094, 297, 1749, 8783, 15717, 526, 2989, 1024, 6031, 313, 29878, 1624, 1135, 6219, 491, 937, 29914, 11922, 1024, 511, 591, 937, 5993, 675, 2983, 313, 1609, 1999, 1331, 322, 1206, 3620, 29897, 322, 769, 16280, 5375, 18897, 297, 1716, 937, 322, 3942, 2983, 10868, 8857, 29889, 32001, 1152, 1269, 1543, 1476, 297, 1024, 8857, 591, 22932, 278, 2058, 4665, 29905, 6661, 1333, 24212, 8875, 491, 278, 1024, 10868, 304, 4017, 263, 5645, 393, 338, 29839, 304, 278, 1353, 310, 12407, 24638, 393, 1024, 313, 6979, 29897, 297, 278, 2702, 2058, 29889, 32001, 320, 6661, 1333, 300, 1062, 29912, 1762, 4017, 4665, 2025, 1338, 5634, 294, 278, 17837, 310, 4954, 6689, 4907, 338, 25745, 23724, 29901, 2989, 10916, 325, 13217, 29879, 29399, 310, 2919, 10916, 805, 9450, 2999, 931, 29920, 2873, 5634, 705, 671, 263, 29544, 310, 7601, 8974, 313, 29872, 29889, 29887, 1696, 5874, 28007, 511, 322, 1661, 29899, 16072, 6743, 313, 29872, 29889, 29887, 1696, 14109, 7456, 467, 29913, 32001, 1334, 2533, 445, 4377, 363, 599, 3161, 304, 4017, 263, 2058, 8158, 29892, 17140, 701, 411, 263, 1051, 310, 779, 6990, 29938, 6689, 29892, 8158, 4535, 5854, 29938, 11000, 29889, 32001, 1334, 769, 8877, 445, 1051, 491, 278, 3186, 5120, 393, 263, 2058, 14393, 304, 322, 2533, 278, 8158, 363, 599, 278, 7600, 297, 1269, 5120, 304, 4017, 385, 12463, 8158, 29892, 6590, 304, 278, 4188, 22342, 393, 278, 9063, 14393, 304, 263, 2183, 3186, 5120, 29889, 32001, 1334, 3566, 278, 6257, 9063, 408, 6421, 515, 278, 3186, 5120, 411, 278, 9939, 8158, 29889, 32001, 450, 4876, 29899, 6707, 11043, 9378, 414, 515, 278, 9078, 322, 443, 5521, 8362, 671, 310, 21759, 29911, 10249, 29879, 29901, 1556, 18777, 671, 10035, 323, 10249, 29879, 1316, 408, 320, 16266, 26139, 510, 1118, 320, 16266, 26139, 990, 1118, 470, 320, 16266, 26139, 1212, 1836, 32001, 12808, 445, 947, 451, 3799, 26018, 4822, 20542, 29901, 3148, 29899, 6707, 18777, 29892, 363, 1342, 29892, 671, 278, 320, 16266, 26139, 375, 29913, 21759, 29911, 10249, 1568, 901, 269, 2495, 290, 368, 1135, 1009, 7824, 6795, 20895, 29889, 32001, 1551, 278, 916, 1361, 278, 9210, 29914, 978, 29899, 6707, 11043, 337, 3687, 373, 278, 17998, 9210, 310, 278, 9063, 5335, 342, 15092, 29889, 32001, 16809, 304, 5780, 22920, 373, 13897, 731, 14340, 29892, 263, 2919, 1353, 310, 25741, 297, 278, 8783, 756, 385, 17998, 9210, 5186, 304, 5225, 29889, 32001, 910, 6602, 29879, 3109, 7786, 25741, 3441, 14573, 5261, 1169, 29911, 29999, 29999, 13985, 1349, 681, 392, 27418, 6478, 29914, 310, 29871, 29906, 29900, 29906, 29900, 29879, 25741, 505, 263, 5225, 9210, 29897, 1135, 9642, 6743, 3441, 14573, 5261, 1169, 29911, 29999, 29999, 13985, 1349, 681, 392, 29914, 297, 29871, 29906, 29900, 29900, 29900, 467, 32001, 1094, 263, 1121, 278, 9210, 29914, 978, 29899, 6707, 11043, 1033, 1095, 701, 6459, 292, 263, 2919, 6232, 310, 9642, 25741, 408, 4148, 287, 491, 11715, 18777, 29892, 322, 304, 263, 3109, 261, 15834, 4092, 550, 29889, 32001, 1763, 6795, 1438, 5626, 591, 14405, 278, 1023, 1737, 324, 10610, 13698, 4208, 491, 15399, 278, 9210, 29914, 978, 29899, 6707, 13698, 304, 599, 25741, 411, 263, 1661, 29899, 9171, 17998, 9210, 29892, 322, 278, 4876, 29899, 6707, 373, 304, 599, 916, 25741, 29889, 32001, 320, 2042, 29912, 12191, 322, 8565, 1558, 291, 29913, 13, 29905, 1643, 29912, 3471, 29901, 9902, 29913, 13, 13, 29905, 463, 29912, 4532, 4044, 13, 29871, 320, 9525, 13, 29871, 320, 7313, 29961, 2103, 2013, 16292, 3199, 1429, 287, 29889, 5140, 29913, 13, 29871, 320, 6671, 29912, 29934, 20819, 310, 25741, 313, 27215, 29897, 322, 6136, 15717, 313, 22503, 29897, 491, 3186, 10640, 975, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29900, 3785, 5003, 32001, 320, 9868, 29961, 14732, 3199, 7264, 287, 2594, 8727, 6445, 278, 3186, 10640, 364, 2219, 359, 363, 25741, 322, 15717, 975, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29900, 3785, 5003, 32001, 320, 1643, 29912, 1003, 29901, 9902, 29913, 13, 29905, 355, 29912, 4532, 4044, 13, 13, 13, 29871, 13, 1762, 1234, 320, 25029, 29912, 29878, 29939, 29901, 479, 397, 24974, 29913, 591, 22229, 278, 1353, 310, 25741, 322, 8359, 15717, 639, 1629, 322, 639, 3186, 10640, 29889, 32001, 1334, 2198, 278, 7625, 2582, 297, 320, 29907, 999, 29912, 1003, 29901, 9902, 29913, 408, 1023, 5096, 287, 2594, 24469, 29892, 6445, 1629, 368, 2867, 3204, 29879, 363, 25741, 322, 15717, 8307, 29889, 32001, 7569, 2594, 11524, 263, 1629, 322, 338, 8877, 287, 297, 269, 29399, 6445, 278, 9063, 29914, 8921, 11959, 363, 1269, 310, 278, 3186, 12786, 310, 320, 29907, 999, 29912, 1003, 29901, 11526, 1958, 29913, 297, 393, 1629, 29889, 32001, 1763, 4772, 714, 27801, 2861, 304, 805, 272, 26538, 17737, 29560, 29892, 297, 278, 4148, 8727, 591, 871, 2050, 15717, 2534, 26869, 472, 3203, 29871, 29945, 25741, 297, 263, 2183, 1629, 29889, 32001, 5806, 5366, 1747, 534, 1975, 297, 278, 24469, 6456, 393, 278, 3001, 3694, 310, 25741, 322, 15717, 6548, 18709, 9247, 975, 931, 29889, 32001, 10133, 363, 278, 937, 2440, 297, 278, 24469, 29892, 278, 1353, 310, 848, 3291, 297, 777, 3186, 12786, 508, 367, 14154, 2319, 29892, 411, 8178, 27721, 373, 278, 25806, 310, 534, 1975, 29889, 32001, 320, 26956, 29912, 7999, 12122, 6894, 537, 975, 931, 29913, 13, 13, 3563, 497, 29892, 278, 2498, 534, 355, 5692, 304, 367, 393, 278, 320, 4534, 29912, 479, 12122, 6894, 537, 297, 970, 775, 338, 10231, 6177, 4644, 6813, 322, 4092, 5136, 630, 1009, 4954, 24130, 749, 4907, 2745, 278, 7256, 310, 278, 29871, 29929, 29900]} Using /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Using /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Using /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Using /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Detected CUDA files, patching ldflags Emitting ninja build file /root/.cache/torch_extensions/py38_cu117/cpu_adam/build.ninja... Building extension module cpu_adaUsing /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Detected CUDA files, patching ldflags Emitting ninja build file /root/.cache/torch_extensions/py38_cu117/cpu_adam/build.ninja... Building extension module cpu_adam... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) Using /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Using /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... ninja: no work to do. Using /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Loading extension module cpu_adam... Time to load cpu_adam op: 1.1967988014221191 seconds Using /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Loading extension module cpu_adam... Time to load cpu_adam op: 0.85666823387146 seconds Using /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Using /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Detected CUDA files, patching ldflags Emitting ninja build file /root/.cache/torch_extensions/py38_cu117/cpu_adam/build.ninja... Building extension module cpu_adam... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. Loading extension module cpu_adam... Time to load cpu_adam op: 1.349351406097412 seconds Loading extension module cpu_adam... Time to load cpu_adam Adam Optimizer #0 is created with AVX2 arithmetic capability. Config: alpha=0.000020, betas=(0.900000, 0.999000), weight_decay=0.000000, adam_w=1 cpu_adam... Time to load cpu_adam op: 1.5626463890075684 seconds Loading extension module cpu_adam... Time to load cpu_adam op: 1.461012840270996 seconds ds Loading extension module cpu_adam... Time to load cpu_adam op: 1.3767824172973633 seconds Loading extension module cpu_adam... Time to load cpu_adam op: 1.3856804370880127 seconds Adam Optimizer #0 is created with AVX2 arithmetic capability. Config: alpha=0.000020, betas=(0.900000, 0.999000), weight_decay=0.000000, adam_w=1 [INFO|trainer.py:1755] 2024-01-29 10:29:07,631 >> ***** Running training ***** [INFO|trainer.py:1756] 2024-01-29 10:29:07,631 >> Num examples = 930514 [INFO|trainer.py:1757] 2024-01-29 10:29:07,631 >> Num Epochs = 1 [INFO|trainer.py:1758] 2024-01-29 10:29:07,632 >> Instantaneous batch size per device = 2 [INFO|trainer.py:1759] 2024-01-29 10:29:07,632 >> Total train batch size (w. parallel, distributed & accumulation) = 512 [INFO|trainer.py:1760] 2024-01-29 10:29:07,632 >> Gradient Accumulation steps = 8 [INFO|trainer.py:1761] 2024-01-29 10:29:07,632 >> Total optimization steps = 1817 [INFO|trainer.py:1762] 2024-01-29 10:29:07,634 >> Number of trainable parameters = 13015884800 0%| | 0/1817 [00:00 [2024-01-29 10:29:07,837] [INFO] [config.py:972:print] communication_data_type ...... None [2024-01-29 10:29:07,837] [INFO] [config.py:972:print] compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}} [2024-01-29 10:29:07,837] [INFO] [config.py:972:print] curriculum_enabled_legacy .... False [2024-01-29 10:29:07,837] [INFO] [config.py:972:print] curriculum_params_legacy ..... False [2024-01-29 10:29:07,837] [INFO] [config.py:972:print] data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}} [2024-01-29 10:29:07,837] [INFO] [config.py:972:print] data_efficiency_enabled ...... False [2024-01-29 10:29:07,837] [INFO] [config.py:972:print] dataloader_drop_last ......... False [2024-01-29 10:29:07,837] [INFO] [config.py:972:print] disable_allgather ............ False [2024-01-29 10:29:07,837] [INFO] [config.py:972:print] dump_state ................... False [2024-01-29 10:29:07,837] [INFO] [config.py:972:print] dynamic_loss_scale_args ...... None [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] eigenvalue_enabled ........... False [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] eigenvalue_gas_boundary_resolution 1 [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] eigenvalue_layer_name ........ bert.encoder.layer [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] eigenvalue_layer_num ......... 0 [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] eigenvalue_max_iter .......... 100 [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] eigenvalue_stability ......... 1e-06 [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] eigenvalue_tol ............... 0.01 [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] eigenvalue_verbose ........... False [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] elasticity_enabled ........... False [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] flops_profiler_config ........ { "enabled": false, "recompute_fwd_factor": 0.0, "profile_step": 1, "module_depth": -1, "top_modules": 1, "detailed": true, "output_file": null } [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] fp16_auto_cast ............... None [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] fp16_enabled ................. False [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] fp16_master_weights_and_gradients False [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] global_rank .................. 0 [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] grad_accum_dtype ............. None [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] gradient_accumulation_steps .. 8 [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] gradient_clipping ............ 1.0 [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] gradient_predivide_factor .... 1.0 [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8 [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] initial_dynamic_scale ........ 1 [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] load_universal_checkpoint .... False [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] loss_scale ................... 1.0 [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] memory_breakdown ............. False [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] mics_hierarchial_params_gather False [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] mics_shard_size .............. -1 [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=False [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] nebula_config ................ { "enabled": false, "persistent_storage_path": null, "persistent_time_interval": 100, "num_of_version_in_retention": 2, "enable_nebula_load": true, "load_path": null } [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] optimizer_legacy_fusion ...... False [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] optimizer_name ............... adam [2024-01-29 10:29:07,838] [INFO] [config.py:972:print] optimizer_params ............. {'lr': 2e-05, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.0} [2024-01-29 10:29:07,839] [INFO] [config.py:972:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0} [2024-01-29 10:29:07,839] [INFO] [config.py:972:print] pld_enabled .................. False [2024-01-29 10:29:07,839] [INFO] [config.py:972:print] pld_params ................... False [2024-01-29 10:29:07,839] [INFO] [config.py:972:print] prescale_gradients ........... False [2024-01-29 10:29:07,839] [INFO] [config.py:972:print] scheduler_name ............... None [2024-01-29 10:29:07,839] [INFO] [config.py:972:print] scheduler_params ............. None [2024-01-29 10:29:07,839] [INFO] [config.py:972:print] sparse_attention ............. None [2024-01-29 10:29:07,839] [INFO] [config.py:972:print] sparse_gradients_enabled ..... False [2024-01-29 10:29:07,839] [INFO] [config.py:972:print] steps_per_print .............. 1000 [2024-01-29 10:29:07,839] [INFO] [config.py:972:print] train_batch_size ............. 512 [2024-01-29 10:29:07,839] [INFO] [config.py:972:print] train_micro_batch_size_per_gpu 2 [2024-01-29 10:29:07,839] [INFO] [config.py:972:print] use_node_local_storage ....... False [2024-01-29 10:29:07,839] [INFO] [config.py:972:print] wall_clock_breakdown ......... False [2024-01-29 10:29:07,839] [INFO] [config.py:972:print] weight_quantization_config ... None [2024-01-29 10:29:07,839] [INFO] [config.py:972:print] world_size ................... 32 [2024-01-29 10:29:07,839] [INFO] [config.py:972:print] zero_allow_untested_optimizer False [2024-01-29 10:29:07,839] [INFO] [config.py:972:print] zero_config .................. stage=3 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=26214400 allgather_partitions=True allgather_bucket_size=500,000,000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=DeepSpeedZeroOffloadParamConfig(device='cpu', nvme_path=None, buffer_count=5, buffer_size=100,000,000, max_in_cpu=1,000,000,000, pin_memory=True) offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='cpu', nvme_path=None, buffer_count=4, pin_memory=True, pipeline=False, pipeline_read=False, pipeline_write=False, fast_init=False) sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=23592960 param_persistence_threshold=51200 model_persistence_threshold=sys.maxsize max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=True stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True [2024-01-29 10:29:07,839] [INFO] [config.py:972:print] zero_enabled ................. True [2024-01-29 10:29:07,839] [INFO] [config.py:972:print] zero_force_ds_cpu_optimizer .. True [2024-01-29 10:29:07,839] [INFO] [config.py:972:print] zero_optimization_stage ...... 3 [2024-01-29 10:29:07,839] [INFO] [config.py:958:print_user_config] json = { "optimizer": { "type": "Adam", "params": { "lr": 2e-05, "betas": [0.9, 0.999], "eps": 1e-08, "weight_decay": 0.0 } }, "bf16": { "enabled": true }, "zero_optimization": { "stage": 3, "offload_optimizer": { "device": "cpu", "pin_memory": true }, "offload_param": { "device": "cpu", "pin_memory": true }, "overlap_comm": true, "contiguous_gradients": true, "reduce_bucket_size": 2.621440e+07, "stage3_prefetch_bucket_size": 2.359296e+07, "stage3_param_persistence_threshold": 5.120000e+04, "sub_group_size": 1.000000e+09, "stage3_max_live_parameters": 1.000000e+09, "stage3_max_reuse_distance": 1.000000e+09, "stage3_gather_16bit_weights_on_model_save": true }, "gradient_accumulation_steps": 8, "gradient_clipping": 1.0, "steps_per_print": 1000, "train_batch_size": 512, "train_micro_batch_size_per_gpu": 2, "wall_clock_breakdown": false } [INFO|trainer.py:1755] 2024-01-29 10:29:07,841 >> ***** Running training ***** [INFO|trainer.py:1756] 2024-01-29 10:29:07,841 >> Num examples = 930514 [INFO|trainer.py:1757] 2024-01-29 10:29:07,841 >> Num Epochs = 1 [INFO|trainer.py:1758] 2024-01-29 10:29:07,841 >> Instantaneous batch size per device = 2 [INFO|trainer.py:1759] 2024-01-29 10:29:07,841 >> Total train batch size (w. parallel, distributed & accumulation) = 512 [INFO|trainer.py:1760] 2024-01-29 10:29:07,841 >> Gradient Accumulation steps = 8 [INFO|trainer.py:1761] 2024-01-29 10:29:07,841 >> Total optimization steps = 1817 [INFO|trainer.py:1762] 2024-01-29 10:29:07,843 >> Number of trainable parameters = 13015884800 0%| | 0/1817 [00:0031->30 [1] 24/-1/-1->31->30 ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:69944:82212 [6] NCCL INFO Trees [0] 31/-1/-1->30->29 [1] 31/-1/-ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95682:5856 [5] NCCL INFO Trees [0] 22/-1/-1->21->20 [1] 22/-1/-1->21->20 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95681:5849 [4] NCCL INFO Trees [0] 21/-1/-1->20->19 [1] 21/-1/-1->20->19 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95674:5855 [2] NCCL INFO Trees [0] 19/26/-1->18->2 [1] 19/-1/-1->18->11 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95676:5851 [3] NCCL INFO Trees [0] 20/10/-1->19->18 [1] 20/-1/-1->19->18 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95673:5854 [1] NCCL INFO Trees [0] -1/-1/-1->17->16 [1] -1/-1/-1->17->16 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95672:5853 [0] NCCL INFO Trees [0] 17/-1/-1->16->23 [1] 17/-1/-1->16->23 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95674:5855 [2] NCCL INFO Channel 00/0 : 18[4b000] -> 23[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95681:5849 [4] NCCL INFO Channel 00/0 : 20[93000] -> 17[13000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95672:5853 [0] NCCL INFO Channel 00/0 : 16[e000] -> 19[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95674:5855 [2] NCCL INFO Channel 01/0 : 18[4b000] -> 23[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95681:5849 [4] NCCL INFO Channel 01/0 : 20[93000] -> 17[13000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95672:5853 [0] NCCL INFO Channel 01/0 : 16[e000] -> 19[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95683:5ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118723:130976 [6] NCCL INFO ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95682:5856ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118722:130977 [5] NCCL INts-cbba87c5e7504a249f5127ts-cbba87c5e7504a249f5127103d9ce4ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118723:130976 [6] NCCLts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95682:5856 [5] NCCL INFO Channel 01/0 : 21[99000] -> 20[93000] via P2P/IPC/read ts-cbba87c5e7504a249f512710ts-cbba87c5e7504a249f5127103d9ce40f-wts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118718:130974 [1] NCCL INFO Channel 00/0 : 9[13000] -> 8[e000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118724:130978 [ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95673:5854 [1] NCCL INFts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118718:1309ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95684:5852 [7] NCCL INFO Channel 01/0 : 23[d0000] -> 22[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95681:5849 [4] NCCL INFO Cots-cbba87c5e7504a2ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95684:5852 [7] NCCL INFO Connets-cbba87c5e7504a249f5127103d9ce40f-worker-0:118721:130975 [4] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118717:130979 [0] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95673:5854 [1] NCCL INFO Connected alts-cbba8ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95672:5853 [0] NCCL INFO Channel 01/0 : ts-cbba87c5e7504a249f5127103d9ce40f-worts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95682:5856 [5] NCCL INFO Connected all rits-cbba87c5e7504a249f5127103d9ce40f-worker-0:118722:130977 [5] NCCL INFO Connected all rings tts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95681:5849 [4] NCCL INFO Channel 00/0 : 20[9300ts-cbba87c5e7504a249f5127103d9ce4ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95681:5849 [4] NCCL INFO Channel 01/0 : 20[93000] ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118721:130975 [4] NCCL INFO Channel 01/0 : 12[93000] -> 13[99000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118718:130974 [1] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118718:130974 [1] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118718:130974 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118722:130977 [5] NCCL INFO Channel 00/0 : 13[99000] -> 14[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118717:130979 [0] NCCL INFO Channel 00/0 : 8[e000] -> 15[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118723:130976 [6] NCCL INFO Channel 00/0 : 14[cb000] -> 15[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118722:130977 [5] NCCL INFO Channel 01/0 : 13[99000] -> 14[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118717:130979 [0] NCCL INFO Channel 01/0 : 8[e000] -> 15[d0000] via P2P/IPC/read ts-cbts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95682:5856 [5] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce4ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118722:130977 [5] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118722:130977 [5] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118722:130977 [5] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118723:130976 [6] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118723:130976 [6] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118723:130976 [6] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p cts-cbba87c5e7504ats-cbba87c5e7504a249f5127103d9ce40f-worker-0:118724:130978 [7] NCCL INFO Channel 00/0 : 15[d0000] -> 8[e000ts-cbba87c5e7504a24ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118724:130978 [7] NCCL INFO Channel 01/0 : 15[d0000] -> 8[e000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118717:130979 [0] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118717:130979 [0] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118717:130979 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118720:130980 [3] NCCL INFO Channel 00/0 : 11[51000] -> 18[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118719:130981 [2] NCCL INFO Channel 00/0 : 3[51000] -> 10[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-wts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95674:5855 [2] NCCL INFO Channel 00/0 : 11[51000] -> 18[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95676:5851 [3] NCCL ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:69940:82213 [2] NCCL INFO Channel 01/0 : 19[51000] -> 26[4ts-cbba87c5e7504a249f5127103d9ce40f-ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:69941:82210 [3] NCCL INFts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95676:5851 [3] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95676:5851 [3] NCCL INFO Channel 00/0 : 19[51000] -> 20[93000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95676:ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118720:130980 [3] NCCL INFO Channel 00/0 : 11[51000] -> 12[93000] vits-cbba87c5e750ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118720:130980 [3] NCCL INFO Channel 01/0 : 11[51000] -> 12[93000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118719:130981 [2] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118719:130981 [2] NCCL INFO Channel 00/0 : 10[4b000] -> 11[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118719:130981 [2] NCCL INFO Channel 01/0 : 10[4b000] -> 11[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118721:130975 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95674:5855 [2] NCCL INFO Channel 00/0 : 18[4b000] -> 26[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95676:5851 [3] NCts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118719:130981 [2] NCCL INFO Chats-cbba87c5e7504a249f5127103d9ce40f-worker-1:95674:5855 [2] NCCL INFO Channel 00/0 : 2[4b000] -> 18[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95674:5855 [2] NCCL INFO Channel 00/0 : 18[4b000] -> 2[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95676:5851 [3] NCCL INFO Channel 00/0 : 19[51000] -> 10[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95674:5855 [2] NCCL INFO Channel 00/0 : 26[4b000] -> 18[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95676:5851 [3] NCCL INFO Channel 00/0 : 19[51000] -> 18[4b000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:95676:5851 [3] NCCL INFO Channel 01/0 : 19[51000]ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118719:130981 [2] NCCL INFO Channel 01/0 : 10[4b000] -> 2[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118720:130980 [3] NCCL INFO Channel 00/0 : 11[51000] -> 10[4b000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118720:130980 [3] NCCL INFO Channel 01/0 : 11[51000] -> 10[4b000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118721:130975 [4] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118721:130975 [4] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118721:130975 [4] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118719:130981 [2] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118719:130981 [2] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118719:130981 [2] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118720:130980 [3] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118720:130980 [3] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118720:130980 [3] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118720:130980 [3] NCCL INFO comm 0x7f018402aaa0 rank 11 nranks 32 cudaDev 3 busId 51000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118719:130981 [2] NCCL INFO comm 0x7f59e802a9b0 rank 10 nranks 32 cudaDev 2 busId 4b000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118722:130977 [5] NCCL INFO comm 0x7f730002a780 rank 13 nranks 32 cudaDev 5 busId 99000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118723:130976 [6] NCCL INFO comm 0x7ef50402a750 rank 14 nranks 32 cudaDev 6 busId cb000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118717:130979 [0] NCCL INFO comm 0x7fdc9802a8b0 rank 8 nranks 32 cudaDev 0 busId e000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118718:130974 [1] NCCL INFO comm 0x7efe0402a8e0 rank 9 nranks 32 cudaDev 1 busId 13000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118721:130975 [4] NCCL INFO comm 0x7fd82c02a670 rank 12 nranks 32 cudaDev 4 busId 93000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:118724:130978 [7] NCCL INFO comm 0x7fd21402aa20 rank 15 nranks 32 cudaDev 7 busId d0000 - Init COMPLETE Traceback (most recent call last): File "/apdcephfs/share_733425/vinnylywang/jianhuipang/gogollm/newmodels/run_allms.py", line 836, in main() File "/apdcephfs/share_733425/vinnylywang/jianhuipang/gogollm/newmodels/run_allms.py", line 784, in main train_result = trainer.train(resume_from_checkpoint=checkpoint) File "/apdcephfs/share_733425/vinnylywang/minghaowu/multi-llama-2/transformers/src/transformers/trainer.py", line 1648, in train return inner_training_loop( File "/apdcephfs/share_733425/vinnylywang/minghaowu/multi-llama-2/transformers/src/transformers/trainer.py", line 1915, in _inner_training_loop tr_loss_step = self.training_step(model, inputs) File "/apdcephfs/share_733425/vinnylywang/minghaowu/multi-llama-2/transformers/src/transformers/trainer.py", line 2677, in training_step loss = self.deepspeed.backward(loss) File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn ret_val = func(*args, **kwargs) File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 1929, in backward self.optimizer.backward(loss, retain_graph=retain_graph) File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn ret_val = func(*args, **kwargs) File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/deepspeed/runtime/zero/stage3.py", line 2091, in backward self.loss_scaler.backward(loss.float(), retain_graph=retain_graph) File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/deepspeed/runtime/fp16/loss_scaler.py", line 63, in backward scaled_loss.backward(retain_graph=retain_graph) File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/_tensor.py", line 488, in backward torch.autograd.backward( File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/autograd/__init__.py", line 197, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/autograd/function.py", line 267, in apply return user_fn(self, *args) File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/utils/checkpoint.py", line 157, in backward torch.autograd.backward(outputs_with_grad, args_with_grad) File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/autograd/__init__.py", line 197, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 5.00 GiB (GPU 7; 39.59 GiB total capacity; 23.15 GiB already allocated; 3.93 GiB free; 34.20 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:69940:82225 [2] NCCL INFO [Service thread] Connection closed by localRank 7 ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:69940:70849 [2] NCCL INFO [Service thread] Connection closed by localRank 7 WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 69938 closing signal SIGTERM WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 69939 closing signal SIGTERM WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 69940 closing signal SIGTERM WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 69941 closing signal SIGTERM WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 69942 closing signal SIGTERM WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 69943 closing signal SIGTERM WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 69944 closing signal SIGTERM ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 7 (pid: 69945) of binary: /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/bin/python ERROR:torch.distributed.elastic.agent.server.api:Error waiting on exit barrier. Elapsed: 303.7959244251251 seconds Traceback (most recent call last): File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/distributed/elastic/agent/server/api.py", line 906, in _exit_barrier store_util.barrier( File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/distributed/elastic/utils/store.py", line 78, in barrier synchronize(store, data, rank, world_size, key_prefix, barrier_timeout) File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/distributed/elastic/utils/store.py", line 64, in synchronize agent_data = get_all(store, rank, key_prefix, world_size) File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/distributed/elastic/utils/store.py", line 34, in get_all data = store.get(f"{prefix}{idx}") RuntimeError: Socket Timeout Traceback (most recent call last): File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/bin/torchrun", line 8, in sys.exit(main()) File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper return f(*args, **kwargs) File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/distributed/run.py", line 762, in main run(args) File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/distributed/run.py", line 753, in run elastic_launch( File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 132, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ /apdcephfs/share_733425/vinnylywang/jianhuipang/gogollm/newmodels/run_allms.py FAILED ------------------------------------------------------------ Failures: ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-01-29_10:29:41 host : ts-cbba87c5e7504a249f5127103d9ce40f-worker-2 rank : 31 (local_rank: 7) exitcode : 1 (pid: 69945) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================ WARNING:torch.distributed.run: ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. ***************************************** WARNING:torch.distributed.run: ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. ***************************************** WARNING:torch.distributed.run: ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. ***************************************** [2024-01-29 11:10:29,542] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 11:10:29,542] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 11:10:29,542] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 11:10:29,542] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 11:10:29,542] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 11:10:29,543] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 11:10:29,543] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 11:10:29,544] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 11:10:31,485] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 11:10:31,497] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 11:10:31,511] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 11:10:31,524] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 11:10:31,532] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 11:10:31,533] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 11:10:31,537] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 11:10:31,538] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 11:10:30,980] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 11:10:30,980] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 11:10:30,980] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 11:10:30,980] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 11:10:30,980] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 11:10:30,980] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 11:10:30,980] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 11:10:30,980] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-01-29 11:10:52,817] [INFO] [comm.py:637:init_distributed] cdb=None [2024-01-29 11:10:52,844] [INFO] [comm.py:637:init_distributed] cdb=None [2024-01-29 11:10:52,929] [INFO] [comm.py:637:init_distributed] cdb=None [2024-01-29 11:10:52,944] [INFO] [comm.py:637:init_distributed] cdb=None [2024-01-29 11:10:52,980] [INFO] [comm.py:637:init_distributed] cdb=None [2024-01-29 11:10:52,980] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl [2024-01-29 11:10:53,004] [INFO] [comm.py:637:init_distributed] cdb=None [2024-01-29 11:10:53,015][2024-01-29 11:10:54,588] [INFO] [comm.py:637:init_distributed] cdb=None [2024-01-29 11:10:54,588] [INFO] [comm.py:637:in01/29/2024 11:10:55 - WARNING - __main__ - Process rank: 1, device: cuda:1, n_gpu: 1distributed training: True, 16-bits training: False /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( 01/29/2024 11:10:55 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False 01/29/2024 11:10:55 - INFO - __main__ - Training/evaluation parameters TrainingArguments( _n_gpu=1, adafactor=False, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, auto_find_batch_size=False, bf16=True, bf16_full_eval=True, data_seed=None, dataloader_drop_last=False, dataloader_num_workers=8, dataloader_pin_memory=True, ddp_bucket_cap_mb=None, ddp_find_unused_parameters=None, ddp_timeout=72000, debug=[], deepspeed=/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/train/deepspeed_config_bf16.json, disable_tqdm=False, do_eval=False, do_predict=False, do_train=True, eval_accumulation_steps=None, eval_delay=0, eval_steps=None, evaluation_strategy=no, fp16=False, fp16_backend=auto, fp16_full_eval=False, fp16_opt_level=O1, fsdp=[], fsdp_config={'fsdp_min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False}, fsdp_min_num_params=0, fsdp_transformer_layer_cls_to_wrap=None, full_determinism=False, gradient_accumulation_steps=16, gradient_checkpointing=True, greater_is_better=None, group_by_length=Fals01/29/2024 11:10:55 - WARNING - __main__ - Process rank: 2, device: cuda:2, n_gpu: 1distributed training: True, 16-bits training: False 01/29/2024 11:10:55 - WARNING - __main__ - Process rank: 6, device: cuda:6, n_gpu: 1distributed training: True, 16-bits training: False 01/29/2024 11:10:55 - WARNING - __main__ - Process rank: 1, device: cuda:1, n_gpu: 1distributed training: True, 16-bits training: False 01/29/2024 11:10:55 - WARNING - __main__ - Process rank: 3, device: cuda:3, n_gpu: 1distributed training: True, 16-bits training: False /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( 01/29/2024 11:10:55 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False 01/29/2024 11:10:55 - INFO - __main__ - Training/evaluation parameters TrainingArguments( _n_gpu=1, adafactor=False, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, auto_find_batch_size=False, bf16=True, bf16_full_eval=True, data_seed=None, dataloader_drop_last=False, dataloader_num_workers=8, dataloader_pin_memory=True, ddp_bucket_cap_mb=None, ddp_find_unused_parameters=None, ddp_timeout=72000, debug=[], deepspeed=/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/train/deepspeed_config_bf16.json, disable_tqdm=False, do_eval=False, do_predict=False, do_train=True, eval_accumulation_steps=None, eval_delay=0, eval_steps=None, evaluation_strategy=no, fp16=False, fp16_backend=auto, fp16_full_eval=False, fp16_opt_level=O1, fsdp=[], fsdp_config={'fsdp_min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False}, fsdp_min_num_params=0, fsdp_transformer_layer_cls_to_wrap=None, full_determinism=False, gradient_accumulation_steps=16, gradient_checkpointing=True, greater_is_better=None, group_by_length=False, half_precision_backend=auto, hub_model_id=None, hub_private_repo=False, hub_strategy=every_save, hub_token=, ignore_data_skip=False, include_inputs_for_metrics=False, jit_mode_eval=False, label_names=None, label_smoothing_factor=0.0, learning_rate=2e-05, length_column_name=length, load_best_model_at_end=False, local_rank=0, log_level=passive, log_level_replica=warning, log_on_each_node=True, logging_dir=./checkpoints_ct/ac/allm-ac-13b/runs/Jan29_11-10-31_ts-cbba87c5e7504a249f5127103d9ce40f-worker-1, logging_first_step=False, logging_nan_inf_filter=True, logging_steps=1, logging_strategy=steps, lr_scheduler_type=constant_with_warmup, max_grad_norm=1.0, max_steps=-1, metric_for_best_model=None, mp_parameters=, no_cuda=False, num_train_epochs=1.0, optim=adamw_hf, optim_args=None, out/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( 01/29/2024 11:10:55 - WARNING - __main__ - Process rank: 2, device: cuda:2, n_gpu: 1distributed training: True, 16-bits training: False 01/29/2024 11:10:55 - WARNING - __main__ - Process rank: 3, device: cuda:3, n_gpu: 1distributed training: True, 16-bits training: False /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( /apdcephfs/share_733425/vinnylyw/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( No config specified, defaulting to the single config: red_pajama-data-1_t-sample/plain_text 01/29/2024 11:10:55 - INFO - datasets.builder - No config specified, defaulting to the single config: red_pajama-data-1_t-sample/plain_text Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/modules/datasets_modules/datasets/RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 11:10:55 - INFO - datasets.info - Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/modules/datasNo config specified, defaulting to the single config: red_pajama-data-1_t-sample/plain_text 01/29/2024 11:10:56 - INFO - datasets.builder - No config specified, defaulting to the single config: red_pajama-data-1_t-sample/plain_text Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/modules/datasets_modules/datasets/RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 11:10:56 - INFO - datasets.info - Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/modules/datasets_modules/datasets/RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 Overwrite dataset info from restored data version if exists. 01/29/2024 11:10:56 - INFO - datasets.builder - Overwrite dataset info from restored data version if exists. Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 11:10:56 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 Found cached dataset red_pajama-data-1_t-sample (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039) 01/29/2024 11:10:56 - INFO - datasets.builder - Found cached dataset red_pajama-data-1_t-sample (/apdcephfs/share_733425/vinnylywang/jianhuNo config specified, defaulting to the single config: red_pajama-data-1_t-sample/plain_text 01/29/2024 11:10:57 - INFO - datasets.builder - No config specified, defaulting to the single config: red_pajama-data-1_t-sample/plain_text Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/modules/datasets_modules/datasets/RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 11:10:57 - INFO - datasets.info - Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/modules/datasets_modules/datasets/RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 Overwrite dataset info from restored data version if exists. 01/29/2024 11:10:58 - INFO - datasets.builder - Overwrite dataset info from restored data version if exists. Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 11:10:58 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 Found cached dataset red_pajama-data-1_t-sample (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039) 01/29/2024 11:10:58 - INFO - datasets.builder - Found cached dataset red_pajama-data-1_t-sample (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039) Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 11:10:58 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 No config specified, defaulting to the single config: red_pajama-data-1_t-sample/plain_text 01/29/2024 11:10:58 - INFO - datasets.builder - No config specified, defaulting to the single config: red_pajama-data-1_t-sample/plain_text Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/modules/datasets_modules/datasets/RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 11:10:58 - INFO - datasets.info - Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/modules/datasets_modules/datasets/RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 - INFO - datasets.builder - No config specified, defaulting to the single config: red_pajama-data-1_t-sample/plain_text Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/modules/datasets_modules/datasets/RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 11:10:57 - INFO - datasets.info - Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/modules/datasets_modules/datasets/RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 Overwrite dataset info from restored data version if exists. 01/29/2024 11:10:57 - INFO - datasets.builder - Overwrite dataset info from restored data version if exists. Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 11:10:57 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 Found cached dataset red_pajama-data-1_t-sample (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039) 01/29/2024 11:10:57 - INFO - datasets.builder - Found cached dataset red_pajama-data-1_t-sample (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039) Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 11:10:57 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 [INFO|configuration_utils.py:666] 2024-01-29 11:10:57,698 >> loading configuration file /apdcephfs/share_733425/vinnylywang/jianhuipang/opensourcellms/llama2/Llama-2-13b-hf/config.json [INFO|configuration_utils.py:720] 2024-01-29 11:10:57,699 >> Model config LlamaConfig { "_name_or_path": "/apdcephfs/share_733425/vinnylywang/jianhuipang/opensourcellms/llama2/Llama-2-13b-hf", "architectures": [ "LlamaForCausalLM" ], "bos_token_id": 1, "eos_token_id": 2, "hidden_act": "silu", "hidden_size": 5120, "initializer_range": 0.02, "intermediate_size": 13824, "max_position_embeddings": 4096, "model_type": "llama", "num_attention_heads": 40, "num_hidden_layers": 40, "num_key_value_heads": 40, "pad_token_id": 0, "pretraining_tp": 1, "rms_norm_eps": 1e-05, "rope_scaling": null, "tie_word_embeddings": false, "torch_dtype": "float16", "transformers_version": "4.28.0.dev0", "use_cache": true, "vocab_size": 32000 } 01/29/2024 11:10:57 - INFO - __main__ - Tokenizer_kwargs: {'cache_dir': None, 'use_fast': True, 'revision': 'main', 'use_auth_token': None} [INFO|tokenization_utils_base.py:1801] 2024-01-29 11:10:57,836 >> loading file tokenizer.model [INFO|tokenization_utils_base.py:1801] 2024-01-29 11:10:57,837 >> loading file added_tokens.json [INFO|tokenization_utils_base.py:1801] 2024-01-29 11:10:57,837 >> loading file special_tokens_map.json [INFO|tokenization_utils_base.py:1801] 2024-01-29 11:10:57,837 >> loading file tokenizer_config.json 01/29/2024 11:10:57 - INFO - __main__ - Loading checkpoints in dtype: None [INFO|modeling_utils.py:2395] 2024-01-29 11:10:57,864 >> loading weights file /apdcephfs/share_733425/vinnylywang/jianhuipang/opensourcellms/llama2/Llama-2-13b-hf/model.safetensors.index.json [INFO|modeling_utils.py:2487] 2024-01-29 11:10:57,865 >> Detected DeepSpeed ZeRO-3: activating zero.init() for this model [INFO|configuration_utils.py:575] 2024-01-29 11:10:57,869 >> Generate config GenerationConfig { "_from_model_config": true, "bos_token_id": 1, "eos_token_id": 2, "pad_token_id": 0, "transformers_version": "4.28.0.dev0" } ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8021:8021 [6] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8017 [2] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8020:8020 [5] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8015:8015 [0] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8021:8021 [6] NCCL INFO Bootstrap : Using eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8017 [2] NCCL INFO Bootstrap : Using eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:8018 [3] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8020:8020 [5] NCCL INFO Bootstrap : Using eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8019:8019 [4] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8016:8016 [1] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8015:8015 [0] NCCL INFO Bootstrap : Using eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8022:8022 [7] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:8018 [3] NCCL INFO Bootstrap : Using eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8019:8019 [4] NCCL INFO Bootstrap : Using eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8016:8016 [1] NCCL INFO Bootstrap : Using eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8022:8022 [7] NCCL INFO Bootstrap : Using eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8017 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8021:8021 [6] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8020:8020 [5] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8015:8015 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:8018 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8019:8019 [4] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8016:8016 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8022:8022 [7] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8908 [2] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8908 [2] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8021:8909 [6] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8021:8909 [6] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8020:8910 [5] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8020:8910 [5] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8022:8915 [7] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8022:8915 [7] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8015:8911 [0] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8015:8911 [0] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8019:8913 [4] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8019:8913 [4] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:8912 [3] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8016:8914 [1] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.64.140<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:8912 [3] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8016:8914 [1] NCCL INFO Using network IB g eth1:11.219.11.45<0> ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:67023 [3] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:67022 [2] NCCL INFO Bootstrap : Using eth1:11.219.11.45<0> ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:67024 [4] NCCL INFO Bootstrap : Using eth1:11.219.11.45<0> ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:67025 [5] NCCL INFO Bootstrap : Using eth1:11.219.11.45<0> ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:67023 [3] NCCL INFO Bootstrap : Using eth1:11.219.11.45<0> ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:67027 [7] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:67021 [1] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:67027 [7] NCCL INFO Bootstrap : Using eth1:11.219.11.45<0> ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:67021 [1] NCCL INFO Bootstrap : Using eth1:11.219.11.45<0> ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:67022 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:67024 [4] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67026:67026 [6] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:67025 [5] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:67023 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:67027 [7] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:67021 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:67951 [0] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.219.11.45<0> ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:67951 [0] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:67952 [2] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.219.11.45<0> ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:67952 [2] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:67953 [4] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.219.11.45<0> ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:67953 [4] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67026:67954 [6] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.219.11.45<0> ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67026:67954 [6] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:67955 [5] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.219.11.45<0> ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:67955 [5] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:67957 [3] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.219.11.45<0> ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:67957 [3] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:67959 [1] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.219.11.45<0> ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:67958 [7] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.219.11.45<0> ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:67959 [1] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:67958 [7] NCCL INFO Using network IB 01/29/2024 11:11:55 - WARNING - __main__ - Process rank: 2, device: cuda:2, n_gpu: 1distributed training: True, 16-bits training: False 01/29/2024 11:11:55 - WARNING - __main__ - Process rank: 4, device: cuda:4, n_gpu: 1distributed training: True, 16-bits training: False 01/29/2024 11:11:55 - WARNING - __main__ - Process rank: 6, device: cuda:6, n_gpu: 1distributed training: True, 16-bits training: False 01/29/2024 11:11:55 - WARNING - __main__ - Process rank: 7, device: cuda:7, n_gpu: 1distributed training: True, 16-bits training: False 01/29/2024 11:11:55 - WARNING - __main__ - Process rank: 3, device: cuda:3, n_gpu: 1distributed training: True, 16-bits training: False /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( 01/29/2024 11:11:55 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False 01/29/2024 11:11:55 - INFO - __main__ - Training/evaluation parameters TrainingArguments( _n_gpu=1, adafactor=False, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, auto_find_batch_size=False, bf16=True, bf16_full_eval=True, data_seed=None, dataloader_drop_last=False, dataloader_num_workers=8, dataloader_pin_memory=True, ddp_bucket_cap_mb=None, ddp_find_unused_parameters=None, ddp_timeout=72000, debug=[], deepspeed=/apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/train/deepspeed_config_bf16.json, disable_tqdm=False, do_eval=False, do_predict=False, do_train=True, eval_accumulation_steps=None, eval_delay=0, eval_steps=None, evaluation_strategy=no, fp16=False, fp16_backend=auto, fp16_full_eval=False, fp16_opt_level=O1, fsdp=[], fsdp_config={'fsdp_min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False}, fsdp_min_num_params=0, fsdp_transformer_layer_cls_to_wrap=None, full_determinism=False, gradient_accumulation_steps=16, gradient_checkpointing=True, greater_is_better=None, group_by_length=False, half_precision_backend=auto, hub_model_id=None, hub_private_repo=False, hub_strategy=every_save, hub_token=, ignore_data_skip=False, include_inputs_for_metrics=False, jit_mode_eval=False, label_names=None, label_smoothing_factor=0.0, learning_rate=2e-05, length_column_name=length, load_best_model_at_end=False, local_rank=0, log_level=passive, log_level_replica=warning, log_on_each_node=True, logging_dir=./checkpoints_ct/ac/allm-ac-13b/runs/Jan29_11-10-30_ts-cbba87c5e7504a249f5127103d9ce40f-worker-2, logging_first_step=False, logging_nan_inf_filter=True, logging_steps=1, logging_strategy=steps, lr_scheduler_type=constant_with_warmup, max_grad_norm=1.0, max_steps=-1, metric_for_best_model=None, mp_parameters=, no_cuda=False, num_train_epochs=1.0, optim=adamw_hf, optim_args=None, output_dir=./checkpoints_ct/ac/allm-ac-13b, overwrite_output_dir=True, past_index=-1, per_device_eval_batch_size=1, per_device_train_batch_size=1, prediction_loss_only=False, push_to_hub=False, push_to_hub_model_id=None, push_to_hub_organization=None, push_to_hub_token=, ray_scope=last, remove_unused_columns=True, report_to=['tensorboard'], resume_from_checkpoint=None, run_name=./checkpoints_ct/ac/allm-ac-13b, save_on_each_node=False, save_steps=500, save_strategy=steps, save_total_limit=1, seed=34, sharded_ddp=[], skip_memory_metrics=True, tf32=None, torch_compile=False, torch_compile_backend=None, torch_compile_mode=None, torchdynamo=None, tpu_metrics_debug=False, tpu_num_cores=None, use_ipex=False, use_legacy_prediction_loop=False, use_mps_device=False, warmup_ratio=0.0, warmup_steps=20, weight_decay=0.0, xpu_backend=None, ) 01/29/2024 11:11:55 - WARNING - __main__ - Process rank: 5, device: cuda:5, n_gpu: 1distributed training: True, 16-bits training: False /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( 01/29/2024 11:11:55 - WARNING - __main__ - Process rank: 1, device: cuda:1, n_gpu: 1distributed training: True, 16-bits training: False /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=None' instead. warnings.warn( No config specified, defaulting to the single config: red_pajama-data-1_t-sample/plain_text 01/29/2024 11:11:56 - INFO - datasets.builder - No config specified, defaulting to the single config: red_pajama-data-1_t-sample/plain_text Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/modules/datasets_modules/datasets/RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 11:11:56 - INFO - datasets.info - Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/modules/datasets_modules/datasets/RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133153:133153 [4] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133153:133153 [4] NCCL INFO Bootstrap : Using eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133153:133153 [4] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133152:133152 [3] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133152:133152 [3] NCCL INFO Bootstrap : Using eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133153:134076 [4] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133153:134076 [4] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133152:133152 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133152:134078 [3] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133152:134078 [3] NCCL INFO Using network IB Overwrite dataset info from restored data version if exists. 01/29/2024 11:11:59 - INFO - datasets.builder - Overwrite dataset info from restored data version if exists. Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 11:11:59 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 Found cached dataset red_pajama-data-1_t-sample (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039) 01/29/2024 11:11:59 - INFO - datasets.builder - Found cached dataset red_pajama-data-1_t-sample (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039) Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 11:11:59 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 No config specified, defaulting to the single config: red_pajama-data-1_t-sample/plain_text 01/29/2024 11:11:59 - INFO - datasets.builder - No config specified, defaulting to the single config: red_pajama-data-1_t-sample/plain_text Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/modules/datasets_modules/datasets/RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 11:11:59 - INFO - datasets.info - Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/modules/datasets_modules/datasets/RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133155:133155 [6] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133155:133155 [6] NCCL INFO Bootstrap : Using eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133155:133155 [6] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133155:134080 [6] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133155:134080 [6] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133150:133150 [1] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133150:133150 [1] NCCL INFO Bootstrap : Using eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133150:133150 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation Overwrite dataset info from restored data version if exists. 01/29/2024 11:11:59 - INFO - datasets.builder - Overwrite dataset info from restored data version if exists. Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 11:11:59 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133150:134083 [1] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133150:134083 [1] NCCL INFO Using network IB Found cached dataset red_pajama-data-1_t-sample (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039) 01/29/2024 11:11:59 - INFO - datasets.builder - Found cached dataset red_pajama-data-1_t-sample (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039) Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 11:11:59 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 No config specified, defaulting to the single config: red_pajama-data-1_t-sample/plain_text 01/29/2024 11:12:00 - INFO - datasets.builder - No config specified, defaulting to the single config: red_pajama-data-1_t-sample/plain_text Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/modules/datasets_modules/datasets/RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 11:12:00 - INFO - datasets.info - Loading Dataset Infos from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/modules/datasets_modules/datasets/RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 Overwrite dataset info from restored data version if exists. 01/29/2024 11:12:00 - INFO - datasets.builder - Overwrite dataset info from restored data version if exists. Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 01/29/2024 11:12:00 - INFO - datasets.info - Loading Dataset info from /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133156:133156 [7] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133156:133156 [7] NCCL INFO Bootstrap : Using eth1:11.218.9.169<0> Found cached dataset red_pajama-data-1_t-sample (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039) 01/29/2024 11:12:00 - INFO - datasets.builder - Found cached dataset red_pajama-data-1_t-sample (/apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039) Loading Dataset infts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84376:84376 [2] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84378:84378 [4] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84379:84379 [5] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84376:84376 [2] NCCL INFO Bootstrap : Using eth1:11.215.57.242<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84378:84378 [4] NCCL INFO Bootstrap : Using eth1:11.215.57.242<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84379:84379 [5] NCCL INFO Bootstrap : Using eth1:11.215.57.242<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84376:84376 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84378:84378 [4] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84379:84379 [5] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84376:85316 [2] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.57.242<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84376:85316 [2] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84379:85318 [5] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.57.242<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84378:85317 [4] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.57.242<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84379:85318 [5] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84378:85317 [4] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84377:84377 [3] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84377:84377 [3] NCCL INFO Bootstrap : Using eth1:11.215.57.242<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84377:84377 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84377:85322 [3] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.57.242<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84377:85322 [3] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84380:84380 [6] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84380:84380 [6] NCCL INFO Bootstrap : Using eth1:11.215.57.242<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84380:84380 [6] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84380:85324 [6] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.215.57.242<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84380:85324 [6] NCCL INFO Using network IB transformers_version": "4.28.0.dev0" } ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133154:133154 [5] NCCL INFO cudaDriverVersion 1107tsts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133154:133154 [5] NCCL INFO Bootstrap : Using eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133154:133154 [5] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133154:134094 [5] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133154:134094 [5] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133151:133151 [2] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133151:133151 [2] NCCL INFO Bootstrap : Using eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133151:133151 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133151:134096 [2] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133151:134096 [2] NCCL INFO Using network IB ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133149:133149 [0] NCCL INFO cudaDriverVersion 11070 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133149:133149 [0] NCCL INFO Bootstrap : Using eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133149:133149 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133149:134098 [0] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth1:11.218.9.169<0> ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133149:134098 [0] NCCts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:67958 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,ffff0000,00000000,ffffffff,ffff0000,00000000 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:67952 [2] NCCL INFO Setting affinity for GPU 2 to ffff,ffffffff,00000000,0000ffff,ffffffff ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:67951 [0] NCCL INFO Setting affinity for GPU 0 to ffff,ffffffff,00000000,0000ffff,ffffffff ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:67959 [1] NCCL INFO Setting affinity for GPU 1 to ffff,ffffffff,00000000,0000ffff,ffffffff ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:67953 [4] NCCL INFO Setting affinity for GPU 4 to ffffffff,ffff0000,00000000,ffffffff,ffff0000,00000000 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67026:67954 [6] NCCL INFO Setting affinity for GPU 6 to ffffffff,ffff0000,00000000,ffffffff,ffff0000,00000000 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:67957 [3] NCCL INFO Setting affinity for GPU 3 to ffff,ffffffff,00000000,0000ffff,ffffffff ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:67955 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffff0000,00000000,ffffffff,ffff0000,00000000 ts-cbba8ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8022:8915 [7] NCCL INFO Trees [0] 16/-1/-1->23->22 [1] 16/-1/-1->23->22 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8908 [2] NCCL INFO Trees [0] 19/26/-1->18->2 [1] 19/-1/-1->18->11 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8015:8911 [0] NCCL INFO Trees [0] 17/-1/-1->16->23 [1] 17/-1/-1->16->23 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8016:8914 [1] NCCL INFO Trees [0] -1/-1/-1->17->16 [1] -1/-1/-1->17->16 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8021:8909 [6] NCCL INFO Trees [0] 23/-1/-1->22->21 [1] 23/-1/-1->22->21 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:8912 [3] NCCL INFO Trees [0] 20/10/-1->19->18 [1] 20/-1/-1->19->18 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8020:8910 [5] NCCL INFO Trees [0] 22/-1/-1->21->20 [1] 22/-1/-1->21->20 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8019:8913 [4] NCCL INFO Trees [0] 21/-1/-1->20->19 [1] 21/-1/-1->20->19 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8908 [2] NCCL INFO Channel 00/0 : 18[4b000] -> 23[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8015:8911 [0] NCCL INFO Channel 00/0 : 16[e000] -> 19[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8019:8913 [4] NCCL INFO Channel 00/0 : 20[93000] -> 17[13000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8908 [2] NCCL INFO Channel 01/0 : 18[4b000] -> 23[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8015:8911 [0] NCCL INFO Channel 01/0 : 16[e000] -> 19[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8019:8913 [4] NCCL INFO Channel 01/0 : 20[93000] -> 17[13000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8021:8909 [6] NCCL INFO Channel 00/0 : 22[cb000] -> 21[99000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8020:8910 [5] NCCL INFO Channel 00/0 : 21[99000] -> 20[93000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8021:8909 [6] NCCL INFO Channel 01/0 : 22[cb000] -> 21[99000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8020:8910 [5] NCCL INFO Channel 01/0 : 21[99000] -> 20[93000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8022:8915 [7] NCCL INFO Channel 00/0 : 23[d0000] -> 22[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:8912 [3] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8016:8914 [1] NCCL INFO Channel 00/0 : 17[13000] -> 16[e000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8022:8915 [7] NCCL INFO Channel 01/0 : 23[d0000] -> 22[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8016:8914 [1] NCCL INFO Channel 01/0 : 17[13000] -> 16[e000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8022:8915 [7] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8015:8911 [0] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8015:8911 [0] NCCL INFO Channel 00/0 : 16[e000] -> 17[13000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8016:8914 [1] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8019:8913 [4] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8015:8911 [0] NCCL INFO Channel 01/0 : 16[e000] -> 17[13000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8021:8909 [6] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8020:8910 [5] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8019:8913 [4] NCCL INFO Channel 00/0 : 20[93000] -> 21[99000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8019:8913 [4] NCCL INFO Channel 01/0 : 20[93000] -> 21[99000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8021:8909 [6] NCCL INFO Channel 00/0 : 22[cb000] -> 23[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8016:8914 [1] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8016:8914 [1] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8016:8914 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8015:8911 [0] NCCL INFO Channel 00/0 : 16[e000] -> 23[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8020:8910 [5] NCCL INFO Channel 00/0 : 21[99000] -> 22[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8021:8909 [6] NCCL INFO Channel 01/0 : 22[cb000] -> 23[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8015:8911 [0] NCCL INFO Channel 01/0 : 16[e000] -> 23[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8908 [2] NCCL INFO Channel 00/0 : 11[51000] -> 18[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8020:8910 [5] NCCL INFO Channel 01/0 : 21[99000] -> 22[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:8912 [3] NCCL INFO Channel 00/0 : 19[51000] -> 26[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8021:8909 [6] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8021:8909 [6] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8021:8909 [6] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8020:8910 [5] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8020:8910 [5] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8020:8910 [5] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8022:8915 [7] NCCL INFO Channel 00/0 : 23[d0000] -> 16[e000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8022:8915 [7] NCCL INFO Channel 01/0 : 23[d0000] -> 16[e000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8022:8915 [7] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8022:8915 [7] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8022:8915 [7] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8015:8911 [0] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8015:8911 [0] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8015:8911 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8016:8914 [1] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8908 [2] NCCL INFO Channel 01/0 : 11[51000] -> 18[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:8912 [3] NCCL INFO Channel 01/0 : 19[51000] -> 26[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8982 [2] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:8988 [3] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8982 [2] NCCL INFO NCCL_IB_SL set by environment to 3. ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8021:8909 [6] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8020:8910 [5] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8022:8915 [7] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8015:8911 [0] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:8912 [3] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:8912 [3] NCCL INFO Channel 00/0 : 19[51000] -> 20[93000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8908 [2] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8908 [2] NCCL INFO Channel 00/0 : 18[4b000] -> 19[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:67957 [3] NCCL INFO Channel 00/0 : 3[51000]ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8908 [2] NCCL INFO Channel 01/0 : 18[4b000] -> 19[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8019:8913 [4] NCCL INFO Channel 00/0 : 20[93000] -> 19[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8019:8913 [4] NCCL INFO Channel 01/0 : 20[93000] -> 19[51000] via P2P/IPC/rets-ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8908 [2] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8908 [2] NCCL INFO Channel 00/0 : 18[4b000] -> 26[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:8912 [3] NCCL INFO Channel 00/0 : 10[4b000] -> 19[51000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8908 [2] NCCL INFO Channel 00/0 : 2[4b000] -> 18[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:8988 [3] NCCL INFO NCCL_IB_SL set by environment to 3. ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8908 [2] NCCL INFO Channel 00/0 : 18[4b000] -> 2[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:8912 [3] NCCL INFO Channel 00/0 : 19[51000] -> 10[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8908 [2] NCCL INFO Channel 00/0 : 26[4b000] -> 18[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8908 [2] NCCL INFO Channel 01/0 : 18[4b000] ts-cbba87c5e7504a249f5127103d9ce40f-workts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:8912 [3] NCCL INFO Channel 00/0 : 19[51000] -> 18[4b000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:8912 [3] NCCL INFO Channel 01/0 : 19[51000] -> 18[4b000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8908 [2] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8908 [2] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8908 [2] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8019:8913 [4] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8019:8913 [4] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8019:8913 [4] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8019:8913 [4] NCCL INFO NCCL_NET_GDR_READ set by environment to 1. ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:8912 [3] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:8912 [3] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:8912 [3] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8015:8911 [0] NCCL INFO comm 0x44ea48c0 rank 16 nranks 32 cudaDev 0 busId e000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:8912 [3] NCCL INFO comm 0x439c6e10 rank 19 nranks 32 cudaDev 3 busId 51000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8022:8915 [7] NCCL INFO comm 0x45c4b830 rank 23 nranks 32 cudaDev 7 busId d0000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8021:8909 [6] NCCL INFO comm 0x4411e5c0 rank 22 nranks 32 cudaDev 6 busId cb000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8020:8910 [5] NCCL INFO comm 0x43e1d460 rank 21 nranks 32 cudaDev 5 busId 99000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8019:8913 [4] NCCL INFO comm 0x442726c0 rank 20 nranks 32 cudaDev 4 busId 93000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8016:8914 [1] NCCL INFO comm 0x44e39400 rank 17 nranks 32 cudaDev 1 busId 13000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:8908 [2] NCCL INFO comm 0x44c0e980 rank 18 nranks 32 cudaDev 2 busId 4b000 - Init COMPLETE . 7504a249f5127103d9ce40f-worker-0:133153:134076 [4] NCCL INFO comm 0x44a033c0 rank 12 nranks 32 cudaDev 4 busId 93000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133156:134086 [7] NCCL INFO comm 0x44b216f0 rank 15 nranks 32 cudaDev 7 busId d0000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133151:134096 [2] NCCL INFO comm 0x40f33bd0 rank 10 nranks 32 cudaDev 2 busId 4b000 - Init COMPLETE 30 rank 31 nranks 32 cudaDev 7 busId d0000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84377:85322 [3] NCCL INFO comm 0x45e84f80 rank 27 nranks 32 cudaDev 3 busId 51000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84376:85316 [2] NCCL INFO comm 0x43fd4740 rank 26 nranks 32 cudaDev 2 busId 4b000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84375:85333 [1] NCCL INFO comm 0x451ffa30 rank 25 nranks 32 cudaDev 1 busId 13000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84377:85346 [3] NCCL INFO NCCL_IB_SL set by environment to 3. Loading checkpoint shards: 0%| | 0/3 [00:00> Using pad_token, but it is not set yet. [ERROR|tokenization_utils_base.py:1042] 2024-01-29 11:13:13,889 >> Using pad_token, but it is not set yet. [ERROR|tokenization_utils_base.py:1042] 2024-01-29 11:13:13,894 >> Using pad_token, but it is not set yet. Loading checkpoint shards: 100%|██████████| 3/3 [01:03<00:00, 19.94s/it] Loading checkpoint shards: 100%|██████████| 3/3 [01:03<00:00, 21.27s/it] [ERROR|tokenization_utils_base.py:1042] 2024-01-29 11:13:13,882 >> Using pad_token, but it is not set yet. Loading checkpoint shards: 100%|██████████| 3/3 [01:03<00:00, 19.94s/it] Loading checkpoint shards: 100%|██████████| 3/3 [01:03<00:00, 21.27s/it] [ERROR|tokenization_utils_base.py:1042] 2024-01-29 11:13:13,895 >> Using pad_token, but it is not set yet. Loading checkpoint shards: 100%|██████████| 3/3 [01:03<00:00, 19.95s/it] Loading checkpoint shards: 100%|██████████| 3/3 [01:03<00:00, 21.29s/it] [ERROR|tokenization_utils_base.py:1042] 2024-01-29 11:13:13,960 >> Using pad_token, but it is not set yet. Loading checkpoint shards: 100%|██████████| 3/3 [01:03<00:00, 19.96s/it] Loading checkpoint shards: 100%|██████████| 3/3 [01:03<00:00, 21.29s/it] [ERROR|tokenization_utils_base.py:1042] 2024-01-29 11:13:13,972 >> Using pad_token, but it is not set yet. Loading checkpoint shards: 100%|██████████| 3/3 [01:03<00:00, 19.95s/it] Loading checkpoint shards: 100%|██████████| 3/3 [01:03<00:00, 21.32s/it] [INFO|modeling_utils.py:3029] 2024-01-29 11:13:14,036 >> All model checkpoint weights were used when initializing LlamaForCausalLM. [INFO|modeling_utils.py:3037] 2024-01-29 11:13:14,036 >> All the weights of LlamaForCausalLM were initialized from the model checkpoint at /apdcephfs/share_733425/vinnylywang/jianhuipang/opensourcellms/llama2/Llama-2-13b-hf. If your task is similar to the task the model of the checkpoint was trained on, you can already use LlamaForCausalLM for predictions without further training. [INFO|configuration_utils.py:535] 2024-01-29 11:13:14,045 >> loading configuration file /apdcephfs/share_733425/vinnylywang/jianhuipang/opensourcellms/llama2/Llama-2-13b-hf/generation_config.json [INFO|configuration_utils.py:575] 2024-01-29 11:13:14,045 >> Generate config GenerationConfig { "bos_token_id": 1, "do_sample": true, "eos_token_id": 2, "max_length": 4096, "pad_token_id": 0, "temperature": 0.6, "top_p": 0.9, "transformers_version": "4.28.0.dev0" } [ERROR|tokenization_utils_base.py:1042] 2024-01-29 11:13:14,045 >> Using pad_token, but it is not set yet. [INFO|tokenization_utils_base.py:907] 2024-01-29 11:13:14,045 >> Assigning [PAD] to the pad_token key of the tokenizer [INFO|tokenization_utils.py:426] 2024-01-29 11:13:14,045 >> Adding [PAD] to the vocabulary [INFO|tokenization_utils_base.py:907] 2024-01-29 11:13:17,361 >> Assigning to the eos_token key of the tokenizer [INFO|tokenization_utils_base.py:907] 2024-01-29 11:13:17,362 >> Assigning to the bos_token key of the tokenizer [INFO|tokenization_utils_base.py:907] 2024-01-29 11:13:17,362 >> Assigning to the unk_token key of the tokenizer [INFO|tokenization_utils.py:426] 2024-01-29 11:13:17,452 >> Adding to the vocabulary 01/29/2024 11:13:17 - INFO - __main__ - We have added new 1 token as an anchor Process #0 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00000_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #0 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00000_of_00032.arrow Process #1 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00001_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #1 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00001_of_00032.arrow Process #2 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00002_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #2 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00002_of_00032.arrow Process #3 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00003_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #3 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00003_of_00032.arrow Process #4 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00004_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #4 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00004_of_00032.arrow Process #5 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00005_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #5 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00005_of_00032.arrow Process #6 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00006_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #6 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00006_of_00032.arrow Process #7 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00007_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #7 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00007_of_00032.arrow Process #8 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00008_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #8 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00008_of_00032.arrow Process #9 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00009_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #9 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00009_of_00032.arrow Process #10 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00010_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #10 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00010_of_00032.arrow Process #11 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00011_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #11 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00011_of_00032.arrow Process #12 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00012_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #12 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00012_of_00032.arrow Process #13 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00013_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #13 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00013_of_00032.arrow Process #14 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00014_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #14 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00014_of_00032.arrow Process #15 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-77d0271b9daa8aa8_00015_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #15 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00015_of_00032.arrow Process #16 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00016_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #16 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00016_of_00032.arrow Process #17 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00017_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #17 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00017_of_00032.arrow Process #18 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00018_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #18 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00018_of_00032.arrow Process #19 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00019_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #19 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00019_of_00032.arrow Process #20 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00020_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #20 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00020_of_00032.arrow Process #21 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00021_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #21 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00021_of_00032.arrow Process #22 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00022_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #22 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00022_of_00032.arrow Process #23 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00023_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #23 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00023_of_00032.arrow Process #24 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00024_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #24 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00024_of_00032.arrow Process #25 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00025_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #25 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00025_of_00032.arrow Process #26 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00026_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #26 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00026_of_00032.arrow Process #27 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00027_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #27 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00027_of_00032.arrow Process #28 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00028_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #28 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00028_of_00032.arrow Process #29 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00029_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #29 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00029_of_00032.arrow Process #30 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00030_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #30 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00030_of_00032.arrow Process #31 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00031_of_00032.arrow 01/29/2024 11:13:20 - INFO - datasets.arrow_dataset - Process #31 will write at /apdcephfs/share_733425/vinnylywang/jianhuipang/hf_cache2/datasets/red_pajama-data-1_t-sample/plain_text/1.0.0/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039/cache-de620f8b1ff7998e_00031_of_00032.arrow Spawning 32 processes 01/29/2024 11:13:21 - INFO - datasets.arrow_dataset - Spawning 32 processes Map (num_proc=32): 0%| | 0/930514 [00:00> Using cuda_amp half precision backend Map (num_proc=32): 0%| | 0/930514 [00:00 [2024-01-29 11:45:26,038] [INFO] [logging.py:96:log_dist] [Rank 0] Creating fp16 ZeRO stage 3 optimizer, MiCS is enabled False, Hierarchical params gather False [2024-01-29 11:45:26,038] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 3 optimizer [2024-01-29 11:45:26,148] [INFO] [utils.py:802:see_memory_usage] Stage 3 initialize beginning [2024-01-29 11:45:26,149] [INFO] [utils.py:803:see_memory_usage] MA 0.65 GB Max_MA 1.3 GB CA 1.61 GB Max_CA 2 GB [2024-01-29 11:45:26,149] [INFO] [utils.py:810:see_memory_usage] CPU Virtual Memory: used = 58.9 GB, percent = 5.9% [2024-01-29 11:45:26,153] [INFO] [stage3.py:126:__init__] Reduce bucket size 26214400 [2024-01-29 11:45:26,153] [INFO] [stage3.py:127:__init__] Prefetch bucket size 23592960 [2024-01-29 11:45:26,256] [INFO] [utils.py:802:see_memory_usage] DeepSpeedZeRoOffload initialize [begin] [2024-01-29 11:45:26,257] [INFO] [utils.py:803:see_memory_usage] MA 0.65 GB Max_MA 0.65 GB CA 1.61 GB Max_CA 2 GB [2024-01-29 11:45:26,257] [INFO] [utils.py:810:see_memory_usage] CPU Virtual Memory: used = 59.47 GB, percent = 5.9% 29875, 4569, 29906, 29900, 29896, 29955, 29913, 322, 27599, 515, 372, 13, 29906, 29889, 29906, 24464, 29905, 13007, 25741, 3190, 2347, 515, 29871, 29896, 29953, 29900, 7284, 29905, 13007, 9279, 322, 4148, 287, 491, 13, 29946, 29941, 7284, 29905, 13007, 15717, 2645, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29896, 931, 3785, 29889, 32001, 1334, 1737, 324, 542, 403, 18777, 304, 13, 29905, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 29892, 773, 408, 18470, 4876, 4234, 775, 2246, 29899, 5563, 21904, 313, 617, 29911, 10249, 29879, 29897, 322, 29871, 13, 8921, 313, 4102, 29914, 4230, 29897, 2983, 9401, 411, 1024, 18822, 2820, 278, 3186, 29892, 322, 17998, 1283, 7224, 29871, 13, 1195, 287, 515, 9063, 15562, 29889, 32001, 1334, 1284, 10757, 310, 278, 4688, 8022, 749, 310, 4644, 6813, 297, 1722, 2752, 13, 20415, 29892, 2678, 8772, 491, 4092, 29889, 32001, 2860, 393, 3785, 29892, 278, 1737, 12122, 6894, 537, 29871, 13, 262, 970, 775, 756, 1063, 21003, 10231, 29889, 32001, 1334, 884, 12439, 8018, 15839, 528, 17741, 13, 12817, 304, 278, 1095, 310, 278, 8291, 6415, 29129, 322, 278, 7910, 310, 14137, 4631, 4135, 297, 13, 23369, 1705, 322, 4275, 14325, 29892, 408, 1532, 408, 310, 2545, 1664, 17292, 28342, 763, 25539, 1608, 322, 13, 25719, 10298, 4822, 10916, 313, 6727, 16783, 29914, 331, 16783, 467, 32001, 320, 26956, 29912, 1469, 20847, 3097, 5003, 32001, 319, 1634, 1414, 3577, 363, 445, 5650, 338, 3625, 515, 796, 264, 8144, 472, 13, 29905, 2271, 29912, 991, 597, 1867, 29875, 29889, 990, 29914, 29896, 29900, 29889, 29945, 29906, 29947, 29896, 29914, 2256, 8144, 29889, 29953, 29941, 29929, 29900, 29941, 29945, 29945, 29913, 2651, 2036, 29912, 3445, 1414, 29899, 5113, 1836, 32001, 320, 2042, 29912, 9662, 630, 5244, 29913, 13, 29905, 1643, 29912, 3471, 29901, 12817, 29913, 13, 13, 29933, 720, 4688, 322, 7786, 1736, 2651, 2036, 29912, 12443, 10578, 29906, 29900, 29900, 29945, 5062, 11235, 29892, 13260, 333, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 13, 29871, 696, 7586, 29906, 29900, 29896, 29946, 7610, 345, 2941, 271, 24541, 29892, 697, 309, 29906, 29900, 29896, 29953, 311, 5365, 550, 332, 6950, 29913, 505, 2931, 1891, 278, 13, 479, 5275, 310, 12362, 29914, 6585, 7562, 18540, 313, 5800, 1799, 29897, 773, 320, 7278, 29912, 6734, 26946, 952, 1118, 13, 4716, 3867, 1880, 29899, 29567, 6089, 541, 526, 9078, 297, 2159, 313, 29906, 29899, 29945, 5940, 29968, 18777, 29897, 13, 392, 508, 367, 4768, 1463, 491, 5221, 424, 23460, 29889, 32001, 512, 29871, 29906, 29900, 29900, 29947, 2261, 801, 2681, 634, 394, 8770, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29913, 18043, 263, 3031, 979, 13, 16961, 29899, 7052, 313, 1454, 278, 931, 29897, 6559, 373, 18322, 1799, 320, 7278, 29912, 479, 5275, 773, 1375, 292, 7047, 13, 29871, 28914, 313, 4345, 29934, 29897, 13698, 1836, 32001, 2688, 29537, 287, 278, 3978, 310, 29871, 29896, 5940, 29924, 17737, 29560, 13, 4746, 278, 7562, 2831, 479, 1404, 2566, 322, 611, 6504, 1051, 3190, 3145, 975, 278, 13, 29896, 29929, 29929, 29929, 489, 29906, 29900, 29900, 29945, 3785, 29892, 773, 408, 18470, 2472, 2788, 304, 1749, 29879, 29901, 4876, 21904, 13, 392, 17998, 1283, 7224, 29889, 32001, 450, 12399, 3785, 313, 29955, 2440, 29897, 297, 2651, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29913, 338, 20511, 1135, 29871, 13, 5816, 338, 12399, 297, 278, 2198, 5650, 313, 29945, 29900, 2440, 29897, 322, 278, 848, 8974, 526, 29871, 13, 27489, 873, 1422, 29936, 411, 393, 297, 3458, 29892, 1749, 2582, 1510, 263, 10029, 7200, 14978, 310, 29871, 13, 15654, 273, 325, 13217, 29940, 2072, 3082, 20706, 29889, 32001, 7280, 29190, 936, 664, 515, 29871, 29906, 29900, 29896, 29900, 491, 14619, 29882, 371, 4099, 29894, 322, 13, 29950, 309, 1372, 2651, 2036, 29912, 29873, 19426, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29913, 4023, 29894, 2868, 1583, 29899, 7099, 433, 1127, 1737, 12122, 13, 2029, 800, 310, 25492, 15303, 8304, 3598, 1494, 1009, 12368, 29892, 13, 15914, 292, 2472, 363, 779, 14850, 4535, 29892, 29955, 29900, 5940, 29968, 25492, 4160, 29889, 32001, 319, 1407, 7786, 13, 1287, 2651, 2036, 29912, 21686, 29879, 29906, 29900, 29906, 29896, 2209, 479, 5275, 29913, 491, 399, 496, 29879, 634, 394, 13217, 5349, 1737, 324, 542, 630, 4203, 263, 7284, 13, 28712, 16046, 4160, 29892, 2534, 26869, 472, 3203, 29871, 29896, 29900, 29900, 25741, 1269, 29892, 322, 1058, 13, 1311, 29899, 7099, 8663, 14354, 373, 1009, 25492, 28723, 29889, 32001, 5806, 278, 6559, 338, 13, 3149, 29899, 262, 29899, 2230, 408, 310, 29871, 29906, 29900, 29906, 29896, 29892, 278, 15717, 7252, 1009, 1284, 886, 13, 351, 475, 303, 2651, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29892, 1850, 29882, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29913, 304, 13, 18609, 675, 278, 14675, 310, 18322, 1799, 1737, 5275, 975, 278, 931, 15101, 845, 1862, 4586, 491, 13, 1552, 2211, 11898, 29889, 32001, 3831, 1965, 411, 3517, 29190, 936, 1736, 29892, 1749, 6559, 338, 1568, 7200, 6287, 5634, 29882, 5555, 13, 7054, 12339, 287, 29871, 29946, 29941, 7284, 29905, 13007, 15717, 310, 29871, 29906, 29889, 29906, 24464, 29905, 13007, 25741, 515, 29871, 29896, 29953, 29900, 7284, 29905, 13007, 13, 16418, 5634, 5426, 11267, 979, 975, 29871, 29945, 29900, 2440, 310, 970, 775, 20706, 3265, 1135, 13, 3149, 297, 931, 29892, 322, 884, 901, 2691, 29899, 3874, 1312, 313, 2541, 1629, 29899, 1609, 29899, 6360, 3803, 1070, 537, 975, 13, 1552, 8900, 3785, 467, 32001, 8108, 1189, 1711, 29892, 1749, 6559, 337, 3687, 373, 10079, 11264, 13, 3924, 313, 8257, 29903, 29897, 9063, 848, 3265, 1135, 7481, 29899, 7099, 433, 1127, 4423, 2472, 29889, 32001, 5901, 1736, 5634, 262, 3153, 278, 664, 491, 8432, 2651, 2036, 29912, 18386, 709, 29906, 29900, 29896, 29941, 2209, 29881, 24974, 29913, 13, 392, 29892, 901, 10325, 29892, 390, 579, 468, 29875, 634, 394, 8770, 2036, 29912, 29878, 579, 468, 29875, 29906, 29900, 29896, 29953, 479, 711, 3173, 29892, 13, 29871, 364, 579, 468, 29875, 29906, 29900, 29896, 29947, 479, 711, 3173, 29892, 544, 1648, 29906, 29900, 29906, 29896, 479, 468, 1581, 29881, 24974, 29913, 5634, 17532, 12399, 1737, 12122, 13, 29905, 7278, 29912, 29881, 24974, 322, 24003, 1118, 474, 29889, 29872, 1696, 278, 15834, 304, 607, 278, 3978, 310, 18322, 1799, 13, 17426, 6602, 1009, 11465, 1230, 14137, 14188, 29889, 32001, 512, 445, 664, 591, 2931, 1891, 1737, 12122, 6894, 537, 297, 970, 775, 363, 278, 937, 13, 2230, 472, 445, 6287, 29892, 1716, 297, 4958, 310, 17737, 29560, 322, 15500, 3785, 29889, 32001, 1334, 437, 13, 1333, 22002, 280, 278, 24003, 10696, 29892, 541, 3867, 29190, 936, 848, 322, 1284, 886, 393, 508, 367, 13, 280, 369, 4063, 304, 393, 1095, 408, 5434, 664, 29889, 32001, 320, 7278, 29912, 12756, 7047, 21639, 29913, 2651, 2036, 29912, 2276, 5824, 19982, 29906, 29900, 29900, 29955, 23705, 1338, 29893, 996, 29913, 338, 278, 13, 1491, 29899, 2671, 310, 7047, 21639, 393, 756, 29537, 287, 278, 18066, 267, 310, 21640, 13, 6734, 24771, 13149, 635, 29892, 3704, 278, 2702, 5932, 310, 920, 304, 5376, 13, 2541, 1737, 12122, 6894, 537, 2651, 2036, 29912, 13376, 303, 456, 29906, 29900, 29900, 29953, 10945, 3359, 29892, 1424, 29440, 29906, 29900, 29896, 29946, 23027, 5933, 1836, 32001, 3826, 3076, 2678, 278, 2198, 6559, 8128, 10757, 393, 508, 367, 1304, 29892, 297, 278, 13, 14940, 1206, 310, 970, 775, 322, 472, 263, 1407, 2919, 6287, 29892, 304, 11539, 607, 13, 14032, 4637, 310, 5534, 7047, 21639, 505, 9820, 484, 15774, 29889, 32001, 320, 2042, 29912, 4062, 3002, 29913, 13, 29905, 1643, 29912, 3471, 29901, 5696, 29913, 13, 13, 13, 29905, 1482, 361, 29905, 361, 29887, 798, 386, 1003, 29871, 320, 29887, 798, 386, 1003, 3009, 13, 29905, 361, 29887, 798, 386, 1003, 13, 29905, 463, 29912, 4532, 29913, 13, 29871, 320, 7313, 29961, 2103, 2013, 4914, 2103, 3199, 6360, 368, 29899, 2055, 1169, 29913, 13, 29871, 320, 6671, 29912, 12883, 368, 970, 25741, 975, 931, 313, 1188, 6287, 467, 13, 29913, 32001, 320, 1643, 29912, 1003, 29901, 29887, 798, 386, 29913, 13, 29905, 355, 29912, 4532, 29913, 13, 29905, 7241, 13, 13, 29905, 26956, 29912, 16390, 24541, 29913, 13, 13, 4806, 27387, 515, 320, 23066, 29950, 29914, 2651, 2036, 29912, 29879, 1332, 29899, 1516, 29878, 29906, 29900, 29896, 29929, 29899, 24713, 29913, 599, 25741, 3190, 2347, 2745, 320, 25832, 1964, 579, 1523, 2415, 2539, 6294, 32001, 2688, 5253, 304, 320, 14573, 5261, 1169, 22131, 29914, 25741, 29892, 5412, 491, 317, 15715, 29896, 15882, 29892, 4023, 29894, 2868, 515, 320, 25832, 1299, 7288, 5261, 1169, 797, 7068, 29914, 970, 9279, 6421, 515, 4655, 5849, 363, 2710, 313, 28712, 16046, 29892, 11786, 28632, 29892, 2992, 1846, 32001, 322, 3577, 28914, 313, 10251, 713, 29892, 10772, 2227, 29892, 405, 13427, 29892, 2992, 6250, 32001, 1876, 1169, 297, 278, 8783, 526, 491, 320, 25832, 6344, 2806, 943, 22131, 29914, 15717, 29892, 5412, 491, 779, 6990, 29938, 978, 29892, 4876, 4535, 5854, 29938, 11000, 29889, 32001, 450, 8783, 2996, 408, 1023, 1104, 1288, 6131, 29892, 697, 363, 25741, 322, 697, 363, 15717, 29892, 411, 278, 4642, 29371, 278, 7480, 3025, 263, 9117, 1820, 29889, 32001, 320, 361, 5426, 13, 9760, 1948, 297, 278, 9063, 1591, 3743, 278, 1494, 4235, 29901, 9063, 317, 15715, 29896, 15882, 29892, 4148, 322, 844, 5171, 5335, 342, 15092, 29892, 4148, 322, 844, 5171, 2893, 14903, 313, 20275, 16750, 278, 4148, 1591, 467, 32001, 450, 21578, 1546, 9063, 15717, 322, 9063, 2153, 2041, 515, 11786, 29892, 607, 6511, 304, 9063, 263, 1735, 4148, 287, 491, 4856, 1683, 29889, 32001, 1152, 445, 6559, 591, 21309, 373, 15717, 322, 17262, 9063, 2153, 29892, 408, 278, 4328, 1546, 278, 1023, 338, 451, 8018, 363, 1749, 5925, 5155, 322, 278, 5253, 310, 25741, 411, 263, 844, 5171, 916, 1135, 967, 4148, 338, 3480, 3473, 1821, 29889, 32001, 320, 7241, 13, 2831, 1269, 6251, 297, 278, 4148, 1591, 591, 505, 4148, 2989, 1024, 322, 4876, 408, 1023, 5004, 6031, 310, 10650, 6262, 29889, 32001, 1334, 6206, 2411, 15273, 1821, 470, 18325, 519, 2983, 393, 29901, 526, 451, 1602, 397, 519, 408, 18351, 29899, 29947, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12283, 7099, 397, 519, 29914, 4148, 2983, 6206, 511, 526, 4876, 14157, 2012, 310, 2983, 3441, 25832, 6344, 2806, 943, 29934, 29885, 9823, 29914, 4954, 7039, 4907, 511, 5718, 310, 871, 9654, 4890, 3441, 25832, 6344, 2806, 943, 29934, 29885, 10358, 804, 23201, 1712, 901, 1135, 29871, 29896, 29900, 8958, 1661, 29899, 1026, 2153, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12283, 15670, 23201, 526, 5520, 1135, 29871, 29896, 29900, 29900, 4890, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12229, 549, 12495, 32001, 2860, 21166, 29892, 1048, 320, 25832, 6344, 2806, 943, 29925, 15273, 1821, 2052, 307, 29916, 29914, 15717, 3441, 25832, 6344, 2806, 943, 29925, 15273, 1821, 29925, 312, 29914, 310, 278, 2847, 8783, 29897, 9488, 363, 4340, 7418, 29889, 32001, 3940, 393, 278, 5253, 310, 970, 775, 25741, 313, 392, 15717, 29897, 11122, 297, 278, 13, 11228, 8783, 25088, 18709, 9247, 975, 13, 2230, 2651, 2036, 29912, 29879, 1332, 29899, 771, 854, 749, 29899, 331, 344, 1012, 361, 29887, 798, 386, 1003, 29892, 408, 4318, 363, 25741, 297, 13, 29905, 29907, 999, 29912, 1003, 29901, 29887, 798, 386, 1012, 2870, 29901, 515, 395, 29896, 29900, 29985, 29946, 29938, 25741, 297, 29871, 29896, 29929, 29955, 29896, 29892, 304, 395, 29896, 29900, 29985, 29953, 29938, 297, 29871, 29896, 29929, 29929, 29947, 29892, 304, 13, 284, 3242, 395, 29896, 29900, 29985, 29929, 29938, 297, 29871, 29906, 29900, 29906, 29900, 29905, 7241, 29889, 32001, 1094, 263, 17004, 278, 8900, 534, 1975, 10331, 304, 367, 901, 13, 13844, 297, 7786, 1602, 3076, 1135, 297, 29871, 29946, 29900, 29974, 1629, 29899, 1025, 6743, 29892, 2861, 304, 13964, 4586, 373, 13, 735, 1112, 9247, 7200, 23093, 29889, 32001, 320, 26956, 29912, 7999, 324, 10610, 29913, 13, 13, 29905, 463, 29912, 4532, 29913, 13, 29871, 320, 9525, 13, 29871, 320, 7313, 29961, 24049, 29892, 15450, 29922, 29953, 4912, 29871, 29953, 4912, 29871, 29900, 29871, 29900, 29892, 2103, 2013, 16292, 3199, 1491, 1727, 1080, 29899, 2470, 29913, 13, 29871, 320, 6671, 29912, 1576, 320, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 1304, 408, 1737, 324, 10610, 22525, 5003, 32001, 320, 1643, 29912, 1003, 29901, 11526, 1958, 29913, 13, 29905, 355, 29912, 4532, 29913, 13, 13, 2887, 1737, 324, 10610, 22525, 591, 671, 11758, 3186, 12786, 10723, 515, 278, 3303, 18269, 1737, 359, 305, 2004, 2651, 2036, 29912, 348, 29896, 29929, 29929, 29929, 479, 359, 305, 2004, 1836, 32001, 1763, 4772, 2432, 3381, 491, 2919, 10916, 313, 29872, 29889, 29887, 1696, 7551, 470, 12710, 29897, 2629, 11758, 12786, 29892, 591, 19412, 322, 6219, 777, 12786, 2729, 373, 1737, 12122, 23203, 537, 322, 278, 19383, 310, 758, 331, 8946, 16375, 29769, 5680, 29892, 1316, 408, 19182, 4086, 29889, 32001, 320, 29907, 999, 29912, 1003, 29901, 11526, 1958, 29913, 3697, 278, 2186, 1051, 310, 320, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 1304, 408, 1737, 324, 10610, 22525, 297, 445, 6559, 29889, 32001, 1879, 324, 10610, 310, 9063, 15717, 304, 3186, 12786, 3913, 278, 1023, 19595, 653, 13698, 9129, 297, 2651, 2036, 29912, 293, 344, 29899, 344, 275, 29899, 29906, 29900, 29906, 29906, 29899, 26098, 1118, 23359, 337, 13998, 2400, 29889, 32001, 450, 937, 697, 337, 3687, 373, 278, 4234, 775, 2246, 29899, 5563, 5354, 313, 617, 29911, 10249, 29897, 310, 4876, 14157, 23892, 515, 9063, 15562, 29892, 321, 29889, 29887, 1696, 320, 16266, 26139, 1341, 1118, 320, 16266, 26139, 582, 1118, 320, 16266, 26139, 18038, 1118, 2992, 29889, 32001, 1334, 4687, 515, 278, 306, 2190, 29909, 1051, 310, 13548, 2931, 21759, 29911, 10249, 29879, 2651, 2036, 29912, 6011, 29899, 29883, 312, 430, 29913, 322, 7522, 20545, 1269, 6590, 20123, 304, 263, 3646, 3186, 5120, 29889, 32001, 450, 1473, 1737, 324, 10610, 11043, 3913, 278, 17998, 9210, 310, 9063, 5335, 342, 15092, 313, 29872, 29889, 29887, 1696, 17998, 29899, 29900, 29945, 29901, 29900, 29900, 29897, 322, 4148, 2983, 304, 8161, 278, 1556, 5517, 3186, 5120, 310, 278, 9063, 4148, 29889, 32001, 1152, 1269, 17998, 9210, 591, 8161, 263, 1051, 310, 15878, 7600, 313, 13509, 29892, 2106, 29892, 470, 14278, 20123, 29897, 297, 278, 3186, 393, 29892, 472, 278, 931, 310, 393, 9063, 29892, 750, 393, 17998, 9210, 29936, 9063, 931, 338, 1820, 1244, 29892, 408, 4234, 17998, 1283, 7224, 13100, 975, 931, 2861, 304, 29431, 3620, 29889, 32001, 1763, 1207, 445, 3683, 3381, 591, 671, 278, 306, 2190, 29909, 931, 10640, 2566, 2651, 2036, 29912, 17559, 1272, 1836, 32001, 1987, 591, 3566, 304, 1269, 2058, 263, 8158, 393, 4332, 1973, 278, 4188, 22342, 393, 263, 2183, 4148, 1024, 338, 17443, 310, 372, 29889, 32001, 1763, 445, 1095, 591, 671, 278, 28297, 29890, 15451, 8783, 310, 278, 29511, 310, 278, 1556, 3619, 937, 322, 3942, 2983, 607, 29892, 439, 11427, 515, 2651, 2036, 29912, 1079, 29890, 799, 29899, 7039, 6177, 2802, 277, 12181, 4954, 16123, 2247, 278, 26368, 5528, 5084, 310, 363, 264, 1280, 322, 269, 595, 1280, 7371, 515, 263, 2566, 310, 320, 1949, 29912, 29946, 29871, 29900, 29946, 29946, 29871, 29945, 29946, 29953, 29871, 29929, 29941, 29947, 29913, 2305, 313, 29945, 29945, 29889, 29945, 8958, 310, 8471, 2305, 297, 29871, 29906, 29900, 29896, 29946, 467, 32001, 1094, 310, 3839, 29871, 29906, 29900, 29896, 29929, 372, 18469, 320, 1949, 29912, 29906, 29955, 29871, 29953, 29953, 29906, 29871, 29947, 29900, 29896, 29913, 363, 264, 1280, 322, 320, 1949, 29912, 29906, 29955, 29871, 29906, 29900, 29953, 29871, 29947, 29906, 29896, 29913, 269, 595, 1280, 297, 29871, 29906, 29941, 29953, 24894, 8977, 1080, 29889, 4907, 29913, 32001, 1094, 297, 1749, 8783, 15717, 526, 2989, 1024, 6031, 313, 29878, 1624, 1135, 6219, 491, 937, 29914, 11922, 1024, 511, 591, 937, 5993, 675, 2983, 313, 1609, 1999, 1331, 322, 1206, 3620, 29897, 322, 769, 16280, 5375, 18897, 297, 1716, 937, 322, 3942, 2983, 10868, 8857, 29889, 32001, 1152, 1269, 1543, 1476, 297, 1024, 8857, 591, 22932, 278, 2058, 4665, 29905, 6661, 1333, 24212, 8875, 491, 278, 1024, 10868, 304, 4017, 263, 5645, 393, 338, 29839, 304, 278, 1353, 310, 12407, 24638, 393, 1024, 313, 6979, 29897, 297, 278, 2702, 2058, 29889, 32001, 320, 6661, 1333, 300, 1062, 29912, 1762, 4017, 4665, 2025, 1338, 5634, 294, 278, 17837, 310, 4954, 6689, 4907, 338, 25745, 23724, 29901, 2989, 10916, 325, 13217, 29879, 29399, 310, 2919, 10916, 805, 9450, 2999, 931, 29920, 2873, 5634, 705, 671, 263, 29544, 310, 7601, 8974, 313, 29872, 29889, 29887, 1696, 5874, 28007, 511, 322, 1661, 29899, 16072, 6743, 313, 29872, 29889, 29887, 1696, 14109, 7456, 467, 29913, 32001, 1334, 2533, 445, 4377, 363, 599, 3161, 304, 4017, 263, 2058, 8158, 29892, 17140, 701, 411, 263, 1051, 310, 779, 6990, 29938, 6689, 29892, 8158, 4535, 5854, 29938, 11000, 29889, 32001, 1334, 769, 8877, 445, 1051, 491, 278, 3186, 5120, 393, 263, 2058, 14393, 304, 322, 2533, 278, 8158, 363, 599, 278, 7600, 297, 1269, 5120, 304, 4017, 385, 12463, 8158, 29892, 6590, 304, 278, 4188, 22342, 393, 278, 9063, 14393, 304, 263, 2183, 3186, 5120, 29889, 32001, 1334, 3566, 278, 6257, 9063, 408, 6421, 515, 278, 3186, 5120, 411, 278, 9939, 8158, 29889, 32001, 450, 4876, 29899, 6707, 11043, 9378, 414, 515, 278, 9078, 322, 443, 5521, 8362, 671, 310, 21759, 29911, 10249, 29879, 29901, 1556, 18777, 671, 10035, 323, 10249, 29879, 1316, 408, 320, 16266, 26139, 510, 1118, 320, 16266, 26139, 990, 1118, 470, 320, 16266, 26139, 1212, 1836, 32001, 12808, 445, 947, 451, 3799, 26018, 4822, 20542, 29901, 3148, 29899, 6707, 18777, 29892, 363, 1342, 29892, 671, 278, 320, 16266, 26139, 375, 29913, 21759, 29911, 10249, 1568, 901, 269, 2495, 290, 368, 1135, 1009, 7824, 6795, 20895, 29889, 32001, 1551, 278, 916, 1361, 278, 9210, 29914, 978, 29899, 6707, 11043, 337, 3687, 373, 278, 17998, 9210, 310, 278, 9063, 5335, 342, 15092, 29889, 32001, 16809, 304, 5780, 22920, 373, 13897, 731, 14340, 29892, 263, 2919, 1353, 310, 25741, 297, 278, 8783, 756, 385, 17998, 9210, 5186, 304, 5225, 29889, 32001, 910, 6602, 29879, 3109, 7786, 25741, 3441, 14573, 5261, 1169, 29911, 29999, 29999, 13985, 1349, 681, 392, 27418, 6478, 29914, 310, 29871, 29906, 29900, 29906, 29900, 29879, 25741, 505, 263, 5225, 9210, 29897, 1135, 9642, 6743, 3441, 14573, 5261, 1169, 29911, 29999, 29999, 13985, 1349, 681, 392, 29914, 297, 29871, 29906, 29900, 29900, 29900, 467, 32001, 1094, 263, 1121, 278, 9210, 29914, 978, 29899, 6707, 11043, 1033, 1095, 701, 6459, 292, 263, 2919, 6232, 310, 9642, 25741, 408, 4148, 287, 491, 11715, 18777, 29892, 322, 304, 263, 3109, 261, 15834, 4092, 550, 29889, 32001, 1763, 6795, 1438, 5626, 591, 14405, 278, 1023, 1737, 324, 10610, 13698, 4208, 491, 15399, 278, 9210, 29914, 978, 29899, 6707, 13698, 304, 599, 25741, 411, 263, 1661, 29899, 9171, 17998, 9210, 29892, 322, 278, 4876, 29899, 6707, 373, 304, 599, 916, 25741, 29889, 32001, 320, 2042, 29912, 12191, 322, 8565, 1558, 291, 29913, 13, 29905, 1643, 29912, 3471, 29901, 9902, 29913, 13, 13, 29905, 463, 29912, 4532, 4044, 13, 29871, 320, 9525, 13, 29871, 320, 7313, 29961, 2103, 2013, 16292, 3199, 1429, 287, 29889, 5140, 29913, 13, 29871, 320, 6671, 29912, 29934, 20819, 310, 25741, 313, 27215, 29897, 322, 6136, 15717, 313, 22503, 29897, 491, 3186, 10640, 975, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29900, 3785, 5003, 32001, 320, 9868, 29961, 14732, 3199, 7264, 287, 2594, 8727, 6445, 278, 3186, 10640, 364, 2219, 359, 363, 25741, 322, 15717, 975, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29900, 3785, 5003, 32001, 320, 1643, 29912, 1003, 29901, 9902, 29913, 13, 29905, 355, 29912, 4532, 4044, 13, 13, 13, 29871, 13, 1762, 1234, 320, 25029, 29912, 29878, 29939, 29901, 479, 397, 24974, 29913, 591, 22229, 278, 1353, 310, 25741, 322, 8359, 15717, 639, 1629, 322, 639, 3186, 10640, 29889, 32001, 1334, 2198, 278, 7625, 2582, 297, 320, 29907, 999, 29912, 1003, 29901, 9902, 29913, 408, 1023, 5096, 287, 2594, 24469, 29892, 6445, 1629, 368, 2867, 3204, 29879, 363, 25741, 322, 15717, 8307, 29889, 32001, 7569, 2594, 11524, 263, 1629, 322, 338, 8877, 287, 297, 269, 29399, 6445, 278, 9063, 29914, 8921, 11959, 363, 1269, 310, 278, 3186, 12786, 310, 320, 29907, 999, 29912, 1003, 29901, 11526, 1958, 29913, 297, 393, 1629, 29889, 32001, 1763, 4772, 714, 27801, 2861, 304, 805, 272, 26538, 17737, 29560, 29892, 297, 278, 4148, 8727, 591, 871, 2050, 15717, 2534, 26869, 472, 3203, 29871, 29945, 25741, 297, 263, 2183, 1629, 29889, 32001, 5806, 5366, 1747, 534, 1975, 297, 278, 24469, 6456, 393, 278, 3001, 3694, 310, 25741, 322, 15717, 6548, 18709, 9247, 975, 931, 29889, 32001, 10133, 363, 278, 937, 2440, 297, 278, 24469, 29892, 278, 1353, 310, 848, 3291, 297, 777, 3186, 12786, 508, 367, 14154, 2319, 29892, 411, 8178, 27721, 373, 278, 25806, 310, 534, 1975, 29889, 32001, 320, 26956, 29912, 7999, 12122, 6894, 537, 975, 931, 29913, 13, 13, 3563, 497, 29892, 278, 2498, 534, 355, 5692, 304, 367, 393, 278, 320, 4534, 29912, 479, 12122, 6894, 537, 297, 970, 775, 338, 10231, 6177, 4644, 6813, 322, 4092, 5136, 630, 1009, 4954, 24130, 749, 4907, 2745, 278, 7256, 310, 278, 29871, 29929, 29900], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [1, 320, 2042, 29912, 25898, 29913, 13, 29905, 1643, 29912, 3471, 29901, 23333, 29913, 13, 13, 29905, 7278, 29912, 29954, 1581, 6894, 537, 1118, 470, 901, 4049, 967, 10225, 727, 974, 29892, 4249, 27138, 304, 13, 20415, 5849, 14188, 756, 1063, 26606, 12399, 297, 7786, 2440, 29889, 32001, 512, 13, 1595, 16311, 29892, 278, 10122, 310, 29892, 9545, 310, 29892, 322, 6795, 1004, 25414, 363, 320, 7278, 29912, 26098, 13, 29871, 24003, 29913, 297, 12362, 29914, 6585, 7562, 18540, 313, 5800, 1799, 29897, 505, 4520, 263, 3287, 310, 8570, 13, 957, 278, 4940, 316, 6332, 2651, 2036, 29912, 29881, 16093, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 3855, 5871, 29906, 29900, 29896, 29900, 29895, 311, 29893, 2770, 29892, 13, 29871, 1055, 29888, 375, 29906, 29900, 29896, 29906, 5041, 267, 29892, 413, 434, 305, 1358, 29906, 29900, 29896, 29906, 26098, 29888, 2209, 29892, 19723, 5475, 4979, 29906, 29900, 29896, 29946, 26098, 29892, 13, 29871, 697, 309, 29906, 29900, 29896, 29953, 311, 5365, 550, 332, 6950, 29892, 696, 7586, 29906, 29900, 29896, 29953, 29893, 2770, 7192, 2209, 29892, 1935, 15044, 29906, 29900, 29896, 29955, 26098, 29892, 13, 29871, 503, 562, 305, 3350, 492, 29906, 29900, 29906, 29896, 26098, 1836, 32001, 320, 7278, 29912, 7999, 12122, 6894, 537, 29913, 338, 373, 278, 916, 1361, 278, 13, 14380, 310, 6894, 537, 393, 380, 1567, 515, 27138, 297, 777, 5534, 6354, 6421, 13, 3166, 1422, 3186, 12786, 322, 4185, 1973, 29889, 32001, 26304, 6894, 537, 297, 18322, 1799, 756, 4520, 13774, 2217, 8570, 297, 21344, 368, 13, 13129, 29889, 32001, 512, 3153, 29892, 1550, 3031, 979, 18994, 29899, 6707, 322, 13, 3149, 29899, 262, 29899, 2230, 18350, 29899, 7052, 11898, 310, 278, 1737, 12122, 1677, 1144, 310, 18322, 1799, 13, 21570, 29560, 1863, 2651, 2036, 29912, 12443, 10578, 29906, 29900, 29900, 29945, 5062, 11235, 29892, 13260, 333, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 13, 29871, 2594, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29892, 1850, 29882, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29892, 696, 7586, 29906, 29900, 29896, 29946, 7610, 345, 2941, 271, 24541, 29892, 13, 29871, 281, 496, 29879, 29906, 29900, 29906, 29896, 2209, 479, 5275, 1118, 2919, 29899, 7052, 25579, 979, 11898, 310, 278, 1737, 12122, 13, 12574, 310, 18322, 1799, 17737, 29560, 526, 1603, 10225, 292, 29889, 32001, 10506, 263, 4323, 23378, 13, 18609, 2133, 723, 367, 5407, 304, 1871, 1602, 12112, 4475, 304, 5534, 13, 25431, 10907, 2651, 2036, 29912, 2276, 5824, 19982, 29906, 29900, 29900, 29955, 23705, 1338, 29893, 996, 29913, 322, 298, 8491, 16650, 583, 297, 278, 13, 19678, 15483, 313, 1806, 29897, 9999, 29892, 408, 1532, 408, 29126, 2114, 950, 2472, 13, 517, 278, 2553, 1078, 373, 278, 17407, 10879, 322, 5374, 3002, 310, 18322, 1799, 2820, 278, 3186, 29889, 32001, 320, 26956, 29912, 1323, 3224, 29879, 29913, 13, 13, 3047, 445, 664, 591, 29126, 304, 3802, 445, 17261, 491, 7512, 292, 320, 4534, 29912, 1552, 937, 13, 29871, 25579, 979, 6559, 310, 278, 1737, 12122, 3978, 310, 17737, 29560, 304, 970, 775, 13, 29871, 975, 29871, 29945, 29900, 2440, 5003, 32001, 26321, 29892, 591, 3867, 263, 758, 2576, 3821, 1234, 304, 278, 13, 23031, 292, 5925, 1139, 29901, 13, 29905, 463, 29912, 690, 2842, 12470, 29913, 13, 29871, 3645, 607, 3186, 12786, 437, 15717, 310, 970, 368, 3625, 25741, 2041, 515, 13, 29871, 322, 920, 756, 372, 3939, 975, 278, 4940, 29871, 29945, 29900, 2440, 29973, 32001, 320, 1643, 29912, 29878, 29939, 29901, 479, 397, 24974, 29913, 13, 29905, 355, 29912, 690, 2842, 12470, 29913, 13, 4806, 671, 408, 8783, 278, 320, 23066, 29950, 29914, 18871, 2651, 2036, 29912, 29879, 1332, 29875, 4569, 29906, 29900, 29896, 29955, 29913, 322, 27599, 515, 372, 13, 29906, 29889, 29906, 24464, 29905, 13007, 25741, 3190, 2347, 515, 29871, 29896, 29953, 29900, 7284, 29905, 13007, 9279, 322, 4148, 287, 491, 13, 29946, 29941, 7284, 29905, 13007, 15717, 2645, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29896, 931, 3785, 29889, 32001, 1334, 1737, 324, 542, 403, 18777, 304, 13, 29905, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 29892, 773, 408, 18470, 4876, 4234, 775, 2246, 29899, 5563, 21904, 313, 617, 29911, 10249, 29879, 29897, 322, 29871, 13, 8921, 313, 4102, 29914, 4230, 29897, 2983, 9401, 411, 1024, 18822, 2820, 278, 3186, 29892, 322, 17998, 1283, 7224, 29871, 13, 1195, 287, 515, 9063, 15562, 29889, 32001, 1334, 1284, 10757, 310, 278, 4688, 8022, 749, 310, 4644, 6813, 297, 1722, 2752, 13, 20415, 29892, 2678, 8772, 491, 4092, 29889, 32001, 2860, 393, 3785, 29892, 278, 1737, 12122, 6894, 537, 29871, 13, 262, 970, 775, 756, 1063, 21003, 10231, 29889, 32001, 1334, 884, 12439, 8018, 15839, 528, 17741, 13, 12817, 304, 278, 1095, 310, 278, 8291, 6415, 29129, 322, 278, 7910, 310, 14137, 4631, 4135, 297, 13, 23369, 1705, 322, 4275, 14325, 29892, 408, 1532, 408, 310, 2545, 1664, 17292, 28342, 763, 25539, 1608, 322, 13, 25719, 10298, 4822, 10916, 313, 6727, 16783, 29914, 331, 16783, 467, 32001, 320, 26956, 29912, 1469, 20847, 3097, 5003, 32001, 319, 1634, 1414, 3577, 363, 445, 5650, 338, 3625, 515, 796, 264, 8144, 472, 13, 29905, 2271, 29912, 991, 597, 1867, 29875, 29889, 990, 29914, 29896, 29900, 29889, 29945, 29906, 29947, 29896, 29914, 2256, 8144, 29889, 29953, 29941, 29929, 29900, 29941, 29945, 29945, 29913, 2651, 2036, 29912, 3445, 1414, 29899, 5113, 1836, 32001, 320, 2042, 29912, 9662, 630, 5244, 29913, 13, 29905, 1643, 29912, 3471, 29901, 12817, 29913, 13, 13, 29933, 720, 4688, 322, 7786, 1736, 2651, 2036, 29912, 12443, 10578, 29906, 29900, 29900, 29945, 5062, 11235, 29892, 13260, 333, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 13, 29871, 696, 7586, 29906, 29900, 29896, 29946, 7610, 345, 2941, 271, 24541, 29892, 697, 309, 29906, 29900, 29896, 29953, 311, 5365, 550, 332, 6950, 29913, 505, 2931, 1891, 278, 13, 479, 5275, 310, 12362, 29914, 6585, 7562, 18540, 313, 5800, 1799, 29897, 773, 320, 7278, 29912, 6734, 26946, 952, 1118, 13, 4716, 3867, 1880, 29899, 29567, 6089, 541, 526, 9078, 297, 2159, 313, 29906, 29899, 29945, 5940, 29968, 18777, 29897, 13, 392, 508, 367, 4768, 1463, 491, 5221, 424, 23460, 29889, 32001, 512, 29871, 29906, 29900, 29900, 29947, 2261, 801, 2681, 634, 394, 8770, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29913, 18043, 263, 3031, 979, 13, 16961, 29899, 7052, 313, 1454, 278, 931, 29897, 6559, 373, 18322, 1799, 320, 7278, 29912, 479, 5275, 773, 1375, 292, 7047, 13, 29871, 28914, 313, 4345, 29934, 29897, 13698, 1836, 32001, 2688, 29537, 287, 278, 3978, 310, 29871, 29896, 5940, 29924, 17737, 29560, 13, 4746, 278, 7562, 2831, 479, 1404, 2566, 322, 611, 6504, 1051, 3190, 3145, 975, 278, 13, 29896, 29929, 29929, 29929, 489, 29906, 29900, 29900, 29945, 3785, 29892, 773, 408, 18470, 2472, 2788, 304, 1749, 29879, 29901, 4876, 21904, 13, 392, 17998, 1283, 7224, 29889, 32001, 450, 12399, 3785, 313, 29955, 2440, 29897, 297, 2651, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29913, 338, 20511, 1135, 29871, 13, 5816, 338, 12399, 297, 278, 2198, 5650, 313, 29945, 29900, 2440, 29897, 322, 278, 848, 8974, 526, 29871, 13, 27489, 873, 1422, 29936, 411, 393, 297, 3458, 29892, 1749, 2582, 1510, 263, 10029, 7200, 14978, 310, 29871, 13, 15654, 273, 325, 13217, 29940, 2072, 3082, 20706, 29889, 32001, 7280, 29190, 936, 664, 515, 29871, 29906, 29900, 29896, 29900, 491, 14619, 29882, 371, 4099, 29894, 322, 13, 29950, 309, 1372, 2651, 2036, 29912, 29873, 19426, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29913, 4023, 29894, 2868, 1583, 29899, 7099, 433, 1127, 1737, 12122, 13, 2029, 800, 310, 25492, 15303, 8304, 3598, 1494, 1009, 12368, 29892, 13, 15914, 292, 2472, 363, 779, 14850, 4535, 29892, 29955, 29900, 5940, 29968, 25492, 4160, 29889, 32001, 319, 1407, 7786, 13, 1287, 2651, 2036, 29912, 21686, 29879, 29906, 29900, 29906, 29896, 2209, 479, 5275, 29913, 491, 399, 496, 29879, 634, 394, 13217, 5349, 1737, 324, 542, 630, 4203, 263, 7284, 13, 28712, 16046, 4160, 29892, 2534, 26869, 472, 3203, 29871, 29896, 29900, 29900, 25741, 1269, 29892, 322, 1058, 13, 1311, 29899, 7099, 8663, 14354, 373, 1009, 25492, 28723, 29889, 32001, 5806, 278, 6559, 338, 13, 3149, 29899, 262, 29899, 2230, 408, 310, 29871, 29906, 29900, 29906, 29896, 29892, 278, 15717, 7252, 1009, 1284, 886, 13, 351, 475, 303, 2651, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29892, 1850, 29882, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29913, 304, 13, 18609, 675, 278, 14675, 310, 18322, 1799, 1737, 5275, 975, 278, 931, 15101, 845, 1862, 4586, 491, 13, 1552, 2211, 11898, 29889, 32001, 3831, 1965, 411, 3517, 29190, 936, 1736, 29892, 1749, 6559, 338, 1568, 7200, 6287, 5634, 29882, 5555, 13, 7054, 12339, 287, 29871, 29946, 29941, 7284, 29905, 13007, 15717, 310, 29871, 29906, 29889, 29906, 24464, 29905, 13007, 25741, 515, 29871, 29896, 29953, 29900, 7284, 29905, 13007, 13, 16418, 5634, 5426, 11267, 979, 975, 29871, 29945, 29900, 2440, 310, 970, 775, 20706, 3265, 1135, 13, 3149, 297, 931, 29892, 322, 884, 901, 2691, 29899, 3874, 1312, 313, 2541, 1629, 29899, 1609, 29899, 6360, 3803, 1070, 537, 975, 13, 1552, 8900, 3785, 467, 32001, 8108, 1189, 1711, 29892, 1749, 6559, 337, 3687, 373, 10079, 11264, 13, 3924, 313, 8257, 29903, 29897, 9063, 848, 3265, 1135, 7481, 29899, 7099, 433, 1127, 4423, 2472, 29889, 32001, 5901, 1736, 5634, 262, 3153, 278, 664, 491, 8432, 2651, 2036, 29912, 18386, 709, 29906, 29900, 29896, 29941, 2209, 29881, 24974, 29913, 13, 392, 29892, 901, 10325, 29892, 390, 579, 468, 29875, 634, 394, 8770, 2036, 29912, 29878, 579, 468, 29875, 29906, 29900, 29896, 29953, 479, 711, 3173, 29892, 13, 29871, 364, 579, 468, 29875, 29906, 29900, 29896, 29947, 479, 711, 3173, 29892, 544, 1648, 29906, 29900, 29906, 29896, 479, 468, 1581, 29881, 24974, 29913, 5634, 17532, 12399, 1737, 12122, 13, 29905, 7278, 29912, 29881, 24974, 322, 24003, 1118, 474, 29889, 29872, 1696, 278, 15834, 304, 607, 278, 3978, 310, 18322, 1799, 13, 17426, 6602, 1009, 11465, 1230, 14137, 14188, 29889, 32001, 512, 445, 664, 591, 2931, 1891, 1737, 12122, 6894, 537, 297, 970, 775, 363, 278, 937, 13, 2230, 472, 445, 6287, 29892, 1716, 297, 4958, 310, 17737, 29560, 322, 15500, 3785, 29889, 32001, 1334, 437, 13, 1333, 22002, 280, 278, 24003, 10696, 29892, 541, 3867, 29190, 936, 848, 322, 1284, 886, 393, 508, 367, 13, 280, 369, 4063, 304, 393, 1095, 408, 5434, 664, 29889, 32001, 320, 7278, 29912, 12756, 7047, 21639, 29913, 2651, 2036, 29912, 2276, 5824, 19982, 29906, 29900, 29900, 29955, 23705, 1338, 29893, 996, 29913, 338, 278, 13, 1491, 29899, 2671, 310, 7047, 21639, 393, 756, 29537, 287, 278, 18066, 267, 310, 21640, 13, 6734, 24771, 13149, 635, 29892, 3704, 278, 2702, 5932, 310, 920, 304, 5376, 13, 2541, 1737, 12122, 6894, 537, 2651, 2036, 29912, 13376, 303, 456, 29906, 29900, 29900, 29953, 10945, 3359, 29892, 1424, 29440, 29906, 29900, 29896, 29946, 23027, 5933, 1836, 32001, 3826, 3076, 2678, 278, 2198, 6559, 8128, 10757, 393, 508, 367, 1304, 29892, 297, 278, 13, 14940, 1206, 310, 970, 775, 322, 472, 263, 1407, 2919, 6287, 29892, 304, 11539, 607, 13, 14032, 4637, 310, 5534, 7047, 21639, 505, 9820, 484, 15774, 29889, 32001, 320, 2042, 29912, 4062, 3002, 29913, 13, 29905, 1643, 29912, 3471, 29901, 5696, 29913, 13, 13, 13, 29905, 1482, 361, 29905, 361, 29887, 798, 386, 1003, 29871, 320, 29887, 798, 386, 1003, 3009, 13, 29905, 361, 29887, 798, 386, 1003, 13, 29905, 463, 29912, 4532, 29913, 13, 29871, 320, 7313, 29961, 2103, 2013, 4914, 2103, 3199, 6360, 368, 29899, 2055, 1169, 29913, 13, 29871, 320, 6671, 29912, 12883, 368, 970, 25741, 975, 931, 313, 1188, 6287, 467, 13, 29913, 32001, 320, 1643, 29912, 1003, 29901, 29887, 798, 386, 29913, 13, 29905, 355, 29912, 4532, 29913, 13, 29905, 7241, 13, 13, 29905, 26956, 29912, 16390, 24541, 29913, 13, 13, 4806, 27387, 515, 320, 23066, 29950, 29914, 2651, 2036, 29912, 29879, 1332, 29899, 1516, 29878, 29906, 29900, 29896, 29929, 29899, 24713, 29913, 599, 25741, 3190, 2347, 2745, 320, 25832, 1964, 579, 1523, 2415, 2539, 6294, 32001, 2688, 5253, 304, 320, 14573, 5261, 1169, 22131, 29914, 25741, 29892, 5412, 491, 317, 15715, 29896, 15882, 29892, 4023, 29894, 2868, 515, 320, 25832, 1299, 7288, 5261, 1169, 797, 7068, 29914, 970, 9279, 6421, 515, 4655, 5849, 363, 2710, 313, 28712, 16046, 29892, 11786, 28632, 29892, 2992, 1846, 32001, 322, 3577, 28914, 313, 10251, 713, 29892, 10772, 2227, 29892, 405, 13427, 29892, 2992, 6250, 32001, 1876, 1169, 297, 278, 8783, 526, 491, 320, 25832, 6344, 2806, 943, 22131, 29914, 15717, 29892, 5412, 491, 779, 6990, 29938, 978, 29892, 4876, 4535, 5854, 29938, 11000, 29889, 32001, 450, 8783, 2996, 408, 1023, 1104, 1288, 6131, 29892, 697, 363, 25741, 322, 697, 363, 15717, 29892, 411, 278, 4642, 29371, 278, 7480, 3025, 263, 9117, 1820, 29889, 32001, 320, 361, 5426, 13, 9760, 1948, 297, 278, 9063, 1591, 3743, 278, 1494, 4235, 29901, 9063, 317, 15715, 29896, 15882, 29892, 4148, 322, 844, 5171, 5335, 342, 15092, 29892, 4148, 322, 844, 5171, 2893, 14903, 313, 20275, 16750, 278, 4148, 1591, 467, 32001, 450, 21578, 1546, 9063, 15717, 322, 9063, 2153, 2041, 515, 11786, 29892, 607, 6511, 304, 9063, 263, 1735, 4148, 287, 491, 4856, 1683, 29889, 32001, 1152, 445, 6559, 591, 21309, 373, 15717, 322, 17262, 9063, 2153, 29892, 408, 278, 4328, 1546, 278, 1023, 338, 451, 8018, 363, 1749, 5925, 5155, 322, 278, 5253, 310, 25741, 411, 263, 844, 5171, 916, 1135, 967, 4148, 338, 3480, 3473, 1821, 29889, 32001, 320, 7241, 13, 2831, 1269, 6251, 297, 278, 4148, 1591, 591, 505, 4148, 2989, 1024, 322, 4876, 408, 1023, 5004, 6031, 310, 10650, 6262, 29889, 32001, 1334, 6206, 2411, 15273, 1821, 470, 18325, 519, 2983, 393, 29901, 526, 451, 1602, 397, 519, 408, 18351, 29899, 29947, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12283, 7099, 397, 519, 29914, 4148, 2983, 6206, 511, 526, 4876, 14157, 2012, 310, 2983, 3441, 25832, 6344, 2806, 943, 29934, 29885, 9823, 29914, 4954, 7039, 4907, 511, 5718, 310, 871, 9654, 4890, 3441, 25832, 6344, 2806, 943, 29934, 29885, 10358, 804, 23201, 1712, 901, 1135, 29871, 29896, 29900, 8958, 1661, 29899, 1026, 2153, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12283, 15670, 23201, 526, 5520, 1135, 29871, 29896, 29900, 29900, 4890, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12229, 549, 12495, 32001, 2860, 21166, 29892, 1048, 320, 25832, 6344, 2806, 943, 29925, 15273, 1821, 2052, 307, 29916, 29914, 15717, 3441, 25832, 6344, 2806, 943, 29925, 15273, 1821, 29925, 312, 29914, 310, 278, 2847, 8783, 29897, 9488, 363, 4340, 7418, 29889, 32001, 3940, 393, 278, 5253, 310, 970, 775, 25741, 313, 392, 15717, 29897, 11122, 297, 278, 13, 11228, 8783, 25088, 18709, 9247, 975, 13, 2230, 2651, 2036, 29912, 29879, 1332, 29899, 771, 854, 749, 29899, 331, 344, 1012, 361, 29887, 798, 386, 1003, 29892, 408, 4318, 363, 25741, 297, 13, 29905, 29907, 999, 29912, 1003, 29901, 29887, 798, 386, 1012, 2870, 29901, 515, 395, 29896, 29900, 29985, 29946, 29938, 25741, 297, 29871, 29896, 29929, 29955, 29896, 29892, 304, 395, 29896, 29900, 29985, 29953, 29938, 297, 29871, 29896, 29929, 29929, 29947, 29892, 304, 13, 284, 3242, 395, 29896, 29900, 29985, 29929, 29938, 297, 29871, 29906, 29900, 29906, 29900, 29905, 7241, 29889, 32001, 1094, 263, 17004, 278, 8900, 534, 1975, 10331, 304, 367, 901, 13, 13844, 297, 7786, 1602, 3076, 1135, 297, 29871, 29946, 29900, 29974, 1629, 29899, 1025, 6743, 29892, 2861, 304, 13964, 4586, 373, 13, 735, 1112, 9247, 7200, 23093, 29889, 32001, 320, 26956, 29912, 7999, 324, 10610, 29913, 13, 13, 29905, 463, 29912, 4532, 29913, 13, 29871, 320, 9525, 13, 29871, 320, 7313, 29961, 24049, 29892, 15450, 29922, 29953, 4912, 29871, 29953, 4912, 29871, 29900, 29871, 29900, 29892, 2103, 2013, 16292, 3199, 1491, 1727, 1080, 29899, 2470, 29913, 13, 29871, 320, 6671, 29912, 1576, 320, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 1304, 408, 1737, 324, 10610, 22525, 5003, 32001, 320, 1643, 29912, 1003, 29901, 11526, 1958, 29913, 13, 29905, 355, 29912, 4532, 29913, 13, 13, 2887, 1737, 324, 10610, 22525, 591, 671, 11758, 3186, 12786, 10723, 515, 278, 3303, 18269, 1737, 359, 305, 2004, 2651, 2036, 29912, 348, 29896, 29929, 29929, 29929, 479, 359, 305, 2004, 1836, 32001, 1763, 4772, 2432, 3381, 491, 2919, 10916, 313, 29872, 29889, 29887, 1696, 7551, 470, 12710, 29897, 2629, 11758, 12786, 29892, 591, 19412, 322, 6219, 777, 12786, 2729, 373, 1737, 12122, 23203, 537, 322, 278, 19383, 310, 758, 331, 8946, 16375, 29769, 5680, 29892, 1316, 408, 19182, 4086, 29889, 32001, 320, 29907, 999, 29912, 1003, 29901, 11526, 1958, 29913, 3697, 278, 2186, 1051, 310, 320, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 1304, 408, 1737, 324, 10610, 22525, 297, 445, 6559, 29889, 32001, 1879, 324, 10610, 310, 9063, 15717, 304, 3186, 12786, 3913, 278, 1023, 19595, 653, 13698, 9129, 297, 2651, 2036, 29912, 293, 344, 29899, 344, 275, 29899, 29906, 29900, 29906, 29906, 29899, 26098, 1118, 23359, 337, 13998, 2400, 29889, 32001, 450, 937, 697, 337, 3687, 373, 278, 4234, 775, 2246, 29899, 5563, 5354, 313, 617, 29911, 10249, 29897, 310, 4876, 14157, 23892, 515, 9063, 15562, 29892, 321, 29889, 29887, 1696, 320, 16266, 26139, 1341, 1118, 320, 16266, 26139, 582, 1118, 320, 16266, 26139, 18038, 1118, 2992, 29889, 32001, 1334, 4687, 515, 278, 306, 2190, 29909, 1051, 310, 13548, 2931, 21759, 29911, 10249, 29879, 2651, 2036, 29912, 6011, 29899, 29883, 312, 430, 29913, 322, 7522, 20545, 1269, 6590, 20123, 304, 263, 3646, 3186, 5120, 29889, 32001, 450, 1473, 1737, 324, 10610, 11043, 3913, 278, 17998, 9210, 310, 9063, 5335, 342, 15092, 313, 29872, 29889, 29887, 1696, 17998, 29899, 29900, 29945, 29901, 29900, 29900, 29897, 322, 4148, 2983, 304, 8161, 278, 1556, 5517, 3186, 5120, 310, 278, 9063, 4148, 29889, 32001, 1152, 1269, 17998, 9210, 591, 8161, 263, 1051, 310, 15878, 7600, 313, 13509, 29892, 2106, 29892, 470, 14278, 20123, 29897, 297, 278, 3186, 393, 29892, 472, 278, 931, 310, 393, 9063, 29892, 750, 393, 17998, 9210, 29936, 9063, 931, 338, 1820, 1244, 29892, 408, 4234, 17998, 1283, 7224, 13100, 975, 931, 2861, 304, 29431, 3620, 29889, 32001, 1763, 1207, 445, 3683, 3381, 591, 671, 278, 306, 2190, 29909, 931, 10640, 2566, 2651, 2036, 29912, 17559, 1272, 1836, 32001, 1987, 591, 3566, 304, 1269, 2058, 263, 8158, 393, 4332, 1973, 278, 4188, 22342, 393, 263, 2183, 4148, 1024, 338, 17443, 310, 372, 29889, 32001, 1763, 445, 1095, 591, 671, 278, 28297, 29890, 15451, 8783, 310, 278, 29511, 310, 278, 1556, 3619, 937, 322, 3942, 2983, 607, 29892, 439, 11427, 515, 2651, 2036, 29912, 1079, 29890, 799, 29899, 7039, 6177, 2802, 277, 12181, 4954, 16123, 2247, 278, 26368, 5528, 5084, 310, 363, 264, 1280, 322, 269, 595, 1280, 7371, 515, 263, 2566, 310, 320, 1949, 29912, 29946, 29871, 29900, 29946, 29946, 29871, 29945, 29946, 29953, 29871, 29929, 29941, 29947, 29913, 2305, 313, 29945, 29945, 29889, 29945, 8958, 310, 8471, 2305, 297, 29871, 29906, 29900, 29896, 29946, 467, 32001, 1094, 310, 3839, 29871, 29906, 29900, 29896, 29929, 372, 18469, 320, 1949, 29912, 29906, 29955, 29871, 29953, 29953, 29906, 29871, 29947, 29900, 29896, 29913, 363, 264, 1280, 322, 320, 1949, 29912, 29906, 29955, 29871, 29906, 29900, 29953, 29871, 29947, 29906, 29896, 29913, 269, 595, 1280, 297, 29871, 29906, 29941, 29953, 24894, 8977, 1080, 29889, 4907, 29913, 32001, 1094, 297, 1749, 8783, 15717, 526, 2989, 1024, 6031, 313, 29878, 1624, 1135, 6219, 491, 937, 29914, 11922, 1024, 511, 591, 937, 5993, 675, 2983, 313, 1609, 1999, 1331, 322, 1206, 3620, 29897, 322, 769, 16280, 5375, 18897, 297, 1716, 937, 322, 3942, 2983, 10868, 8857, 29889, 32001, 1152, 1269, 1543, 1476, 297, 1024, 8857, 591, 22932, 278, 2058, 4665, 29905, 6661, 1333, 24212, 8875, 491, 278, 1024, 10868, 304, 4017, 263, 5645, 393, 338, 29839, 304, 278, 1353, 310, 12407, 24638, 393, 1024, 313, 6979, 29897, 297, 278, 2702, 2058, 29889, 32001, 320, 6661, 1333, 300, 1062, 29912, 1762, 4017, 4665, 2025, 1338, 5634, 294, 278, 17837, 310, 4954, 6689, 4907, 338, 25745, 23724, 29901, 2989, 10916, 325, 13217, 29879, 29399, 310, 2919, 10916, 805, 9450, 2999, 931, 29920, 2873, 5634, 705, 671, 263, 29544, 310, 7601, 8974, 313, 29872, 29889, 29887, 1696, 5874, 28007, 511, 322, 1661, 29899, 16072, 6743, 313, 29872, 29889, 29887, 1696, 14109, 7456, 467, 29913, 32001, 1334, 2533, 445, 4377, 363, 599, 3161, 304, 4017, 263, 2058, 8158, 29892, 17140, 701, 411, 263, 1051, 310, 779, 6990, 29938, 6689, 29892, 8158, 4535, 5854, 29938, 11000, 29889, 32001, 1334, 769, 8877, 445, 1051, 491, 278, 3186, 5120, 393, 263, 2058, 14393, 304, 322, 2533, 278, 8158, 363, 599, 278, 7600, 297, 1269, 5120, 304, 4017, 385, 12463, 8158, 29892, 6590, 304, 278, 4188, 22342, 393, 278, 9063, 14393, 304, 263, 2183, 3186, 5120, 29889, 32001, 1334, 3566, 278, 6257, 9063, 408, 6421, 515, 278, 3186, 5120, 411, 278, 9939, 8158, 29889, 32001, 450, 4876, 29899, 6707, 11043, 9378, 414, 515, 278, 9078, 322, 443, 5521, 8362, 671, 310, 21759, 29911, 10249, 29879, 29901, 1556, 18777, 671, 10035, 323, 10249, 29879, 1316, 408, 320, 16266, 26139, 510, 1118, 320, 16266, 26139, 990, 1118, 470, 320, 16266, 26139, 1212, 1836, 32001, 12808, 445, 947, 451, 3799, 26018, 4822, 20542, 29901, 3148, 29899, 6707, 18777, 29892, 363, 1342, 29892, 671, 278, 320, 16266, 26139, 375, 29913, 21759, 29911, 10249, 1568, 901, 269, 2495, 290, 368, 1135, 1009, 7824, 6795, 20895, 29889, 32001, 1551, 278, 916, 1361, 278, 9210, 29914, 978, 29899, 6707, 11043, 337, 3687, 373, 278, 17998, 9210, 310, 278, 9063, 5335, 342, 15092, 29889, 32001, 16809, 304, 5780, 22920, 373, 13897, 731, 14340, 29892, 263, 2919, 1353, 310, 25741, 297, 278, 8783, 756, 385, 17998, 9210, 5186, 304, 5225, 29889, 32001, 910, 6602, 29879, 3109, 7786, 25741, 3441, 14573, 5261, 1169, 29911, 29999, 29999, 13985, 1349, 681, 392, 27418, 6478, 29914, 310, 29871, 29906, 29900, 29906, 29900, 29879, 25741, 505, 263, 5225, 9210, 29897, 1135, 9642, 6743, 3441, 14573, 5261, 1169, 29911, 29999, 29999, 13985, 1349, 681, 392, 29914, 297, 29871, 29906, 29900, 29900, 29900, 467, 32001, 1094, 263, 1121, 278, 9210, 29914, 978, 29899, 6707, 11043, 1033, 1095, 701, 6459, 292, 263, 2919, 6232, 310, 9642, 25741, 408, 4148, 287, 491, 11715, 18777, 29892, 322, 304, 263, 3109, 261, 15834, 4092, 550, 29889, 32001, 1763, 6795, 1438, 5626, 591, 14405, 278, 1023, 1737, 324, 10610, 13698, 4208, 491, 15399, 278, 9210, 29914, 978, 29899, 6707, 13698, 304, 599, 25741, 411, 263, 1661, 29899, 9171, 17998, 9210, 29892, 322, 278, 4876, 29899, 6707, 373, 304, 599, 916, 25741, 29889, 32001, 320, 2042, 29912, 12191, 322, 8565, 1558, 291, 29913, 13, 29905, 1643, 29912, 3471, 29901, 9902, 29913, 13, 13, 29905, 463, 29912, 4532, 4044, 13, 29871, 320, 9525, 13, 29871, 320, 7313, 29961, 2103, 2013, 16292, 3199, 1429, 287, 29889, 5140, 29913, 13, 29871, 320, 6671, 29912, 29934, 20819, 310, 25741, 313, 27215, 29897, 322, 6136, 15717, 313, 22503, 29897, 491, 3186, 10640, 975, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29900, 3785, 5003, 32001, 320, 9868, 29961, 14732, 3199, 7264, 287, 2594, 8727, 6445, 278, 3186, 10640, 364, 2219, 359, 363, 25741, 322, 15717, 975, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29900, 3785, 5003, 32001, 320, 1643, 29912, 1003, 29901, 9902, 29913, 13, 29905, 355, 29912, 4532, 4044, 13, 13, 13, 29871, 13, 1762, 1234, 320, 25029, 29912, 29878, 29939, 29901, 479, 397, 24974, 29913, 591, 22229, 278, 1353, 310, 25741, 322, 8359, 15717, 639, 1629, 322, 639, 3186, 10640, 29889, 32001, 1334, 2198, 278, 7625, 2582, 297, 320, 29907, 999, 29912, 1003, 29901, 9902, 29913, 408, 1023, 5096, 287, 2594, 24469, 29892, 6445, 1629, 368, 2867, 3204, 29879, 363, 25741, 322, 15717, 8307, 29889, 32001, 7569, 2594, 11524, 263, 1629, 322, 338, 8877, 287, 297, 269, 29399, 6445, 278, 9063, 29914, 8921, 11959, 363, 1269, 310, 278, 3186, 12786, 310, 320, 29907, 999, 29912, 1003, 29901, 11526, 1958, 29913, 297, 393, 1629, 29889, 32001, 1763, 4772, 714, 27801, 2861, 304, 805, 272, 26538, 17737, 29560, 29892, 297, 278, 4148, 8727, 591, 871, 2050, 15717, 2534, 26869, 472, 3203, 29871, 29945, 25741, 297, 263, 2183, 1629, 29889, 32001, 5806, 5366, 1747, 534, 1975, 297, 278, 24469, 6456, 393, 278, 3001, 3694, 310, 25741, 322, 15717, 6548, 18709, 9247, 975, 931, 29889, 32001, 10133, 363, 278, 937, 2440, 297, 278, 24469, 29892, 278, 1353, 310, 848, 3291, 297, 777, 3186, 12786, 508, 367, 14154, 2319, 29892, 411, 8178, 27721, 373, 278, 25806, 310, 534, 1975, 29889, 32001, 320, 26956, 29912, 7999, 12122, 6894, 537, 975, 931, 29913, 13, 13, 3563, 497, 29892, 278, 2498, 534, 355, 5692, 304, 367, 393, 278, 320, 4534, 29912, 479, 12122, 6894, 537, 297, 970, 775, 338, 10231, 6177, 4644, 6813, 322, 4092, 5136, 630, 1009, 4954, 24130, 749, 4907, 2745, 278, 7256, 310, 278, 29871, 29929, 29900]} {'input_ids': [1, 320, 2042, 29912, 25898, 29913, 13, 29905, 1643, 29912, 3471, 29901, 23333, 29913, 13, 13, 29905, 7278, 29912, 29954, 1581, 6894, 537, 1118, 470, 901, 4049, 967, 10225, 727, 974, 29892, 4249, 27138, 304, 13, 20415, 5849, 14188, 756, 1063, 26606, 12399, 297, 7786, 2440, 29889, 32001, 512, 13, 1595, 16311, 29892, 278, 10122, 310, 29892, 9545, 310, 29892, 322, 6795, 1004, 25414, 363, 320, 7278, 29912, 26098, 13, 29871, 24003, 29913, 297, 12362, 29914, 6585, 7562, 18540, 313, 5800, 1799, 29897, 505, 4520, 263, 3287, 310, 8570, 13, 957, 278, 4940, 316, 6332, 2651, 2036, 29912, 29881, 16093, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 3855, 5871, 29906, 29900, 29896, 29900, 29895, 311, 29893, 2770, 29892, 13, 29871, 1055, 29888, 375, 29906, 29900, 29896, 29906, 5041, 267, 29892, 413, 434, 305, 1358, 29906, 29900, 29896, 29906, 26098, 29888, 2209, 29892, 19723, 5475, 4979, 29906, 29900, 29896, 29946, 26098, 29892, 13, 29871, 697, 309, 29906, 29900, 29896, 29953, 311, 5365, 550, 332, 6950, 29892, 696, 7586, 29906, 29900, 29896, 29953, 29893, 2770, 7192, 2209, 29892, 1935, 15044, 29906, 29900, 29896, 29955, 26098, 29892, 13, 29871, 503, 562, 305, 3350, 492, 29906, 29900, 29906, 29896, 26098, 1836, 32001, 320, 7278, 29912, 7999, 12122, 6894, 537, 29913, 338, 373, 278, 916, 1361, 278, 13, 14380, 310, 6894, 537, 393, 380, 1567, 515, 27138, 297, 777, 5534, 6354, 6421, 13, 3166, 1422, 3186, 12786, 322, 4185, 1973, 29889, 32001, 26304, 6894, 537, 297, 18322, 1799, 756, 4520, 13774, 2217, 8570, 297, 21344, 368, 13, 13129, 29889, 32001, 512, 3153, 29892, 1550, 3031, 979, 18994, 29899, 6707, 322, 13, 3149, 29899, 262, 29899, 2230, 18350, 29899, 7052, 11898, 310, 278, 1737, 12122, 1677, 1144, 310, 18322, 1799, 13, 21570, 29560, 1863, 2651, 2036, 29912, 12443, 10578, 29906, 29900, 29900, 29945, 5062, 11235, 29892, 13260, 333, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 13, 29871, 2594, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29892, 1850, 29882, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29892, 696, 7586, 29906, 29900, 29896, 29946, 7610, 345, 2941, 271, 24541, 29892, 13, 29871, 281, 496, 29879, 29906, 29900, 29906, 29896, 2209, 479, 5275, 1118, 2919, 29899, 7052, 25579, 979, 11898, 310, 278, 1737, 12122, 13, 12574, 310, 18322, 1799, 17737, 29560, 526, 1603, 10225, 292, 29889, 32001, 10506, 263, 4323, 23378, 13, 18609, 2133, 723, 367, 5407, 304, 1871, 1602, 12112, 4475, 304, 5534, 13, 25431, 10907, 2651, 2036, 29912, 2276, 5824, 19982, 29906, 29900, 29900, 29955, 23705, 1338, 29893, 996, 29913, 322, 298, 8491, 16650, 583, 297, 278, 13, 19678, 15483, 313, 1806, 29897, 9999, 29892, 408, 1532, 408, 29126, 2114, 950, 2472, 13, 517, 278, 2553, 1078, 373, 278, 17407, 10879, 322, 5374, 3002, 310, 18322, 1799, 2820, 278, 3186, 29889, 32001, 320, 26956, 29912, 1323, 3224, 29879, 29913, 13, 13, 3047, 445, 664, 591, 29126, 304, 3802, 445, 17261, 491, 7512, 292, 320, 4534, 29912, 1552, 937, 13, 29871, 25579, 979, 6559, 310, 278, 1737, 12122, 3978, 310, 17737, 29560, 304, 970, 775, 13, 29871, 975, 29871, 29945, 29900, 2440, 5003, 32001, 26321, 29892, 591, 3867, 263, 758, 2576, 3821, 1234, 304, 278, 13, 23031, 292, 5925, 1139, 29901, 13, 29905, 463, 29912, 690, 2842, 12470, 29913, 13, 29871, 3645, 607, 3186, 12786, 437, 15717, 310, 970, 368, 3625, 25741, 2041, 515, 13, 29871, 322, 920, 756, 372, 3939, 975, 278, 4940, 29871, 29945, 29900, 2440, 29973, 32001, 320, 1643, 29912, 29878, 29939, 29901, 479, 397, 24974, 29913, 13, 29905, 355, 29912, 690, 2842, 12470, 29913, 13, 4806, 671, 408, 8783, 278, 320, 23066, 29950, 29914, 18871, 2651, 2036, 29912, 29879, 1332, 29875, 4569, 29906, 29900, 29896, 29955, 29913, 322, 27599, 515, 372, 13, 29906, 29889, 29906, 24464, 29905, 13007, 25741, 3190, 2347, 515, 29871, 29896, 29953, 29900, 7284, 29905, 13007, 9279, 322, 4148, 287, 491, 13, 29946, 29941, 7284, 29905, 13007, 15717, 2645, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29896, 931, 3785, 29889, 32001, 1334, 1737, 324, 542, 403, 18777, 304, 13, 29905, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 29892, 773, 408, 18470, 4876, 4234, 775, 2246, 29899, 5563, 21904, 313, 617, 29911, 10249, 29879, 29897, 322, 29871, 13, 8921, 313, 4102, 29914, 4230, 29897, 2983, 9401, 411, 1024, 18822, 2820, 278, 3186, 29892, 322, 17998, 1283, 7224, 29871, 13, 1195, 287, 515, 9063, 15562, 29889, 32001, 1334, 1284, 10757, 310, 278, 4688, 8022, 749, 310, 4644, 6813, 297, 1722, 2752, 13, 20415, 29892, 2678, 8772, 491, 4092, 29889, 32001, 2860, 393, 3785, 29892, 278, 1737, 12122, 6894, 537, 29871, 13, 262, 970, 775, 756, 1063, 21003, 10231, 29889, 32001, 1334, 884, 12439, 8018, 15839, 528, 17741, 13, 12817, 304, 278, 1095, 310, 278, 8291, 6415, 29129, 322, 278, 7910, 310, 14137, 4631, 4135, 297, 13, 23369, 1705, 322, 4275, 14325, 29892, 408, 1532, 408, 310, 2545, 1664, 17292, 28342, 763, 25539, 1608, 322, 13, 25719, 10298, 4822, 10916, 313, 6727, 16783, 29914, 331, 16783, 467, 32001, 320, 26956, 29912, 1469, 20847, 3097, 5003, 32001, 319, 1634, 1414, 3577, 363, 445, 5650, 338, 3625, 515, 796, 264, 8144, 472, 13, 29905, 2271, 29912, 991, 597, 1867, 29875, 29889, 990, 29914, 29896, 29900, 29889, 29945, 29906, 29947, 29896, 29914, 2256, 8144, 29889, 29953, 29941, 29929, 29900, 29941, 29945, 29945, 29913, 2651, 2036, 29912, 3445, 1414, 29899, 5113, 1836, 32001, 320, 2042, 29912, 9662, 630, 5244, 29913, 13, 29905, 1643, 29912, 3471, 29901, 12817, 29913, 13, 13, 29933, 720, 4688, 322, 7786, 1736, 2651, 2036, 29912, 12443, 10578, 29906, 29900, 29900, 29945, 5062, 11235, 29892, 13260, 333, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 13, 29871, 696, 7586, 29906, 29900, 29896, 29946, 7610, 345, 2941, 271, 24541, 29892, 697, 309, 29906, 29900, 29896, 29953, 311, 5365, 550, 332, 6950, 29913, 505, 2931, 1891, 278, 13, 479, 5275, 310, 12362, 29914, 6585, 7562, 18540, 313, 5800, 1799, 29897, 773, 320, 7278, 29912, 6734, 26946, 952, 1118, 13, 4716, 3867, 1880, 29899, 29567, 6089, 541, 526, 9078, 297, 2159, 313, 29906, 29899, 29945, 5940, 29968, 18777, 29897, 13, 392, 508, 367, 4768, 1463, 491, 5221, 424, 23460, 29889, 32001, 512, 29871, 29906, 29900, 29900, 29947, 2261, 801, 2681, 634, 394, 8770, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29913, 18043, 263, 3031, 979, 13, 16961, 29899, 7052, 313, 1454, 278, 931, 29897, 6559, 373, 18322, 1799, 320, 7278, 29912, 479, 5275, 773, 1375, 292, 7047, 13, 29871, 28914, 313, 4345, 29934, 29897, 13698, 1836, 32001, 2688, 29537, 287, 278, 3978, 310, 29871, 29896, 5940, 29924, 17737, 29560, 13, 4746, 278, 7562, 2831, 479, 1404, 2566, 322, 611, 6504, 1051, 3190, 3145, 975, 278, 13, 29896, 29929, 29929, 29929, 489, 29906, 29900, 29900, 29945, 3785, 29892, 773, 408, 18470, 2472, 2788, 304, 1749, 29879, 29901, 4876, 21904, 13, 392, 17998, 1283, 7224, 29889, 32001, 450, 12399, 3785, 313, 29955, 2440, 29897, 297, 2651, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29913, 338, 20511, 1135, 29871, 13, 5816, 338, 12399, 297, 278, 2198, 5650, 313, 29945, 29900, 2440, 29897, 322, 278, 848, 8974, 526, 29871, 13, 27489, 873, 1422, 29936, 411, 393, 297, 3458, 29892, 1749, 2582, 1510, 263, 10029, 7200, 14978, 310, 29871, 13, 15654, 273, 325, 13217, 29940, 2072, 3082, 20706, 29889, 32001, 7280, 29190, 936, 664, 515, 29871, 29906, 29900, 29896, 29900, 491, 14619, 29882, 371, 4099, 29894, 322, 13, 29950, 309, 1372, 2651, 2036, 29912, 29873, 19426, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29913, 4023, 29894, 2868, 1583, 29899, 7099, 433, 1127, 1737, 12122, 13, 2029, 800, 310, 25492, 15303, 8304, 3598, 1494, 1009, 12368, 29892, 13, 15914, 292, 2472, 363, 779, 14850, 4535, 29892, 29955, 29900, 5940, 29968, 25492, 4160, 29889, 32001, 319, 1407, 7786, 13, 1287, 2651, 2036, 29912, 21686, 29879, 29906, 29900, 29906, 29896, 2209, 479, 5275, 29913, 491, 399, 496, 29879, 634, 394, 13217, 5349, 1737, 324, 542, 630, 4203, 263, 7284, 13, 28712, 16046, 4160, 29892, 2534, 26869, 472, 3203, 29871, 29896, 29900, 29900, 25741, 1269, 29892, 322, 1058, 13, 1311, 29899, 7099, 8663, 14354, 373, 1009, 25492, 28723, 29889, 32001, 5806, 278, 6559, 338, 13, 3149, 29899, 262, 29899, 2230, 408, 310, 29871, 29906, 29900, 29906, 29896, 29892, 278, 15717, 7252, 1009, 1284, 886, 13, 351, 475, 303, 2651, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29892, 1850, 29882, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29913, 304, 13, 18609, 675, 278, 14675, 310, 18322, 1799, 1737, 5275, 975, 278, 931, 15101, 845, 1862, 4586, 491, 13, 1552, 2211, 11898, 29889, 32001, 3831, 1965, 411, 3517, 29190, 936, 1736, 29892, 1749, 6559, 338, 1568, 7200, 6287, 5634, 29882, 5555, 13, 7054, 12339, 287, 29871, 29946, 29941, 7284, 29905, 13007, 15717, 310, 29871, 29906, 29889, 29906, 24464, 29905, 13007, 25741, 515, 29871, 29896, 29953, 29900, 7284, 29905, 13007, 13, 16418, 5634, 5426, 11267, 979, 975, 29871, 29945, 29900, 2440, 310, 970, 775, 20706, 3265, 1135, 13, 3149, 297, 931, 29892, 322, 884, 901, 2691, 29899, 3874, 1312, 313, 2541, 1629, 29899, 1609, 29899, 6360, 3803, 1070, 537, 975, 13, 1552, 8900, 3785, 467, 32001, 8108, 1189, 1711, 29892, 1749, 6559, 337, 3687, 373, 10079, 11264, 13, 3924, 313, 8257, 29903, 29897, 9063, 848, 3265, 1135, 7481, 29899, 7099, 433, 1127, 4423, 2472, 29889, 32001, 5901, 1736, 5634, 262, 3153, 278, 664, 491, 8432, 2651, 2036, 29912, 18386, 709, 29906, 29900, 29896, 29941, 2209, 29881, 24974, 29913, 13, 392, 29892, 901, 10325, 29892, 390, 579, 468, 29875, 634, 394, 8770, 2036, 29912, 29878, 579, 468, 29875, 29906, 29900, 29896, 29953, 479, 711, 3173, 29892, 13, 29871, 364, 579, 468, 29875, 29906, 29900, 29896, 29947, 479, 711, 3173, 29892, 544, 1648, 29906, 29900, 29906, 29896, 479, 468, 1581, 29881, 24974, 29913, 5634, 17532, 12399, 1737, 12122, 13, 29905, 7278, 29912, 29881, 24974, 322, 24003, 1118, 474, 29889, 29872, 1696, 278, 15834, 304, 607, 278, 3978, 310, 18322, 1799, 13, 17426, 6602, 1009, 11465, 1230, 14137, 14188, 29889, 32001, 512, 445, 664, 591, 2931, 1891, 1737, 12122, 6894, 537, 297, 970, 775, 363, 278, 937, 13, 2230, 472, 445, 6287, 29892, 1716, 297, 4958, 310, 17737, 29560, 322, 15500, 3785, 29889, 32001, 1334, 437, 13, 1333, 22002, 280, 278, 24003, 10696, 29892, 541, 3867, 29190, 936, 848, 322, 1284, 886, 393, 508, 367, 13, 280, 369, 4063, 304, 393, 1095, 408, 5434, 664, 29889, 32001, 320, 7278, 29912, 12756, 7047, 21639, 29913, 2651, 2036, 29912, 2276, 5824, 19982, 29906, 29900, 29900, 29955, 23705, 1338, 29893, 996, 29913, 338, 278, 13, 1491, 29899, 2671, 310, 7047, 21639, 393, 756, 29537, 287, 278, 18066, 267, 310, 21640, 13, 6734, 24771, 13149, 635, 29892, 3704, 278, 2702, 5932, 310, 920, 304, 5376, 13, 2541, 1737, 12122, 6894, 537, 2651, 2036, 29912, 13376, 303, 456, 29906, 29900, 29900, 29953, 10945, 3359, 29892, 1424, 29440, 29906, 29900, 29896, 29946, 23027, 5933, 1836, 32001, 3826, 3076, 2678, 278, 2198, 6559, 8128, 10757, 393, 508, 367, 1304, 29892, 297, 278, 13, 14940, 1206, 310, 970, 775, 322, 472, 263, 1407, 2919, 6287, 29892, 304, 11539, 607, 13, 14032, 4637, 310, 5534, 7047, 21639, 505, 9820, 484, 15774, 29889, 32001, 320, 2042, 29912, 4062, 3002, 29913, 13, 29905, 1643, 29912, 3471, 29901, 5696, 29913, 13, 13, 13, 29905, 1482, 361, 29905, 361, 29887, 798, 386, 1003, 29871, 320, 29887, 798, 386, 1003, 3009, 13, 29905, 361, 29887, 798, 386, 1003, 13, 29905, 463, 29912, 4532, 29913, 13, 29871, 320, 7313, 29961, 2103, 2013, 4914, 2103, 3199, 6360, 368, 29899, 2055, 1169, 29913, 13, 29871, 320, 6671, 29912, 12883, 368, 970, 25741, 975, 931, 313, 1188, 6287, 467, 13, 29913, 32001, 320, 1643, 29912, 1003, 29901, 29887, 798, 386, 29913, 13, 29905, 355, 29912, 4532, 29913, 13, 29905, 7241, 13, 13, 29905, 26956, 29912, 16390, 24541, 29913, 13, 13, 4806, 27387, 515, 320, 23066, 29950, 29914, 2651, 2036, 29912, 29879, 1332, 29899, 1516, 29878, 29906, 29900, 29896, 29929, 29899, 24713, 29913, 599, 25741, 3190, 2347, 2745, 320, 25832, 1964, 579, 1523, 2415, 2539, 6294, 32001, 2688, 5253, 304, 320, 14573, 5261, 1169, 22131, 29914, 25741, 29892, 5412, 491, 317, 15715, 29896, 15882, 29892, 4023, 29894, 2868, 515, 320, 25832, 1299, 7288, 5261, 1169, 797, 7068, 29914, 970, 9279, 6421, 515, 4655, 5849, 363, 2710, 313, 28712, 16046, 29892, 11786, 28632, 29892, 2992, 1846, 32001, 322, 3577, 28914, 313, 10251, 713, 29892, 10772, 2227, 29892, 405, 13427, 29892, 2992, 6250, 32001, 1876, 1169, 297, 278, 8783, 526, 491, 320, 25832, 6344, 2806, 943, 22131, 29914, 15717, 29892, 5412, 491, 779, 6990, 29938, 978, 29892, 4876, 4535, 5854, 29938, 11000, 29889, 32001, 450, 8783, 2996, 408, 1023, 1104, 1288, 6131, 29892, 697, 363, 25741, 322, 697, 363, 15717, 29892, 411, 278, 4642, 29371, 278, 7480, 3025, 263, 9117, 1820, 29889, 32001, 320, 361, 5426, 13, 9760, 1948, 297, 278, 9063, 1591, 3743, 278, 1494, 4235, 29901, 9063, 317, 15715, 29896, 15882, 29892, 4148, 322, 844, 5171, 5335, 342, 15092, 29892, 4148, 322, 844, 5171, 2893, 14903, 313, 20275, 16750, 278, 4148, 1591, 467, 32001, 450, 21578, 1546, 9063, 15717, 322, 9063, 2153, 2041, 515, 11786, 29892, 607, 6511, 304, 9063, 263, 1735, 4148, 287, 491, 4856, 1683, 29889, 32001, 1152, 445, 6559, 591, 21309, 373, 15717, 322, 17262, 9063, 2153, 29892, 408, 278, 4328, 1546, 278, 1023, 338, 451, 8018, 363, 1749, 5925, 5155, 322, 278, 5253, 310, 25741, 411, 263, 844, 5171, 916, 1135, 967, 4148, 338, 3480, 3473, 1821, 29889, 32001, 320, 7241, 13, 2831, 1269, 6251, 297, 278, 4148, 1591, 591, 505, 4148, 2989, 1024, 322, 4876, 408, 1023, 5004, 6031, 310, 10650, 6262, 29889, 32001, 1334, 6206, 2411, 15273, 1821, 470, 18325, 519, 2983, 393, 29901, 526, 451, 1602, 397, 519, 408, 18351, 29899, 29947, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12283, 7099, 397, 519, 29914, 4148, 2983, 6206, 511, 526, 4876, 14157, 2012, 310, 2983, 3441, 25832, 6344, 2806, 943, 29934, 29885, 9823, 29914, 4954, 7039, 4907, 511, 5718, 310, 871, 9654, 4890, 3441, 25832, 6344, 2806, 943, 29934, 29885, 10358, 804, 23201, 1712, 901, 1135, 29871, 29896, 29900, 8958, 1661, 29899, 1026, 2153, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12283, 15670, 23201, 526, 5520, 1135, 29871, 29896, 29900, 29900, 4890, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12229, 549, 12495, 32001, 2860, 21166, 29892, 1048, 320, 25832, 6344, 2806, 943, 29925, 15273, 1821, 2052, 307, 29916, 29914, 15717, 3441, 25832, 6344, 2806, 943, 29925, 15273, 1821, 29925, 312, 29914, 310, 278, 2847, 8783, 29897, 9488, 363, 4340, 7418, 29889, 32001, 3940, 393, 278, 5253, 310, 970, 775, 25741, 313, 392, 15717, 29897, 11122, 297, 278, 13, 11228, 8783, 25088, 18709, 9247, 975, 13, 2230, 2651, 2036, 29912, 29879, 1332, 29899, 771, 854, 749, 29899, 331, 344, 1012, 361, 29887, 798, 386, 1003, 29892, 408, 4318, 363, 25741, 297, 13, 29905, 29907, 999, 29912, 1003, 29901, 29887, 798, 386, 1012, 2870, 29901, 515, 395, 29896, 29900, 29985, 29946, 29938, 25741, 297, 29871, 29896, 29929, 29955, 29896, 29892, 304, 395, 29896, 29900, 29985, 29953, 29938, 297, 29871, 29896, 29929, 29929, 29947, 29892, 304, 13, 284, 3242, 395, 29896, 29900, 29985, 29929, 29938, 297, 29871, 29906, 29900, 29906, 29900, 29905, 7241, 29889, 32001, 1094, 263, 17004, 278, 8900, 534, 1975, 10331, 304, 367, 901, 13, 13844, 297, 7786, 1602, 3076, 1135, 297, 29871, 29946, 29900, 29974, 1629, 29899, 1025, 6743, 29892, 2861, 304, 13964, 4586, 373, 13, 735, 1112, 9247, 7200, 23093, 29889, 32001, 320, 26956, 29912, 7999, 324, 10610, 29913, 13, 13, 29905, 463, 29912, 4532, 29913, 13, 29871, 320, 9525, 13, 29871, 320, 7313, 29961, 24049, 29892, 15450, 29922, 29953, 4912, 29871, 29953, 4912, 29871, 29900, 29871, 29900, 29892, 2103, 2013, 16292, 3199, 1491, 1727, 1080, 29899, 2470, 29913, 13, 29871, 320, 6671, 29912, 1576, 320, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 1304, 408, 1737, 324, 10610, 22525, 5003, 32001, 320, 1643, 29912, 1003, 29901, 11526, 1958, 29913, 13, 29905, 355, 29912, 4532, 29913, 13, 13, 2887, 1737, 324, 10610, 22525, 591, 671, 11758, 3186, 12786, 10723, 515, 278, 3303, 18269, 1737, 359, 305, 2004, 2651, 2036, 29912, 348, 29896, 29929, 29929, 29929, 479, 359, 305, 2004, 1836, 32001, 1763, 4772, 2432, 3381, 491, 2919, 10916, 313, 29872, 29889, 29887, 1696, 7551, 470, 12710, 29897, 2629, 11758, 12786, 29892, 591, 19412, 322, 6219, 777, 12786, 2729, 373, 1737, 12122, 23203, 537, 322, 278, 19383, 310, 758, 331, 8946, 16375, 29769, 5680, 29892, 1316, 408, 19182, 4086, 29889, 32001, 320, 29907, 999, 29912, 1003, 29901, 11526, 1958, 29913, 3697, 278, 2186, 1051, 310, 320, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 1304, 408, 1737, 324, 10610, 22525, 297, 445, 6559, 29889, 32001, 1879, 324, 10610, 310, 9063, 15717, 304, 3186, 12786, 3913, 278, 1023, 19595, 653, 13698, 9129, 297, 2651, 2036, 29912, 293, 344, 29899, 344, 275, 29899, 29906, 29900, 29906, 29906, 29899, 26098, 1118, 23359, 337, 13998, 2400, 29889, 32001, 450, 937, 697, 337, 3687, 373, 278, 4234, 775, 2246, 29899, 5563, 5354, 313, 617, 29911, 10249, 29897, 310, 4876, 14157, 23892, 515, 9063, 15562, 29892, 321, 29889, 29887, 1696, 320, 16266, 26139, 1341, 1118, 320, 16266, 26139, 582, 1118, 320, 16266, 26139, 18038, 1118, 2992, 29889, 32001, 1334, 4687, 515, 278, 306, 2190, 29909, 1051, 310, 13548, 2931, 21759, 29911, 10249, 29879, 2651, 2036, 29912, 6011, 29899, 29883, 312, 430, 29913, 322, 7522, 20545, 1269, 6590, 20123, 304, 263, 3646, 3186, 5120, 29889, 32001, 450, 1473, 1737, 324, 10610, 11043, 3913, 278, 17998, 9210, 310, 9063, 5335, 342, 15092, 313, 29872, 29889, 29887, 1696, 17998, 29899, 29900, 29945, 29901, 29900, 29900, 29897, 322, 4148, 2983, 304, 8161, 278, 1556, 5517, 3186, 5120, 310, 278, 9063, 4148, 29889, 32001, 1152, 1269, 17998, 9210, 591, 8161, 263, 1051, 310, 15878, 7600, 313, 13509, 29892, 2106, 29892, 470, 14278, 20123, 29897, 297, 278, 3186, 393, 29892, 472, 278, 931, 310, 393, 9063, 29892, 750, 393, 17998, 9210, 29936, 9063, 931, 338, 1820, 1244, 29892, 408, 4234, 17998, 1283, 7224, 13100, 975, 931, 2861, 304, 29431, 3620, 29889, 32001, 1763, 1207, 445, 3683, 3381, 591, 671, 278, 306, 2190, 29909, 931, 10640, 2566, 2651, 2036, 29912, 17559, 1272, 1836, 32001, 1987, 591, 3566, 304, 1269, 2058, 263, 8158, 393, 4332, 1973, 278, 4188, 22342, 393, 263, 2183, 4148, 1024, 338, 17443, 310, 372, 29889, 32001, 1763, 445, 1095, 591, 671, 278, 28297, 29890, 15451, 8783, 310, 278, 29511, 310, 278, 1556, 3619, 937, 322, 3942, 2983, 607, 29892, 439, 11427, 515, 2651, 2036, 29912, 1079, 29890, 799, 29899, 7039, 6177, 2802, 277, 12181, 4954, 16123, 2247, 278, 26368, 5528, 5084, 310, 363, 264, 1280, 322, 269, 595, 1280, 7371, 515, 263, 2566, 310, 320, 1949, 29912, 29946, 29871, 29900, 29946, 29946, 29871, 29945, 29946, 29953, 29871, 29929, 29941, 29947, 29913, 2305, 313, 29945, 29945, 29889, 29945, 8958, 310, 8471, 2305, 297, 29871, 29906, 29900, 29896, 29946, 467, 32001, 1094, 310, 3839, 29871, 29906, 29900, 29896, 29929, 372, 18469, 320, 1949, 29912, 29906, 29955, 29871, 29953, 29953, 29906, 29871, 29947, 29900, 29896, 29913, 363, 264, 1280, 322, 320, 1949, 29912, 29906, 29955, 29871, 29906, 29900, 29953, 29871, 29947, 29906, 29896, 29913, 269, 595, 1280, 297, 29871, 29906, 29941, 29953, 24894, 8977, 1080, 29889, 4907, 29913, 32001, 1094, 297, 1749, 8783, 15717, 526, 2989, 1024, 6031, 313, 29878, 1624, 1135, 6219, 491, 937, 29914, 11922, 1024, 511, 591, 937, 5993, 675, 2983, 313, 1609, 1999, 1331, 322, 1206, 3620, 29897, 322, 769, 16280, 5375, 18897, 297, 1716, 937, 322, 3942, 2983, 10868, 8857, 29889, 32001, 1152, 1269, 1543, 1476, 297, 1024, 8857, 591, 22932, 278, 2058, 4665, 29905, 6661, 1333, 24212, 8875, 491, 278, 1024, 10868, 304, 4017, 263, 5645, 393, 338, 29839, 304, 278, 1353, 310, 12407, 24638, 393, 1024, 313, 6979, 29897, 297, 278, 2702, 2058, 29889, 32001, 320, 6661, 1333, 300, 1062, 29912, 1762, 4017, 4665, 2025, 1338, 5634, 294, 278, 17837, 310, 4954, 6689, 4907, 338, 25745, 23724, 29901, 2989, 10916, 325, 13217, 29879, 29399, 310, 2919, 10916, 805, 9450, 2999, 931, 29920, 2873, 5634, 705, 671, 263, 29544, 310, 7601, 8974, 313, 29872, 29889, 29887, 1696, 5874, 28007, 511, 322, 1661, 29899, 16072, 6743, 313, 29872, 29889, 29887, 1696, 14109, 7456, 467, 29913, 32001, 1334, 2533, 445, 4377, 363, 599, 3161, 304, 4017, 263, 2058, 8158, 29892, 17140, 701, 411, 263, 1051, 310, 779, 6990, 29938, 6689, 29892, 8158, 4535, 5854, 29938, 11000, 29889, 32001, 1334, 769, 8877, 445, 1051, 491, 278, 3186, 5120, 393, 263, 2058, 14393, 304, 322, 2533, 278, 8158, 363, 599, 278, 7600, 297, 1269, 5120, 304, 4017, 385, 12463, 8158, 29892, 6590, 304, 278, 4188, 22342, 393, 278, 9063, 14393, 304, 263, 2183, 3186, 5120, 29889, 32001, 1334, 3566, 278, 6257, 9063, 408, 6421, 515, 278, 3186, 5120, 411, 278, 9939, 8158, 29889, 32001, 450, 4876, 29899, 6707, 11043, 9378, 414, 515, 278, 9078, 322, 443, 5521, 8362, 671, 310, 21759, 29911, 10249, 29879, 29901, 1556, 18777, 671, 10035, 323, 10249, 29879, 1316, 408, 320, 16266, 26139, 510, 1118, 320, 16266, 26139, 990, 1118, 470, 320, 16266, 26139, 1212, 1836, 32001, 12808, 445, 947, 451, 3799, 26018, 4822, 20542, 29901, 3148, 29899, 6707, 18777, 29892, 363, 1342, 29892, 671, 278, 320, 16266, 26139, 375, 29913, 21759, 29911, 10249, 1568, 901, 269, 2495, 290, 368, 1135, 1009, 7824, 6795, 20895, 29889, 32001, 1551, 278, 916, 1361, 278, 9210, 29914, 978, 29899, 6707, 11043, 337, 3687, 373, 278, 17998, 9210, 310, 278, 9063, 5335, 342, 15092, 29889, 32001, 16809, 304, 5780, 22920, 373, 13897, 731, 14340, 29892, 263, 2919, 1353, 310, 25741, 297, 278, 8783, 756, 385, 17998, 9210, 5186, 304, 5225, 29889, 32001, 910, 6602, 29879, 3109, 7786, 25741, 3441, 14573, 5261, 1169, 29911, 29999, 29999, 13985, 1349, 681, 392, 27418, 6478, 29914, 310, 29871, 29906, 29900, 29906, 29900, 29879, 25741, 505, 263, 5225, 9210, 29897, 1135, 9642, 6743, 3441, 14573, 5261, 1169, 29911, 29999, 29999, 13985, 1349, 681, 392, 29914, 297, 29871, 29906, 29900, 29900, 29900, 467, 32001, 1094, 263, 1121, 278, 9210, 29914, 978, 29899, 6707, 11043, 1033, 1095, 701, 6459, 292, 263, 2919, 6232, 310, 9642, 25741, 408, 4148, 287, 491, 11715, 18777, 29892, 322, 304, 263, 3109, 261, 15834, 4092, 550, 29889, 32001, 1763, 6795, 1438, 5626, 591, 14405, 278, 1023, 1737, 324, 10610, 13698, 4208, 491, 15399, 278, 9210, 29914, 978, 29899, 6707, 13698, 304, 599, 25741, 411, 263, 1661, 29899, 9171, 17998, 9210, 29892, 322, 278, 4876, 29899, 6707, 373, 304, 599, 916, 25741, 29889, 32001, 320, 2042, 29912, 12191, 322, 8565, 1558, 291, 29913, 13, 29905, 1643, 29912, 3471, 29901, 9902, 29913, 13, 13, 29905, 463, 29912, 4532, 4044, 13, 29871, 320, 9525, 13, 29871, 320, 7313, 29961, 2103, 2013, 16292, 3199, 1429, 287, 29889, 5140, 29913, 13, 29871, 320, 6671, 29912, 29934, 20819, 310, 25741, 313, 27215, 29897, 322, 6136, 15717, 313, 22503, 29897, 491, 3186, 10640, 975, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29900, 3785, 5003, 32001, 320, 9868, 29961, 14732, 3199, 7264, 287, 2594, 8727, 6445, 278, 3186, 10640, 364, 2219, 359, 363, 25741, 322, 15717, 975, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29900, 3785, 5003, 32001, 320, 1643, 29912, 1003, 29901, 9902, 29913, 13, 29905, 355, 29912, 4532, 4044, 13, 13, 13, 29871, 13, 1762, 1234, 320, 25029, 29912, 29878, 29939, 29901, 479, 397, 24974, 29913, 591, 22229, 278, 1353, 310, 25741, 322, 8359, 15717, 639, 1629, 322, 639, 3186, 10640, 29889, 32001, 1334, 2198, 278, 7625, 2582, 297, 320, 29907, 999, 29912, 1003, 29901, 9902, 29913, 408, 1023, 5096, 287, 2594, 24469, 29892, 6445, 1629, 368, 2867, 3204, 29879, 363, 25741, 322, 15717, 8307, 29889, 32001, 7569, 2594, 11524, 263, 1629, 322, 338, 8877, 287, 297, 269, 29399, 6445, 278, 9063, 29914, 8921, 11959, 363, 1269, 310, 278, 3186, 12786, 310, 320, 29907, 999, 29912, 1003, 29901, 11526, 1958, 29913, 297, 393, 1629, 29889, 32001, 1763, 4772, 714, 27801, 2861, 304, 805, 272, 26538, 17737, 29560, 29892, 297, 278, 4148, 8727, 591, 871, 2050, 15717, 2534, 26869, 472, 3203, 29871, 29945, 25741, 297, 263, 2183, 1629, 29889, 32001, 5806, 5366, 1747, 534, 1975, 297, 278, 24469, 6456, 393, 278, 3001, 3694, 310, 25741, 322, 15717, 6548, 18709, 9247, 975, 931, 29889, 32001, 10133, 363, 278, 937, 2440, 297, 278, 24469, 29892, 278, 1353, 310, 848, 3291, 297, 777, 3186, 12786, 508, 367, 14154, 2319, 29892, 411, 8178, 27721, 373, 278, 25806, 310, 534, 1975, 29889, 32001, 320, 26956, 29912, 7999, 12122, 6894, 537, 975, 931, 29913, 13, 13, 3563, 497, 29892, 278, 2498, 534, 355, 5692, 304, 367, 393, 278, 320, 4534, 29912, 479, 12122, 6894, 537, 297, 970, 775, 338, 10231, 6177, 4644, 6813, 322, 4092, 5136, 630, 1009, 4954, 24130, 749, 4907, 2745, 278, 7256, 310, 278, 29871, 29929, 29900], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [1, 320, 2042, 29912, 25898, 29913, 13, 29905, 1643, 29912, 3471, 29901, 23333, 29913, 13, 13, 29905, 7278, 29912, 29954, 1581, 6894, 537, 1118, 470, 901, 4049, 967, 10225, 727, 974, 29892, 4249, 27138, 304, 13, 20415, 5849, 14188, 756, 1063, 26606, 12399, 297, 7786, 2440, 29889, 32001, 512, 13, 1595, 16311, 29892, 278, 10122, 310, 29892, 9545, 310, 29892, 322, 6795, 1004, 25414, 363, 320, 7278, 29912, 26098, 13, 29871, 24003, 29913, 297, 12362, 29914, 6585, 7562, 18540, 313, 5800, 1799, 29897, 505, 4520, 263, 3287, 310, 8570, 13, 957, 278, 4940, 316, 6332, 2651, 2036, 29912, 29881, 16093, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 3855, 5871, 29906, 29900, 29896, 29900, 29895, 311, 29893, 2770, 29892, 13, 29871, 1055, 29888, 375, 29906, 29900, 29896, 29906, 5041, 267, 29892, 413, 434, 305, 1358, 29906, 29900, 29896, 29906, 26098, 29888, 2209, 29892, 19723, 5475, 4979, 29906, 29900, 29896, 29946, 26098, 29892, 13, 29871, 697, 309, 29906, 29900, 29896, 29953, 311, 5365, 550, 332, 6950, 29892, 696, 7586, 29906, 29900, 29896, 29953, 29893, 2770, 7192, 2209, 29892, 1935, 15044, 29906, 29900, 29896, 29955, 26098, 29892, 13, 29871, 503, 562, 305, 3350, 492, 29906, 29900, 29906, 29896, 26098, 1836, 32001, 320, 7278, 29912, 7999, 12122, 6894, 537, 29913, 338, 373, 278, 916, 1361, 278, 13, 14380, 310, 6894, 537, 393, 380, 1567, 515, 27138, 297, 777, 5534, 6354, 6421, 13, 3166, 1422, 3186, 12786, 322, 4185, 1973, 29889, 32001, 26304, 6894, 537, 297, 18322, 1799, 756, 4520, 13774, 2217, 8570, 297, 21344, 368, 13, 13129, 29889, 32001, 512, 3153, 29892, 1550, 3031, 979, 18994, 29899, 6707, 322, 13, 3149, 29899, 262, 29899, 2230, 18350, 29899, 7052, 11898, 310, 278, 1737, 12122, 1677, 1144, 310, 18322, 1799, 13, 21570, 29560, 1863, 2651, 2036, 29912, 12443, 10578, 29906, 29900, 29900, 29945, 5062, 11235, 29892, 13260, 333, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 13, 29871, 2594, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29892, 1850, 29882, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29892, 696, 7586, 29906, 29900, 29896, 29946, 7610, 345, 2941, 271, 24541, 29892, 13, 29871, 281, 496, 29879, 29906, 29900, 29906, 29896, 2209, 479, 5275, 1118, 2919, 29899, 7052, 25579, 979, 11898, 310, 278, 1737, 12122, 13, 12574, 310, 18322, 1799, 17737, 29560, 526, 1603, 10225, 292, 29889, 32001, 10506, 263, 4323, 23378, 13, 18609, 2133, 723, 367, 5407, 304, 1871, 1602, 12112, 4475, 304, 5534, 13, 25431, 10907, 2651, 2036, 29912, 2276, 5824, 19982, 29906, 29900, 29900, 29955, 23705, 1338, 29893, 996, 29913, 322, 298, 8491, 16650, 583, 297, 278, 13, 19678, 15483, 313, 1806, 29897, 9999, 29892, 408, 1532, 408, 29126, 2114, 950, 2472, 13, 517, 278, 2553, 1078, 373, 278, 17407, 10879, 322, 5374, 3002, 310, 18322, 1799, 2820, 278, 3186, 29889, 32001, 320, 26956, 29912, 1323, 3224, 29879, 29913, 13, 13, 3047, 445, 664, 591, 29126, 304, 3802, 445, 17261, 491, 7512, 292, 320, 4534, 29912, 1552, 937, 13, 29871, 25579, 979, 6559, 310, 278, 1737, 12122, 3978, 310, 17737, 29560, 304, 970, 775, 13, 29871, 975, 29871, 29945, 29900, 2440, 5003, 32001, 26321, 29892, 591, 3867, 263, 758, 2576, 3821, 1234, 304, 278, 13, 23031, 292, 5925, 1139, 29901, 13, 29905, 463, 29912, 690, 2842, 12470, 29913, 13, 29871, 3645, 607, 3186, 12786, 437, 15717, 310, 970, 368, 3625, 25741, 2041, 515, 13, 29871, 322, 920, 756, 372, 3939, 975, 278, 4940, 29871, 29945, 29900, 2440, 29973, 32001, 320, 1643, 29912, 29878, 29939, 29901, 479, 397, 24974, 29913, 13, 29905, 355, 29912, 690, 2842, 12470, 29913, 13, 4806, 671, 408, 8783, 278, 320, 23066, 29950, 29914, 18871, 2651, 2036, 29912, 29879, 1332, 29875, 4569, 29906, 29900, 29896, 29955, 29913, 322, 27599, 515, 372, 13, 29906, 29889, 29906, 24464, 29905, 13007, 25741, 3190, 2347, 515, 29871, 29896, 29953, 29900, 7284, 29905, 13007, 9279, 322, 4148, 287, 491, 13, 29946, 29941, 7284, 29905, 13007, 15717, 2645, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29896, 931, 3785, 29889, 32001, 1334, 1737, 324, 542, 403, 18777, 304, 13, 29905, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 29892, 773, 408, 18470, 4876, 4234, 775, 2246, 29899, 5563, 21904, 313, 617, 29911, 10249, 29879, 29897, 322, 29871, 13, 8921, 313, 4102, 29914, 4230, 29897, 2983, 9401, 411, 1024, 18822, 2820, 278, 3186, 29892, 322, 17998, 1283, 7224, 29871, 13, 1195, 287, 515, 9063, 15562, 29889, 32001, 1334, 1284, 10757, 310, 278, 4688, 8022, 749, 310, 4644, 6813, 297, 1722, 2752, 13, 20415, 29892, 2678, 8772, 491, 4092, 29889, 32001, 2860, 393, 3785, 29892, 278, 1737, 12122, 6894, 537, 29871, 13, 262, 970, 775, 756, 1063, 21003, 10231, 29889, 32001, 1334, 884, 12439, 8018, 15839, 528, 17741, 13, 12817, 304, 278, 1095, 310, 278, 8291, 6415, 29129, 322, 278, 7910, 310, 14137, 4631, 4135, 297, 13, 23369, 1705, 322, 4275, 14325, 29892, 408, 1532, 408, 310, 2545, 1664, 17292, 28342, 763, 25539, 1608, 322, 13, 25719, 10298, 4822, 10916, 313, 6727, 16783, 29914, 331, 16783, 467, 32001, 320, 26956, 29912, 1469, 20847, 3097, 5003, 32001, 319, 1634, 1414, 3577, 363, 445, 5650, 338, 3625, 515, 796, 264, 8144, 472, 13, 29905, 2271, 29912, 991, 597, 1867, 29875, 29889, 990, 29914, 29896, 29900, 29889, 29945, 29906, 29947, 29896, 29914, 2256, 8144, 29889, 29953, 29941, 29929, 29900, 29941, 29945, 29945, 29913, 2651, 2036, 29912, 3445, 1414, 29899, 5113, 1836, 32001, 320, 2042, 29912, 9662, 630, 5244, 29913, 13, 29905, 1643, 29912, 3471, 29901, 12817, 29913, 13, 13, 29933, 720, 4688, 322, 7786, 1736, 2651, 2036, 29912, 12443, 10578, 29906, 29900, 29900, 29945, 5062, 11235, 29892, 13260, 333, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 13, 29871, 696, 7586, 29906, 29900, 29896, 29946, 7610, 345, 2941, 271, 24541, 29892, 697, 309, 29906, 29900, 29896, 29953, 311, 5365, 550, 332, 6950, 29913, 505, 2931, 1891, 278, 13, 479, 5275, 310, 12362, 29914, 6585, 7562, 18540, 313, 5800, 1799, 29897, 773, 320, 7278, 29912, 6734, 26946, 952, 1118, 13, 4716, 3867, 1880, 29899, 29567, 6089, 541, 526, 9078, 297, 2159, 313, 29906, 29899, 29945, 5940, 29968, 18777, 29897, 13, 392, 508, 367, 4768, 1463, 491, 5221, 424, 23460, 29889, 32001, 512, 29871, 29906, 29900, 29900, 29947, 2261, 801, 2681, 634, 394, 8770, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29913, 18043, 263, 3031, 979, 13, 16961, 29899, 7052, 313, 1454, 278, 931, 29897, 6559, 373, 18322, 1799, 320, 7278, 29912, 479, 5275, 773, 1375, 292, 7047, 13, 29871, 28914, 313, 4345, 29934, 29897, 13698, 1836, 32001, 2688, 29537, 287, 278, 3978, 310, 29871, 29896, 5940, 29924, 17737, 29560, 13, 4746, 278, 7562, 2831, 479, 1404, 2566, 322, 611, 6504, 1051, 3190, 3145, 975, 278, 13, 29896, 29929, 29929, 29929, 489, 29906, 29900, 29900, 29945, 3785, 29892, 773, 408, 18470, 2472, 2788, 304, 1749, 29879, 29901, 4876, 21904, 13, 392, 17998, 1283, 7224, 29889, 32001, 450, 12399, 3785, 313, 29955, 2440, 29897, 297, 2651, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29913, 338, 20511, 1135, 29871, 13, 5816, 338, 12399, 297, 278, 2198, 5650, 313, 29945, 29900, 2440, 29897, 322, 278, 848, 8974, 526, 29871, 13, 27489, 873, 1422, 29936, 411, 393, 297, 3458, 29892, 1749, 2582, 1510, 263, 10029, 7200, 14978, 310, 29871, 13, 15654, 273, 325, 13217, 29940, 2072, 3082, 20706, 29889, 32001, 7280, 29190, 936, 664, 515, 29871, 29906, 29900, 29896, 29900, 491, 14619, 29882, 371, 4099, 29894, 322, 13, 29950, 309, 1372, 2651, 2036, 29912, 29873, 19426, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29913, 4023, 29894, 2868, 1583, 29899, 7099, 433, 1127, 1737, 12122, 13, 2029, 800, 310, 25492, 15303, 8304, 3598, 1494, 1009, 12368, 29892, 13, 15914, 292, 2472, 363, 779, 14850, 4535, 29892, 29955, 29900, 5940, 29968, 25492, 4160, 29889, 32001, 319, 1407, 7786, 13, 1287, 2651, 2036, 29912, 21686, 29879, 29906, 29900, 29906, 29896, 2209, 479, 5275, 29913, 491, 399, 496, 29879, 634, 394, 13217, 5349, 1737, 324, 542, 630, 4203, 263, 7284, 13, 28712, 16046, 4160, 29892, 2534, 26869, 472, 3203, 29871, 29896, 29900, 29900, 25741, 1269, 29892, 322, 1058, 13, 1311, 29899, 7099, 8663, 14354, 373, 1009, 25492, 28723, 29889, 32001, 5806, 278, 6559, 338, 13, 3149, 29899, 262, 29899, 2230, 408, 310, 29871, 29906, 29900, 29906, 29896, 29892, 278, 15717, 7252, 1009, 1284, 886, 13, 351, 475, 303, 2651, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29892, 1850, 29882, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29913, 304, 13, 18609, 675, 278, 14675, 310, 18322, 1799, 1737, 5275, 975, 278, 931, 15101, 845, 1862, 4586, 491, 13, 1552, 2211, 11898, 29889, 32001, 3831, 1965, 411, 3517, 29190, 936, 1736, 29892, 1749, 6559, 338, 1568, 7200, 6287, 5634, 29882, 5555, 13, 7054, 12339, 287, 29871, 29946, 29941, 7284, 29905, 13007, 15717, 310, 29871, 29906, 29889, 29906, 24464, 29905, 13007, 25741, 515, 29871, 29896, 29953, 29900, 7284, 29905, 13007, 13, 16418, 5634, 5426, 11267, 979, 975, 29871, 29945, 29900, 2440, 310, 970, 775, 20706, 3265, 1135, 13, 3149, 297, 931, 29892, 322, 884, 901, 2691, 29899, 3874, 1312, 313, 2541, 1629, 29899, 1609, 29899, 6360, 3803, 1070, 537, 975, 13, 1552, 8900, 3785, 467, 32001, 8108, 1189, 1711, 29892, 1749, 6559, 337, 3687, 373, 10079, 11264, 13, 3924, 313, 8257, 29903, 29897, 9063, 848, 3265, 1135, 7481, 29899, 7099, 433, 1127, 4423, 2472, 29889, 32001, 5901, 1736, 5634, 262, 3153, 278, 664, 491, 8432, 2651, 2036, 29912, 18386, 709, 29906, 29900, 29896, 29941, 2209, 29881, 24974, 29913, 13, 392, 29892, 901, 10325, 29892, 390, 579, 468, 29875, 634, 394, 8770, 2036, 29912, 29878, 579, 468, 29875, 29906, 29900, 29896, 29953, 479, 711, 3173, 29892, 13, 29871, 364, 579, 468, 29875, 29906, 29900, 29896, 29947, 479, 711, 3173, 29892, 544, 1648, 29906, 29900, 29906, 29896, 479, 468, 1581, 29881, 24974, 29913, 5634, 17532, 12399, 1737, 12122, 13, 29905, 7278, 29912, 29881, 24974, 322, 24003, 1118, 474, 29889, 29872, 1696, 278, 15834, 304, 607, 278, 3978, 310, 18322, 1799, 13, 17426, 6602, 1009, 11465, 1230, 14137, 14188, 29889, 32001, 512, 445, 664, 591, 2931, 1891, 1737, 12122, 6894, 537, 297, 970, 775, 363, 278, 937, 13, 2230, 472, 445, 6287, 29892, 1716, 297, 4958, 310, 17737, 29560, 322, 15500, 3785, 29889, 32001, 1334, 437, 13, 1333, 22002, 280, 278, 24003, 10696, 29892, 541, 3867, 29190, 936, 848, 322, 1284, 886, 393, 508, 367, 13, 280, 369, 4063, 304, 393, 1095, 408, 5434, 664, 29889, 32001, 320, 7278, 29912, 12756, 7047, 21639, 29913, 2651, 2036, 29912, 2276, 5824, 19982, 29906, 29900, 29900, 29955, 23705, 1338, 29893, 996, 29913, 338, 278, 13, 1491, 29899, 2671, 310, 7047, 21639, 393, 756, 29537, 287, 278, 18066, 267, 310, 21640, 13, 6734, 24771, 13149, 635, 29892, 3704, 278, 2702, 5932, 310, 920, 304, 5376, 13, 2541, 1737, 12122, 6894, 537, 2651, 2036, 29912, 13376, 303, 456, 29906, 29900, 29900, 29953, 10945, 3359, 29892, 1424, 29440, 29906, 29900, 29896, 29946, 23027, 5933, 1836, 32001, 3826, 3076, 2678, 278, 2198, 6559, 8128, 10757, 393, 508, 367, 1304, 29892, 297, 278, 13, 14940, 1206, 310, 970, 775, 322, 472, 263, 1407, 2919, 6287, 29892, 304, 11539, 607, 13, 14032, 4637, 310, 5534, 7047, 21639, 505, 9820, 484, 15774, 29889, 32001, 320, 2042, 29912, 4062, 3002, 29913, 13, 29905, 1643, 29912, 3471, 29901, 5696, 29913, 13, 13, 13, 29905, 1482, 361, 29905, 361, 29887, 798, 386, 1003, 29871, 320, 29887, 798, 386, 1003, 3009, 13, 29905, 361, 29887, 798, 386, 1003, 13, 29905, 463, 29912, 4532, 29913, 13, 29871, 320, 7313, 29961, 2103, 2013, 4914, 2103, 3199, 6360, 368, 29899, 2055, 1169, 29913, 13, 29871, 320, 6671, 29912, 12883, 368, 970, 25741, 975, 931, 313, 1188, 6287, 467, 13, 29913, 32001, 320, 1643, 29912, 1003, 29901, 29887, 798, 386, 29913, 13, 29905, 355, 29912, 4532, 29913, 13, 29905, 7241, 13, 13, 29905, 26956, 29912, 16390, 24541, 29913, 13, 13, 4806, 27387, 515, 320, 23066, 29950, 29914, 2651, 2036, 29912, 29879, 1332, 29899, 1516, 29878, 29906, 29900, 29896, 29929, 29899, 24713, 29913, 599, 25741, 3190, 2347, 2745, 320, 25832, 1964, 579, 1523, 2415, 2539, 6294, 32001, 2688, 5253, 304, 320, 14573, 5261, 1169, 22131, 29914, 25741, 29892, 5412, 491, 317, 15715, 29896, 15882, 29892, 4023, 29894, 2868, 515, 320, 25832, 1299, 7288, 5261, 1169, 797, 7068, 29914, 970, 9279, 6421, 515, 4655, 5849, 363, 2710, 313, 28712, 16046, 29892, 11786, 28632, 29892, 2992, 1846, 32001, 322, 3577, 28914, 313, 10251, 713, 29892, 10772, 2227, 29892, 405, 13427, 29892, 2992, 6250, 32001, 1876, 1169, 297, 278, 8783, 526, 491, 320, 25832, 6344, 2806, 943, 22131, 29914, 15717, 29892, 5412, 491, 779, 6990, 29938, 978, 29892, 4876, 4535, 5854, 29938, 11000, 29889, 32001, 450, 8783, 2996, 408, 1023, 1104, 1288, 6131, 29892, 697, 363, 25741, 322, 697, 363, 15717, 29892, 411, 278, 4642, 29371, 278, 7480, 3025, 263, 9117, 1820, 29889, 32001, 320, 361, 5426, 13, 9760, 1948, 297, 278, 9063, 1591, 3743, 278, 1494, 4235, 29901, 9063, 317, 15715, 29896, 15882, 29892, 4148, 322, 844, 5171, 5335, 342, 15092, 29892, 4148, 322, 844, 5171, 2893, 14903, 313, 20275, 16750, 278, 4148, 1591, 467, 32001, 450, 21578, 1546, 9063, 15717, 322, 9063, 2153, 2041, 515, 11786, 29892, 607, 6511, 304, 9063, 263, 1735, 4148, 287, 491, 4856, 1683, 29889, 32001, 1152, 445, 6559, 591, 21309, 373, 15717, 322, 17262, 9063, 2153, 29892, 408, 278, 4328, 1546, 278, 1023, 338, 451, 8018, 363, 1749, 5925, 5155, 322, 278, 5253, 310, 25741, 411, 263, 844, 5171, 916, 1135, 967, 4148, 338, 3480, 3473, 1821, 29889, 32001, 320, 7241, 13, 2831, 1269, 6251, 297, 278, 4148, 1591, 591, 505, 4148, 2989, 1024, 322, 4876, 408, 1023, 5004, 6031, 310, 10650, 6262, 29889, 32001, 1334, 6206, 2411, 15273, 1821, 470, 18325, 519, 2983, 393, 29901, 526, 451, 1602, 397, 519, 408, 18351, 29899, 29947, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12283, 7099, 397, 519, 29914, 4148, 2983, 6206, 511, 526, 4876, 14157, 2012, 310, 2983, 3441, 25832, 6344, 2806, 943, 29934, 29885, 9823, 29914, 4954, 7039, 4907, 511, 5718, 310, 871, 9654, 4890, 3441, 25832, 6344, 2806, 943, 29934, 29885, 10358, 804, 23201, 1712, 901, 1135, 29871, 29896, 29900, 8958, 1661, 29899, 1026, 2153, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12283, 15670, 23201, 526, 5520, 1135, 29871, 29896, 29900, 29900, 4890, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12229, 549, 12495, 32001, 2860, 21166, 29892, 1048, 320, 25832, 6344, 2806, 943, 29925, 15273, 1821, 2052, 307, 29916, 29914, 15717, 3441, 25832, 6344, 2806, 943, 29925, 15273, 1821, 29925, 312, 29914, 310, 278, 2847, 8783, 29897, 9488, 363, 4340, 7418, 29889, 32001, 3940, 393, 278, 5253, 310, 970, 775, 25741, 313, 392, 15717, 29897, 11122, 297, 278, 13, 11228, 8783, 25088, 18709, 9247, 975, 13, 2230, 2651, 2036, 29912, 29879, 1332, 29899, 771, 854, 749, 29899, 331, 344, 1012, 361, 29887, 798, 386, 1003, 29892, 408, 4318, 363, 25741, 297, 13, 29905, 29907, 999, 29912, 1003, 29901, 29887, 798, 386, 1012, 2870, 29901, 515, 395, 29896, 29900, 29985, 29946, 29938, 25741, 297, 29871, 29896, 29929, 29955, 29896, 29892, 304, 395, 29896, 29900, 29985, 29953, 29938, 297, 29871, 29896, 29929, 29929, 29947, 29892, 304, 13, 284, 3242, 395, 29896, 29900, 29985, 29929, 29938, 297, 29871, 29906, 29900, 29906, 29900, 29905, 7241, 29889, 32001, 1094, 263, 17004, 278, 8900, 534, 1975, 10331, 304, 367, 901, 13, 13844, 297, 7786, 1602, 3076, 1135, 297, 29871, 29946, 29900, 29974, 1629, 29899, 1025, 6743, 29892, 2861, 304, 13964, 4586, 373, 13, 735, 1112, 9247, 7200, 23093, 29889, 32001, 320, 26956, 29912, 7999, 324, 10610, 29913, 13, 13, 29905, 463, 29912, 4532, 29913, 13, 29871, 320, 9525, 13, 29871, 320, 7313, 29961, 24049, 29892, 15450, 29922, 29953, 4912, 29871, 29953, 4912, 29871, 29900, 29871, 29900, 29892, 2103, 2013, 16292, 3199, 1491, 1727, 1080, 29899, 2470, 29913, 13, 29871, 320, 6671, 29912, 1576, 320, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 1304, 408, 1737, 324, 10610, 22525, 5003, 32001, 320, 1643, 29912, 1003, 29901, 11526, 1958, 29913, 13, 29905, 355, 29912, 4532, 29913, 13, 13, 2887, 1737, 324, 10610, 22525, 591, 671, 11758, 3186, 12786, 10723, 515, 278, 3303, 18269, 1737, 359, 305, 2004, 2651, 2036, 29912, 348, 29896, 29929, 29929, 29929, 479, 359, 305, 2004, 1836, 32001, 1763, 4772, 2432, 3381, 491, 2919, 10916, 313, 29872, 29889, 29887, 1696, 7551, 470, 12710, 29897, 2629, 11758, 12786, 29892, 591, 19412, 322, 6219, 777, 12786, 2729, 373, 1737, 12122, 23203, 537, 322, 278, 19383, 310, 758, 331, 8946, 16375, 29769, 5680, 29892, 1316, 408, 19182, 4086, 29889, 32001, 320, 29907, 999, 29912, 1003, 29901, 11526, 1958, 29913, 3697, 278, 2186, 1051, 310, 320, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 1304, 408, 1737, 324, 10610, 22525, 297, 445, 6559, 29889, 32001, 1879, 324, 10610, 310, 9063, 15717, 304, 3186, 12786, 3913, 278, 1023, 19595, 653, 13698, 9129, 297, 2651, 2036, 29912, 293, 344, 29899, 344, 275, 29899, 29906, 29900, 29906, 29906, 29899, 26098, 1118, 23359, 337, 13998, 2400, 29889, 32001, 450, 937, 697, 337, 3687, 373, 278, 4234, 775, 2246, 29899, 5563, 5354, 313, 617, 29911, 10249, 29897, 310, 4876, 14157, 23892, 515, 9063, 15562, 29892, 321, 29889, 29887, 1696, 320, 16266, 26139, 1341, 1118, 320, 16266, 26139, 582, 1118, 320, 16266, 26139, 18038, 1118, 2992, 29889, 32001, 1334, 4687, 515, 278, 306, 2190, 29909, 1051, 310, 13548, 2931, 21759, 29911, 10249, 29879, 2651, 2036, 29912, 6011, 29899, 29883, 312, 430, 29913, 322, 7522, 20545, 1269, 6590, 20123, 304, 263, 3646, 3186, 5120, 29889, 32001, 450, 1473, 1737, 324, 10610, 11043, 3913, 278, 17998, 9210, 310, 9063, 5335, 342, 15092, 313, 29872, 29889, 29887, 1696, 17998, 29899, 29900, 29945, 29901, 29900, 29900, 29897, 322, 4148, 2983, 304, 8161, 278, 1556, 5517, 3186, 5120, 310, 278, 9063, 4148, 29889, 32001, 1152, 1269, 17998, 9210, 591, 8161, 263, 1051, 310, 15878, 7600, 313, 13509, 29892, 2106, 29892, 470, 14278, 20123, 29897, 297, 278, 3186, 393, 29892, 472, 278, 931, 310, 393, 9063, 29892, 750, 393, 17998, 9210, 29936, 9063, 931, 338, 1820, 1244, 29892, 408, 4234, 17998, 1283, 7224, 13100, 975, 931, 2861, 304, 29431, 3620, 29889, 32001, 1763, 1207, 445, 3683, 3381, 591, 671, 278, 306, 2190, 29909, 931, 10640, 2566, 2651, 2036, 29912, 17559, 1272, 1836, 32001, 1987, 591, 3566, 304, 1269, 2058, 263, 8158, 393, 4332, 1973, 278, 4188, 22342, 393, 263, 2183, 4148, 1024, 338, 17443, 310, 372, 29889, 32001, 1763, 445, 1095, 591, 671, 278, 28297, 29890, 15451, 8783, 310, 278, 29511, 310, 278, 1556, 3619, 937, 322, 3942, 2983, 607, 29892, 439, 11427, 515, 2651, 2036, 29912, 1079, 29890, 799, 29899, 7039, 6177, 2802, 277, 12181, 4954, 16123, 2247, 278, 26368, 5528, 5084, 310, 363, 264, 1280, 322, 269, 595, 1280, 7371, 515, 263, 2566, 310, 320, 1949, 29912, 29946, 29871, 29900, 29946, 29946, 29871, 29945, 29946, 29953, 29871, 29929, 29941, 29947, 29913, 2305, 313, 29945, 29945, 29889, 29945, 8958, 310, 8471, 2305, 297, 29871, 29906, 29900, 29896, 29946, 467, 32001, 1094, 310, 3839, 29871, 29906, 29900, 29896, 29929, 372, 18469, 320, 1949, 29912, 29906, 29955, 29871, 29953, 29953, 29906, 29871, 29947, 29900, 29896, 29913, 363, 264, 1280, 322, 320, 1949, 29912, 29906, 29955, 29871, 29906, 29900, 29953, 29871, 29947, 29906, 29896, 29913, 269, 595, 1280, 297, 29871, 29906, 29941, 29953, 24894, 8977, 1080, 29889, 4907, 29913, 32001, 1094, 297, 1749, 8783, 15717, 526, 2989, 1024, 6031, 313, 29878, 1624, 1135, 6219, 491, 937, 29914, 11922, 1024, 511, 591, 937, 5993, 675, 2983, 313, 1609, 1999, 1331, 322, 1206, 3620, 29897, 322, 769, 16280, 5375, 18897, 297, 1716, 937, 322, 3942, 2983, 10868, 8857, 29889, 32001, 1152, 1269, 1543, 1476, 297, 1024, 8857, 591, 22932, 278, 2058, 4665, 29905, 6661, 1333, 24212, 8875, 491, 278, 1024, 10868, 304, 4017, 263, 5645, 393, 338, 29839, 304, 278, 1353, 310, 12407, 24638, 393, 1024, 313, 6979, 29897, 297, 278, 2702, 2058, 29889, 32001, 320, 6661, 1333, 300, 1062, 29912, 1762, 4017, 4665, 2025, 1338, 5634, 294, 278, 17837, 310, 4954, 6689, 4907, 338, 25745, 23724, 29901, 2989, 10916, 325, 13217, 29879, 29399, 310, 2919, 10916, 805, 9450, 2999, 931, 29920, 2873, 5634, 705, 671, 263, 29544, 310, 7601, 8974, 313, 29872, 29889, 29887, 1696, 5874, 28007, 511, 322, 1661, 29899, 16072, 6743, 313, 29872, 29889, 29887, 1696, 14109, 7456, 467, 29913, 32001, 1334, 2533, 445, 4377, 363, 599, 3161, 304, 4017, 263, 2058, 8158, 29892, 17140, 701, 411, 263, 1051, 310, 779, 6990, 29938, 6689, 29892, 8158, 4535, 5854, 29938, 11000, 29889, 32001, 1334, 769, 8877, 445, 1051, 491, 278, 3186, 5120, 393, 263, 2058, 14393, 304, 322, 2533, 278, 8158, 363, 599, 278, 7600, 297, 1269, 5120, 304, 4017, 385, 12463, 8158, 29892, 6590, 304, 278, 4188, 22342, 393, 278, 9063, 14393, 304, 263, 2183, 3186, 5120, 29889, 32001, 1334, 3566, 278, 6257, 9063, 408, 6421, 515, 278, 3186, 5120, 411, 278, 9939, 8158, 29889, 32001, 450, 4876, 29899, 6707, 11043, 9378, 414, 515, 278, 9078, 322, 443, 5521, 8362, 671, 310, 21759, 29911, 10249, 29879, 29901, 1556, 18777, 671, 10035, 323, 10249, 29879, 1316, 408, 320, 16266, 26139, 510, 1118, 320, 16266, 26139, 990, 1118, 470, 320, 16266, 26139, 1212, 1836, 32001, 12808, 445, 947, 451, 3799, 26018, 4822, 20542, 29901, 3148, 29899, 6707, 18777, 29892, 363, 1342, 29892, 671, 278, 320, 16266, 26139, 375, 29913, 21759, 29911, 10249, 1568, 901, 269, 2495, 290, 368, 1135, 1009, 7824, 6795, 20895, 29889, 32001, 1551, 278, 916, 1361, 278, 9210, 29914, 978, 29899, 6707, 11043, 337, 3687, 373, 278, 17998, 9210, 310, 278, 9063, 5335, 342, 15092, 29889, 32001, 16809, 304, 5780, 22920, 373, 13897, 731, 14340, 29892, 263, 2919, 1353, 310, 25741, 297, 278, 8783, 756, 385, 17998, 9210, 5186, 304, 5225, 29889, 32001, 910, 6602, 29879, 3109, 7786, 25741, 3441, 14573, 5261, 1169, 29911, 29999, 29999, 13985, 1349, 681, 392, 27418, 6478, 29914, 310, 29871, 29906, 29900, 29906, 29900, 29879, 25741, 505, 263, 5225, 9210, 29897, 1135, 9642, 6743, 3441, 14573, 5261, 1169, 29911, 29999, 29999, 13985, 1349, 681, 392, 29914, 297, 29871, 29906, 29900, 29900, 29900, 467, 32001, 1094, 263, 1121, 278, 9210, 29914, 978, 29899, 6707, 11043, 1033, 1095, 701, 6459, 292, 263, 2919, 6232, 310, 9642, 25741, 408, 4148, 287, 491, 11715, 18777, 29892, 322, 304, 263, 3109, 261, 15834, 4092, 550, 29889, 32001, 1763, 6795, 1438, 5626, 591, 14405, 278, 1023, 1737, 324, 10610, 13698, 4208, 491, 15399, 278, 9210, 29914, 978, 29899, 6707, 13698, 304, 599, 25741, 411, 263, 1661, 29899, 9171, 17998, 9210, 29892, 322, 278, 4876, 29899, 6707, 373, 304, 599, 916, 25741, 29889, 32001, 320, 2042, 29912, 12191, 322, 8565, 1558, 291, 29913, 13, 29905, 1643, 29912, 3471, 29901, 9902, 29913, 13, 13, 29905, 463, 29912, 4532, 4044, 13, 29871, 320, 9525, 13, 29871, 320, 7313, 29961, 2103, 2013, 16292, 3199, 1429, 287, 29889, 5140, 29913, 13, 29871, 320, 6671, 29912, 29934, 20819, 310, 25741, 313, 27215, 29897, 322, 6136, 15717, 313, 22503, 29897, 491, 3186, 10640, 975, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29900, 3785, 5003, 32001, 320, 9868, 29961, 14732, 3199, 7264, 287, 2594, 8727, 6445, 278, 3186, 10640, 364, 2219, 359, 363, 25741, 322, 15717, 975, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29900, 3785, 5003, 32001, 320, 1643, 29912, 1003, 29901, 9902, 29913, 13, 29905, 355, 29912, 4532, 4044, 13, 13, 13, 29871, 13, 1762, 1234, 320, 25029, 29912, 29878, 29939, 29901, 479, 397, 24974, 29913, 591, 22229, 278, 1353, 310, 25741, 322, 8359, 15717, 639, 1629, 322, 639, 3186, 10640, 29889, 32001, 1334, 2198, 278, 7625, 2582, 297, 320, 29907, 999, 29912, 1003, 29901, 9902, 29913, 408, 1023, 5096, 287, 2594, 24469, 29892, 6445, 1629, 368, 2867, 3204, 29879, 363, 25741, 322, 15717, 8307, 29889, 32001, 7569, 2594, 11524, 263, 1629, 322, 338, 8877, 287, 297, 269, 29399, 6445, 278, 9063, 29914, 8921, 11959, 363, 1269, 310, 278, 3186, 12786, 310, 320, 29907, 999, 29912, 1003, 29901, 11526, 1958, 29913, 297, 393, 1629, 29889, 32001, 1763, 4772, 714, 27801, 2861, 304, 805, 272, 26538, 17737, 29560, 29892, 297, 278, 4148, 8727, 591, 871, 2050, 15717, 2534, 26869, 472, 3203, 29871, 29945, 25741, 297, 263, 2183, 1629, 29889, 32001, 5806, 5366, 1747, 534, 1975, 297, 278, 24469, 6456, 393, 278, 3001, 3694, 310, 25741, 322, 15717, 6548, 18709, 9247, 975, 931, 29889, 32001, 10133, 363, 278, 937, 2440, 297, 278, 24469, 29892, 278, 1353, 310, 848, 3291, 297, 777, 3186, 12786, 508, 367, 14154, 2319, 29892, 411, 8178, 27721, 373, 278, 25806, 310, 534, 1975, 29889, 32001, 320, 26956, 29912, 7999, 12122, 6894, 537, 975, 931, 29913, 13, 13, 3563, 497, 29892, 278, 2498, 534, 355, 5692, 304, 367, 393, 278, 320, 4534, 29912, 479, 12122, 6894, 537, 297, 970, 775, 338, 10231, 6177, 4644, 6813, 322, 4092, 5136, 630, 1009, 4954, 24130, 749, 4907, 2745, 278, 7256, 310, 278, 29871, 29929, 29900]} {'input_ids': [1, 320, 2042, 29912, 25898, 29913, 13, 29905, 1643, 29912, 3471, 29901, 23333, 29913, 13, 13, 29905, 7278, 29912, 29954, 1581, 6894, 537, 1118, 470, 901, 4049, 967, 10225, 727, 974, 29892, 4249, 27138, 304, 13, 20415, 5849, 14188, 756, 1063, 26606, 12399, 297, 7786, 2440, 29889, 32001, 512, 13, 1595, 16311, 29892, 278, 10122, 310, 29892, 9545, 310, 29892, 322, 6795, 1004, 25414, 363, 320, 7278, 29912, 26098, 13, 29871, 24003, 29913, 297, 12362, 29914, 6585, 7562, 18540, 313, 5800, 1799, 29897, 505, 4520, 263, 3287, 310, 8570, 13, 957, 278, 4940, 316, 6332, 2651, 2036, 29912, 29881, 16093, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 3855, 5871, 29906, 29900, 29896, 29900, 29895, 311, 29893, 2770, 29892, 13, 29871, 1055, 29888, 375, 29906, 29900, 29896, 29906, 5041, 267, 29892, 413, 434, 305, 1358, 29906, 29900, 29896, 29906, 26098, 29888, 2209, 29892, 19723, 5475, 4979, 29906, 29900, 29896, 29946, 26098, 29892, 13, 29871, 697, 309, 29906, 29900, 29896, 29953, 311, 5365, 550, 332, 6950, 29892, 696, 7586, 29906, 29900, 29896, 29953, 29893, 2770, 7192, 2209, 29892, 1935, 15044, 29906, 29900, 29896, 29955, 26098, 29892, 13, 29871, 503, 562, 305, 3350, 492, 29906, 29900, 29906, 29896, 26098, 1836, 32001, 320, 7278, 29912, 7999, 12122, 6894, 537, 29913, 338, 373, 278, 916, 1361, 278, 13, 14380, 310, 6894, 537, 393, 380, 1567, 515, 27138, 297, 777, 5534, 6354, 6421, 13, 3166, 1422, 3186, 12786, 322, 4185, 1973, 29889, 32001, 26304, 6894, 537, 297, 18322, 1799, 756, 4520, 13774, 2217, 8570, 297, 21344, 368, 13, 13129, 29889, 32001, 512, 3153, 29892, 1550, 3031, 979, 18994, 29899, 6707, 322, 13, 3149, 29899, 262, 29899, 2230, 18350, 29899, 7052, 11898, 310, 278, 1737, 12122, 1677, 1144, 310, 18322, 1799, 13, 21570, 29560, 1863, 2651, 2036, 29912, 12443, 10578, 29906, 29900, 29900, 29945, 5062, 11235, 29892, 13260, 333, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 13, 29871, 2594, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29892, 1850, 29882, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29892, 696, 7586, 29906, 29900, 29896, 29946, 7610, 345, 2941, 271, 24541, 29892, 13, 29871, 281, 496, 29879, 29906, 29900, 29906, 29896, 2209, 479, 5275, 1118, 2919, 29899, 7052, 25579, 979, 11898, 310, 278, 1737, 12122, 13, 12574, 310, 18322, 1799, 17737, 29560, 526, 1603, 10225, 292, 29889, 32001, 10506, 263, 4323, 23378, 13, 18609, 2133, 723, 367, 5407, 304, 1871, 1602, 12112, 4475, 304, 5534, 13, 25431, 10907, 2651, 2036, 29912, 2276, 5824, 19982, 29906, 29900, 29900, 29955, 23705, 1338, 29893, 996, 29913, 322, 298, 8491, 16650, 583, 297, 278, 13, 19678, 15483, 313, 1806, 29897, 9999, 29892, 408, 1532, 408, 29126, 2114, 950, 2472, 13, 517, 278, 2553, 1078, 373, 278, 17407, 10879, 322, 5374, 3002, 310, 18322, 1799, 2820, 278, 3186, 29889, 32001, 320, 26956, 29912, 1323, 3224, 29879, 29913, 13, 13, 3047, 445, 664, 591, 29126, 304, 3802, 445, 17261, 491, 7512, 292, 320, 4534, 29912, 1552, 937, 13, 29871, 25579, 979, 6559, 310, 278, 1737, 12122, 3978, 310, 17737, 29560, 304, 970, 775, 13, 29871, 975, 29871, 29945, 29900, 2440, 5003, 32001, 26321, 29892, 591, 3867, 263, 758, 2576, 3821, 1234, 304, 278, 13, 23031, 292, 5925, 1139, 29901, 13, 29905, 463, 29912, 690, 2842, 12470, 29913, 13, 29871, 3645, 607, 3186, 12786, 437, 15717, 310, 970, 368, 3625, 25741, 2041, 515, 13, 29871, 322, 920, 756, 372, 3939, 975, 278, 4940, 29871, 29945, 29900, 2440, 29973, 32001, 320, 1643, 29912, 29878, 29939, 29901, 479, 397, 24974, 29913, 13, 29905, 355, 29912, 690, 2842, 12470, 29913, 13, 4806, 671, 408, 8783, 278, 320, 23066, 29950, 29914, 18871, 2651, 2036, 29912, 29879, 1332, 29875, 4569, 29906, 29900, 29896, 29955, 29913, 322, 27599, 515, 372, 13, 29906, 29889, 29906, 24464, 29905, 13007, 25741, 3190, 2347, 515, 29871, 29896, 29953, 29900, 7284, 29905, 13007, 9279, 322, 4148, 287, 491, 13, 29946, 29941, 7284, 29905, 13007, 15717, 2645, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29896, 931, 3785, 29889, 32001, 1334, 1737, 324, 542, 403, 18777, 304, 13, 29905, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 29892, 773, 408, 18470, 4876, 4234, 775, 2246, 29899, 5563, 21904, 313, 617, 29911, 10249, 29879, 29897, 322, 29871, 13, 8921, 313, 4102, 29914, 4230, 29897, 2983, 9401, 411, 1024, 18822, 2820, 278, 3186, 29892, 322, 17998, 1283, 7224, 29871, 13, 1195, 287, 515, 9063, 15562, 29889, 32001, 1334, 1284, 10757, 310, 278, 4688, 8022, 749, 310, 4644, 6813, 297, 1722, 2752, 13, 20415, 29892, 2678, 8772, 491, 4092, 29889, 32001, 2860, 393, 3785, 29892, 278, 1737, 12122, 6894, 537, 29871, 13, 262, 970, 775, 756, 1063, 21003, 10231, 29889, 32001, 1334, 884, 12439, 8018, 15839, 528, 17741, 13, 12817, 304, 278, 1095, 310, 278, 8291, 6415, 29129, 322, 278, 7910, 310, 14137, 4631, 4135, 297, 13, 23369, 1705, 322, 4275, 14325, 29892, 408, 1532, 408, 310, 2545, 1664, 17292, 28342, 763, 25539, 1608, 322, 13, 25719, 10298, 4822, 10916, 313, 6727, 16783, 29914, 331, 16783, 467, 32001, 320, 26956, 29912, 1469, 20847, 3097, 5003, 32001, 319, 1634, 1414, 3577, 363, 445, 5650, 338, 3625, 515, 796, 264, 8144, 472, 13, 29905, 2271, 29912, 991, 597, 1867, 29875, 29889, 990, 29914, 29896, 29900, 29889, 29945, 29906, 29947, 29896, 29914, 2256, 8144, 29889, 29953, 29941, 29929, 29900, 29941, 29945, 29945, 29913, 2651, 2036, 29912, 3445, 1414, 29899, 5113, 1836, 32001, 320, 2042, 29912, 9662, 630, 5244, 29913, 13, 29905, 1643, 29912, 3471, 29901, 12817, 29913, 13, 13, 29933, 720, 4688, 322, 7786, 1736, 2651, 2036, 29912, 12443, 10578, 29906, 29900, 29900, 29945, 5062, 11235, 29892, 13260, 333, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 13, 29871, 696, 7586, 29906, 29900, 29896, 29946, 7610, 345, 2941, 271, 24541, 29892, 697, 309, 29906, 29900, 29896, 29953, 311, 5365, 550, 332, 6950, 29913, 505, 2931, 1891, 278, 13, 479, 5275, 310, 12362, 29914, 6585, 7562, 18540, 313, 5800, 1799, 29897, 773, 320, 7278, 29912, 6734, 26946, 952, 1118, 13, 4716, 3867, 1880, 29899, 29567, 6089, 541, 526, 9078, 297, 2159, 313, 29906, 29899, 29945, 5940, 29968, 18777, 29897, 13, 392, 508, 367, 4768, 1463, 491, 5221, 424, 23460, 29889, 32001, 512, 29871, 29906, 29900, 29900, 29947, 2261, 801, 2681, 634, 394, 8770, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29913, 18043, 263, 3031, 979, 13, 16961, 29899, 7052, 313, 1454, 278, 931, 29897, 6559, 373, 18322, 1799, 320, 7278, 29912, 479, 5275, 773, 1375, 292, 7047, 13, 29871, 28914, 313, 4345, 29934, 29897, 13698, 1836, 32001, 2688, 29537, 287, 278, 3978, 310, 29871, 29896, 5940, 29924, 17737, 29560, 13, 4746, 278, 7562, 2831, 479, 1404, 2566, 322, 611, 6504, 1051, 3190, 3145, 975, 278, 13, 29896, 29929, 29929, 29929, 489, 29906, 29900, 29900, 29945, 3785, 29892, 773, 408, 18470, 2472, 2788, 304, 1749, 29879, 29901, 4876, 21904, 13, 392, 17998, 1283, 7224, 29889, 32001, 450, 12399, 3785, 313, 29955, 2440, 29897, 297, 2651, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29913, 338, 20511, 1135, 29871, 13, 5816, 338, 12399, 297, 278, 2198, 5650, 313, 29945, 29900, 2440, 29897, 322, 278, 848, 8974, 526, 29871, 13, 27489, 873, 1422, 29936, 411, 393, 297, 3458, 29892, 1749, 2582, 1510, 263, 10029, 7200, 14978, 310, 29871, 13, 15654, 273, 325, 13217, 29940, 2072, 3082, 20706, 29889, 32001, 7280, 29190, 936, 664, 515, 29871, 29906, 29900, 29896, 29900, 491, 14619, 29882, 371, 4099, 29894, 322, 13, 29950, 309, 1372, 2651, 2036, 29912, 29873, 19426, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29913, 4023, 29894, 2868, 1583, 29899, 7099, 433, 1127, 1737, 12122, 13, 2029, 800, 310, 25492, 15303, 8304, 3598, 1494, 1009, 12368, 29892, 13, 15914, 292, 2472, 363, 779, 14850, 4535, 29892, 29955, 29900, 5940, 29968, 25492, 4160, 29889, 32001, 319, 1407, 7786, 13, 1287, 2651, 2036, 29912, 21686, 29879, 29906, 29900, 29906, 29896, 2209, 479, 5275, 29913, 491, 399, 496, 29879, 634, 394, 13217, 5349, 1737, 324, 542, 630, 4203, 263, 7284, 13, 28712, 16046, 4160, 29892, 2534, 26869, 472, 3203, 29871, 29896, 29900, 29900, 25741, 1269, 29892, 322, 1058, 13, 1311, 29899, 7099, 8663, 14354, 373, 1009, 25492, 28723, 29889, 32001, 5806, 278, 6559, 338, 13, 3149, 29899, 262, 29899, 2230, 408, 310, 29871, 29906, 29900, 29906, 29896, 29892, 278, 15717, 7252, 1009, 1284, 886, 13, 351, 475, 303, 2651, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29892, 1850, 29882, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29913, 304, 13, 18609, 675, 278, 14675, 310, 18322, 1799, 1737, 5275, 975, 278, 931, 15101, 845, 1862, 4586, 491, 13, 1552, 2211, 11898, 29889, 32001, 3831, 1965, 411, 3517, 29190, 936, 1736, 29892, 1749, 6559, 338, 1568, 7200, 6287, 5634, 29882, 5555, 13, 7054, 12339, 287, 29871, 29946, 29941, 7284, 29905, 13007, 15717, 310, 29871, 29906, 29889, 29906, 24464, 29905, 13007, 25741, 515, 29871, 29896, 29953, 29900, 7284, 29905, 13007, 13, 16418, 5634, 5426, 11267, 979, 975, 29871, 29945, 29900, 2440, 310, 970, 775, 20706, 3265, 1135, 13, 3149, 297, 931, 29892, 322, 884, 901, 2691, 29899, 3874, 1312, 313, 2541, 1629, 29899, 1609, 29899, 6360, 3803, 1070, 537, 975, 13, 1552, 8900, 3785, 467, 32001, 8108, 1189, 1711, 29892, 1749, 6559, 337, 3687, 373, 10079, 11264, 13, 3924, 313, 8257, 29903, 29897, 9063, 848, 3265, 1135, 7481, 29899, 7099, 433, 1127, 4423, 2472, 29889, 32001, 5901, 1736, 5634, 262, 3153, 278, 664, 491, 8432, 2651, 2036, 29912, 18386, 709, 29906, 29900, 29896, 29941, 2209, 29881, 24974, 29913, 13, 392, 29892, 901, 10325, 29892, 390, 579, 468, 29875, 634, 394, 8770, 2036, 29912, 29878, 579, 468, 29875, 29906, 29900, 29896, 29953, 479, 711, 3173, 29892, 13, 29871, 364, 579, 468, 29875, 29906, 29900, 29896, 29947, 479, 711, 3173, 29892, 544, 1648, 29906, 29900, 29906, 29896, 479, 468, 1581, 29881, 24974, 29913, 5634, 17532, 12399, 1737, 12122, 13, 29905, 7278, 29912, 29881, 24974, 322, 24003, 1118, 474, 29889, 29872, 1696, 278, 15834, 304, 607, 278, 3978, 310, 18322, 1799, 13, 17426, 6602, 1009, 11465, 1230, 14137, 14188, 29889, 32001, 512, 445, 664, 591, 2931, 1891, 1737, 12122, 6894, 537, 297, 970, 775, 363, 278, 937, 13, 2230, 472, 445, 6287, 29892, 1716, 297, 4958, 310, 17737, 29560, 322, 15500, 3785, 29889, 32001, 1334, 437, 13, 1333, 22002, 280, 278, 24003, 10696, 29892, 541, 3867, 29190, 936, 848, 322, 1284, 886, 393, 508, 367, 13, 280, 369, 4063, 304, 393, 1095, 408, 5434, 664, 29889, 32001, 320, 7278, 29912, 12756, 7047, 21639, 29913, 2651, 2036, 29912, 2276, 5824, 19982, 29906, 29900, 29900, 29955, 23705, 1338, 29893, 996, 29913, 338, 278, 13, 1491, 29899, 2671, 310, 7047, 21639, 393, 756, 29537, 287, 278, 18066, 267, 310, 21640, 13, 6734, 24771, 13149, 635, 29892, 3704, 278, 2702, 5932, 310, 920, 304, 5376, 13, 2541, 1737, 12122, 6894, 537, 2651, 2036, 29912, 13376, 303, 456, 29906, 29900, 29900, 29953, 10945, 3359, 29892, 1424, 29440, 29906, 29900, 29896, 29946, 23027, 5933, 1836, 32001, 3826, 3076, 2678, 278, 2198, 6559, 8128, 10757, 393, 508, 367, 1304, 29892, 297, 278, 13, 14940, 1206, 310, 970, 775, 322, 472, 263, 1407, 2919, 6287, 29892, 304, 11539, 607, 13, 14032, 4637, 310, 5534, 7047, 21639, 505, 9820, 484, 15774, 29889, 32001, 320, 2042, 29912, 4062, 3002, 29913, 13, 29905, 1643, 29912, 3471, 29901, 5696, 29913, 13, 13, 13, 29905, 1482, 361, 29905, 361, 29887, 798, 386, 1003, 29871, 320, 29887, 798, 386, 1003, 3009, 13, 29905, 361, 29887, 798, 386, 1003, 13, 29905, 463, 29912, 4532, 29913, 13, 29871, 320, 7313, 29961, 2103, 2013, 4914, 2103, 3199, 6360, 368, 29899, 2055, 1169, 29913, 13, 29871, 320, 6671, 29912, 12883, 368, 970, 25741, 975, 931, 313, 1188, 6287, 467, 13, 29913, 32001, 320, 1643, 29912, 1003, 29901, 29887, 798, 386, 29913, 13, 29905, 355, 29912, 4532, 29913, 13, 29905, 7241, 13, 13, 29905, 26956, 29912, 16390, 24541, 29913, 13, 13, 4806, 27387, 515, 320, 23066, 29950, 29914, 2651, 2036, 29912, 29879, 1332, 29899, 1516, 29878, 29906, 29900, 29896, 29929, 29899, 24713, 29913, 599, 25741, 3190, 2347, 2745, 320, 25832, 1964, 579, 1523, 2415, 2539, 6294, 32001, 2688, 5253, 304, 320, 14573, 5261, 1169, 22131, 29914, 25741, 29892, 5412, 491, 317, 15715, 29896, 15882, 29892, 4023, 29894, 2868, 515, 320, 25832, 1299, 7288, 5261, 1169, 797, 7068, 29914, 970, 9279, 6421, 515, 4655, 5849, 363, 2710, 313, 28712, 16046, 29892, 11786, 28632, 29892, 2992, 1846, 32001, 322, 3577, 28914, 313, 10251, 713, 29892, 10772, 2227, 29892, 405, 13427, 29892, 2992, 6250, 32001, 1876, 1169, 297, 278, 8783, 526, 491, 320, 25832, 6344, 2806, 943, 22131, 29914, 15717, 29892, 5412, 491, 779, 6990, 29938, 978, 29892, 4876, 4535, 5854, 29938, 11000, 29889, 32001, 450, 8783, 2996, 408, 1023, 1104, 1288, 6131, 29892, 697, 363, 25741, 322, 697, 363, 15717, 29892, 411, 278, 4642, 29371, 278, 7480, 3025, 263, 9117, 1820, 29889, 32001, 320, 361, 5426, 13, 9760, 1948, 297, 278, 9063, 1591, 3743, 278, 1494, 4235, 29901, 9063, 317, 15715, 29896, 15882, 29892, 4148, 322, 844, 5171, 5335, 342, 15092, 29892, 4148, 322, 844, 5171, 2893, 14903, 313, 20275, 16750, 278, 4148, 1591, 467, 32001, 450, 21578, 1546, 9063, 15717, 322, 9063, 2153, 2041, 515, 11786, 29892, 607, 6511, 304, 9063, 263, 1735, 4148, 287, 491, 4856, 1683, 29889, 32001, 1152, 445, 6559, 591, 21309, 373, 15717, 322, 17262, 9063, 2153, 29892, 408, 278, 4328, 1546, 278, 1023, 338, 451, 8018, 363, 1749, 5925, 5155, 322, 278, 5253, 310, 25741, 411, 263, 844, 5171, 916, 1135, 967, 4148, 338, 3480, 3473, 1821, 29889, 32001, 320, 7241, 13, 2831, 1269, 6251, 297, 278, 4148, 1591, 591, 505, 4148, 2989, 1024, 322, 4876, 408, 1023, 5004, 6031, 310, 10650, 6262, 29889, 32001, 1334, 6206, 2411, 15273, 1821, 470, 18325, 519, 2983, 393, 29901, 526, 451, 1602, 397, 519, 408, 18351, 29899, 29947, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12283, 7099, 397, 519, 29914, 4148, 2983, 6206, 511, 526, 4876, 14157, 2012, 310, 2983, 3441, 25832, 6344, 2806, 943, 29934, 29885, 9823, 29914, 4954, 7039, 4907, 511, 5718, 310, 871, 9654, 4890, 3441, 25832, 6344, 2806, 943, 29934, 29885, 10358, 804, 23201, 1712, 901, 1135, 29871, 29896, 29900, 8958, 1661, 29899, 1026, 2153, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12283, 15670, 23201, 526, 5520, 1135, 29871, 29896, 29900, 29900, 4890, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12229, 549, 12495, 32001, 2860, 21166, 29892, 1048, 320, 25832, 6344, 2806, 943, 29925, 15273, 1821, 2052, 307, 29916, 29914, 15717, 3441, 25832, 6344, 2806, 943, 29925, 15273, 1821, 29925, 312, 29914, 310, 278, 2847, 8783, 29897, 9488, 363, 4340, 7418, 29889, 32001, 3940, 393, 278, 5253, 310, 970, 775, 25741, 313, 392, 15717, 29897, 11122, 297, 278, 13, 11228, 8783, 25088, 18709, 9247, 975, 13, 2230, 2651, 2036, 29912, 29879, 1332, 29899, 771, 854, 749, 29899, 331, 344, 1012, 361, 29887, 798, 386, 1003, 29892, 408, 4318, 363, 25741, 297, 13, 29905, 29907, 999, 29912, 1003, 29901, 29887, 798, 386, 1012, 2870, 29901, 515, 395, 29896, 29900, 29985, 29946, 29938, 25741, 297, 29871, 29896, 29929, 29955, 29896, 29892, 304, 395, 29896, 29900, 29985, 29953, 29938, 297, 29871, 29896, 29929, 29929, 29947, 29892, 304, 13, 284, 3242, 395, 29896, 29900, 29985, 29929, 29938, 297, 29871, 29906, 29900, 29906, 29900, 29905, 7241, 29889, 32001, 1094, 263, 17004, 278, 8900, 534, 1975, 10331, 304, 367, 901, 13, 13844, 297, 7786, 1602, 3076, 1135, 297, 29871, 29946, 29900, 29974, 1629, 29899, 1025, 6743, 29892, 2861, 304, 13964, 4586, 373, 13, 735, 1112, 9247, 7200, 23093, 29889, 32001, 320, 26956, 29912, 7999, 324, 10610, 29913, 13, 13, 29905, 463, 29912, 4532, 29913, 13, 29871, 320, 9525, 13, 29871, 320, 7313, 29961, 24049, 29892, 15450, 29922, 29953, 4912, 29871, 29953, 4912, 29871, 29900, 29871, 29900, 29892, 2103, 2013, 16292, 3199, 1491, 1727, 1080, 29899, 2470, 29913, 13, 29871, 320, 6671, 29912, 1576, 320, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 1304, 408, 1737, 324, 10610, 22525, 5003, 32001, 320, 1643, 29912, 1003, 29901, 11526, 1958, 29913, 13, 29905, 355, 29912, 4532, 29913, 13, 13, 2887, 1737, 324, 10610, 22525, 591, 671, 11758, 3186, 12786, 10723, 515, 278, 3303, 18269, 1737, 359, 305, 2004, 2651, 2036, 29912, 348, 29896, 29929, 29929, 29929, 479, 359, 305, 2004, 1836, 32001, 1763, 4772, 2432, 3381, 491, 2919, 10916, 313, 29872, 29889, 29887, 1696, 7551, 470, 12710, 29897, 2629, 11758, 12786, 29892, 591, 19412, 322, 6219, 777, 12786, 2729, 373, 1737, 12122, 23203, 537, 322, 278, 19383, 310, 758, 331, 8946, 16375, 29769, 5680, 29892, 1316, 408, 19182, 4086, 29889, 32001, 320, 29907, 999, 29912, 1003, 29901, 11526, 1958, 29913, 3697, 278, 2186, 1051, 310, 320, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 1304, 408, 1737, 324, 10610, 22525, 297, 445, 6559, 29889, 32001, 1879, 324, 10610, 310, 9063, 15717, 304, 3186, 12786, 3913, 278, 1023, 19595, 653, 13698, 9129, 297, 2651, 2036, 29912, 293, 344, 29899, 344, 275, 29899, 29906, 29900, 29906, 29906, 29899, 26098, 1118, 23359, 337, 13998, 2400, 29889, 32001, 450, 937, 697, 337, 3687, 373, 278, 4234, 775, 2246, 29899, 5563, 5354, 313, 617, 29911, 10249, 29897, 310, 4876, 14157, 23892, 515, 9063, 15562, 29892, 321, 29889, 29887, 1696, 320, 16266, 26139, 1341, 1118, 320, 16266, 26139, 582, 1118, 320, 16266, 26139, 18038, 1118, 2992, 29889, 32001, 1334, 4687, 515, 278, 306, 2190, 29909, 1051, 310, 13548, 2931, 21759, 29911, 10249, 29879, 2651, 2036, 29912, 6011, 29899, 29883, 312, 430, 29913, 322, 7522, 20545, 1269, 6590, 20123, 304, 263, 3646, 3186, 5120, 29889, 32001, 450, 1473, 1737, 324, 10610, 11043, 3913, 278, 17998, 9210, 310, 9063, 5335, 342, 15092, 313, 29872, 29889, 29887, 1696, 17998, 29899, 29900, 29945, 29901, 29900, 29900, 29897, 322, 4148, 2983, 304, 8161, 278, 1556, 5517, 3186, 5120, 310, 278, 9063, 4148, 29889, 32001, 1152, 1269, 17998, 9210, 591, 8161, 263, 1051, 310, 15878, 7600, 313, 13509, 29892, 2106, 29892, 470, 14278, 20123, 29897, 297, 278, 3186, 393, 29892, 472, 278, 931, 310, 393, 9063, 29892, 750, 393, 17998, 9210, 29936, 9063, 931, 338, 1820, 1244, 29892, 408, 4234, 17998, 1283, 7224, 13100, 975, 931, 2861, 304, 29431, 3620, 29889, 32001, 1763, 1207, 445, 3683, 3381, 591, 671, 278, 306, 2190, 29909, 931, 10640, 2566, 2651, 2036, 29912, 17559, 1272, 1836, 32001, 1987, 591, 3566, 304, 1269, 2058, 263, 8158, 393, 4332, 1973, 278, 4188, 22342, 393, 263, 2183, 4148, 1024, 338, 17443, 310, 372, 29889, 32001, 1763, 445, 1095, 591, 671, 278, 28297, 29890, 15451, 8783, 310, 278, 29511, 310, 278, 1556, 3619, 937, 322, 3942, 2983, 607, 29892, 439, 11427, 515, 2651, 2036, 29912, 1079, 29890, 799, 29899, 7039, 6177, 2802, 277, 12181, 4954, 16123, 2247, 278, 26368, 5528, 5084, 310, 363, 264, 1280, 322, 269, 595, 1280, 7371, 515, 263, 2566, 310, 320, 1949, 29912, 29946, 29871, 29900, 29946, 29946, 29871, 29945, 29946, 29953, 29871, 29929, 29941, 29947, 29913, 2305, 313, 29945, 29945, 29889, 29945, 8958, 310, 8471, 2305, 297, 29871, 29906, 29900, 29896, 29946, 467, 32001, 1094, 310, 3839, 29871, 29906, 29900, 29896, 29929, 372, 18469, 320, 1949, 29912, 29906, 29955, 29871, 29953, 29953, 29906, 29871, 29947, 29900, 29896, 29913, 363, 264, 1280, 322, 320, 1949, 29912, 29906, 29955, 29871, 29906, 29900, 29953, 29871, 29947, 29906, 29896, 29913, 269, 595, 1280, 297, 29871, 29906, 29941, 29953, 24894, 8977, 1080, 29889, 4907, 29913, 32001, 1094, 297, 1749, 8783, 15717, 526, 2989, 1024, 6031, 313, 29878, 1624, 1135, 6219, 491, 937, 29914, 11922, 1024, 511, 591, 937, 5993, 675, 2983, 313, 1609, 1999, 1331, 322, 1206, 3620, 29897, 322, 769, 16280, 5375, 18897, 297, 1716, 937, 322, 3942, 2983, 10868, 8857, 29889, 32001, 1152, 1269, 1543, 1476, 297, 1024, 8857, 591, 22932, 278, 2058, 4665, 29905, 6661, 1333, 24212, 8875, 491, 278, 1024, 10868, 304, 4017, 263, 5645, 393, 338, 29839, 304, 278, 1353, 310, 12407, 24638, 393, 1024, 313, 6979, 29897, 297, 278, 2702, 2058, 29889, 32001, 320, 6661, 1333, 300, 1062, 29912, 1762, 4017, 4665, 2025, 1338, 5634, 294, 278, 17837, 310, 4954, 6689, 4907, 338, 25745, 23724, 29901, 2989, 10916, 325, 13217, 29879, 29399, 310, 2919, 10916, 805, 9450, 2999, 931, 29920, 2873, 5634, 705, 671, 263, 29544, 310, 7601, 8974, 313, 29872, 29889, 29887, 1696, 5874, 28007, 511, 322, 1661, 29899, 16072, 6743, 313, 29872, 29889, 29887, 1696, 14109, 7456, 467, 29913, 32001, 1334, 2533, 445, 4377, 363, 599, 3161, 304, 4017, 263, 2058, 8158, 29892, 17140, 701, 411, 263, 1051, 310, 779, 6990, 29938, 6689, 29892, 8158, 4535, 5854, 29938, 11000, 29889, 32001, 1334, 769, 8877, 445, 1051, 491, 278, 3186, 5120, 393, 263, 2058, 14393, 304, 322, 2533, 278, 8158, 363, 599, 278, 7600, 297, 1269, 5120, 304, 4017, 385, 12463, 8158, 29892, 6590, 304, 278, 4188, 22342, 393, 278, 9063, 14393, 304, 263, 2183, 3186, 5120, 29889, 32001, 1334, 3566, 278, 6257, 9063, 408, 6421, 515, 278, 3186, 5120, 411, 278, 9939, 8158, 29889, 32001, 450, 4876, 29899, 6707, 11043, 9378, 414, 515, 278, 9078, 322, 443, 5521, 8362, 671, 310, 21759, 29911, 10249, 29879, 29901, 1556, 18777, 671, 10035, 323, 10249, 29879, 1316, 408, 320, 16266, 26139, 510, 1118, 320, 16266, 26139, 990, 1118, 470, 320, 16266, 26139, 1212, 1836, 32001, 12808, 445, 947, 451, 3799, 26018, 4822, 20542, 29901, 3148, 29899, 6707, 18777, 29892, 363, 1342, 29892, 671, 278, 320, 16266, 26139, 375, 29913, 21759, 29911, 10249, 1568, 901, 269, 2495, 290, 368, 1135, 1009, 7824, 6795, 20895, 29889, 32001, 1551, 278, 916, 1361, 278, 9210, 29914, 978, 29899, 6707, 11043, 337, 3687, 373, 278, 17998, 9210, 310, 278, 9063, 5335, 342, 15092, 29889, 32001, 16809, 304, 5780, 22920, 373, 13897, 731, 14340, 29892, 263, 2919, 1353, 310, 25741, 297, 278, 8783, 756, 385, 17998, 9210, 5186, 304, 5225, 29889, 32001, 910, 6602, 29879, 3109, 7786, 25741, 3441, 14573, 5261, 1169, 29911, 29999, 29999, 13985, 1349, 681, 392, 27418, 6478, 29914, 310, 29871, 29906, 29900, 29906, 29900, 29879, 25741, 505, 263, 5225, 9210, 29897, 1135, 9642, 6743, 3441, 14573, 5261, 1169, 29911, 29999, 29999, 13985, 1349, 681, 392, 29914, 297, 29871, 29906, 29900, 29900, 29900, 467, 32001, 1094, 263, 1121, 278, 9210, 29914, 978, 29899, 6707, 11043, 1033, 1095, 701, 6459, 292, 263, 2919, 6232, 310, 9642, 25741, 408, 4148, 287, 491, 11715, 18777, 29892, 322, 304, 263, 3109, 261, 15834, 4092, 550, 29889, 32001, 1763, 6795, 1438, 5626, 591, 14405, 278, 1023, 1737, 324, 10610, 13698, 4208, 491, 15399, 278, 9210, 29914, 978, 29899, 6707, 13698, 304, 599, 25741, 411, 263, 1661, 29899, 9171, 17998, 9210, 29892, 322, 278, 4876, 29899, 6707, 373, 304, 599, 916, 25741, 29889, 32001, 320, 2042, 29912, 12191, 322, 8565, 1558, 291, 29913, 13, 29905, 1643, 29912, 3471, 29901, 9902, 29913, 13, 13, 29905, 463, 29912, 4532, 4044, 13, 29871, 320, 9525, 13, 29871, 320, 7313, 29961, 2103, 2013, 16292, 3199, 1429, 287, 29889, 5140, 29913, 13, 29871, 320, 6671, 29912, 29934, 20819, 310, 25741, 313, 27215, 29897, 322, 6136, 15717, 313, 22503, 29897, 491, 3186, 10640, 975, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29900, 3785, 5003, 32001, 320, 9868, 29961, 14732, 3199, 7264, 287, 2594, 8727, 6445, 278, 3186, 10640, 364, 2219, 359, 363, 25741, 322, 15717, 975, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29900, 3785, 5003, 32001, 320, 1643, 29912, 1003, 29901, 9902, 29913, 13, 29905, 355, 29912, 4532, 4044, 13, 13, 13, 29871, 13, 1762, 1234, 320, 25029, 29912, 29878, 29939, 29901, 479, 397, 24974, 29913, 591, 22229, 278, 1353, 310, 25741, 322, 8359, 15717, 639, 1629, 322, 639, 3186, 10640, 29889, 32001, 1334, 2198, 278, 7625, 2582, 297, 320, 29907, 999, 29912, 1003, 29901, 9902, 29913, 408, 1023, 5096, 287, 2594, 24469, 29892, 6445, 1629, 368, 2867, 3204, 29879, 363, 25741, 322, 15717, 8307, 29889, 32001, 7569, 2594, 11524, 263, 1629, 322, 338, 8877, 287, 297, 269, 29399, 6445, 278, 9063, 29914, 8921, 11959, 363, 1269, 310, 278, 3186, 12786, 310, 320, 29907, 999, 29912, 1003, 29901, 11526, 1958, 29913, 297, 393, 1629, 29889, 32001, 1763, 4772, 714, 27801, 2861, 304, 805, 272, 26538, 17737, 29560, 29892, 297, 278, 4148, 8727, 591, 871, 2050, 15717, 2534, 26869, 472, 3203, 29871, 29945, 25741, 297, 263, 2183, 1629, 29889, 32001, 5806, 5366, 1747, 534, 1975, 297, 278, 24469, 6456, 393, 278, 3001, 3694, 310, 25741, 322, 15717, 6548, 18709, 9247, 975, 931, 29889, 32001, 10133, 363, 278, 937, 2440, 297, 278, 24469, 29892, 278, 1353, 310, 848, 3291, 297, 777, 3186, 12786, 508, 367, 14154, 2319, 29892, 411, 8178, 27721, 373, 278, 25806, 310, 534, 1975, 29889, 32001, 320, 26956, 29912, 7999, 12122, 6894, 537, 975, 931, 29913, 13, 13, 3563, 497, 29892, 278, 2498, 534, 355, 5692, 304, 367, 393, 278, 320, 4534, 29912, 479, 12122, 6894, 537, 297, 970, 775, 338, 10231, 6177, 4644, 6813, 322, 4092, 5136, 630, 1009, 4954, 24130, 749, 4907, 2745, 278, 7256, 310, 278, 29871, 29929, 29900], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [1, 320, 2042, 29912, 25898, 29913, 13, 29905, 1643, 29912, 3471, 29901, 23333, 29913, 13, 13, 29905, 7278, 29912, 29954, 1581, 6894, 537, 1118, 470, 901, 4049, 967, 10225, 727, 974, 29892, 4249, 27138, 304, 13, 20415, 5849, 14188, 756, 1063, 26606, 12399, 297, 7786, 2440, 29889, 32001, 512, 13, 1595, 16311, 29892, 278, 10122, 310, 29892, 9545, 310, 29892, 322, 6795, 1004, 25414, 363, 320, 7278, 29912, 26098, 13, 29871, 24003, 29913, 297, 12362, 29914, 6585, 7562, 18540, 313, 5800, 1799, 29897, 505, 4520, 263, 3287, 310, 8570, 13, 957, 278, 4940, 316, 6332, 2651, 2036, 29912, 29881, 16093, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 3855, 5871, 29906, 29900, 29896, 29900, 29895, 311, 29893, 2770, 29892, 13, 29871, 1055, 29888, 375, 29906, 29900, 29896, 29906, 5041, 267, 29892, 413, 434, 305, 1358, 29906, 29900, 29896, 29906, 26098, 29888, 2209, 29892, 19723, 5475, 4979, 29906, 29900, 29896, 29946, 26098, 29892, 13, 29871, 697, 309, 29906, 29900, 29896, 29953, 311, 5365, 550, 332, 6950, 29892, 696, 7586, 29906, 29900, 29896, 29953, 29893, 2770, 7192, 2209, 29892, 1935, 15044, 29906, 29900, 29896, 29955, 26098, 29892, 13, 29871, 503, 562, 305, 3350, 492, 29906, 29900, 29906, 29896, 26098, 1836, 32001, 320, 7278, 29912, 7999, 12122, 6894, 537, 29913, 338, 373, 278, 916, 1361, 278, 13, 14380, 310, 6894, 537, 393, 380, 1567, 515, 27138, 297, 777, 5534, 6354, 6421, 13, 3166, 1422, 3186, 12786, 322, 4185, 1973, 29889, 32001, 26304, 6894, 537, 297, 18322, 1799, 756, 4520, 13774, 2217, 8570, 297, 21344, 368, 13, 13129, 29889, 32001, 512, 3153, 29892, 1550, 3031, 979, 18994, 29899, 6707, 322, 13, 3149, 29899, 262, 29899, 2230, 18350, 29899, 7052, 11898, 310, 278, 1737, 12122, 1677, 1144, 310, 18322, 1799, 13, 21570, 29560, 1863, 2651, 2036, 29912, 12443, 10578, 29906, 29900, 29900, 29945, 5062, 11235, 29892, 13260, 333, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 13, 29871, 2594, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29892, 1850, 29882, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29892, 696, 7586, 29906, 29900, 29896, 29946, 7610, 345, 2941, 271, 24541, 29892, 13, 29871, 281, 496, 29879, 29906, 29900, 29906, 29896, 2209, 479, 5275, 1118, 2919, 29899, 7052, 25579, 979, 11898, 310, 278, 1737, 12122, 13, 12574, 310, 18322, 1799, 17737, 29560, 526, 1603, 10225, 292, 29889, 32001, 10506, 263, 4323, 23378, 13, 18609, 2133, 723, 367, 5407, 304, 1871, 1602, 12112, 4475, 304, 5534, 13, 25431, 10907, 2651, 2036, 29912, 2276, 5824, 19982, 29906, 29900, 29900, 29955, 23705, 1338, 29893, 996, 29913, 322, 298, 8491, 16650, 583, 297, 278, 13, 19678, 15483, 313, 1806, 29897, 9999, 29892, 408, 1532, 408, 29126, 2114, 950, 2472, 13, 517, 278, 2553, 1078, 373, 278, 17407, 10879, 322, 5374, 3002, 310, 18322, 1799, 2820, 278, 3186, 29889, 32001, 320, 26956, 29912, 1323, 3224, 29879, 29913, 13, 13, 3047, 445, 664, 591, 29126, 304, 3802, 445, 17261, 491, 7512, 292, 320, 4534, 29912, 1552, 937, 13, 29871, 25579, 979, 6559, 310, 278, 1737, 12122, 3978, 310, 17737, 29560, 304, 970, 775, 13, 29871, 975, 29871, 29945, 29900, 2440, 5003, 32001, 26321, 29892, 591, 3867, 263, 758, 2576, 3821, 1234, 304, 278, 13, 23031, 292, 5925, 1139, 29901, 13, 29905, 463, 29912, 690, 2842, 12470, 29913, 13, 29871, 3645, 607, 3186, 12786, 437, 15717, 310, 970, 368, 3625, 25741, 2041, 515, 13, 29871, 322, 920, 756, 372, 3939, 975, 278, 4940, 29871, 29945, 29900, 2440, 29973, 32001, 320, 1643, 29912, 29878, 29939, 29901, 479, 397, 24974, 29913, 13, 29905, 355, 29912, 690, 2842, 12470, 29913, 13, 4806, 671, 408, 8783, 278, 320, 23066, 29950, 29914, 18871, 2651, 2036, 29912, 29879, 1332, 29875, 4569, 29906, 29900, 29896, 29955, 29913, 322, 27599, 515, 372, 13, 29906, 29889, 29906, 24464, 29905, 13007, 25741, 3190, 2347, 515, 29871, 29896, 29953, 29900, 7284, 29905, 13007, 9279, 322, 4148, 287, 491, 13, 29946, 29941, 7284, 29905, 13007, 15717, 2645, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29896, 931, 3785, 29889, 32001, 1334, 1737, 324, 542, 403, 18777, 304, 13, 29905, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 29892, 773, 408, 18470, 4876, 4234, 775, 2246, 29899, 5563, 21904, 313, 617, 29911, 10249, 29879, 29897, 322, 29871, 13, 8921, 313, 4102, 29914, 4230, 29897, 2983, 9401, 411, 1024, 18822, 2820, 278, 3186, 29892, 322, 17998, 1283, 7224, 29871, 13, 1195, 287, 515, 9063, 15562, 29889, 32001, 1334, 1284, 10757, 310, 278, 4688, 8022, 749, 310, 4644, 6813, 297, 1722, 2752, 13, 20415, 29892, 2678, 8772, 491, 4092, 29889, 32001, 2860, 393, 3785, 29892, 278, 1737, 12122, 6894, 537, 29871, 13, 262, 970, 775, 756, 1063, 21003, 10231, 29889, 32001, 1334, 884, 12439, 8018, 15839, 528, 17741, 13, 12817, 304, 278, 1095, 310, 278, 8291, 6415, 29129, 322, 278, 7910, 310, 14137, 4631, 4135, 297, 13, 23369, 1705, 322, 4275, 14325, 29892, 408, 1532, 408, 310, 2545, 1664, 17292, 28342, 763, 25539, 1608, 322, 13, 25719, 10298, 4822, 10916, 313, 6727, 16783, 29914, 331, 16783, 467, 32001, 320, 26956, 29912, 1469, 20847, 3097, 5003, 32001, 319, 1634, 1414, 3577, 363, 445, 5650, 338, 3625, 515, 796, 264, 8144, 472, 13, 29905, 2271, 29912, 991, 597, 1867, 29875, 29889, 990, 29914, 29896, 29900, 29889, 29945, 29906, 29947, 29896, 29914, 2256, 8144, 29889, 29953, 29941, 29929, 29900, 29941, 29945, 29945, 29913, 2651, 2036, 29912, 3445, 1414, 29899, 5113, 1836, 32001, 320, 2042, 29912, 9662, 630, 5244, 29913, 13, 29905, 1643, 29912, 3471, 29901, 12817, 29913, 13, 13, 29933, 720, 4688, 322, 7786, 1736, 2651, 2036, 29912, 12443, 10578, 29906, 29900, 29900, 29945, 5062, 11235, 29892, 13260, 333, 29906, 29900, 29900, 29947, 29888, 2209, 3359, 29879, 29892, 13, 29871, 696, 7586, 29906, 29900, 29896, 29946, 7610, 345, 2941, 271, 24541, 29892, 697, 309, 29906, 29900, 29896, 29953, 311, 5365, 550, 332, 6950, 29913, 505, 2931, 1891, 278, 13, 479, 5275, 310, 12362, 29914, 6585, 7562, 18540, 313, 5800, 1799, 29897, 773, 320, 7278, 29912, 6734, 26946, 952, 1118, 13, 4716, 3867, 1880, 29899, 29567, 6089, 541, 526, 9078, 297, 2159, 313, 29906, 29899, 29945, 5940, 29968, 18777, 29897, 13, 392, 508, 367, 4768, 1463, 491, 5221, 424, 23460, 29889, 32001, 512, 29871, 29906, 29900, 29900, 29947, 2261, 801, 2681, 634, 394, 8770, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29913, 18043, 263, 3031, 979, 13, 16961, 29899, 7052, 313, 1454, 278, 931, 29897, 6559, 373, 18322, 1799, 320, 7278, 29912, 479, 5275, 773, 1375, 292, 7047, 13, 29871, 28914, 313, 4345, 29934, 29897, 13698, 1836, 32001, 2688, 29537, 287, 278, 3978, 310, 29871, 29896, 5940, 29924, 17737, 29560, 13, 4746, 278, 7562, 2831, 479, 1404, 2566, 322, 611, 6504, 1051, 3190, 3145, 975, 278, 13, 29896, 29929, 29929, 29929, 489, 29906, 29900, 29900, 29945, 3785, 29892, 773, 408, 18470, 2472, 2788, 304, 1749, 29879, 29901, 4876, 21904, 13, 392, 17998, 1283, 7224, 29889, 32001, 450, 12399, 3785, 313, 29955, 2440, 29897, 297, 2651, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29913, 338, 20511, 1135, 29871, 13, 5816, 338, 12399, 297, 278, 2198, 5650, 313, 29945, 29900, 2440, 29897, 322, 278, 848, 8974, 526, 29871, 13, 27489, 873, 1422, 29936, 411, 393, 297, 3458, 29892, 1749, 2582, 1510, 263, 10029, 7200, 14978, 310, 29871, 13, 15654, 273, 325, 13217, 29940, 2072, 3082, 20706, 29889, 32001, 7280, 29190, 936, 664, 515, 29871, 29906, 29900, 29896, 29900, 491, 14619, 29882, 371, 4099, 29894, 322, 13, 29950, 309, 1372, 2651, 2036, 29912, 29873, 19426, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29913, 4023, 29894, 2868, 1583, 29899, 7099, 433, 1127, 1737, 12122, 13, 2029, 800, 310, 25492, 15303, 8304, 3598, 1494, 1009, 12368, 29892, 13, 15914, 292, 2472, 363, 779, 14850, 4535, 29892, 29955, 29900, 5940, 29968, 25492, 4160, 29889, 32001, 319, 1407, 7786, 13, 1287, 2651, 2036, 29912, 21686, 29879, 29906, 29900, 29906, 29896, 2209, 479, 5275, 29913, 491, 399, 496, 29879, 634, 394, 13217, 5349, 1737, 324, 542, 630, 4203, 263, 7284, 13, 28712, 16046, 4160, 29892, 2534, 26869, 472, 3203, 29871, 29896, 29900, 29900, 25741, 1269, 29892, 322, 1058, 13, 1311, 29899, 7099, 8663, 14354, 373, 1009, 25492, 28723, 29889, 32001, 5806, 278, 6559, 338, 13, 3149, 29899, 262, 29899, 2230, 408, 310, 29871, 29906, 29900, 29906, 29896, 29892, 278, 15717, 7252, 1009, 1284, 886, 13, 351, 475, 303, 2651, 2036, 29912, 1646, 801, 2681, 29906, 29900, 29900, 29947, 479, 397, 24974, 29892, 1850, 29882, 371, 4099, 29894, 29906, 29900, 29896, 29900, 2209, 479, 5275, 29913, 304, 13, 18609, 675, 278, 14675, 310, 18322, 1799, 1737, 5275, 975, 278, 931, 15101, 845, 1862, 4586, 491, 13, 1552, 2211, 11898, 29889, 32001, 3831, 1965, 411, 3517, 29190, 936, 1736, 29892, 1749, 6559, 338, 1568, 7200, 6287, 5634, 29882, 5555, 13, 7054, 12339, 287, 29871, 29946, 29941, 7284, 29905, 13007, 15717, 310, 29871, 29906, 29889, 29906, 24464, 29905, 13007, 25741, 515, 29871, 29896, 29953, 29900, 7284, 29905, 13007, 13, 16418, 5634, 5426, 11267, 979, 975, 29871, 29945, 29900, 2440, 310, 970, 775, 20706, 3265, 1135, 13, 3149, 297, 931, 29892, 322, 884, 901, 2691, 29899, 3874, 1312, 313, 2541, 1629, 29899, 1609, 29899, 6360, 3803, 1070, 537, 975, 13, 1552, 8900, 3785, 467, 32001, 8108, 1189, 1711, 29892, 1749, 6559, 337, 3687, 373, 10079, 11264, 13, 3924, 313, 8257, 29903, 29897, 9063, 848, 3265, 1135, 7481, 29899, 7099, 433, 1127, 4423, 2472, 29889, 32001, 5901, 1736, 5634, 262, 3153, 278, 664, 491, 8432, 2651, 2036, 29912, 18386, 709, 29906, 29900, 29896, 29941, 2209, 29881, 24974, 29913, 13, 392, 29892, 901, 10325, 29892, 390, 579, 468, 29875, 634, 394, 8770, 2036, 29912, 29878, 579, 468, 29875, 29906, 29900, 29896, 29953, 479, 711, 3173, 29892, 13, 29871, 364, 579, 468, 29875, 29906, 29900, 29896, 29947, 479, 711, 3173, 29892, 544, 1648, 29906, 29900, 29906, 29896, 479, 468, 1581, 29881, 24974, 29913, 5634, 17532, 12399, 1737, 12122, 13, 29905, 7278, 29912, 29881, 24974, 322, 24003, 1118, 474, 29889, 29872, 1696, 278, 15834, 304, 607, 278, 3978, 310, 18322, 1799, 13, 17426, 6602, 1009, 11465, 1230, 14137, 14188, 29889, 32001, 512, 445, 664, 591, 2931, 1891, 1737, 12122, 6894, 537, 297, 970, 775, 363, 278, 937, 13, 2230, 472, 445, 6287, 29892, 1716, 297, 4958, 310, 17737, 29560, 322, 15500, 3785, 29889, 32001, 1334, 437, 13, 1333, 22002, 280, 278, 24003, 10696, 29892, 541, 3867, 29190, 936, 848, 322, 1284, 886, 393, 508, 367, 13, 280, 369, 4063, 304, 393, 1095, 408, 5434, 664, 29889, 32001, 320, 7278, 29912, 12756, 7047, 21639, 29913, 2651, 2036, 29912, 2276, 5824, 19982, 29906, 29900, 29900, 29955, 23705, 1338, 29893, 996, 29913, 338, 278, 13, 1491, 29899, 2671, 310, 7047, 21639, 393, 756, 29537, 287, 278, 18066, 267, 310, 21640, 13, 6734, 24771, 13149, 635, 29892, 3704, 278, 2702, 5932, 310, 920, 304, 5376, 13, 2541, 1737, 12122, 6894, 537, 2651, 2036, 29912, 13376, 303, 456, 29906, 29900, 29900, 29953, 10945, 3359, 29892, 1424, 29440, 29906, 29900, 29896, 29946, 23027, 5933, 1836, 32001, 3826, 3076, 2678, 278, 2198, 6559, 8128, 10757, 393, 508, 367, 1304, 29892, 297, 278, 13, 14940, 1206, 310, 970, 775, 322, 472, 263, 1407, 2919, 6287, 29892, 304, 11539, 607, 13, 14032, 4637, 310, 5534, 7047, 21639, 505, 9820, 484, 15774, 29889, 32001, 320, 2042, 29912, 4062, 3002, 29913, 13, 29905, 1643, 29912, 3471, 29901, 5696, 29913, 13, 13, 13, 29905, 1482, 361, 29905, 361, 29887, 798, 386, 1003, 29871, 320, 29887, 798, 386, 1003, 3009, 13, 29905, 361, 29887, 798, 386, 1003, 13, 29905, 463, 29912, 4532, 29913, 13, 29871, 320, 7313, 29961, 2103, 2013, 4914, 2103, 3199, 6360, 368, 29899, 2055, 1169, 29913, 13, 29871, 320, 6671, 29912, 12883, 368, 970, 25741, 975, 931, 313, 1188, 6287, 467, 13, 29913, 32001, 320, 1643, 29912, 1003, 29901, 29887, 798, 386, 29913, 13, 29905, 355, 29912, 4532, 29913, 13, 29905, 7241, 13, 13, 29905, 26956, 29912, 16390, 24541, 29913, 13, 13, 4806, 27387, 515, 320, 23066, 29950, 29914, 2651, 2036, 29912, 29879, 1332, 29899, 1516, 29878, 29906, 29900, 29896, 29929, 29899, 24713, 29913, 599, 25741, 3190, 2347, 2745, 320, 25832, 1964, 579, 1523, 2415, 2539, 6294, 32001, 2688, 5253, 304, 320, 14573, 5261, 1169, 22131, 29914, 25741, 29892, 5412, 491, 317, 15715, 29896, 15882, 29892, 4023, 29894, 2868, 515, 320, 25832, 1299, 7288, 5261, 1169, 797, 7068, 29914, 970, 9279, 6421, 515, 4655, 5849, 363, 2710, 313, 28712, 16046, 29892, 11786, 28632, 29892, 2992, 1846, 32001, 322, 3577, 28914, 313, 10251, 713, 29892, 10772, 2227, 29892, 405, 13427, 29892, 2992, 6250, 32001, 1876, 1169, 297, 278, 8783, 526, 491, 320, 25832, 6344, 2806, 943, 22131, 29914, 15717, 29892, 5412, 491, 779, 6990, 29938, 978, 29892, 4876, 4535, 5854, 29938, 11000, 29889, 32001, 450, 8783, 2996, 408, 1023, 1104, 1288, 6131, 29892, 697, 363, 25741, 322, 697, 363, 15717, 29892, 411, 278, 4642, 29371, 278, 7480, 3025, 263, 9117, 1820, 29889, 32001, 320, 361, 5426, 13, 9760, 1948, 297, 278, 9063, 1591, 3743, 278, 1494, 4235, 29901, 9063, 317, 15715, 29896, 15882, 29892, 4148, 322, 844, 5171, 5335, 342, 15092, 29892, 4148, 322, 844, 5171, 2893, 14903, 313, 20275, 16750, 278, 4148, 1591, 467, 32001, 450, 21578, 1546, 9063, 15717, 322, 9063, 2153, 2041, 515, 11786, 29892, 607, 6511, 304, 9063, 263, 1735, 4148, 287, 491, 4856, 1683, 29889, 32001, 1152, 445, 6559, 591, 21309, 373, 15717, 322, 17262, 9063, 2153, 29892, 408, 278, 4328, 1546, 278, 1023, 338, 451, 8018, 363, 1749, 5925, 5155, 322, 278, 5253, 310, 25741, 411, 263, 844, 5171, 916, 1135, 967, 4148, 338, 3480, 3473, 1821, 29889, 32001, 320, 7241, 13, 2831, 1269, 6251, 297, 278, 4148, 1591, 591, 505, 4148, 2989, 1024, 322, 4876, 408, 1023, 5004, 6031, 310, 10650, 6262, 29889, 32001, 1334, 6206, 2411, 15273, 1821, 470, 18325, 519, 2983, 393, 29901, 526, 451, 1602, 397, 519, 408, 18351, 29899, 29947, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12283, 7099, 397, 519, 29914, 4148, 2983, 6206, 511, 526, 4876, 14157, 2012, 310, 2983, 3441, 25832, 6344, 2806, 943, 29934, 29885, 9823, 29914, 4954, 7039, 4907, 511, 5718, 310, 871, 9654, 4890, 3441, 25832, 6344, 2806, 943, 29934, 29885, 10358, 804, 23201, 1712, 901, 1135, 29871, 29896, 29900, 8958, 1661, 29899, 1026, 2153, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12283, 15670, 23201, 526, 5520, 1135, 29871, 29896, 29900, 29900, 4890, 3441, 25832, 6344, 2806, 943, 29934, 29885, 12229, 549, 12495, 32001, 2860, 21166, 29892, 1048, 320, 25832, 6344, 2806, 943, 29925, 15273, 1821, 2052, 307, 29916, 29914, 15717, 3441, 25832, 6344, 2806, 943, 29925, 15273, 1821, 29925, 312, 29914, 310, 278, 2847, 8783, 29897, 9488, 363, 4340, 7418, 29889, 32001, 3940, 393, 278, 5253, 310, 970, 775, 25741, 313, 392, 15717, 29897, 11122, 297, 278, 13, 11228, 8783, 25088, 18709, 9247, 975, 13, 2230, 2651, 2036, 29912, 29879, 1332, 29899, 771, 854, 749, 29899, 331, 344, 1012, 361, 29887, 798, 386, 1003, 29892, 408, 4318, 363, 25741, 297, 13, 29905, 29907, 999, 29912, 1003, 29901, 29887, 798, 386, 1012, 2870, 29901, 515, 395, 29896, 29900, 29985, 29946, 29938, 25741, 297, 29871, 29896, 29929, 29955, 29896, 29892, 304, 395, 29896, 29900, 29985, 29953, 29938, 297, 29871, 29896, 29929, 29929, 29947, 29892, 304, 13, 284, 3242, 395, 29896, 29900, 29985, 29929, 29938, 297, 29871, 29906, 29900, 29906, 29900, 29905, 7241, 29889, 32001, 1094, 263, 17004, 278, 8900, 534, 1975, 10331, 304, 367, 901, 13, 13844, 297, 7786, 1602, 3076, 1135, 297, 29871, 29946, 29900, 29974, 1629, 29899, 1025, 6743, 29892, 2861, 304, 13964, 4586, 373, 13, 735, 1112, 9247, 7200, 23093, 29889, 32001, 320, 26956, 29912, 7999, 324, 10610, 29913, 13, 13, 29905, 463, 29912, 4532, 29913, 13, 29871, 320, 9525, 13, 29871, 320, 7313, 29961, 24049, 29892, 15450, 29922, 29953, 4912, 29871, 29953, 4912, 29871, 29900, 29871, 29900, 29892, 2103, 2013, 16292, 3199, 1491, 1727, 1080, 29899, 2470, 29913, 13, 29871, 320, 6671, 29912, 1576, 320, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 1304, 408, 1737, 324, 10610, 22525, 5003, 32001, 320, 1643, 29912, 1003, 29901, 11526, 1958, 29913, 13, 29905, 355, 29912, 4532, 29913, 13, 13, 2887, 1737, 324, 10610, 22525, 591, 671, 11758, 3186, 12786, 10723, 515, 278, 3303, 18269, 1737, 359, 305, 2004, 2651, 2036, 29912, 348, 29896, 29929, 29929, 29929, 479, 359, 305, 2004, 1836, 32001, 1763, 4772, 2432, 3381, 491, 2919, 10916, 313, 29872, 29889, 29887, 1696, 7551, 470, 12710, 29897, 2629, 11758, 12786, 29892, 591, 19412, 322, 6219, 777, 12786, 2729, 373, 1737, 12122, 23203, 537, 322, 278, 19383, 310, 758, 331, 8946, 16375, 29769, 5680, 29892, 1316, 408, 19182, 4086, 29889, 32001, 320, 29907, 999, 29912, 1003, 29901, 11526, 1958, 29913, 3697, 278, 2186, 1051, 310, 320, 14573, 14058, 4597, 1080, 29914, 3186, 12786, 1304, 408, 1737, 324, 10610, 22525, 297, 445, 6559, 29889, 32001, 1879, 324, 10610, 310, 9063, 15717, 304, 3186, 12786, 3913, 278, 1023, 19595, 653, 13698, 9129, 297, 2651, 2036, 29912, 293, 344, 29899, 344, 275, 29899, 29906, 29900, 29906, 29906, 29899, 26098, 1118, 23359, 337, 13998, 2400, 29889, 32001, 450, 937, 697, 337, 3687, 373, 278, 4234, 775, 2246, 29899, 5563, 5354, 313, 617, 29911, 10249, 29897, 310, 4876, 14157, 23892, 515, 9063, 15562, 29892, 321, 29889, 29887, 1696, 320, 16266, 26139, 1341, 1118, 320, 16266, 26139, 582, 1118, 320, 16266, 26139, 18038, 1118, 2992, 29889, 32001, 1334, 4687, 515, 278, 306, 2190, 29909, 1051, 310, 13548, 2931, 21759, 29911, 10249, 29879, 2651, 2036, 29912, 6011, 29899, 29883, 312, 430, 29913, 322, 7522, 20545, 1269, 6590, 20123, 304, 263, 3646, 3186, 5120, 29889, 32001, 450, 1473, 1737, 324, 10610, 11043, 3913, 278, 17998, 9210, 310, 9063, 5335, 342, 15092, 313, 29872, 29889, 29887, 1696, 17998, 29899, 29900, 29945, 29901, 29900, 29900, 29897, 322, 4148, 2983, 304, 8161, 278, 1556, 5517, 3186, 5120, 310, 278, 9063, 4148, 29889, 32001, 1152, 1269, 17998, 9210, 591, 8161, 263, 1051, 310, 15878, 7600, 313, 13509, 29892, 2106, 29892, 470, 14278, 20123, 29897, 297, 278, 3186, 393, 29892, 472, 278, 931, 310, 393, 9063, 29892, 750, 393, 17998, 9210, 29936, 9063, 931, 338, 1820, 1244, 29892, 408, 4234, 17998, 1283, 7224, 13100, 975, 931, 2861, 304, 29431, 3620, 29889, 32001, 1763, 1207, 445, 3683, 3381, 591, 671, 278, 306, 2190, 29909, 931, 10640, 2566, 2651, 2036, 29912, 17559, 1272, 1836, 32001, 1987, 591, 3566, 304, 1269, 2058, 263, 8158, 393, 4332, 1973, 278, 4188, 22342, 393, 263, 2183, 4148, 1024, 338, 17443, 310, 372, 29889, 32001, 1763, 445, 1095, 591, 671, 278, 28297, 29890, 15451, 8783, 310, 278, 29511, 310, 278, 1556, 3619, 937, 322, 3942, 2983, 607, 29892, 439, 11427, 515, 2651, 2036, 29912, 1079, 29890, 799, 29899, 7039, 6177, 2802, 277, 12181, 4954, 16123, 2247, 278, 26368, 5528, 5084, 310, 363, 264, 1280, 322, 269, 595, 1280, 7371, 515, 263, 2566, 310, 320, 1949, 29912, 29946, 29871, 29900, 29946, 29946, 29871, 29945, 29946, 29953, 29871, 29929, 29941, 29947, 29913, 2305, 313, 29945, 29945, 29889, 29945, 8958, 310, 8471, 2305, 297, 29871, 29906, 29900, 29896, 29946, 467, 32001, 1094, 310, 3839, 29871, 29906, 29900, 29896, 29929, 372, 18469, 320, 1949, 29912, 29906, 29955, 29871, 29953, 29953, 29906, 29871, 29947, 29900, 29896, 29913, 363, 264, 1280, 322, 320, 1949, 29912, 29906, 29955, 29871, 29906, 29900, 29953, 29871, 29947, 29906, 29896, 29913, 269, 595, 1280, 297, 29871, 29906, 29941, 29953, 24894, 8977, 1080, 29889, 4907, 29913, 32001, 1094, 297, 1749, 8783, 15717, 526, 2989, 1024, 6031, 313, 29878, 1624, 1135, 6219, 491, 937, 29914, 11922, 1024, 511, 591, 937, 5993, 675, 2983, 313, 1609, 1999, 1331, 322, 1206, 3620, 29897, 322, 769, 16280, 5375, 18897, 297, 1716, 937, 322, 3942, 2983, 10868, 8857, 29889, 32001, 1152, 1269, 1543, 1476, 297, 1024, 8857, 591, 22932, 278, 2058, 4665, 29905, 6661, 1333, 24212, 8875, 491, 278, 1024, 10868, 304, 4017, 263, 5645, 393, 338, 29839, 304, 278, 1353, 310, 12407, 24638, 393, 1024, 313, 6979, 29897, 297, 278, 2702, 2058, 29889, 32001, 320, 6661, 1333, 300, 1062, 29912, 1762, 4017, 4665, 2025, 1338, 5634, 294, 278, 17837, 310, 4954, 6689, 4907, 338, 25745, 23724, 29901, 2989, 10916, 325, 13217, 29879, 29399, 310, 2919, 10916, 805, 9450, 2999, 931, 29920, 2873, 5634, 705, 671, 263, 29544, 310, 7601, 8974, 313, 29872, 29889, 29887, 1696, 5874, 28007, 511, 322, 1661, 29899, 16072, 6743, 313, 29872, 29889, 29887, 1696, 14109, 7456, 467, 29913, 32001, 1334, 2533, 445, 4377, 363, 599, 3161, 304, 4017, 263, 2058, 8158, 29892, 17140, 701, 411, 263, 1051, 310, 779, 6990, 29938, 6689, 29892, 8158, 4535, 5854, 29938, 11000, 29889, 32001, 1334, 769, 8877, 445, 1051, 491, 278, 3186, 5120, 393, 263, 2058, 14393, 304, 322, 2533, 278, 8158, 363, 599, 278, 7600, 297, 1269, 5120, 304, 4017, 385, 12463, 8158, 29892, 6590, 304, 278, 4188, 22342, 393, 278, 9063, 14393, 304, 263, 2183, 3186, 5120, 29889, 32001, 1334, 3566, 278, 6257, 9063, 408, 6421, 515, 278, 3186, 5120, 411, 278, 9939, 8158, 29889, 32001, 450, 4876, 29899, 6707, 11043, 9378, 414, 515, 278, 9078, 322, 443, 5521, 8362, 671, 310, 21759, 29911, 10249, 29879, 29901, 1556, 18777, 671, 10035, 323, 10249, 29879, 1316, 408, 320, 16266, 26139, 510, 1118, 320, 16266, 26139, 990, 1118, 470, 320, 16266, 26139, 1212, 1836, 32001, 12808, 445, 947, 451, 3799, 26018, 4822, 20542, 29901, 3148, 29899, 6707, 18777, 29892, 363, 1342, 29892, 671, 278, 320, 16266, 26139, 375, 29913, 21759, 29911, 10249, 1568, 901, 269, 2495, 290, 368, 1135, 1009, 7824, 6795, 20895, 29889, 32001, 1551, 278, 916, 1361, 278, 9210, 29914, 978, 29899, 6707, 11043, 337, 3687, 373, 278, 17998, 9210, 310, 278, 9063, 5335, 342, 15092, 29889, 32001, 16809, 304, 5780, 22920, 373, 13897, 731, 14340, 29892, 263, 2919, 1353, 310, 25741, 297, 278, 8783, 756, 385, 17998, 9210, 5186, 304, 5225, 29889, 32001, 910, 6602, 29879, 3109, 7786, 25741, 3441, 14573, 5261, 1169, 29911, 29999, 29999, 13985, 1349, 681, 392, 27418, 6478, 29914, 310, 29871, 29906, 29900, 29906, 29900, 29879, 25741, 505, 263, 5225, 9210, 29897, 1135, 9642, 6743, 3441, 14573, 5261, 1169, 29911, 29999, 29999, 13985, 1349, 681, 392, 29914, 297, 29871, 29906, 29900, 29900, 29900, 467, 32001, 1094, 263, 1121, 278, 9210, 29914, 978, 29899, 6707, 11043, 1033, 1095, 701, 6459, 292, 263, 2919, 6232, 310, 9642, 25741, 408, 4148, 287, 491, 11715, 18777, 29892, 322, 304, 263, 3109, 261, 15834, 4092, 550, 29889, 32001, 1763, 6795, 1438, 5626, 591, 14405, 278, 1023, 1737, 324, 10610, 13698, 4208, 491, 15399, 278, 9210, 29914, 978, 29899, 6707, 13698, 304, 599, 25741, 411, 263, 1661, 29899, 9171, 17998, 9210, 29892, 322, 278, 4876, 29899, 6707, 373, 304, 599, 916, 25741, 29889, 32001, 320, 2042, 29912, 12191, 322, 8565, 1558, 291, 29913, 13, 29905, 1643, 29912, 3471, 29901, 9902, 29913, 13, 13, 29905, 463, 29912, 4532, 4044, 13, 29871, 320, 9525, 13, 29871, 320, 7313, 29961, 2103, 2013, 16292, 3199, 1429, 287, 29889, 5140, 29913, 13, 29871, 320, 6671, 29912, 29934, 20819, 310, 25741, 313, 27215, 29897, 322, 6136, 15717, 313, 22503, 29897, 491, 3186, 10640, 975, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29900, 3785, 5003, 32001, 320, 9868, 29961, 14732, 3199, 7264, 287, 2594, 8727, 6445, 278, 3186, 10640, 364, 2219, 359, 363, 25741, 322, 15717, 975, 278, 29871, 29896, 29929, 29955, 29896, 489, 29906, 29900, 29906, 29900, 3785, 5003, 32001, 320, 1643, 29912, 1003, 29901, 9902, 29913, 13, 29905, 355, 29912, 4532, 4044, 13, 13, 13, 29871, 13, 1762, 1234, 320, 25029, 29912, 29878, 29939, 29901, 479, 397, 24974, 29913, 591, 22229, 278, 1353, 310, 25741, 322, 8359, 15717, 639, 1629, 322, 639, 3186, 10640, 29889, 32001, 1334, 2198, 278, 7625, 2582, 297, 320, 29907, 999, 29912, 1003, 29901, 9902, 29913, 408, 1023, 5096, 287, 2594, 24469, 29892, 6445, 1629, 368, 2867, 3204, 29879, 363, 25741, 322, 15717, 8307, 29889, 32001, 7569, 2594, 11524, 263, 1629, 322, 338, 8877, 287, 297, 269, 29399, 6445, 278, 9063, 29914, 8921, 11959, 363, 1269, 310, 278, 3186, 12786, 310, 320, 29907, 999, 29912, 1003, 29901, 11526, 1958, 29913, 297, 393, 1629, 29889, 32001, 1763, 4772, 714, 27801, 2861, 304, 805, 272, 26538, 17737, 29560, 29892, 297, 278, 4148, 8727, 591, 871, 2050, 15717, 2534, 26869, 472, 3203, 29871, 29945, 25741, 297, 263, 2183, 1629, 29889, 32001, 5806, 5366, 1747, 534, 1975, 297, 278, 24469, 6456, 393, 278, 3001, 3694, 310, 25741, 322, 15717, 6548, 18709, 9247, 975, 931, 29889, 32001, 10133, 363, 278, 937, 2440, 297, 278, 24469, 29892, 278, 1353, 310, 848, 3291, 297, 777, 3186, 12786, 508, 367, 14154, 2319, 29892, 411, 8178, 27721, 373, 278, 25806, 310, 534, 1975, 29889, 32001, 320, 26956, 29912, 7999, 12122, 6894, 537, 975, 931, 29913, 13, 13, 3563, 497, 29892, 278, 2498, 534, 355, 5692, 304, 367, 393, 278, 320, 4534, 29912, 479, 12122, 6894, 537, 297, 970, 775, 338, 10231, 6177, 4644, 6813, 322, 4092, 5136, 630, 1009, 4954, 24130, 749, 4907, 2745, 278, 7256, 310, 278, 29871, 29929, 29900]} Using /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Using /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Using /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Detected CUDA files, patching ldflags Emitting ninja build file /root/.cache/torch_extensions/py38_cu117/cpu_adam/build.ninja... Building extension moduleDetected CUDA files, patching ldflags Emitting ninja build file /root/.cache/torch_extensions/py38_cu117/cpu_adam/build.ninja..ninja: no work to do. Loading extension module cpu_adam... Time to load cpu_adam op: 1.2693920135498047 seconds Using /root/.cache/torch_extensions/py38_cninja: no work to do. Loading extension module cpu_adam... Time to load cpu_adam op: 1.283562421798706 seconds LUsing /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Using /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Using /root/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Detected CUDA files, patching ldflags Emitting ninja build file /root/.cache/torch_extensions/py38_cu117/cpu_adam/build.ninja... Building extension module cpu_adam... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. Loading extension module cpu_Detected CUDA files, patching ldflags Emitting ninja build file /root/.cache/torch_extensions/py38_cu117/cpu_adam/build.ninja... Building extension module cpu_adam... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. Loading extension module cpu_adam... Loading extension module cpu_adam... Time to load cpu_adam op: 1.4235050678253174 seconds Time to load cpu_adam op: 1.4118576049804688 seconds Loading extension module cpu_adam... Time to load cpu_adam op: 1.435286045074463 seconds Loading extension module cAdam Optimizer #0 is created with AVX2 arithmetic capability. Config: alpha=0.000020, betas=(0.900000, 0.999000), weight_decay=0.000000, adam_w=1 =1 nds Adam Optimizer #0 is created with AVX2 arithmetic capability. Config: alpha=0.000020, betas=(0.900000, 0.999000), weight_decay=0.000000, adam_w=1 Parameter Offload: Total persistent parameters: 414720 in 81 params [2024-01-29 11:45:27,328] [INFO] [utils.py:802:see_memory_usage] DeepSpeedZeRoOffload initialize [end] [2024-01-29 11:45:27,329] [INFO] [utils.py:803:see_memory_usage] MA 0.04 GB Max_MA 0.65 GB CA 1.61 GB Max_CA 2 GB [2024-01-29 11:45:27,329] [INFO] [utils.py:810:see_memory_usage] CPU Virtual Memory: used = 60.57 GB, percent = 6.0% [2024-01-29 11:45:27,436] [INFO] [utils.py:802:see_memory_usage] Before creating fp16 partitions [2024-01-29 11:45:27,437] [INFO] [utils.py:803:see_memory_usage] MA 0.04 GB Max_MA 0.04 GB CA 1.61 GB Max_CA 2 GB [2024-01-29 11:45:27,437] [INFO] [utils.py:810:see_memory_usage] CPU Virtual Memory: used = 60.6 GB, percent = 6.0% [2024-01-29 11:45:28,252] [INFO] [utils.py:802:see_memory_usage] After creating fp16 partitions: 1 [2024-01-29 11:45:28,253] [INFO] [utils.py:803:see_memory_usage] MA 0.04 GB Max_MA 0.04 GB CA 1.61 GB Max_CA 2 GB [2024-01-29 11:45:28,253] [INFO] [utils.py:810:see_memory_usage] CPU Virtual Memory: used = 71.17 GB, percent = 7.1% [2024-01-29 11:45:28,370] [INFO] [utils.py:802:see_memory_usage] Before creating fp32 partitions [2024-01-29 11:45:28,370] [INFO] [utils.py:803:see_memory_usage] MA 0.04 GB Max_MA 0.04 GB CA 1.61 GB Max_CA 2 GB [2024-01-29 11:45:28,371] [INFO] [utils.py:810:see_memory_usage] CPU Virtual Memory: used = 73.71 GB, percent = 7.3% [2024-01-29 11:45:29,205] [INFO] [utils.py:802:see_memory_usage] After creating fp32 partitions [2024-01-29 11:45:29,206] [INFO] [utils.py:803:see_memory_usage] MA 0.04 GB Max_MA 0.04 GB CA 1.61 GB Max_CA 2 GB [2024-01-29 11:45:29,206] [INFO] [utils.py:810:see_memory_usage] CPU Virtual Memory: used = 80.87 GB, percent = 8.0% [2024-01-29 11:45:29,317] [INFO] [utils.py:802:see_memory_usage] Before initializing optimizer states [2024-01-29 11:45:29,318] [INFO] [utils.py:803:see_memory_usage] MA 0.04 GB Max_MA 0.04 GB CA 1.61 GB Max_CA 2 GB [2024-01-29 11:45:29,318] [INFO] [utils.py:810:see_memory_usage] CPU Virtual Memory: used = 83.72 GB, percent = 8.3% [2024-01-29 11:45:32,840] [INFO] [utils.py:802:see_memory_usage] After initializing optimizer states [2024-01-29 11:45:32,841] [INFO] [utils.py:803:see_memory_usage] MA 0.04 GB Max_MA 0.04 GB CA 1.61 GB Max_CA 2 GB [2024-01-29 11:45:32,841] [INFO] [utils.py:810:see_memory_usage] CPU Virtual Memory: used = 121.44 GB, percent = 12.1% [2024-01-29 11:45:32,946] [INFO] [stage3.py:459:_setup_for_real_optimizer] optimizer state initialized [2024-01-29 11:45:36,614] [INFO] [utils.py:802:see_memory_usage] After initializing ZeRO optimizer [2024-01-29 11:45:36,615] [INFO] [utils.py:803:see_memory_usage] MA 0.09 GB Max_MA 0.7 GB CA 1.61 GB Max_CA 2 GB [2024-01-29 11:45:36,615] [INFO] [utils.py:810:see_memory_usage] CPU Virtual Memory: used = 129.78 GB, percent = 12.9% [2024-01-29 11:45:36,615] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Final Optimizer = adam [2024-01-29 11:45:36,615] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed using client callable to create LR scheduler [2024-01-29 11:45:36,616] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed LR Scheduler = [2024-01-201/29/2024 11:45:37 - WARNING - llama_sft_forward_thisversion_new - `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... 01/29/2024 11:45:37 - WARNING - llama_sft_forward_thisversion_new - `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... 01/29/2024 11:45:37 - WARNING - llama_sft_forward_thisversion_new - `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... 01/29/2024 11:45:37 - WARNING - llama_sft_forward_thisversion_new - `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... 01/29/2024 11:45:37 - WARNING - llama_sft_forward_thisversion_new - `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... 01/29/2024 11:45:37 - WARNING - llama_sft_forward_thisversion_new - `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... 01/29/2024 11:45:37 - WARNING - llama_sft_forward_thisversion_new - `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... 01/29/2024 11:45:37 - WARNING - llama_sft_forward_thisversion_new - `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... e, "start_profile_step": 3, "end_profile_step": 5, "tuner_type": "gridsearch", "tuner_early_stopping": 5, "tuner_num_trials": 50, "model_info_path": null, "mp_size": 1, "max_train_batch_size": null, "min_train_batch_size": 1, "max_train_micro_batch_size_per_gpu": 1.024000e+03, "min_train_micro_batch_size_per_gpu": 1, "num_tuning_micro_batch_sizes": 3 } [2024-01-29 11:45:36,618] [INFO] [config.py:972:print] bfloat16_enabled ............. True [2024-01-29 11:45:36,618] [INFO] [config.py:972:print] checkpoint_parallel_write_pipeline False [2024-01-29 11:45:36,618] [INFO] [config.py:972:print] checkpoint_tag_validation_enabled True [2024-01-29 11:45:36,618] [INFO] [config.py:972:print] checkpoint_tag_validation_fail False [2024-01-29 11:45:36,618] [INFO] [config.py:972:print] comms_config ................. [2024-01-29 11:45:36,618] [INFO] [config.py:972:print] communication_data_type ...... None [2024-01-29 11:45:36,618] [INFO] [config.py:972:print] compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}} [2024-01-29 11:45:36,618] [INFO] [config.py:972:print] curriculum_enabled_legacy .... False [2024-01-29 11:45:36,618] [INFO] [config.py:972:print] curriculum_params_legacy ..... False [2024-01-29 11:45:36,618] [INFO] [config.py:972:print] data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}} [2024-01-29 11:45:36,618] [INFO] [config.py:972:print] data_efficiency_enabled ...... False [2024-01-29 11:45:36,618] [INFO] [config.py:972:print] dataloader_drop_last ......... False [2024-01-29 11:45:36,618] [INFO] [config.py:972:print] disable_allgather ............ False [2024-01-29 11:45:36,618] [INFO] [config.py:972:print] dump_state ................... False [2024-01-29 11:45:36,618] [INFO] [config.py:972:print] dynamic_loss_scale_args ...... None [2024-01-29 11:45:36,618] [INFO] [config.py:972:print] eigenvalue_enabled ........... False [2024-01-29 11:45:36,618] [INFO] [config.py:972:print] eigenvalue_gas_boundary_resolution 1 [2024-01-29 11:45:36,618] [INFO] [config.py:972:print] eigenvalue_layer_name ........ bert.encoder.layer [2024-01-29 11:45:36,618] [INFO] [config.py:972:print] eigenvalue_layer_num ......... 0 [2024-01-29 11:45:36,618] [INFO] [config.py:972:print] eigenvalue_max_iter .......... 100 [2024-01-29 11:45:36,618] [INFO] [config.py:972:print] eigenvalue_stability ......... 1e-06 [2024-01-29 11:45:36,618] [INFO] [config.py:972:print] eigenvalue_tol ............... 0.01 [2024-01-29 11:45:36,618] [INFO] [config.py:972:print] eigenvalue_verbose ........... False [2024-01-29 11:45:36,618] [INFO] [config.py:972:print] elasticity_enabled ........... False [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] flops_profiler_config ........ { "enabled": false, "recompute_fwd_factor": 0.0, "profile_step": 1, "module_depth": -1, "top_modules": 1, "detailed": true, "output_file": null } [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] fp16_auto_cast ............... None [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] fp16_enabled ................. False [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] fp16_master_weights_and_gradients False [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] global_rank .................. 0 [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] grad_accum_dtype ............. None [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] gradient_accumulation_steps .. 16 [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] gradient_clipping ............ 1.0 [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] gradient_predivide_factor .... 1.0 [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8 [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] initial_dynamic_scale ........ 1 [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] load_universal_checkpoint .... False [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] loss_scale ................... 1.0 [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] memory_breakdown ............. False [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] mics_hierarchial_params_gather False [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] mics_shard_size .............. -1 [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=False [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] nebula_config ................ { "enabled": false, "persistent_storage_path": null, "persistent_time_interval": 100, "num_of_version_in_retention": 2, "enable_nebula_load": true, "load_path": null } [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] optimizer_legacy_fusion ...... False [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] optimizer_name ............... adam [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] optimizer_params ............. {'lr': 2e-05, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.0} [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0} [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] pld_enabled .................. False [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] pld_params ................... False [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] prescale_gradients ........... False [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] scheduler_name ............... None [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] scheduler_params ............. None [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] sparse_attention ............. None [2024-01-29 11:45:36,619] [INFO] [config.py:972:print] sparse_gradients_enabled ..... False [2024-01-29 11:45:36,620] [INFO] [config.py:972:print] steps_per_print .............. 1000 [2024-01-29 11:45:36,620] [INFO] [config.py:972:print] train_batch_size ............. 512 [2024-01-29 11:45:36,620] [INFO] [config.py:972:print] train_micro_batch_size_per_gpu 1 [2024-01-29 11:45:36,620] [INFO] [config.py:972:print] use_node_local_storage ....... False [2024-01-29 11:45:36,620] [INFO] [config.py:972:print] wall_clock_breakdown ......... False [2024-01-29 11:45:36,620] [INFO] [config.py:972:print] weight_quantization_config ... None [2024-01-29 11:45:36,620] [INFO] [config.py:972:print] world_size ................... 32 [2024-01-29 11:45:36,620] [INFO] [config.py:972:print] zero_allow_untested_optimizer False [2024-01-29 11:45:36,620] [INFO] [config.py:972:print] zero_config .................. stage=3 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=26214400 allgather_partitions=True allgather_bucket_size=500,000,000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=DeepSpeedZeroOffloadParamConfig(device='cpu', nvme_path=None, buffer_count=5, buffer_size=100,000,000, max_in_cpu=1,000,000,000, pin_memory=True) offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='cpu', nvme_path=None, buffer_count=4, pin_memory=True, pipeline=False, pipeline_read=False, pipeline_write=False, fast_init=False) sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=23592960 param_persistence_threshold=51200 model_persistence_threshold=sys.maxsize max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=True stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True [2024-01-29 11:45:36,620] [INFO] [config.py:972:print] zero_enabled ................. True [2024-01-29 11:45:36,620] [INFO] [config.py:972:print] zero_force_ds_cpu_optimizer .. True [2024-01-29 11:45:36,620] [INFO] [config.py:972:print] zero_optimization_stage ...... 3 [2024-01-29 11:45:36,620] [INFO] [config.py:958:print_user_config] json = { "optimizer": { "type": "Adam", "params": { "lr": 2e-05, "betas": [0.9, 0.999], "eps": 1e-08, "weight_decay": 0.0 } }, "bf16": { "enabled": true }, "zero_optimization": { "stage": 3, "offload_optimizer": { "device": "cpu", "pin_memory": true }, "offload_param": { "device": "cpu", "pin_memory": true }, "overlap_comm": true, "contiguous_gradients": true, "reduce_bucket_size": 2.621440e+07, "stage3_prefetch_bucket_size": 2.359296e+07, "stage3_param_persistence_threshold": 5.120000e+04, "sub_group_size": 1.000000e+09, "stage3_max_live_parameters": 1.000000e+09, "stage3_max_reuse_distance": 1.000000e+09, "stage3_gather_16bit_weights_on_model_save": true }, "gradient_accumulation_steps": 16, "gradient_clipping": 1.0, "steps_per_print": 1000, "train_batch_size": 512, "train_micro_batch_size_per_gpu": 1, "wall_clock_breakdown": false } [INFO|trainer.py:1755] 2024-01-29 11:45:36,622 >> ***** Running training ***** [INFO|trainer.py:1756] 2024-01-29 11:45:36,622 >> Num examples = 930514 [INFO|trainer.py:1757] 2024-01-29 11:45:36,622 >> Num Epochs = 1 [INFO|trainer.py:1758] 2024-01-29 11:45:36,622 >> Instantaneous batch size per device = 1 [INFO|trainer.py:1759] 2024-01-29 11:45:36,622 >> Total train batch size (w. parallel, distributed & accumulation) = 512 [INFO|trainer.py:1760] 2024-01-29 11:45:36,622 >> Gradient Accumulation steps = 16 [INFO|trainer.py:1761] 2024-01-29 11:45:36,622 >> Total optimization steps = 1817 [INFO|trainer.py:1762] 2024-01-29 11:45:36,624 >> Number of trainable parameters = 13015884800 0%| | 0/1817 [00:007->6 [1] 0/-ts-cbba87c5ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67026:81154 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:81152 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81155 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:81151 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:81149 [0] NCCL INFO Channel 00/02 : 0 3 10 15 14 13 12 9 8 11 18 23 22 21 20 17 16 19 26 31 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81153 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81156 [2] NCCL INFO Trees [0] 3/18/-1->2->-1 [1] 3/-1/-1->2->10 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:81149 [0] NCCL INFO Channel 01/02 : 0 3 10 15 14 13 12 9 8 11 18 23 22 21 20 17 16 19 26 31 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:81149 [0] NCCL INFO Trees [0] 1/-1/-1->0->7 [1] 1/-1/-1->0->7 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81155 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:20367 [2] NCCL INFO Channel 01/0 : 18[4b000] -> 23[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8015:20366 [0] NCCL INFO Channel 01/0 : 16[e000] -> 19[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8019:20368 [4] NCCL INFO Channel 01/0 : 20[93000] -> 17[13000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8020:20363 [5] NCCL INFO Channel 00/0 : 21[99000] -> 20[93000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8021:20364 [6] NCCL INFO Channel 00/0 : 22[cb000] -> 21[99000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8020:20363 [5] NCCL INFO Channel 01/0 : 21[99000] -> 20[93000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8021:20364 [6] NCCL INFO Channel 01/0 : 22[cb000] -> 21[99000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8022:20365 [7] NCCL INFO Channel 00/0 : 23[ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133156:145475 [7] NCCL INFO ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84375:96672 [1] NCCL INFO Channel 00/0 : 25ts-cbba87c5e7504a249f5127103d9ce40f-wts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84381:9667ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133156:145475 [7] NCCL INFO Channel 01/0 : 15[d0000] -> 14[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103dts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8016:20369 [1] NCCL INFO Connected all rings ts-cbbts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133156:145475 [7] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133153:145472 [4] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133150:145471 [1] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133149:145476 [0] NCCL INFO Connected all rings ts-cbba87c5e7504ts-cbba87c5e7504a249f5127103d9ce40f-worker-1ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84380:96670 [6] NCCLts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133149:145476 [0] NCCL INFO Channel 01/0 : 8[e000] -> 9[13000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133155:145473 [6] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133154:145474 [5] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133153:145472 [4] NCCL INFO Channel 00/0 : 12[93000] -> 13[99000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133153:145472 [4] NCCL INFO Channel 01/0 : 12[93000] -> 13[99000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133155:145473 [6] NCCL INFO Channel 00/0 : 14[cb000] -> 15[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133149:145476 [0] NCCL INFO Channel 00/0 : 8[e000] -> 15[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133154:145474 [5] NCCL INFO Channel 00/0 : 13[99000] -> 14[cb000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133150:145471 [1] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133150:145471 [1] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133150:145471 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133155:145473 [6] NCCL INFO Channel 01/0 : 14[cb000] -> 15[d0000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133154:145474 [5] NCCL INFO Channel 01/0 : 13[99000] -> 14[cb0ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8021:20364 [6] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8021:20364 [ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133154:145474 [5] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133154:145474 [5] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133154:145474 [5] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133155:145473 [6] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133155:145473 [6] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133155:145473 [6] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133156:145475 [7] NCCL INFO Channel 00/0 : 15[d0000] -> 8[e000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133156:145475 [7] NCCL INFO Channel 01/0 : 15[d0000] -> 8[e000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133156:145475 [7] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133156:145475 [7] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133156:145475 [7] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133149:145476 [0] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133149:145476 [0] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133149:145476 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133152:145478 [3] NCCL INFO Channel 00/0 ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:20367 [2] NCCL INFO Channel 01/0 : 11[51000] -> 18[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:20370 [3] NCCL INFO Channel 01/0 : 19[51000] -> 26[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:20370 [3] NCCL INFO Connts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84377:96671 [3] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84377:96671 [3] NCCL INFO Channel 00/0 : 27[51000] -> 28[93000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-2:84376:96673 [2] NCCL INFO Connected all rings ts-cbba87c5e7504a249fts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81153 [3] NCCL INFO Channel 00/0 : 3[51000] -> 4[93000] via P2P/IPC/read ts-cbba87c5e7504a24ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133151:145477 [2] NCCL INFO Connected all rings ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133151:145477 [2] NCCL INFO Channel 00/0 : 10[4b000] -> 11[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133151:145477 [2] NCCL INFO Channel 01/0 : 10[4b000] -> 11[51000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40fts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:20367 [2] NCCL INFO Channel 00/0 : 18[4b000] -> 26[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:20370 [3] NCCL INFO Channel 00/0 : 1ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133151:ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:20367 [2] NCCL INFO Channel 00/0 : 2[4b000] -> 18[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:20367 [2] NCCL INFO Channel 00/0 : 18[4b000] -> 2[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:20370 [3] NCCL INFO Channel 00/0 : 19[51000] -ts-cbba87c5e7504a249f5127103d9ce40f-worts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8017:20367 [2] NCCL INFO Channel 00/0 : 26[4b000] -> 18[4b000] [receive] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:20370 [3] NCCL INFO Channel 00/0 : 19[51000] -> 18[4b000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-1:8018:20370 [3] NCCL INFO Channel 01/0 : 19[51000] -> 18[4b000] via P2P/IPts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133151:145477 [2] NCCL INFO Channel 01/0 : 10[4b000] -> 2[4b000] [send] via NET/IB/0/GDRDMA ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133152:145478 [3] NCCL INFO Channel 00/0 : 11[51000] -> 10[4b000] via P2P/IPC/read ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133152:145478 [3] NCCL INFO Channel 01/0 : 11[51000] -> 10[4b000] viats-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81156 [2] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81156 [2] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81156 [2] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81153 [3] NCCL INFO Connected all trees ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81153 [3] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 512 | 512 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81153 [3] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:81149 [0] NCCL INFO comm 0x7f4ecc02abf0 rank 0 nranks 32 cudaDev 0 busId e000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81153 [3] NCCL INFO comm 0x7fade802a6c0 rank 3 nranks 32 cudaDev 3 busId 51000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81156 [2] NCCL INFO comm 0xts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133152:145478 [3] NCCL INFO comm 0x7fdd3402a6a0 rank 11 nranks 32 cudaDev 3 busId 51000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133151:145477 [2] NCCL INFO comm 0x7f484402a6c0 rank 10 nranks 32 cudaDev 2 busId 4b000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133149:145476 [0] NCCL INFO comm 0x7fa0d802aa40 rank 8 nranks 32 cudaDev 0 busId e000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133150:145471 [1] NCCL INFO comm 0x7fadf002a790 rank 9 nranks 32 cudaDev 1 busId 13000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133154:145474 [5] NCCL INFO comm 0x7f192c02a6c0 rank 13 nranks 32 cudaDev 5 busId 99000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133155:145473 [6] NCCL INFO comm 0x7ef57002a7f0 rank 14 nranks 32 cudaDev 6 busId cb000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133153:145472 [4] NCCL INFO comm 0x7f5b3002a6a0 rank 12 nranks 32 cudaDev 4 busId 93000 - Init COMPLETE ts-cbba87c5e7504a249f5127103d9ce40f-worker-0:133156:145475 [7] NCCL INFO comm 0x7fd83402a820 rank 15 nranks 32 cudaDev 7 busId d0000 - Init COMPLETE 0%| | 1/1817 [02:22<72:01:32, 142.78s/it] {'loss': 5.1648, 'learning_rate': 1.0000000000000002e-06, 'epoch': 0.0} 0%| | 1/1817 [02:22<71:54:20, 142.54s/it] 0%| | 2/1817 [04:33<68:31:16, 135.91s/it] {'loss': 5.1865, 'learning_rate': 2.0000000000000003e-06, 'epoch': 0.0} 0%| | 2/1817 [04:33<68:28:19, 135.81s/it] 0%| | 3/1817 [06:43<67:04:35, 133.12s/it] {'loss': 5.1278, 'learning_rate': 3e-06, 'epoch': 0.0} 0%| | 3/1817 [06:43<67:02:59, 133.06s/it] 0%| | 4/1817 [08:55<66:44:37, 132.53s/it] {'loss': 5.156, 'learning_rate': 4.000000000000001e-06, 'epoch': 0.0} 0%| | 4/1817 [08:55<66:43:39, 132.50s/it] 0%| | 5/1817 [11:06<66:31:34, 132.17s/it] {'loss': 4.9573, 'learning_rate': 5e-06, 'epoch': 0.0} 0%| | 5/1817 [11:06<66:30:57, 132.15s/it] 0%| | 6/1817 [13:18<66:25:17, 132.04s/it] {'loss': 4.708, 'learning_rate': 6e-06, 'epoch': 0.0} 0%| | 6/1817 [13:18<66:24:53, 132.02s/it] 0%| | 7/1817 [15:29<66:13:59, 131.73s/it] {'loss': 4.54, 'learning_rate': 7e-06, 'epoch': 0.0} 0%| | 7/1817 [15:29<66:13:42, 131.73s/it] 0%| | 8/1817 [17:41<66:12:57, 131.77s/it] {'loss': 4.3333, 'learning_rate': 8.000000000000001e-06, 'epoch': 0.0} 0%| | 8/1817 [17:41<66:12:45, 131.77s/it] 0%| | 9/1817 [19:51<65:49:40, 131.07s/it] {'loss': 4.2478, 'learning_rate': 9e-06, 'epoch': 0.0} 0%| | 9/1817 [19:50<65:49:32, 131.07s/it] 1%| | 10/1817 [22:04<66:06:28, 131.70s/it] {'loss': 3.9929, 'learning_rate': 1e-05, 'epoch': 0.01} 1%| | 10/1817 [22:04<66:06:28, 131.70s/it] 1%| | 11/1817 [24:15<66:00:25, 131.58s/it] {'loss': 3.7728, 'learning_rate': 1.1000000000000001e-05, 'epoch': 0.01} 1%| | 11/1817 [24:15<66:00:25, 131.58s/it] 1%| | 12/1817 [26:25<65:48:49, 131.26s/it] {'loss': 3.7259, 'learning_rate': 1.2e-05, 'epoch': 0.01} 1%| | 12/1817 [26:25<65:48:49, 131.26s/it] 1%| | 13/1817 [28:37<65:53:03, 131.48s/it] {'loss': 3.5329, 'learning_rate': 1.3000000000000001e-05, 'epoch': 0.01} 1%| | 13/1817 [28:37<65:53:03, 131.48s/it] 1%| | 14/1817 [30:46<65:25:04, 130.62s/it] {'loss': 3.3213, 'learning_rate': 1.4e-05, 'epoch': 0.01} 1%| | 14/1817 [30:46<65:25:04, 130.62s/it] 1%| | 15/1817 [32:55<65:07:56, 130.12s/it] {'loss': 3.2695, 'learning_rate': 1.5000000000000002e-05, 'epoch': 0.01} 1%| | 15/1817 [32:55<65:07:54, 130.12s/it] 1%| | 16/1817 [35:06<65:13:52, 130.39s/it] {'loss': 3.1222, 'learning_rate': 1.6000000000000003e-05, 'epoch': 0.01} 1%| | 16/1817 [35:06<65:13:51, 130.39s/it] 1%| | 17/1817 [37:17<65:18:04, 130.60s/it] {'loss': 3.0797, 'learning_rate': 1.7e-05, 'epoch': 0.01} 1%| | 17/1817 [37:17<65:18:03, 130.60s/it] 1%| | 18/1817 [39:28<65:16:00, 130.61s/it] {'loss': 3.0067, 'learning_rate': 1.8e-05, 'epoch': 0.01} 1%| | 18/1817 [39:28<65:16:03, 130.61s/it] 1%| | 19/1817 [41:37<64:57:38, 130.07s/it] {'loss': 2.9285, 'learning_rate': 1.9e-05, 'epoch': 0.01} 1%| | 19/1817 [41:36<64:57:37, 130.07s/it] 1%| | 20/1817 [43:49<65:11:14, 130.59s/it] {'loss': 2.7844, 'learning_rate': 2e-05, 'epoch': 0.01} 1%| | 20/1817 [43:48<65:11:17, 130.59s/it] 1%| | 21/1817 [46:02<65:37:53, 131.56s/it] {'loss': 2.7014, 'learning_rate': 2e-05, 'epoch': 0.01} 1%| | 21/1817 [46:02<65:37:53, 131.56s/it] 1%| | 22/1817 [48:15<65:41:47, 131.76s/it] {'loss': 2.6373, 'learning_rate': 2e-05, 'epoch': 0.01} 1%| | 22/1817 [48:14<65:41:47, 131.76s/it] 1%|▏ | 23/1817 [50:25<65:28:48, 131.40s/it] {'loss': 2.6143, 'learning_rate': 2e-05, 'epoch': 0.01} 1%|▏ | 23/1817 [50:25<65:28:48, 131.40s/it] 1%|▏ | 24/1817 [52:38<65:37:17, 131.76s/it] {'loss': 2.5501, 'learning_rate': 2e-05, 'epoch': 0.01} 1%|▏ | 24/1817 [52:37<65:37:21, 131.76s/it] 1%|▏ | 25/1817 [54:50<65:39:38, 131.91s/it] {'loss': 2.496, 'learning_rate': 2e-05, 'epoch': 0.01} 1%|▏ | 25/1817 [54:50<65:39:42, 131.91s/it] 1%|▏ | 26/1817 [57:04<65:52:51, 132.42s/it] {'loss': 2.4313, 'learning_rate': 2e-05, 'epoch': 0.01} 1%|▏ | 26/1817 [57:03<65:52:51, 132.42s/it] 1%|▏ | 27/1817 [59:14<65:35:06, 131.90s/it] {'loss': 2.3946, 'learning_rate': 2e-05, 'epoch': 0.01} 1%|▏ | 27/1817 [59:14<65:35:07, 131.90s/it] 2%|▏ | 28/1817 [1:01:25<65:22:58, 131.57s/it] {'loss': 2.3586, 'learning_rate': 2e-05, 'epoch': 0.02} 2%|▏ | 28/1817 [1:01:25<65:23:03, 131.57s/it] 2%|▏ | 29/1817 [1:03:34<65:01:15, 130.91s/it] {'loss': 2.3506, 'learning_rate': 2e-05, 'epoch': 0.02} 2%|▏ | 29/1817 [1:03:34<65:01:15, 130.91s/it] 2%|▏ | 30/1817 [1:05:46<65:02:52, 131.04s/it] {'loss': 2.3029, 'learning_rate': 2e-05, 'epoch': 0.02} 2%|▏ | 30/1817 [1:05:46<65:02:52, 131.04s/it] 2%|▏ | 31/1817 [1:08:00<65:30:20, 132.04s/it] {'loss': 2.278, 'learning_rate': 2e-05, 'epoch': 0.02} 2%|▏ | 31/1817 [1:08:00<65:30:19, 132.04s/it] 2%|▏ | 32/1817 [1:10:13<65:38:14, 132.38s/it] {'loss': 2.2981, 'learning_rate': 2e-05, 'epoch': 0.02} 2%|▏ | 32/1817 [1:10:13<65:38:14, 132.38s/it] 2%|▏ | 33/1817 [1:12:27<65:51:31, 132.90s/it] {'loss': 2.2715, 'learning_rate': 2e-05, 'epoch': 0.02} 2%|▏ | 33/1817 [1:12:27<65:51:32, 132.90s/it] 2%|▏ | 34/1817 [1:14:38<65:29:04, 132.22s/it] {'loss': 2.2713, 'learning_rate': 2e-05, 'epoch': 0.02} 2%|▏ | 34/1817 [1:14:38<65:29:04, 132.22s/it] 2%|▏ | 35/1817 [1:16:50<65:22:52, 132.08s/it] {'loss': 2.1992, 'learning_rate': 2e-05, 'epoch': 0.02} 2%|▏ | 35/1817 [1:16:50<65:22:56, 132.09s/it] 2%|▏ | 36/1817 [1:18:59<64:51:18, 131.09s/it] {'loss': 2.2279, 'learning_rate': 2e-05, 'epoch': 0.02} 2%|▏ | 36/1817 [1:18:58<64:51:18, 131.09s/it] 2%|▏ | 37/1817 [1:21:08<64:37:30, 130.70s/it] {'loss': 2.1901, 'learning_rate': 2e-05, 'epoch': 0.02} 2%|▏ | 37/1817 [1:21:08<64:37:30, 130.70s/it] 2%|▏ | 38/1817 [1:23:19<64:32:33, 130.61s/it] {'loss': 2.1996, 'learning_rate': 2e-05, 'epoch': 0.02} 2%|▏ | 38/1817 [1:23:19<64:32:33, 130.61s/it] 2%|▏ | 39/1817 [1:25:30<64:36:29, 130.82s/it] {'loss': 2.1535, 'learning_rate': 2e-05, 'epoch': 0.02} 2%|▏ | 39/1817 [1:25:30<64:36:28, 130.81s/it] 2%|▏ | 40/1817 [1:27:44<64:57:26, 131.60s/it] {'loss': 2.1511, 'learning_rate': 2e-05, 'epoch': 0.02} 2%|▏ | 40/1817 [1:27:43<64:57:25, 131.60s/it] 2%|▏ | 41/1817 [1:29:54<64:45:40, 131.27s/it] {'loss': 2.1268, 'learning_rate': 2e-05, 'epoch': 0.02} 2%|▏ | 41/1817 [1:29:54<64:45:40, 131.27s/it] 2%|▏ | 42/1817 [1:32:05<64:37:45, 131.08s/it] {'loss': 2.1241, 'learning_rate': 2e-05, 'epoch': 0.02} 2%|▏ | 42/1817 [1:32:04<64:37:45, 131.08s/it] 2%|▏ | 43/1817 [1:34:15<64:25:00, 130.72s/it] {'loss': 2.1105, 'learning_rate': 2e-05, 'epoch': 0.02} 2%|▏ | 43/1817 [1:34:14<64:25:00, 130.72s/it] 2%|▏ | 44/1817 [1:36:26<64:29:55, 130.96s/it] {'loss': 2.1339, 'learning_rate': 2e-05, 'epoch': 0.02} 2%|▏ | 44/1817 [1:36:26<64:29:55, 130.96s/it] 2%|▏ | 45/1817 [1:38:38<64:37:22, 131.29s/it] {'loss': 2.1267, 'learning_rate': 2e-05, 'epoch': 0.02} 2%|▏ | 45/1817 [1:38:38<64:37:22, 131.29s/it] 3%|▎ | 46/1817 [1:40:53<65:12:17, 132.55s/it] {'loss': 2.1362, 'learning_rate': 2e-05, 'epoch': 0.03} 3%|▎ | 46/1817 [1:40:53<65:12:17, 132.55s/it] 3%|▎ | 47/1817 [1:43:06<65:07:20, 132.45s/it] {'loss': 2.1089, 'learning_rate': 2e-05, 'epoch': 0.03} 3%|▎ | 47/1817 [1:43:06<65:07:20, 132.45s/it] 3%|▎ | 48/1817 [1:45:15<64:38:43, 131.56s/it] {'loss': 2.1016, 'learning_rate': 2e-05, 'epoch': 0.03} 3%|▎ | 48/1817 [1:45:15<64:38:44, 131.56s/it] 3%|▎ | 49/1817 [1:47:25<64:21:53, 131.06s/it] {'loss': 2.1397, 'learning_rate': 2e-05, 'epoch': 0.03} 3%|▎ | 49/1817 [1:47:25<64:21:53, 131.06s/it] 3%|▎ | 50/1817 [1:49:34<64:02:12, 130.47s/it] {'loss': 2.1109, 'learning_rate': 2e-05, 'epoch': 0.03} 3%|▎ | 50/1817 [1:49:34<64:02:12, 130.47s/it] 3%|▎ | 51/1817 [1:51:44<63:57:38, 130.38s/it] {'loss': 2.0972, 'learning_rate': 2e-05, 'epoch': 0.03} 3%|▎ | 51/1817 [1:51:44<63:57:38, 130.38s/it] 3%|▎ | 52/1817 [1:53:54<63:48:03, 130.13s/it] {'loss': 2.077, 'learning_rate': 2e-05, 'epoch': 0.03} 3%|▎ | 52/1817 [1:53:54<63:48:02, 130.13s/it] 3%|▎ | 53/1817 [1:56:03<63:36:30, 129.81s/it] {'loss': 2.0827, 'learning_rate': 2e-05, 'epoch': 0.03} 3%|▎ | 53/1817 [1:56:03<63:36:29, 129.81s/it] 3%|▎ | 54/1817 [1:58:16<64:02:32, 130.77s/it] {'loss': 2.1239, 'learning_rate': 2e-05, 'epoch': 0.03} 3%|▎ | 54/1817 [1:58:16<64:02:32, 130.77s/it] 3%|▎ | 55/1817 [2:00:30<64:29:21, 131.76s/it] {'loss': 2.0598, 'learning_rate': 2e-05, 'epoch': 0.03} 3%|▎ | 55/1817 [2:00:30<64:29:24, 131.76s/it] 3%|▎ | 56/1817 [2:02:39<64:05:35, 131.03s/it] {'loss': 2.0962, 'learning_rate': 2e-05, 'epoch': 0.03} 3%|▎ | 56/1817 [2:02:39<64:05:35, 131.03s/it] 3%|▎ | 57/1817 [2:04:53<64:24:28, 131.74s/it] {'loss': 2.084, 'learning_rate': 2e-05, 'epoch': 0.03} 3%|▎ | 57/1817 [2:04:53<64:24:28, 131.74s/it] 3%|▎ | 58/1817 [2:07:05<64:28:04, 131.94s/it] {'loss': 2.0732, 'learning_rate': 2e-05, 'epoch': 0.03} 3%|▎ | 58/1817 [2:07:05<64:28:04, 131.94s/it] 3%|▎ | 59/1817 [2:09:16<64:16:20, 131.62s/it] {'loss': 2.0596, 'learning_rate': 2e-05, 'epoch': 0.03} 3%|▎ | 59/1817 [2:09:16<64:16:21, 131.62s/it] 3%|▎ | 60/1817 [2:11:29<64:25:24, 132.00s/it] {'loss': 2.0455, 'learning_rate': 2e-05, 'epoch': 0.03} 3%|▎ | 60/1817 [2:11:29<64:25:25, 132.00s/it] 3%|▎ | 61/1817 [2:13:40<64:12:10, 131.62s/it] {'loss': 2.0151, 'learning_rate': 2e-05, 'epoch': 0.03} 3%|▎ | 61/1817 [2:13:40<64:12:06, 131.62s/it] 3%|▎ | 62/1817 [2:15:54<64:35:53, 132.51s/it] {'loss': 2.0724, 'learning_rate': 2e-05, 'epoch': 0.03} 3%|▎ | 62/1817 [2:15:54<64:35:53, 132.51s/it] 3%|▎ | 63/1817 [2:18:02<63:46:37, 130.90s/it] {'loss': 2.0657, 'learning_rate': 2e-05, 'epoch': 0.03} 3%|▎ | 63/1817 [2:18:01<63:46:37, 130.90s/it] 4%|▎ | 64/1817 [2:20:13<63:48:09, 131.03s/it] {'loss': 2.0508, 'learning_rate': 2e-05, 'epoch': 0.04} 4%|▎ | 64/1817 [2:20:13<63:48:09, 131.03s/it] 4%|▎ | 65/1817 [2:22:26<64:02:26, 131.59s/it] {'loss': 2.0655, 'learning_rate': 2e-05, 'epoch': 0.04} 4%|▎ | 65/1817 [2:22:26<64:02:26, 131.59s/it] 4%|▎ | 66/1817 [2:24:35<63:36:01, 130.76s/it] {'loss': 2.0223, 'learning_rate': 2e-05, 'epoch': 0.04} 4%|▎ | 66/1817 [2:24:34<63:36:01, 130.76s/it] 4%|▎ | 67/1817 [2:26:46<63:36:15, 130.84s/it] {'loss': 2.0363, 'learning_rate': 2e-05, 'epoch': 0.04} 4%|▎ | 67/1817 [2:26:45<63:36:16, 130.84s/it] 4%|▎ | 68/1817 [2:28:57<63:41:46, 131.11s/it] {'loss': 2.0409, 'learning_rate': 2e-05, 'epoch': 0.04} 4%|▎ | 68/1817 [2:28:57<63:41:46, 131.11s/it] 4%|▍ | 69/1817 [2:31:07<63:26:37, 130.66s/it] {'loss': 2.0287, 'learning_rate': 2e-05, 'epoch': 0.04} 4%|▍ | 69/1817 [2:31:07<63:26:37, 130.66s/it] 4%|▍ | 70/1817 [2:33:19<63:39:14, 131.17s/it] {'loss': 2.0692, 'learning_rate': 2e-05, 'epoch': 0.04} 4%|▍ | 70/1817 [2:33:19<63:39:18, 131.17s/it] 4%|▍ | 71/1817 [2:35:33<63:58:45, 131.92s/it] {'loss': 2.0614, 'learning_rate': 2e-05, 'epoch': 0.04} 4%|▍ | 71/1817 [2:35:33<63:58:45, 131.92s/it] 4%|▍ | 72/1817 [2:37:44<63:47:18, 131.60s/it] {'loss': 2.0214, 'learning_rate': 2e-05, 'epoch': 0.04} 4%|▍ | 72/1817 [2:37:44<63:47:18, 131.60s/it] 4%|▍ | 73/1817 [2:39:52<63:13:54, 130.52s/it] {'loss': 2.0347, 'learning_rate': 2e-05, 'epoch': 0.04} 4%|▍ | 73/1817 [2:39:52<63:13:54, 130.52s/it] 4%|▍ | 74/1817 [2:42:02<63:10:55, 130.50s/it] {'loss': 2.0306, 'learning_rate': 2e-05, 'epoch': 0.04} 4%|▍ | 74/1817 [2:42:02<63:10:55, 130.50s/it] 4%|▍ | 75/1817 [2:44:11<62:56:18, 130.07s/it] {'loss': 2.0159, 'learning_rate': 2e-05, 'epoch': 0.04} 4%|▍ | 75/1817 [2:44:11<62:56:18, 130.07s/it] 4%|▍ | 76/1817 [2:46:21<62:54:07, 130.07s/it] {'loss': 2.0199, 'learning_rate': 2e-05, 'epoch': 0.04} 4%|▍ | 76/1817 [2:46:21<62:54:07, 130.07s/it] 4%|▍ | 77/1817 [2:48:33<63:03:02, 130.45s/it] {'loss': 2.0329, 'learning_rate': 2e-05, 'epoch': 0.04} 4%|▍ | 77/1817 [2:48:33<63:03:02, 130.45s/it] 4%|▍ | 78/1817 [2:50:44<63:11:41, 130.82s/it] {'loss': 2.0468, 'learning_rate': 2e-05, 'epoch': 0.04} 4%|▍ | 78/1817 [2:50:44<63:11:42, 130.82s/it] 4%|▍ | 79/1817 [2:52:56<63:14:03, 130.98s/it] {'loss': 2.0135, 'learning_rate': 2e-05, 'epoch': 0.04} 4%|▍ | 79/1817 [2:52:56<63:14:03, 130.98s/it] 4%|▍ | 80/1817 [2:55:07<63:16:23, 131.14s/it] {'loss': 2.0397, 'learning_rate': 2e-05, 'epoch': 0.04} 4%|▍ | 80/1817 [2:55:07<63:16:23, 131.14s/it] 4%|▍ | 81/1817 [2:57:14<62:38:38, 129.91s/it] {'loss': 2.0395, 'learning_rate': 2e-05, 'epoch': 0.04} 4%|▍ | 81/1817 [2:57:14<62:38:42, 129.91s/it] 5%|▍ | 82/1817 [2:59:24<62:32:14, 129.76s/it] {'loss': 2.0061, 'learning_rate': 2e-05, 'epoch': 0.05} 5%|▍ | 82/1817 [2:59:24<62:32:18, 129.76s/it] 5%|▍ | 83/1817 [3:01:32<62:20:56, 129.44s/it] {'loss': 1.9889, 'learning_rate': 2e-05, 'epoch': 0.05} 5%|▍ | 83/1817 [3:01:32<62:20:56, 129.44s/it] 5%|▍ | 84/1817 [3:03:42<62:19:41, 129.48s/it] {'loss': 1.9952, 'learning_rate': 2e-05, 'epoch': 0.05} 5%|▍ | 84/1817 [3:03:42<62:19:41, 129.48s/it] 5%|▍ | 85/1817 [3:05:54<62:42:35, 130.34s/it] {'loss': 2.0326, 'learning_rate': 2e-05, 'epoch': 0.05} 5%|▍ | 85/1817 [3:05:54<62:42:35, 130.34s/it] 5%|▍ | 86/1817 [3:08:07<63:06:19, 131.24s/it] {'loss': 2.0359, 'learning_rate': 2e-05, 'epoch': 0.05} 5%|▍ | 86/1817 [3:08:07<63:06:19, 131.24s/it] 5%|▍ | 87/1817 [3:10:24<63:45:25, 132.67s/it] {'loss': 2.0661, 'learning_rate': 2e-05, 'epoch': 0.05} 5%|▍ | 87/1817 [3:10:23<63:45:25, 132.67s/it] 5%|▍ | 88/1817 [3:12:34<63:20:40, 131.89s/it] {'loss': 2.021, 'learning_rate': 2e-05, 'epoch': 0.05} 5%|▍ | 88/1817 [3:12:34<63:20:09, 131.87s/it] 5%|▍ | 89/1817 [3:14:44<63:09:03, 131.56s/it] {'loss': 2.0273, 'learning_rate': 2e-05, 'epoch': 0.05} 5%|▍ | 89/1817 [3:14:44<63:09:03, 131.56s/it] 5%|▍ | 90/1817 [3:16:53<62:42:48, 130.73s/it] {'loss': 1.9696, 'learning_rate': 2e-05, 'epoch': 0.05} 5%|▍ | 90/1817 [3:16:53<62:42:48, 130.73s/it] 5%|▌ | 91/1817 [3:19:02<62:24:12, 130.16s/it] {'loss': 1.9999, 'learning_rate': 2e-05, 'epoch': 0.05} 5%|▌ | 91/1817 [3:19:02<62:24:12, 130.16s/it] 5%|▌ | 92/1817 [3:21:19<63:19:05, 132.14s/it] {'loss': 2.0181, 'learning_rate': 2e-05, 'epoch': 0.05} 5%|▌ | 92/1817 [3:21:19<63:19:10, 132.14s/it] 5%|▌ | 93/1817 [3:23:30<63:05:29, 131.75s/it] {'loss': 1.9673, 'learning_rate': 2e-05, 'epoch': 0.05} 5%|▌ | 93/1817 [3:23:30<63:05:29, 131.75s/it] 5%|▌ | 94/1817 [3:25:42<63:05:54, 131.84s/it] {'loss': 1.9907, 'learning_rate': 2e-05, 'epoch': 0.05} 5%|▌ | 94/1817 [3:25:42<63:05:54, 131.84s/it] 5%|▌ | 95/1817 [3:27:49<62:26:36, 130.54s/it] {'loss': 2.0098, 'learning_rate': 2e-05, 'epoch': 0.05} 5%|▌ | 95/1817 [3:27:49<62:26:36, 130.54s/it] 5%|▌ | 96/1817 [3:29:57<62:03:46, 129.82s/it] {'loss': 1.9847, 'learning_rate': 2e-05, 'epoch': 0.05} 5%|▌ | 96/1817 [3:29:57<62:03:46, 129.82s/it] 5%|▌ | 97/1817 [3:32:07<61:59:34, 129.75s/it] {'loss': 1.9882, 'learning_rate': 2e-05, 'epoch': 0.05} 5%|▌ | 97/1817 [3:32:07<61:59:38, 129.75s/it] 5%|▌ | 98/1817 [3:34:21<62:34:17, 131.04s/it] {'loss': 2.0149, 'learning_rate': 2e-05, 'epoch': 0.05} 5%|▌ | 98/1817 [3:34:21<62:34:17, 131.04s/it] 5%|▌ | 99/1817 [3:36:37<63:16:33, 132.59s/it] {'loss': 2.005, 'learning_rate': 2e-05, 'epoch': 0.05} 5%|▌ | 99/1817 [3:36:37<63:16:33, 132.59s/it] 6%|▌ | 100/1817 [3:38:49<63:10:26, 132.46s/it] {'loss': 1.9619, 'learning_rate': 2e-05, 'epoch': 0.06} 6%|▌ | 100/1817 [3:38:49<63:10:26, 132.46s/it] ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:68073 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:68073 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1886872682 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:68073 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1347896394 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:68073 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:68073 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:68073 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:68073 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:68073 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:68073 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:68073 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:68073 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:68073 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:68073 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:68073 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:68073 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:68073 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:68073 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:68073 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:68073 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1868983913 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:68073 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 721748225 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:68073 [2] NCCL INFO [Service thread] Connection closed by localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:68073 [2] NCCL INFO [Service thread] Connection closed by localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:68073 [2] NCCL INFO [Service thread] Connection closed by localRank -1 6%|▌ | 101/1817 [3:40:57<62:26:27, 131.00s/it] {'loss': 1.9956, 'learning_rate': 2e-05, 'epoch': 0.06} 6%|▌ | 101/1817 [3:40:57<62:26:27, 131.00s/it] 6%|▌ | 102/1817 [3:43:07<62:13:47, 130.63s/it] {'loss': 2.0293, 'learning_rate': 2e-05, 'epoch': 0.06} 6%|▌ | 102/1817 [3:43:07<62:13:48, 130.63s/it] ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81162 [3] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81162 [3] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81162 [3] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81162 [3] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81162 [3] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81162 [3] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81162 [3] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81162 [3] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81162 [3] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81162 [3] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81162 [3] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81162 [3] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81162 [3] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 721748225 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81162 [3] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1868983913 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81162 [3] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81162 [3] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81162 [3] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1886872682 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81162 [3] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1347896394 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81162 [3] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81162 [3] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81162 [3] NCCL INFO [Service thread] Connection closed by localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81162 [3] NCCL INFO [Service thread] Connection closed by localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67023:81162 [3] NCCL INFO [Service thread] Connection closed by localRank -1 6%|▌ | 103/1817 [3:45:18<62:21:35, 130.98s/it] {'loss': 1.9819, 'learning_rate': 2e-05, 'epoch': 0.06} 6%|▌ | 103/1817 [3:45:18<62:21:35, 130.98s/it] 6%|▌ | 104/1817 [3:47:27<61:57:54, 130.22s/it] {'loss': 1.9825, 'learning_rate': 2e-05, 'epoch': 0.06} 6%|▌ | 104/1817 [3:47:27<61:57:53, 130.22s/it] 6%|▌ | 105/1817 [3:49:38<62:03:28, 130.50s/it] {'loss': 1.9436, 'learning_rate': 2e-05, 'epoch': 0.06} 6%|▌ | 105/1817 [3:49:38<62:03:32, 130.50s/it] 6%|▌ | 106/1817 [3:51:48<61:52:56, 130.20s/it] {'loss': 1.9999, 'learning_rate': 2e-05, 'epoch': 0.06} 6%|▌ | 106/1817 [3:51:48<61:52:57, 130.20s/it] ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:68077 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:68077 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:68077 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:68077 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:68077 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:68077 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:68077 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1868983913 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:68077 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:68077 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:68077 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:68077 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:68077 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:68077 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1886872682 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:68077 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1347896394 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:68077 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:68077 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:68077 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:68077 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 721748225 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:68077 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:68077 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:68078 [0] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:68078 [0] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:68078 [0] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:68078 [0] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:68078 [0] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:68078 [0] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:68078 [0] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1868983913 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:68078 [0] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:68078 [0] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1886872682 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:68078 [0] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1347896394 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:68078 [0] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:68078 [0] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:68078 [0] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:68078 [0] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:68078 [0] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:68078 [0] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:68078 [0] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:68078 [0] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:68078 [0] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:68078 [0] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 721748225 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:68077 [5] NCCL INFO [Service thread] Connection closed by localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:68077 [5] NCCL INFO [Service thread] Connection closed by localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:68077 [5] NCCL INFO [Service thread] Connection closed by localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:68078 [0] NCCL INFO [Service thread] Connection closed by localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:68078 [0] NCCL INFO [Service thread] Connection closed by localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67020:68078 [0] NCCL INFO [Service thread] Connection closed by localRank -1 6%|▌ | 107/1817 [3:54:01<62:16:47, 131.12s/it] {'loss': 2.0105, 'learning_rate': 2e-05, 'epoch': 0.06} 6%|▌ | 107/1817 [3:54:01<62:16:48, 131.12s/it] 6%|▌ | 108/1817 [3:56:13<62:21:25, 131.35s/it] {'loss': 2.0088, 'learning_rate': 2e-05, 'epoch': 0.06} 6%|▌ | 108/1817 [3:56:13<62:21:25, 131.35s/it] ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81160 [4] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1868983913 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81160 [4] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81160 [4] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81160 [4] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81160 [4] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81160 [4] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 721748225 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81160 [4] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81160 [4] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81160 [4] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81160 [4] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81160 [4] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81160 [4] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81160 [4] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81160 [4] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1886872682 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81160 [4] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1347896394 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81160 [4] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81160 [4] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81160 [4] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81160 [4] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81160 [4] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81160 [4] NCCL INFO [Service thread] Connection closed by localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81160 [4] NCCL INFO [Service thread] Connection closed by localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67024:81160 [4] NCCL INFO [Service thread] Connection closed by localRank -1 6%|▌ | 109/1817 [3:58:22<61:55:44, 130.53s/it] {'loss': 1.9643, 'learning_rate': 2e-05, 'epoch': 0.06} 6%|▌ | 109/1817 [3:58:21<61:55:45, 130.53s/it] 6%|▌ | 110/1817 [4:00:32<61:57:23, 130.66s/it] {'loss': 1.9859, 'learning_rate': 2e-05, 'epoch': 0.06} 6%|▌ | 110/1817 [4:00:32<61:57:23, 130.66s/it] 6%|▌ | 111/1817 [4:02:42<61:50:56, 130.51s/it] {'loss': 1.9533, 'learning_rate': 2e-05, 'epoch': 0.06} 6%|▌ | 111/1817 [4:02:42<61:50:56, 130.51s/it] 6%|▌ | 112/1817 [4:04:54<61:58:01, 130.84s/it] {'loss': 1.9736, 'learning_rate': 2e-05, 'epoch': 0.06} 6%|▌ | 112/1817 [4:04:54<61:58:01, 130.84s/it] 6%|▌ | 113/1817 [4:07:07<62:16:37, 131.57s/it] {'loss': 1.9908, 'learning_rate': 2e-05, 'epoch': 0.06} 6%|▌ | 113/1817 [4:07:07<62:16:37, 131.57s/it] 6%|▋ | 114/1817 [4:09:19<62:11:52, 131.48s/it] {'loss': 1.9785, 'learning_rate': 2e-05, 'epoch': 0.06} 6%|▋ | 114/1817 [4:09:19<62:11:52, 131.48s/it] ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81158 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81158 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1868983913 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81158 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 721748225 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81158 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81158 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81158 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81158 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81158 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81158 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1886872682 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81158 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1347896394 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81158 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81158 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81158 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81158 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81158 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81158 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81158 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81158 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81158 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81158 [2] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81158 [2] NCCL INFO [Service thread] Connection closed by localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81158 [2] NCCL INFO [Service thread] Connection closed by localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67022:81158 [2] NCCL INFO [Service thread] Connection closed by localRank -1 6%|▋ | 115/1817 [4:11:31<62:18:27, 131.79s/it] {'loss': 1.9596, 'learning_rate': 2e-05, 'epoch': 0.06} 6%|▋ | 115/1817 [4:11:31<62:18:28, 131.79s/it] 6%|▋ | 116/1817 [4:13:42<62:03:11, 131.33s/it] {'loss': 1.979, 'learning_rate': 2e-05, 'epoch': 0.06} 6%|▋ | 116/1817 [4:13:41<62:03:11, 131.33s/it] 6%|▋ | 117/1817 [4:15:54<62:07:37, 131.56s/it] {'loss': 1.9499, 'learning_rate': 2e-05, 'epoch': 0.06} 6%|▋ | 117/1817 [4:15:53<62:07:33, 131.56s/it] 6%|▋ | 118/1817 [4:18:05<62:03:34, 131.50s/it] {'loss': 1.9858, 'learning_rate': 2e-05, 'epoch': 0.06} 6%|▋ | 118/1817 [4:18:05<62:03:34, 131.50s/it] ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:81163 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:81163 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:81163 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:81163 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:81163 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:81163 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:81163 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:81163 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:81163 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:81163 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1868983913 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:81163 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1886872682 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:81163 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1347896394 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:81163 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:81163 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:81163 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:81163 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:81163 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:81163 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:81163 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:81163 [5] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 721748225 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:81163 [5] NCCL INFO [Service thread] Connection closed by localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:81163 [5] NCCL INFO [Service thread] Connection closed by localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67025:81163 [5] NCCL INFO [Service thread] Connection closed by localRank -1 7%|▋ | 119/1817 [4:20:14<61:43:56, 130.88s/it] {'loss': 2.0172, 'learning_rate': 2e-05, 'epoch': 0.07} 7%|▋ | 119/1817 [4:20:14<61:43:57, 130.88s/it] ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:81159 [7] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 721748225 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:81159 [7] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:81159 [7] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:81159 [7] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1868983913 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:81159 [7] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:81159 [7] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:81159 [7] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:81159 [7] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:81159 [7] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:81159 [7] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:81159 [7] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:81159 [7] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:81159 [7] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:81159 [7] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:81159 [7] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:81159 [7] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:81159 [7] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1886872682 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:81159 [7] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1347896394 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:81159 [7] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:81159 [7] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:81159 [7] NCCL INFO [Service thread] Connection closed by localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:81159 [7] NCCL INFO [Service thread] Connection closed by localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67027:81159 [7] NCCL INFO [Service thread] Connection closed by localRank -1 7%|▋ | 120/1817 [4:22:27<61:51:12, 131.22s/it] {'loss': 2.03, 'learning_rate': 2e-05, 'epoch': 0.07} 7%|▋ | 120/1817 [4:22:26<61:51:16, 131.22s/it] ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:68079 [1] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:68079 [1] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:68079 [1] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:68079 [1] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1886872682 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:68079 [1] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1347896394 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:68079 [1] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:68079 [1] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:68079 [1] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:68079 [1] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:68079 [1] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:68079 [1] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:68079 [1] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:68079 [1] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:68079 [1] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:68079 [1] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 1868983913 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:68079 [1] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:68079 [1] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:68079 [1] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 542393671 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:68079 [1] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 721748225 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:68079 [1] proxy.cc:1111 NCCL WARN [Service thread] Unknown command 16843542 from localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:68079 [1] NCCL INFO [Service thread] Connection closed by localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:68079 [1] NCCL INFO [Service thread] Connection closed by localRank -1 ts-cbba87c5e7504a249f5127103d9ce40f-launcher:67021:68079 [1] NCCL INFO [Service thread] Connection closed by localRank -1 7%|▋ | 121/1817 [4:24:37<61:42:18, 130.98s/it] {'loss': 1.9392, 'learning_rate': 2e-05, 'epoch': 0.07} 7%|▋ | 121/1817 [4:24:37<61:42:18, 130.98s/it] 7%|▋ | 122/1817 [4:26:51<62:03:28, 131.80s/it] {'loss': 1.961, 'learning_rate': 2e-05, 'epoch': 0.07} 7%|▋ | 122/1817 [4:26:50<62:03:31, 131.81s/it] 7%|▋ | 123/1817 [4:29:00<61:44:06, 131.20s/it] {'loss': 1.976, 'learning_rate': 2e-05, 'epoch': 0.07} 7%|▋ | 123/1817 [4:29:00<61:44:06, 131.20s/it] 7%|▋ | 124/1817 [4:31:10<61:25:24, 130.61s/it] {'loss': 1.9653, 'learning_rate': 2e-05, 'epoch': 0.07} 7%|▋ | 124/1817 [4:31:09<61:25:24, 130.61s/it] 7%|▋ | 125/1817 [4:33:19<61:11:04, 130.18s/it] {'loss': 1.9847, 'learning_rate': 2e-05, 'epoch': 0.07} 7%|▋ | 125/1817 [4:33:19<61:11:04, 130.18s/it] 7%|▋ | 126/1817 [4:35:34<61:52:34, 131.73s/it] {'loss': 2.0066, 'learning_rate': 2e-05, 'epoch': 0.07} 7%|▋ | 126/1817 [4:35:34<61:52:34, 131.73s/it] 7%|▋ | 127/1817 [4:37:46<61:55:16, 131.90s/it] {'loss': 1.9547, 'learning_rate': 2e-05, 'epoch': 0.07} 7%|▋ | 127/1817 [4:37:46<61:55:15, 131.90s/it] 7%|▋ | 128/1817 [4:39:59<61:56:11, 132.01s/it] {'loss': 1.9901, 'learning_rate': 2e-05, 'epoch': 0.07} 7%|▋ | 128/1817 [4:39:59<61:56:11, 132.01s/it] 7%|▋ | 129/1817 [4:42:10<61:46:41, 131.75s/it] {'loss': 1.9768, 'learning_rate': 2e-05, 'epoch': 0.07} 7%|▋ | 129/1817 [4:42:10<61:46:41, 131.75s/it] 7%|▋ | 130/1817 [4:44:22<61:43:59, 131.74s/it] {'loss': 1.9507, 'learning_rate': 2e-05, 'epoch': 0.07} 7%|▋ | 130/1817 [4:44:21<61:43:59, 131.74s/it] 7%|▋ | 131/1817 [4:46:30<61:17:50, 130.88s/it] {'loss': 1.9582, 'learning_rate': 2e-05, 'epoch': 0.07} 7%|▋ | 131/1817 [4:46:30<61:17:50, 130.88s/it] 7%|▋ | 132/1817 [4:48:41<61:14:48, 130.85s/it] {'loss': 1.951, 'learning_rate': 2e-05, 'epoch': 0.07} 7%|▋ | 132/1817 [4:48:41<61:14:49, 130.85s/it] 7%|▋ | 133/1817 [4:50:53<61:23:41, 131.25s/it] {'loss': 1.9592, 'learning_rate': 2e-05, 'epoch': 0.07} 7%|▋ | 133/1817 [4:50:53<61:23:41, 131.25s/it] 7%|▋ | 134/1817 [4:53:02<61:00:20, 130.49s/it] {'loss': 1.9837, 'learning_rate': 2e-05, 'epoch': 0.07} 7%|▋ | 134/1817 [4:53:02<61:00:20, 130.49s/it] 7%|▋ | 135/1817 [4:55:13<60:59:58, 130.56s/it] {'loss': 1.9941, 'learning_rate': 2e-05, 'epoch': 0.07} 7%|▋ | 135/1817 [4:55:13<60:59:58, 130.56s/it] 7%|▋ | 136/1817 [4:57:23<60:52:36, 130.37s/it] {'loss': 2.0104, 'learning_rate': 2e-05, 'epoch': 0.07} 7%|▋ | 136/1817 [4:57:23<60:52:36, 130.37s/it] 8%|▊ | 137/1817 [4:59:31<60:31:40, 129.70s/it] {'loss': 1.9823, 'learning_rate': 2e-05, 'epoch': 0.08} 8%|▊ | 137/1817 [4:59:31<60:31:40, 129.70s/it] 8%|▊ | 138/1817 [5:01:41<60:32:52, 129.82s/it] {'loss': 1.963, 'learning_rate': 2e-05, 'epoch': 0.08} 8%|▊ | 138/1817 [5:01:41<60:32:52, 129.82s/it] 8%|▊ | 139/1817 [5:03:52<60:38:54, 130.12s/it] {'loss': 1.9311, 'learning_rate': 2e-05, 'epoch': 0.08} 8%|▊ | 139/1817 [5:03:52<60:38:54, 130.12s/it] 8%|▊ | 140/1817 [5:06:04<60:50:17, 130.60s/it] {'loss': 1.9795, 'learning_rate': 2e-05, 'epoch': 0.08} 8%|▊ | 140/1817 [5:06:03<60:50:17, 130.60s/it] 8%|▊ | 141/1817 [5:08:15<60:56:17, 130.89s/it] {'loss': 1.9572, 'learning_rate': 2e-05, 'epoch': 0.08} 8%|▊ | 141/1817 [5:08:15<60:56:17, 130.89s/it] 8%|▊ | 142/1817 [5:10:31<61:31:58, 132.25s/it] {'loss': 1.9607, 'learning_rate': 2e-05, 'epoch': 0.08} 8%|▊ | 142/1817 [5:10:30<61:31:54, 132.25s/it] 8%|▊ | 143/1817 [5:12:40<61:03:43, 131.32s/it] {'loss': 1.9971, 'learning_rate': 2e-05, 'epoch': 0.08} 8%|▊ | 143/1817 [5:12:39<61:03:44, 131.32s/it] 8%|▊ | 144/1817 [5:14:50<60:53:26, 131.03s/it] {'loss': 1.944, 'learning_rate': 2e-05, 'epoch': 0.08} 8%|▊ | 144/1817 [5:14:50<60:53:26, 131.03s/it] 8%|▊ | 145/1817 [5:17:01<60:50:56, 131.01s/it] {'loss': 1.933, 'learning_rate': 2e-05, 'epoch': 0.08} 8%|▊ | 145/1817 [5:17:01<60:50:56, 131.01s/it] 8%|▊ | 146/1817 [5:19:12<60:46:26, 130.93s/it] {'loss': 2.0057, 'learning_rate': 2e-05, 'epoch': 0.08} 8%|▊ | 146/1817 [5:19:12<60:46:26, 130.93s/it] 8%|▊ | 147/1817 [5:21:22<60:35:49, 130.63s/it] {'loss': 1.9599, 'learning_rate': 2e-05, 'epoch': 0.08} 8%|▊ | 147/1817 [5:21:21<60:35:49, 130.63s/it] 8%|▊ | 148/1817 [5:23:31<60:26:05, 130.36s/it] {'loss': 1.9845, 'learning_rate': 2e-05, 'epoch': 0.08} 8%|▊ | 148/1817 [5:23:31<60:26:05, 130.36s/it] 8%|▊ | 149/1817 [5:25:42<60:22:11, 130.29s/it] {'loss': 1.9775, 'learning_rate': 2e-05, 'epoch': 0.08} 8%|▊ | 149/1817 [5:25:41<60:22:14, 130.30s/it] 8%|▊ | 150/1817 [5:27:55<60:45:03, 131.20s/it] {'loss': 1.9685, 'learning_rate': 2e-05, 'epoch': 0.08} 8%|▊ | 150/1817 [5:27:55<60:45:02, 131.20s/it] 8%|▊ | 151/1817 [5:30:05<60:31:59, 130.80s/it] {'loss': 1.9494, 'learning_rate': 2e-05, 'epoch': 0.08} 8%|▊ | 151/1817 [5:30:05<60:31:59, 130.80s/it] 8%|▊ | 152/1817 [5:32:15<60:27:14, 130.71s/it] {'loss': 1.9975, 'learning_rate': 2e-05, 'epoch': 0.08} 8%|▊ | 152/1817 [5:32:15<60:27:14, 130.71s/it] 8%|▊ | 153/1817 [5:34:25<60:20:40, 130.55s/it] {'loss': 1.9855, 'learning_rate': 2e-05, 'epoch': 0.08} 8%|▊ | 153/1817 [5:34:25<60:20:40, 130.55s/it] 8%|▊ | 154/1817 [5:36:35<60:14:00, 130.39s/it] {'loss': 1.9776, 'learning_rate': 2e-05, 'epoch': 0.08} 8%|▊ | 154/1817 [5:36:35<60:14:00, 130.39s/it] 9%|▊ | 155/1817 [5:38:44<60:00:18, 129.98s/it] {'loss': 1.9832, 'learning_rate': 2e-05, 'epoch': 0.09} 9%|▊ | 155/1817 [5:38:44<60:00:18, 129.98s/it] 9%|▊ | 156/1817 [5:40:56<60:14:13, 130.56s/it] {'loss': 1.9789, 'learning_rate': 2e-05, 'epoch': 0.09} 9%|▊ | 156/1817 [5:40:56<60:14:13, 130.56s/it] 9%|▊ | 157/1817 [5:43:06<60:09:16, 130.46s/it] {'loss': 1.9939, 'learning_rate': 2e-05, 'epoch': 0.09} 9%|▊ | 157/1817 [5:43:06<60:09:16, 130.46s/it] 9%|▊ | 158/1817 [5:45:19<60:20:09, 130.93s/it] {'loss': 1.9512, 'learning_rate': 2e-05, 'epoch': 0.09} 9%|▊ | 158/1817 [5:45:18<60:20:09, 130.93s/it] 9%|▉ | 159/1817 [5:47:32<60:42:02, 131.80s/it] {'loss': 1.9702, 'learning_rate': 2e-05, 'epoch': 0.09} 9%|▉ | 159/1817 [5:47:32<60:42:02, 131.80s/it] 9%|▉ | 160/1817 [5:49:42<60:23:12, 131.20s/it] {'loss': 1.9321, 'learning_rate': 2e-05, 'epoch': 0.09} 9%|▉ | 160/1817 [5:49:42<60:23:12, 131.20s/it] 9%|▉ | 161/1817 [5:51:55<60:32:27, 131.61s/it] {'loss': 1.9753, 'learning_rate': 2e-05, 'epoch': 0.09} 9%|▉ | 161/1817 [5:51:55<60:32:27, 131.61s/it] 9%|▉ | 162/1817 [5:54:08<60:44:47, 132.14s/it] {'loss': 1.9336, 'learning_rate': 2e-05, 'epoch': 0.09} 9%|▉ | 162/1817 [5:54:08<60:44:47, 132.14s/it] 9%|▉ | 163/1817 [5:56:26<61:34:20, 134.01s/it] {'loss': 2.0067, 'learning_rate': 2e-05, 'epoch': 0.09} 9%|▉ | 163/1817 [5:56:26<61:34:20, 134.01s/it] 9%|▉ | 164/1817 [5:58:42<61:46:40, 134.54s/it] {'loss': 1.9678, 'learning_rate': 2e-05, 'epoch': 0.09} 9%|▉ | 164/1817 [5:58:42<61:46:40, 134.54s/it] 9%|▉ | 165/1817 [6:00:52<61:02:31, 133.02s/it] {'loss': 1.9205, 'learning_rate': 2e-05, 'epoch': 0.09} 9%|▉ | 165/1817 [6:00:52<61:02:31, 133.02s/it] 9%|▉ | 166/1817 [6:03:01<60:31:26, 131.97s/it] {'loss': 1.9983, 'learning_rate': 2e-05, 'epoch': 0.09} 9%|▉ | 166/1817 [6:03:01<60:31:26, 131.97s/it] 9%|▉ | 167/1817 [6:05:12<60:22:06, 131.71s/it] {'loss': 1.9839, 'learning_rate': 2e-05, 'epoch': 0.09} 9%|▉ | 167/1817 [6:05:12<60:22:06, 131.71s/it] 9%|▉ | 168/1817 [6:07:24<60:23:26, 131.84s/it] {'loss': 1.9636, 'learning_rate': 2e-05, 'epoch': 0.09} 9%|▉ | 168/1817 [6:07:24<60:23:26, 131.84s/it] 9%|▉ | 169/1817 [6:09:32<59:45:28, 130.54s/it] {'loss': 1.9497, 'learning_rate': 2e-05, 'epoch': 0.09} 9%|▉ | 169/1817 [6:09:32<59:45:28, 130.54s/it] 9%|▉ | 170/1817 [6:11:46<60:12:38, 131.61s/it] {'loss': 1.9752, 'learning_rate': 2e-05, 'epoch': 0.09} 9%|▉ | 170/1817 [6:11:46<60:12:04, 131.59s/it] 9%|▉ | 171/1817 [6:14:00<60:27:25, 132.23s/it] {'loss': 2.0004, 'learning_rate': 2e-05, 'epoch': 0.09} 9%|▉ | 171/1817 [6:14:00<60:27:25, 132.23s/it] 9%|▉ | 172/1817 [6:16:10<60:11:36, 131.73s/it] {'loss': 1.9768, 'learning_rate': 2e-05, 'epoch': 0.09} 9%|▉ | 172/1817 [6:16:10<60:11:37, 131.73s/it] 10%|▉ | 173/1817 [6:18:26<60:39:00, 132.81s/it] {'loss': 1.9423, 'learning_rate': 2e-05, 'epoch': 0.1} 10%|▉ | 173/1817 [6:18:26<60:39:00, 132.81s/it] 10%|▉ | 174/1817 [6:20:34<60:01:30, 131.52s/it] {'loss': 1.9642, 'learning_rate': 2e-05, 'epoch': 0.1} 10%|▉ | 174/1817 [6:20:34<60:01:30, 131.52s/it] 10%|▉ | 175/1817 [6:22:45<59:52:53, 131.29s/it] {'loss': 1.8874, 'learning_rate': 2e-05, 'epoch': 0.1} 10%|▉ | 175/1817 [6:22:45<59:52:57, 131.29s/it] 10%|▉ | 176/1817 [6:24:56<59:52:25, 131.35s/it] {'loss': 1.9598, 'learning_rate': 2e-05, 'epoch': 0.1} 10%|▉ | 176/1817 [6:24:56<59:52:25, 131.35s/it] 10%|▉ | 177/1817 [6:27:05<59:29:16, 130.58s/it] {'loss': 1.9566, 'learning_rate': 2e-05, 'epoch': 0.1} 10%|▉ | 177/1817 [6:27:05<59:29:16, 130.58s/it] {'loss': 1.9566, 'learning_rate': 2e-05, 'epoch': 0.1} 10%|▉ | 177/1817 [6:27:05<59:29:17, 130.58s/it] 10%|▉ | 178/1817 [6:29:17<59:38:32, 131.00s/it] {'loss': 1.9285, 'learning_rate': 2e-05, 'epoch': 0.1} 10%|▉ | 178/1817 [6:29:17<59:38:33, 131.00s/it] 10%|▉ | 179/1817 [6:31:25<59:07:45, 129.95s/it] {'loss': 1.9402, 'learning_rate': 2e-05, 'epoch': 0.1} 10%|▉ | 179/1817 [6:31:25<59:07:48, 129.96s/it] 10%|▉ | 180/1817 [6:33:38<59:34:34, 131.02s/it] {'loss': 1.9549, 'learning_rate': 2e-05, 'epoch': 0.1} 10%|▉ | 180/1817 [6:33:38<59:34:37, 131.02s/it] 10%|▉ | 181/1817 [6:35:47<59:14:30, 130.36s/it] {'loss': 1.9637, 'learning_rate': 2e-05, 'epoch': 0.1} 10%|▉ | 181/1817 [6:35:47<59:14:31, 130.36s/it] 10%|█ | 182/1817 [6:37:56<59:05:25, 130.11s/it] {'loss': 1.9558, 'learning_rate': 2e-05, 'epoch': 0.1} 10%|█ | 182/1817 [6:37:56<59:05:25, 130.11s/it] 10%|█ | 183/1817 [6:40:06<59:00:07, 129.99s/it] {'loss': 1.993, 'learning_rate': 2e-05, 'epoch': 0.1} 10%|█ | 183/1817 [6:40:06<59:00:07, 129.99s/it] 10%|█ | 184/1817 [6:42:15<58:46:34, 129.57s/it] {'loss': 1.9812, 'learning_rate': 2e-05, 'epoch': 0.1} 10%|█ | 184/1817 [6:42:15<58:46:34, 129.57s/it] 10%|█ | 185/1817 [6:44:21<58:15:21, 128.51s/it] {'loss': 1.9727, 'learning_rate': 2e-05, 'epoch': 0.1} 10%|█ | 185/1817 [6:44:21<58:15:21, 128.51s/it] 10%|█ | 186/1817 [6:46:34<58:50:25, 129.87s/it] {'loss': 1.9678, 'learning_rate': 2e-05, 'epoch': 0.1} 10%|█ | 186/1817 [6:46:34<58:50:25, 129.87s/it] 10%|█ | 187/1817 [6:48:42<58:36:13, 129.43s/it] {'loss': 1.9619, 'learning_rate': 2e-05, 'epoch': 0.1} 10%|█ | 187/1817 [6:48:42<58:36:16, 129.43s/it] 10%|█ | 188/1817 [6:50:58<59:28:17, 131.43s/it] {'loss': 1.939, 'learning_rate': 2e-05, 'epoch': 0.1} 10%|█ | 188/1817 [6:50:58<59:28:17, 131.43s/it] 10%|█ | 189/1817 [6:53:07<59:01:51, 130.54s/it] {'loss': 1.9806, 'learning_rate': 2e-05, 'epoch': 0.1} 10%|█ | 189/1817 [6:53:07<59:01:51, 130.54s/it] 10%|█ | 190/1817 [6:55:21<59:33:46, 131.79s/it] {'loss': 1.9619, 'learning_rate': 2e-05, 'epoch': 0.1} 10%|█ | 190/1817 [6:55:21<59:33:46, 131.79s/it] 11%|█ | 191/1817 [6:57:37<59:58:55, 132.80s/it] {'loss': 1.9545, 'learning_rate': 2e-05, 'epoch': 0.11} 11%|█ | 191/1817 [6:57:37<59:58:55, 132.80s/it] 11%|█ | 192/1817 [6:59:50<60:03:22, 133.05s/it] {'loss': 1.9216, 'learning_rate': 2e-05, 'epoch': 0.11} 11%|█ | 192/1817 [6:59:50<60:03:22, 133.05s/it] 11%|█ | 193/1817 [7:01:59<59:27:36, 131.81s/it] {'loss': 1.9818, 'learning_rate': 2e-05, 'epoch': 0.11} 11%|█ | 193/1817 [7:01:59<59:27:36, 131.81s/it] 11%|█ | 194/1817 [7:04:04<58:30:24, 129.77s/it] {'loss': 2.0126, 'learning_rate': 2e-05, 'epoch': 0.11} 11%|█ | 194/1817 [7:04:04<58:30:25, 129.78s/it] 11%|█ | 195/1817 [7:06:17<58:49:34, 130.56s/it] {'loss': 1.954, 'learning_rate': 2e-05, 'epoch': 0.11} 11%|█ | 195/1817 [7:06:17<58:49:34, 130.56s/it] 11%|█ | 196/1817 [7:08:29<59:04:42, 131.20s/it] {'loss': 1.9497, 'learning_rate': 2e-05, 'epoch': 0.11} 11%|█ | 196/1817 [7:08:29<59:04:42, 131.20s/it] 11%|█ | 197/1817 [7:10:39<58:49:23, 130.72s/it] {'loss': 1.9395, 'learning_rate': 2e-05, 'epoch': 0.11} 11%|█ | 197/1817 [7:10:39<58:49:23, 130.72s/it] 11%|█ | 198/1817 [7:12:51<59:02:02, 131.27s/it] {'loss': 1.9507, 'learning_rate': 2e-05, 'epoch': 0.11} 11%|█ | 198/1817 [7:12:51<59:02:02, 131.27s/it] 11%|█ | 199/1817 [7:15:00<58:35:27, 130.36s/it] {'loss': 1.9376, 'learning_rate': 2e-05, 'epoch': 0.11} 11%|█ | 199/1817 [7:15:00<58:35:27, 130.36s/it] 11%|█ | 200/1817 [7:17:12<58:50:55, 131.02s/it] {'loss': 1.9479, 'learning_rate': 2e-05, 'epoch': 0.11} 11%|█ | 200/1817 [7:17:12<58:50:55, 131.02s/it] 11%|█ | 201/1817 [7:19:23<58:43:55, 130.84s/it] {'loss': 1.9697, 'learning_rate': 2e-05, 'epoch': 0.11} 11%|█ | 201/1817 [7:19:23<58:43:55, 130.84s/it] 11%|█ | 202/1817 [7:21:35<58:55:44, 131.36s/it] {'loss': 1.9159, 'learning_rate': 2e-05, 'epoch': 0.11} 11%|█ | 202/1817 [7:21:35<58:55:43, 131.36s/it] 11%|█ | 203/1817 [7:23:44<58:31:42, 130.55s/it] {'loss': 1.9402, 'learning_rate': 2e-05, 'epoch': 0.11} 11%|█ | 203/1817 [7:23:44<58:31:42, 130.55s/it] 11%|█ | 204/1817 [7:25:57<58:52:03, 131.38s/it] {'loss': 1.959, 'learning_rate': 2e-05, 'epoch': 0.11} 11%|█ | 204/1817 [7:25:57<58:52:03, 131.38s/it] 11%|█▏ | 205/1817 [7:28:07<58:36:17, 130.88s/it] {'loss': 1.9156, 'learning_rate': 2e-05, 'epoch': 0.11} 11%|█▏ | 205/1817 [7:28:07<58:36:17, 130.88s/it] 11%|█▏ | 206/1817 [7:30:16<58:20:34, 130.38s/it] {'loss': 1.9553, 'learning_rate': 2e-05, 'epoch': 0.11} 11%|█▏ | 206/1817 [7:30:16<58:20:34, 130.38s/it] 11%|█▏ | 207/1817 [7:32:25<58:04:35, 129.86s/it] {'loss': 1.9796, 'learning_rate': 2e-05, 'epoch': 0.11} 11%|█▏ | 207/1817 [7:32:25<58:04:35, 129.86s/it] 11%|█▏ | 208/1817 [7:34:33<57:45:50, 129.24s/it] {'loss': 1.9644, 'learning_rate': 2e-05, 'epoch': 0.11} 11%|█▏ | 208/1817 [7:34:33<57:45:50, 129.24s/it] 12%|█▏ | 209/1817 [7:36:46<58:13:37, 130.36s/it] {'loss': 1.9668, 'learning_rate': 2e-05, 'epoch': 0.11} 12%|█▏ | 209/1817 [7:36:46<58:13:37, 130.36s/it] 12%|█▏ | 210/1817 [7:38:53<57:46:35, 129.43s/it] {'loss': 1.9452, 'learning_rate': 2e-05, 'epoch': 0.12} 12%|█▏ | 210/1817 [7:38:53<57:46:35, 129.43s/it] 12%|█▏ | 211/1817 [7:41:02<57:44:55, 129.45s/it] {'loss': 1.9815, 'learning_rate': 2e-05, 'epoch': 0.12} 12%|█▏ | 211/1817 [7:41:02<57:44:55, 129.45s/it] 12%|█▏ | 212/1817 [7:43:13<57:52:39, 129.82s/it] {'loss': 1.9501, 'learning_rate': 2e-05, 'epoch': 0.12} 12%|█▏ | 212/1817 [7:43:13<57:52:39, 129.82s/it] 12%|█▏ | 213/1817 [7:45:26<58:15:54, 130.77s/it] {'loss': 1.9103, 'learning_rate': 2e-05, 'epoch': 0.12} 12%|█▏ | 213/1817 [7:45:26<58:15:53, 130.77s/it] 12%|█▏ | 214/1817 [7:47:38<58:20:53, 131.04s/it] {'loss': 1.9794, 'learning_rate': 2e-05, 'epoch': 0.12} 12%|█▏ | 214/1817 [7:47:38<58:20:52, 131.04s/it] 12%|█▏ | 215/1817 [7:49:50<58:27:39, 131.37s/it] {'loss': 1.9411, 'learning_rate': 2e-05, 'epoch': 0.12} 12%|█▏ | 215/1817 [7:49:50<58:27:39, 131.37s/it] 12%|█▏ | 216/1817 [7:51:57<57:47:47, 129.96s/it] {'loss': 1.961, 'learning_rate': 2e-05, 'epoch': 0.12} 12%|█▏ | 216/1817 [7:51:56<57:47:47, 129.96s/it] 12%|█▏ | 217/1817 [7:54:12<58:28:12, 131.56s/it] {'loss': 1.9455, 'learning_rate': 2e-05, 'epoch': 0.12} 12%|█▏ | 217/1817 [7:54:12<58:28:11, 131.56s/it] 12%|█▏ | 218/1817 [7:56:26<58:49:08, 132.43s/it] {'loss': 1.9706, 'learning_rate': 2e-05, 'epoch': 0.12} 12%|█▏ | 218/1817 [7:56:26<58:49:07, 132.43s/it] 12%|█▏ | 219/1817 [7:58:37<58:30:20, 131.80s/it] {'loss': 1.9509, 'learning_rate': 2e-05, 'epoch': 0.12} 12%|█▏ | 219/1817 [7:58:37<58:30:20, 131.80s/it] 12%|█▏ | 220/1817 [8:00:47<58:12:36, 131.22s/it] {'loss': 1.968, 'learning_rate': 2e-05, 'epoch': 0.12} 12%|█▏ | 220/1817 [8:00:46<58:12:36, 131.22s/it] 12%|█▏ | 221/1817 [8:03:00<58:27:14, 131.85s/it] {'loss': 1.9208, 'learning_rate': 2e-05, 'epoch': 0.12} 12%|█▏ | 221/1817 [8:03:00<58:27:13, 131.85s/it] 12%|█▏ | 222/1817 [8:05:08<57:51:23, 130.59s/it] {'loss': 1.967, 'learning_rate': 2e-05, 'epoch': 0.12} 12%|█▏ | 222/1817 [8:05:07<57:51:24, 130.59s/it] 12%|█▏ | 223/1817 [8:07:17<57:35:53, 130.08s/it] {'loss': 1.9506, 'learning_rate': 2e-05, 'epoch': 0.12} 12%|█▏ | 223/1817 [8:07:16<57:35:53, 130.08s/it] 12%|█▏ | 224/1817 [8:09:24<57:14:02, 129.34s/it] {'loss': 1.9276, 'learning_rate': 2e-05, 'epoch': 0.12} 12%|█▏ | 224/1817 [8:09:24<57:14:02, 129.34s/it] 12%|█▏ | 225/1817 [8:11:39<57:53:10, 130.90s/it] {'loss': 1.9118, 'learning_rate': 2e-05, 'epoch': 0.12} 12%|█▏ | 225/1817 [8:11:38<57:53:02, 130.89s/it] 12%|█▏ | 226/1817 [8:13:46<57:26:38, 129.98s/it] {'loss': 1.9462, 'learning_rate': 2e-05, 'epoch': 0.12} 12%|█▏ | 226/1817 [8:13:46<57:26:33, 129.98s/it] 12%|█▏ | 227/1817 [8:15:57<57:31:03, 130.23s/it] {'loss': 1.9488, 'learning_rate': 2e-05, 'epoch': 0.12} 12%|█▏ | 227/1817 [8:15:57<57:30:59, 130.23s/it] 13%|█▎ | 228/1817 [8:18:14<58:20:01, 132.16s/it] {'loss': 1.917, 'learning_rate': 2e-05, 'epoch': 0.13} 13%|█▎ | 228/1817 [8:18:14<58:20:02, 132.16s/it] 13%|█▎ | 229/1817 [8:20:23<57:52:00, 131.18s/it] {'loss': 1.9189, 'learning_rate': 2e-05, 'epoch': 0.13} 13%|█▎ | 229/1817 [8:20:23<57:52:01, 131.18s/it] 13%|█▎ | 230/1817 [8:22:35<57:57:11, 131.46s/it] {'loss': 1.9579, 'learning_rate': 2e-05, 'epoch': 0.13} 13%|█▎ | 230/1817 [8:22:35<57:57:11, 131.46s/it] 13%|█▎ | 231/1817 [8:24:45<57:43:20, 131.02s/it] {'loss': 1.9663, 'learning_rate': 2e-05, 'epoch': 0.13} 13%|█▎ | 231/1817 [8:24:45<57:43:21, 131.02s/it] 13%|█▎ | 232/1817 [8:26:55<57:36:38, 130.85s/it] {'loss': 1.9958, 'learning_rate': 2e-05, 'epoch': 0.13} 13%|█▎ | 232/1817 [8:26:55<57:36:39, 130.85s/it] 13%|█▎ | 233/1817 [8:29:05<57:22:27, 130.40s/it] {'loss': 1.9134, 'learning_rate': 2e-05, 'epoch': 0.13} 13%|█▎ | 233/1817 [8:29:05<57:22:26, 130.40s/it] 13%|█▎ | 234/1817 [8:31:15<57:15:14, 130.21s/it] {'loss': 1.9395, 'learning_rate': 2e-05, 'epoch': 0.13} 13%|█▎ | 234/1817 [8:31:14<57:15:15, 130.21s/it] 13%|█▎ | 235/1817 [8:33:24<57:06:49, 129.97s/it] {'loss': 1.9597, 'learning_rate': 2e-05, 'epoch': 0.13} 13%|█▎ | 235/1817 [8:33:24<57:06:49, 129.97s/it] 13%|█▎ | 236/1817 [8:35:33<56:54:57, 129.60s/it] {'loss': 1.9545, 'learning_rate': 2e-05, 'epoch': 0.13} 13%|█▎ | 236/1817 [8:35:32<56:54:57, 129.60s/it] 13%|█▎ | 237/1817 [8:37:44<57:07:06, 130.14s/it] {'loss': 1.9544, 'learning_rate': 2e-05, 'epoch': 0.13} 13%|█▎ | 237/1817 [8:37:44<57:07:07, 130.14s/it] 13%|█▎ | 238/1817 [8:39:59<57:42:50, 131.58s/it] {'loss': 1.9048, 'learning_rate': 2e-05, 'epoch': 0.13} 13%|█▎ | 238/1817 [8:39:59<57:42:50, 131.58s/it] 13%|█▎ | 239/1817 [8:42:10<57:31:56, 131.25s/it] {'loss': 1.9408, 'learning_rate': 2e-05, 'epoch': 0.13} 13%|█▎ | 239/1817 [8:42:09<57:31:56, 131.25s/it] 13%|█▎ | 240/1817 [8:44:20<57:25:37, 131.10s/it] {'loss': 1.9462, 'learning_rate': 2e-05, 'epoch': 0.13} 13%|█▎ | 240/1817 [8:44:20<57:25:37, 131.10s/it] 13%|█▎ | 241/1817 [8:46:30<57:13:18, 130.71s/it] {'loss': 1.9948, 'learning_rate': 2e-05, 'epoch': 0.13} 13%|█▎ | 241/1817 [8:46:30<57:13:19, 130.71s/it] 13%|█▎ | 242/1817 [8:48:40<57:04:57, 130.47s/it] {'loss': 1.8938, 'learning_rate': 2e-05, 'epoch': 0.13} 13%|█▎ | 242/1817 [8:48:40<57:04:57, 130.47s/it] 13%|█▎ | 243/1817 [8:50:52<57:18:29, 131.07s/it] {'loss': 1.9537, 'learning_rate': 2e-05, 'epoch': 0.13} 13%|█▎ | 243/1817 [8:50:52<57:18:32, 131.08s/it] 13%|█▎ | 244/1817 [8:53:07<57:46:21, 132.22s/it] {'loss': 1.9227, 'learning_rate': 2e-05, 'epoch': 0.13} 13%|█▎ | 244/1817 [8:53:07<57:46:24, 132.22s/it] 13%|█▎ | 245/1817 [8:55:17<57:22:06, 131.38s/it] {'loss': 1.9196, 'learning_rate': 2e-05, 'epoch': 0.13} 13%|█▎ | 245/1817 [8:55:17<57:22:09, 131.38s/it] 14%|█▎ | 246/1817 [8:57:29<57:23:46, 131.53s/it] {'loss': 1.8955, 'learning_rate': 2e-05, 'epoch': 0.14} 14%|█▎ | 246/1817 [8:57:28<57:23:47, 131.53s/it] 14%|█▎ | 247/1817 [8:59:40<57:20:40, 131.49s/it] {'loss': 1.9365, 'learning_rate': 2e-05, 'epoch': 0.14} 14%|█▎ | 247/1817 [8:59:40<57:20:44, 131.49s/it] 14%|█▎ | 248/1817 [9:01:54<57:40:21, 132.33s/it] {'loss': 1.9502, 'learning_rate': 2e-05, 'epoch': 0.14} 14%|█▎ | 248/1817 [9:01:54<57:40:22, 132.33s/it] 14%|█▎ | 249/1817 [9:04:05<57:23:23, 131.76s/it] {'loss': 1.9363, 'learning_rate': 2e-05, 'epoch': 0.14} 14%|█▎ | 249/1817 [9:04:05<57:23:26, 131.76s/it] 14%|█▍ | 250/1817 [9:06:13<56:51:07, 130.61s/it] {'loss': 1.9595, 'learning_rate': 2e-05, 'epoch': 0.14} 14%|█▍ | 250/1817 [9:06:12<56:51:08, 130.61s/it] 14%|█▍ | 251/1817 [9:08:25<56:58:56, 130.99s/it] {'loss': 1.9438, 'learning_rate': 2e-05, 'epoch': 0.14} 14%|█▍ | 251/1817 [9:08:24<56:58:56, 130.99s/it] 14%|█▍ | 252/1817 [9:10:35<56:53:24, 130.87s/it] {'loss': 1.9119, 'learning_rate': 2e-05, 'epoch': 0.14} 14%|█▍ | 252/1817 [9:10:35<56:53:18, 130.86s/it] 14%|█▍ | 253/1817 [9:12:45<56:40:21, 130.45s/it] {'loss': 1.9657, 'learning_rate': 2e-05, 'epoch': 0.14} 14%|█▍ | 253/1817 [9:12:44<56:40:16, 130.45s/it] 14%|█▍ | 254/1817 [9:14:53<56:24:09, 129.91s/it] {'loss': 1.9099, 'learning_rate': 2e-05, 'epoch': 0.14} 14%|█▍ | 254/1817 [9:14:53<56:24:05, 129.91s/it] 14%|█▍ | 255/1817 [9:16:59<55:53:16, 128.81s/it] {'loss': 1.9629, 'learning_rate': 2e-05, 'epoch': 0.14} 14%|█▍ | 255/1817 [9:16:59<55:53:16, 128.81s/it] 14%|█▍ | 256/1817 [9:19:06<55:36:00, 128.23s/it] {'loss': 1.9499, 'learning_rate': 2e-05, 'epoch': 0.14} 14%|█▍ | 256/1817 [9:19:06<55:36:05, 128.23s/it] 14%|█▍ | 257/1817 [9:21:18<56:03:33, 129.37s/it] {'loss': 1.9673, 'learning_rate': 2e-05, 'epoch': 0.14} 14%|█▍ | 257/1817 [9:21:18<56:03:37, 129.37s/it] 14%|█▍ | 258/1817 [9:23:28<56:01:49, 129.38s/it] {'loss': 1.9543, 'learning_rate': 2e-05, 'epoch': 0.14} 14%|█▍ | 258/1817 [9:23:28<56:01:53, 129.39s/it] 14%|█▍ | 259/1817 [9:25:38<56:07:28, 129.68s/it] {'loss': 1.9672, 'learning_rate': 2e-05, 'epoch': 0.14} 14%|█▍ | 259/1817 [9:25:38<56:07:32, 129.69s/it] 14%|█▍ | 260/1817 [9:27:47<55:59:43, 129.47s/it] {'loss': 1.9481, 'learning_rate': 2e-05, 'epoch': 0.14} 14%|█▍ | 260/1817 [9:27:47<55:59:46, 129.47s/it] 14%|█▍ | 261/1817 [9:29:57<55:56:54, 129.44s/it] {'loss': 1.9735, 'learning_rate': 2e-05, 'epoch': 0.14} 14%|█▍ | 261/1817 [9:29:56<55:56:57, 129.45s/it] 14%|█▍ | 262/1817 [9:32:07<56:02:37, 129.75s/it] {'loss': 1.9355, 'learning_rate': 2e-05, 'epoch': 0.14} 14%|█▍ | 262/1817 [9:32:07<56:02:37, 129.75s/it] 14%|█▍ | 263/1817 [9:34:18<56:12:13, 130.20s/it] {'loss': 1.8911, 'learning_rate': 2e-05, 'epoch': 0.14} 14%|█▍ | 263/1817 [9:34:18<56:12:13, 130.20s/it] 15%|█▍ | 264/1817 [9:36:29<56:16:49, 130.46s/it] {'loss': 1.94, 'learning_rate': 2e-05, 'epoch': 0.15} 15%|█▍ | 264/1817 [9:36:29<56:16:53, 130.47s/it] 15%|█▍ | 265/1817 [9:38:43<56:40:17, 131.45s/it] {'loss': 1.9752, 'learning_rate': 2e-05, 'epoch': 0.15} 15%|█▍ | 265/1817 [9:38:43<56:40:17, 131.45s/it] 15%|█▍ | 266/1817 [9:40:52<56:19:28, 130.73s/it] {'loss': 1.9546, 'learning_rate': 2e-05, 'epoch': 0.15} 15%|█▍ | 266/1817 [9:40:52<56:19:28, 130.73s/it] 15%|█▍ | 267/1817 [9:43:03<56:17:00, 130.72s/it] {'loss': 1.9282, 'learning_rate': 2e-05, 'epoch': 0.15} 15%|█▍ | 267/1817 [9:43:03<56:17:03, 130.73s/it] 15%|█▍ | 268/1817 [9:45:12<56:01:50, 130.22s/it] {'loss': 1.8975, 'learning_rate': 2e-05, 'epoch': 0.15} 15%|█▍ | 268/1817 [9:45:12<56:01:53, 130.22s/it] 15%|█▍ | 269/1817 [9:47:24<56:10:42, 130.65s/it] {'loss': 1.9477, 'learning_rate': 2e-05, 'epoch': 0.15} 15%|█▍ | 269/1817 [9:47:23<56:10:45, 130.65s/it] 15%|█▍ | 270/1817 [9:49:34<56:09:18, 130.68s/it] {'loss': 1.9196, 'learning_rate': 2e-05, 'epoch': 0.15} 15%|█▍ | 270/1817 [9:49:34<56:09:18, 130.68s/it] 15%|█▍ | 271/1817 [9:51:43<55:48:17, 129.95s/it] {'loss': 1.9826, 'learning_rate': 2e-05, 'epoch': 0.15} 15%|█▍ | 271/1817 [9:51:42<55:48:17, 129.95s/it] 15%|█▍ | 272/1817 [9:53:49<55:17:43, 128.84s/it] {'loss': 1.9169, 'learning_rate': 2e-05, 'epoch': 0.15} 15%|█▍ | 272/1817 [9:53:49<55:17:46, 128.85s/it] 15%|█▌ | 273/1817 [9:56:00<55:32:26, 129.50s/it] {'loss': 1.9188, 'learning_rate': 2e-05, 'epoch': 0.15} 15%|█▌ | 273/1817 [9:56:00<55:32:29, 129.50s/it] 15%|█▌ | 274/1817 [9:58:13<56:01:55, 130.73s/it] {'loss': 1.9373, 'learning_rate': 2e-05, 'epoch': 0.15} 15%|█▌ | 274/1817 [9:58:13<56:01:58, 130.73s/it] 15%|█▌ | 275/1817 [10:00:28<56:26:15, 131.76s/it] {'loss': 1.9809, 'learning_rate': 2e-05, 'epoch': 0.15} 15%|█▌ | 275/1817 [10:00:27<56:26:19, 131.76s/it] 15%|█▌ | 276/1817 [10:02:33<55:37:46, 129.96s/it] {'loss': 1.9022, 'learning_rate': 2e-05, 'epoch': 0.15} 15%|█▌ | 276/1817 [10:02:33<55:37:49, 129.96s/it] 15%|█▌ | 277/1817 [10:04:45<55:45:06, 130.33s/it] {'loss': 1.8952, 'learning_rate': 2e-05, 'epoch': 0.15} 15%|█▌ | 277/1817 [10:04:44<55:45:05, 130.33s/it] 15%|█▌ | 278/1817 [10:06:55<55:46:51, 130.48s/it] {'loss': 1.8861, 'learning_rate': 2e-05, 'epoch': 0.15} 15%|█▌ | 278/1817 [10:06:55<55:46:54, 130.48s/it] 15%|█▌ | 279/1817 [10:09:08<56:01:29, 131.14s/it] {'loss': 1.9292, 'learning_rate': 2e-05, 'epoch': 0.15} 15%|█▌ | 279/1817 [10:09:08<56:01:10, 131.13s/it] 15%|█▌ | 280/1817 [10:11:17<55:44:46, 130.57s/it] {'loss': 1.9068, 'learning_rate': 2e-05, 'epoch': 0.15} 15%|█▌ | 280/1817 [10:11:17<55:44:13, 130.55s/it] 15%|█▌ | 281/1817 [10:13:27<55:32:25, 130.17s/it] {'loss': 1.9268, 'learning_rate': 2e-05, 'epoch': 0.15} 15%|█▌ | 281/1817 [10:13:26<55:32:03, 130.16s/it] 16%|█▌ | 282/1817 [10:15:39<55:47:11, 130.83s/it] {'loss': 1.9485, 'learning_rate': 2e-05, 'epoch': 0.16} 16%|█▌ | 282/1817 [10:15:39<55:47:12, 130.84s/it] 16%|█▌ | 283/1817 [10:17:51<55:51:01, 131.07s/it] {'loss': 1.9, 'learning_rate': 2e-05, 'epoch': 0.16} 16%|█▌ | 283/1817 [10:17:50<55:51:06, 131.07s/it] 16%|█▌ | 284/1817 [10:20:01<55:40:11, 130.73s/it] {'loss': 1.9643, 'learning_rate': 2e-05, 'epoch': 0.16} 16%|█▌ | 284/1817 [10:20:00<55:40:15, 130.73s/it] 16%|█▌ | 285/1817 [10:22:10<55:32:03, 130.50s/it] {'loss': 1.9583, 'learning_rate': 2e-05, 'epoch': 0.16} 16%|█▌ | 285/1817 [10:22:10<55:32:07, 130.50s/it] 16%|█▌ | 286/1817 [10:24:23<55:47:52, 131.20s/it] {'loss': 1.8767, 'learning_rate': 2e-05, 'epoch': 0.16} 16%|█▌ | 286/1817 [10:24:23<55:47:52, 131.20s/it] 16%|█▌ | 287/1817 [10:26:32<55:26:51, 130.46s/it] {'loss': 1.949, 'learning_rate': 2e-05, 'epoch': 0.16} 16%|█▌ | 287/1817 [10:26:32<55:26:54, 130.47s/it] 16%|█▌ | 288/1817 [10:28:41<55:15:09, 130.09s/it] {'loss': 1.9495, 'learning_rate': 2e-05, 'epoch': 0.16} 16%|█▌ | 288/1817 [10:28:41<55:15:13, 130.09s/it] 16%|█▌ | 289/1817 [10:30:52<55:15:46, 130.20s/it] {'loss': 1.9053, 'learning_rate': 2e-05, 'epoch': 0.16} 16%|█▌ | 289/1817 [10:30:51<55:15:46, 130.20s/it] 16%|█▌ | 290/1817 [10:33:06<55:41:24, 131.29s/it] {'loss': 1.9621, 'learning_rate': 2e-05, 'epoch': 0.16} 16%|█▌ | 290/1817 [10:33:05<55:41:27, 131.29s/it] 16%|█▌ | 291/1817 [10:35:18<55:50:44, 131.75s/it] {'loss': 1.9142, 'learning_rate': 2e-05, 'epoch': 0.16} 16%|█▌ | 291/1817 [10:35:18<55:50:47, 131.75s/it] 16%|█▌ | 292/1817 [10:37:29<55:41:08, 131.45s/it] {'loss': 1.9085, 'learning_rate': 2e-05, 'epoch': 0.16} 16%|█▌ | 292/1817 [10:37:29<55:41:12, 131.46s/it] 16%|█▌ | 293/1817 [10:39:41<55:40:09, 131.50s/it] {'loss': 1.9545, 'learning_rate': 2e-05, 'epoch': 0.16} 16%|█▌ | 293/1817 [10:39:41<55:40:09, 131.50s/it] 16%|█▌ | 294/1817 [10:41:56<56:03:38, 132.51s/it] {'loss': 1.903, 'learning_rate': 2e-05, 'epoch': 0.16} 16%|█▌ | 294/1817 [10:41:55<56:03:38, 132.51s/it] 16%|█▌ | 295/1817 [10:44:04<55:26:23, 131.13s/it] {'loss': 1.9391, 'learning_rate': 2e-05, 'epoch': 0.16} 16%|█▌ | 295/1817 [10:44:03<55:26:23, 131.13s/it] 16%|█▋ | 296/1817 [10:46:11<54:57:30, 130.08s/it] {'loss': 1.9512, 'learning_rate': 2e-05, 'epoch': 0.16} 16%|█▋ | 296/1817 [10:46:11<54:57:33, 130.08s/it] 16%|█▋ | 297/1817 [10:48:24<55:13:12, 130.78s/it] {'loss': 1.9373, 'learning_rate': 2e-05, 'epoch': 0.16} 16%|█▋ | 297/1817 [10:48:23<55:13:15, 130.79s/it] 16%|█▋ | 298/1817 [10:50:37<55:28:02, 131.46s/it] {'loss': 1.9162, 'learning_rate': 2e-05, 'epoch': 0.16} 16%|█▋ | 298/1817 [10:50:36<55:28:05, 131.46s/it] 16%|█▋ | 299/1817 [10:52:49<55:34:06, 131.78s/it] {'loss': 1.8918, 'learning_rate': 2e-05, 'epoch': 0.16} 16%|█▋ | 299/1817 [10:52:49<55:34:09, 131.78s/it] 17%|█▋ | 300/1817 [10:55:02<55:37:44, 132.01s/it] {'loss': 1.9439, 'learning_rate': 2e-05, 'epoch': 0.17} 17%|█▋ | 300/1817 [10:55:01<55:37:43, 132.01s/it] 17%|█▋ | 301/1817 [10:57:13<55:29:21, 131.77s/it] {'loss': 1.9044, 'learning_rate': 2e-05, 'epoch': 0.17} 17%|█▋ | 301/1817 [10:57:13<55:29:24, 131.77s/it] 17%|█▋ | 302/1817 [10:59:23<55:10:31, 131.11s/it] {'loss': 1.8871, 'learning_rate': 2e-05, 'epoch': 0.17} 17%|█▋ | 302/1817 [10:59:22<55:10:34, 131.11s/it] 17%|█▋ | 303/1817 [11:01:34<55:07:47, 131.09s/it] {'loss': 1.9389, 'learning_rate': 2e-05, 'epoch': 0.17} 17%|█▋ | 303/1817 [11:01:33<55:07:47, 131.09s/it] 17%|█▋ | 304/1817 [11:03:45<55:08:53, 131.22s/it] {'loss': 1.9595, 'learning_rate': 2e-05, 'epoch': 0.17} 17%|█▋ | 304/1817 [11:03:45<55:08:53, 131.22s/it] 17%|█▋ | 305/1817 [11:05:53<54:41:24, 130.21s/it] {'loss': 1.9291, 'learning_rate': 2e-05, 'epoch': 0.17} 17%|█▋ | 305/1817 [11:05:53<54:41:24, 130.21s/it] 17%|█▋ | 306/1817 [11:08:07<55:04:52, 131.23s/it] {'loss': 1.8655, 'learning_rate': 2e-05, 'epoch': 0.17} 17%|█▋ | 306/1817 [11:08:06<55:04:47, 131.23s/it] 17%|█▋ | 307/1817 [11:10:15<54:43:33, 130.47s/it] {'loss': 1.9569, 'learning_rate': 2e-05, 'epoch': 0.17} 17%|█▋ | 307/1817 [11:10:15<54:43:01, 130.45s/it] 17%|█▋ | 308/1817 [11:12:25<54:38:52, 130.37s/it] {'loss': 1.9349, 'learning_rate': 2e-05, 'epoch': 0.17} 17%|█▋ | 308/1817 [11:12:25<54:38:58, 130.38s/it] {'loss': 1.9349, 'learning_rate': 2e-05, 'epoch': 0.17} 17%|█▋ | 308/1817 [11:12:25<54:38:52, 130.37s/it] 17%|█▋ | 309/1817 [11:14:35<54:28:39, 130.05s/it] {'loss': 1.9086, 'learning_rate': 2e-05, 'epoch': 0.17} 17%|█▋ | 309/1817 [11:14:34<54:28:39, 130.05s/it] 17%|█▋ | 310/1817 [11:16:43<54:13:14, 129.52s/it] {'loss': 1.9066, 'learning_rate': 2e-05, 'epoch': 0.17} 17%|█▋ | 310/1817 [11:16:43<54:13:14, 129.53s/it] 17%|█▋ | 311/1817 [11:18:54<54:24:25, 130.06s/it] {'loss': 1.9347, 'learning_rate': 2e-05, 'epoch': 0.17} 17%|█▋ | 311/1817 [11:18:54<54:24:25, 130.06s/it] 17%|█▋ | 312/1817 [11:21:04<54:20:29, 129.99s/it] {'loss': 1.9328, 'learning_rate': 2e-05, 'epoch': 0.17} 17%|█▋ | 312/1817 [11:21:04<54:20:33, 129.99s/it] 17%|█▋ | 313/1817 [11:23:12<54:05:18, 129.47s/it] {'loss': 1.8877, 'learning_rate': 2e-05, 'epoch': 0.17} 17%|█▋ | 313/1817 [11:23:12<54:05:18, 129.47s/it] 17%|█▋ | 314/1817 [11:25:23<54:14:24, 129.92s/it] {'loss': 1.9743, 'learning_rate': 2e-05, 'epoch': 0.17} 17%|█▋ | 314/1817 [11:25:23<54:14:24, 129.92s/it] 17%|█▋ | 315/1817 [11:27:32<54:05:43, 129.66s/it] {'loss': 1.9331, 'learning_rate': 2e-05, 'epoch': 0.17} 17%|█▋ | 315/1817 [11:27:32<54:05:43, 129.66s/it] 17%|█▋ | 316/1817 [11:29:42<54:04:21, 129.69s/it] {'loss': 1.9266, 'learning_rate': 2e-05, 'epoch': 0.17} 17%|█▋ | 316/1817 [11:29:42<54:04:24, 129.69s/it] 17%|█▋ | 317/1817 [11:31:54<54:17:19, 130.29s/it] {'loss': 1.9407, 'learning_rate': 2e-05, 'epoch': 0.17} 17%|█▋ | 317/1817 [11:31:54<54:17:18, 130.29s/it] 18%|█▊ | 318/1817 [11:34:10<54:57:42, 132.00s/it] {'loss': 1.9479, 'learning_rate': 2e-05, 'epoch': 0.17} 18%|█▊ | 318/1817 [11:34:10<54:57:42, 132.00s/it] 18%|█▊ | 319/1817 [11:36:19<54:32:32, 131.08s/it] {'loss': 1.9474, 'learning_rate': 2e-05, 'epoch': 0.18} 18%|█▊ | 319/1817 [11:36:18<54:32:35, 131.08s/it] 18%|█▊ | 320/1817 [11:38:32<54:49:33, 131.85s/it] {'loss': 1.9499, 'learning_rate': 2e-05, 'epoch': 0.18} 18%|█▊ | 320/1817 [11:38:32<54:49:36, 131.85s/it] 18%|█▊ | 321/1817 [11:40:49<55:24:58, 133.35s/it] {'loss': 1.976, 'learning_rate': 2e-05, 'epoch': 0.18} 18%|█▊ | 321/1817 [11:40:49<55:24:58, 133.35s/it] 18%|█▊ | 322/1817 [11:43:00<55:01:30, 132.50s/it] {'loss': 1.9193, 'learning_rate': 2e-05, 'epoch': 0.18} 18%|█▊ | 322/1817 [11:43:00<55:01:30, 132.50s/it] 18%|█▊ | 323/1817 [11:45:10<54:44:28, 131.91s/it] {'loss': 1.9469, 'learning_rate': 2e-05, 'epoch': 0.18} 18%|█▊ | 323/1817 [11:45:10<54:44:28, 131.91s/it] 18%|█▊ | 324/1817 [11:47:20<54:27:31, 131.31s/it] {'loss': 1.8843, 'learning_rate': 2e-05, 'epoch': 0.18} 18%|█▊ | 324/1817 [11:47:20<54:27:35, 131.32s/it] 18%|█▊ | 325/1817 [11:49:31<54:24:45, 131.29s/it] {'loss': 1.9098, 'learning_rate': 2e-05, 'epoch': 0.18} 18%|█▊ | 325/1817 [11:49:31<54:24:48, 131.29s/it] 18%|█▊ | 326/1817 [11:51:42<54:14:49, 130.98s/it] {'loss': 1.9359, 'learning_rate': 2e-05, 'epoch': 0.18} 18%|█▊ | 326/1817 [11:51:41<54:14:49, 130.98s/it] 18%|█▊ | 327/1817 [11:53:53<54:15:14, 131.08s/it] {'loss': 1.9316, 'learning_rate': 2e-05, 'epoch': 0.18} 18%|█▊ | 327/1817 [11:53:53<54:15:17, 131.09s/it] 18%|█▊ | 328/1817 [11:56:03<54:03:12, 130.69s/it] {'loss': 1.9246, 'learning_rate': 2e-05, 'epoch': 0.18} 18%|█▊ | 328/1817 [11:56:03<54:03:11, 130.69s/it] 18%|█▊ | 329/1817 [11:58:13<53:57:59, 130.56s/it] {'loss': 1.8812, 'learning_rate': 2e-05, 'epoch': 0.18} 18%|█▊ | 329/1817 [11:58:13<53:58:02, 130.57s/it] 18%|█▊ | 330/1817 [12:00:24<53:56:14, 130.58s/it] {'loss': 1.8763, 'learning_rate': 2e-05, 'epoch': 0.18} 18%|█▊ | 330/1817 [12:00:23<53:56:17, 130.58s/it] 18%|█▊ | 331/1817 [12:02:34<53:54:26, 130.60s/it] {'loss': 1.9675, 'learning_rate': 2e-05, 'epoch': 0.18} 18%|█▊ | 331/1817 [12:02:34<53:54:29, 130.60s/it] 18%|█▊ | 332/1817 [12:04:44<53:46:39, 130.37s/it] {'loss': 1.9428, 'learning_rate': 2e-05, 'epoch': 0.18} 18%|█▊ | 332/1817 [12:04:44<53:46:43, 130.37s/it] 18%|█▊ | 333/1817 [12:06:58<54:07:50, 131.31s/it] {'loss': 1.8906, 'learning_rate': 2e-05, 'epoch': 0.18} 18%|█▊ | 333/1817 [12:06:57<54:07:50, 131.31s/it] 18%|█▊ | 334/1817 [12:09:13<54:33:54, 132.46s/it] {'loss': 1.9253, 'learning_rate': 2e-05, 'epoch': 0.18} 18%|█▊ | 334/1817 [12:09:13<54:33:53, 132.46s/it] 18%|█▊ | 335/1817 [12:11:24<54:21:58, 132.06s/it] {'loss': 1.9274, 'learning_rate': 2e-05, 'epoch': 0.18} 18%|█▊ | 335/1817 [12:11:24<54:21:58, 132.06s/it] 18%|█▊ | 336/1817 [12:13:37<54:23:54, 132.23s/it] {'loss': 1.9056, 'learning_rate': 2e-05, 'epoch': 0.18} 18%|█▊ | 336/1817 [12:13:36<54:23:31, 132.22s/it] 19%|█▊ | 337/1817 [12:15:44<53:47:39, 130.85s/it] {'loss': 1.895, 'learning_rate': 2e-05, 'epoch': 0.19} 19%|█▊ | 337/1817 [12:15:44<53:47:39, 130.85s/it] 19%|█▊ | 338/1817 [12:17:56<53:55:27, 131.26s/it] {'loss': 1.9145, 'learning_rate': 2e-05, 'epoch': 0.19} 19%|█▊ | 338/1817 [12:17:56<53:55:27, 131.26s/it] 19%|█▊ | 339/1817 [12:20:11<54:14:23, 132.11s/it] {'loss': 1.8891, 'learning_rate': 2e-05, 'epoch': 0.19} 19%|█▊ | 339/1817 [12:20:10<54:14:27, 132.12s/it] 19%|█▊ | 340/1817 [12:22:20<53:53:07, 131.34s/it] {'loss': 1.8982, 'learning_rate': 2e-05, 'epoch': 0.19} 19%|█▊ | 340/1817 [12:22:20<53:53:07, 131.34s/it] 19%|█▉ | 341/1817 [12:24:28<53:29:17, 130.46s/it] {'loss': 1.9535, 'learning_rate': 2e-05, 'epoch': 0.19} 19%|█▉ | 341/1817 [12:24:28<53:29:17, 130.46s/it] 19%|█▉ | 342/1817 [12:26:38<53:22:05, 130.25s/it] {'loss': 1.9567, 'learning_rate': 2e-05, 'epoch': 0.19} 19%|█▉ | 342/1817 [12:26:38<53:22:05, 130.25s/it] 19%|█▉ | 343/1817 [12:28:48<53:18:12, 130.18s/it] {'loss': 1.9281, 'learning_rate': 2e-05, 'epoch': 0.19} 19%|█▉ | 343/1817 [12:28:48<53:18:12, 130.18s/it] 19%|█▉ | 344/1817 [12:30:59<53:17:51, 130.26s/it] {'loss': 1.9092, 'learning_rate': 2e-05, 'epoch': 0.19} 19%|█▉ | 344/1817 [12:30:58<53:17:50, 130.26s/it] 19%|█▉ | 345/1817 [12:33:12<53:38:17, 131.18s/it] {'loss': 1.8886, 'learning_rate': 2e-05, 'epoch': 0.19} 19%|█▉ | 345/1817 [12:33:12<53:38:17, 131.18s/it] 19%|█▉ | 346/1817 [12:35:26<53:53:40, 131.90s/it] {'loss': 1.9664, 'learning_rate': 2e-05, 'epoch': 0.19} 19%|█▉ | 346/1817 [12:35:25<53:53:40, 131.90s/it] 19%|█▉ | 347/1817 [12:37:34<53:28:27, 130.96s/it] {'loss': 1.9383, 'learning_rate': 2e-05, 'epoch': 0.19} 19%|█▉ | 347/1817 [12:37:34<53:28:27, 130.96s/it] 19%|█▉ | 348/1817 [12:39:43<53:11:06, 130.34s/it] {'loss': 1.9858, 'learning_rate': 2e-05, 'epoch': 0.19} 19%|█▉ | 348/1817 [12:39:43<53:11:07, 130.34s/it] 19%|█▉ | 349/1817 [12:41:56<53:30:04, 131.20s/it] {'loss': 1.9001, 'learning_rate': 2e-05, 'epoch': 0.19} 19%|█▉ | 349/1817 [12:41:56<53:30:04, 131.20s/it] 19%|█▉ | 350/1817 [12:44:13<54:06:54, 132.80s/it] {'loss': 1.9353, 'learning_rate': 2e-05, 'epoch': 0.19} 19%|█▉ | 350/1817 [12:44:13<54:06:54, 132.80s/it] 19%|█▉ | 351/1817 [12:46:26<54:07:39, 132.92s/it] {'loss': 1.9386, 'learning_rate': 2e-05, 'epoch': 0.19} 19%|█▉ | 351/1817 [12:46:26<54:07:39, 132.92s/it] 19%|█▉ | 352/1817 [12:48:36<53:40:32, 131.90s/it] {'loss': 1.907, 'learning_rate': 2e-05, 'epoch': 0.19} 19%|█▉ | 352/1817 [12:48:35<53:40:32, 131.90s/it] 19%|█▉ | 353/1817 [12:50:46<53:29:02, 131.52s/it] {'loss': 1.9296, 'learning_rate': 2e-05, 'epoch': 0.19} 19%|█▉ | 353/1817 [12:50:46<53:29:02, 131.52s/it] 19%|█▉ | 354/1817 [12:53:00<53:44:09, 132.23s/it] {'loss': 1.9351, 'learning_rate': 2e-05, 'epoch': 0.19} 19%|█▉ | 354/1817 [12:53:00<53:44:10, 132.23s/it] 20%|█▉ | 355/1817 [12:55:14<53:55:34, 132.79s/it] {'loss': 1.8881, 'learning_rate': 2e-05, 'epoch': 0.2} 20%|█▉ | 355/1817 [12:55:14<53:55:37, 132.79s/it] 20%|█▉ | 356/1817 [12:57:27<53:49:30, 132.63s/it] {'loss': 1.9738, 'learning_rate': 2e-05, 'epoch': 0.2} 20%|█▉ | 356/1817 [12:57:26<53:49:30, 132.63s/it] 20%|█▉ | 357/1817 [12:59:40<53:50:05, 132.74s/it] {'loss': 1.9434, 'learning_rate': 2e-05, 'epoch': 0.2} 20%|█▉ | 357/1817 [12:59:39<53:50:09, 132.75s/it] 20%|█▉ | 358/1817 [13:01:52<53:44:10, 132.59s/it] {'loss': 1.9527, 'learning_rate': 2e-05, 'epoch': 0.2} 20%|█▉ | 358/1817 [13:01:52<53:44:10, 132.59s/it] 20%|█▉ | 359/1817 [13:04:02<53:24:52, 131.89s/it] {'loss': 1.9727, 'learning_rate': 2e-05, 'epoch': 0.2} 20%|█▉ | 359/1817 [13:04:02<53:24:53, 131.89s/it] 20%|█▉ | 360/1817 [13:06:12<53:07:15, 131.25s/it] {'loss': 1.9641, 'learning_rate': 2e-05, 'epoch': 0.2} 20%|█▉ | 360/1817 [13:06:12<53:07:15, 131.25s/it] 20%|█▉ | 361/1817 [13:08:25<53:20:11, 131.88s/it] {'loss': 1.9478, 'learning_rate': 2e-05, 'epoch': 0.2} 20%|█▉ | 361/1817 [13:08:25<53:20:11, 131.88s/it] 20%|█▉ | 362/1817 [13:10:38<53:22:46, 132.07s/it] {'loss': 1.9147, 'learning_rate': 2e-05, 'epoch': 0.2} 20%|█▉ | 362/1817 [13:10:37<53:22:41, 132.07s/it] 20%|█▉ | 363/1817 [13:12:51<53:31:36, 132.53s/it] {'loss': 1.9257, 'learning_rate': 2e-05, 'epoch': 0.2} 20%|█▉ | 363/1817 [13:12:51<53:31:32, 132.53s/it] 20%|██ | 364/1817 [13:15:02<53:17:35, 132.04s/it] {'loss': 1.9243, 'learning_rate': 2e-05, 'epoch': 0.2} 20%|██ | 364/1817 [13:15:02<53:17:32, 132.04s/it] 20%|██ | 365/1817 [13:17:14<53:19:05, 132.19s/it] {'loss': 1.9099, 'learning_rate': 2e-05, 'epoch': 0.2} 20%|██ | 365/1817 [13:17:15<53:19:05, 132.19s/it] 20%|██ | 366/1817 [13:19:24<52:52:10, 131.17s/it] {'loss': 1.9197, 'learning_rate': 2e-05, 'epoch': 0.2} 20%|██ | 366/1817 [13:19:23<52:52:11, 131.17s/it] 20%|██ | 367/1817 [13:21:33<52:39:48, 130.75s/it] {'loss': 1.9701, 'learning_rate': 2e-05, 'epoch': 0.2} 20%|██ | 367/1817 [13:21:33<52:39:48, 130.75s/it] 20%|██ | 368/1817 [13:23:41<52:15:37, 129.84s/it] {'loss': 1.932, 'learning_rate': 2e-05, 'epoch': 0.2} 20%|██ | 368/1817 [13:23:41<52:15:38, 129.84s/it] 20%|██ | 369/1817 [13:25:52<52:23:39, 130.26s/it] {'loss': 1.9519, 'learning_rate': 2e-05, 'epoch': 0.2} 20%|██ | 369/1817 [13:25:52<52:23:39, 130.26s/it] 20%|██ | 370/1817 [13:28:03<52:23:49, 130.36s/it] {'loss': 1.9506, 'learning_rate': 2e-05, 'epoch': 0.2} 20%|██ | 370/1817 [13:28:03<52:23:50, 130.36s/it] 20%|██ | 371/1817 [13:30:13<52:19:03, 130.25s/it] {'loss': 1.9211, 'learning_rate': 2e-05, 'epoch': 0.2} 20%|██ | 371/1817 [13:30:13<52:19:03, 130.25s/it] 20%|██ | 372/1817 [13:32:23<52:15:48, 130.21s/it] {'loss': 1.972, 'learning_rate': 2e-05, 'epoch': 0.2} 20%|██ | 372/1817 [13:32:23<52:15:49, 130.21s/it] 21%|██ | 373/1817 [13:34:31<51:59:40, 129.63s/it] {'loss': 1.9346, 'learning_rate': 2e-05, 'epoch': 0.21} 21%|██ | 373/1817 [13:34:31<51:59:41, 129.63s/it] 21%|██ | 374/1817 [13:36:43<52:10:03, 130.15s/it] {'loss': 1.9269, 'learning_rate': 2e-05, 'epoch': 0.21} 21%|██ | 374/1817 [13:36:42<52:10:03, 130.15s/it] 21%|██ | 375/1817 [13:38:56<52:32:44, 131.18s/it] {'loss': 1.9373, 'learning_rate': 2e-05, 'epoch': 0.21} 21%|██ | 375/1817 [13:38:56<52:32:44, 131.18s/it] 21%|██ | 376/1817 [13:41:06<52:19:18, 130.71s/it] {'loss': 1.9482, 'learning_rate': 2e-05, 'epoch': 0.21} 21%|██ | 376/1817 [13:41:06<52:19:19, 130.71s/it] 21%|██ | 377/1817 [13:43:17<52:19:10, 130.80s/it] {'loss': 1.9447, 'learning_rate': 2e-05, 'epoch': 0.21} 21%|██ | 377/1817 [13:43:17<52:19:10, 130.80s/it] 21%|██ | 378/1817 [13:45:28<52:20:20, 130.94s/it] {'loss': 1.9138, 'learning_rate': 2e-05, 'epoch': 0.21} 21%|██ | 378/1817 [13:45:28<52:20:20, 130.94s/it] 21%|██ | 379/1817 [13:47:38<52:11:37, 130.67s/it] {'loss': 1.9109, 'learning_rate': 2e-05, 'epoch': 0.21} 21%|██ | 379/1817 [13:47:38<52:11:37, 130.67s/it] 21%|██ | 380/1817 [13:49:50<52:19:08, 131.07s/it] {'loss': 1.9492, 'learning_rate': 2e-05, 'epoch': 0.21} 21%|██ | 380/1817 [13:49:50<52:19:08, 131.07s/it] 21%|██ | 381/1817 [13:52:04<52:39:43, 132.02s/it] {'loss': 1.9191, 'learning_rate': 2e-05, 'epoch': 0.21} 21%|██ | 381/1817 [13:52:04<52:39:42, 132.02s/it] 21%|██ | 382/1817 [13:54:16<52:33:37, 131.86s/it] {'loss': 1.9433, 'learning_rate': 2e-05, 'epoch': 0.21} 21%|██ | 382/1817 [13:54:16<52:33:37, 131.86s/it] 21%|██ | 383/1817 [13:56:29<52:39:33, 132.20s/it] {'loss': 1.9311, 'learning_rate': 2e-05, 'epoch': 0.21} 21%|██ | 383/1817 [13:56:29<52:39:33, 132.20s/it] 21%|██ | 384/1817 [13:58:38<52:17:56, 131.39s/it] {'loss': 1.9495, 'learning_rate': 2e-05, 'epoch': 0.21} 21%|██ | 384/1817 [13:58:38<52:17:57, 131.39s/it] 21%|██ | 385/1817 [14:00:47<51:56:34, 130.58s/it] {'loss': 1.9153, 'learning_rate': 2e-05, 'epoch': 0.21} 21%|██ | 385/1817 [14:00:47<51:56:35, 130.58s/it] 21%|██ | 386/1817 [14:02:57<51:47:01, 130.27s/it] {'loss': 1.8873, 'learning_rate': 2e-05, 'epoch': 0.21} 21%|██ | 386/1817 [14:02:56<51:47:02, 130.27s/it] 21%|██▏ | 387/1817 [14:05:10<52:06:50, 131.20s/it] {'loss': 1.9039, 'learning_rate': 2e-05, 'epoch': 0.21} 21%|██▏ | 387/1817 [14:05:10<52:06:50, 131.20s/it] 21%|██▏ | 388/1817 [14:07:20<51:55:56, 130.83s/it] {'loss': 1.9385, 'learning_rate': 2e-05, 'epoch': 0.21} 21%|██▏ | 388/1817 [14:07:20<51:55:56, 130.83s/it] 21%|██▏ | 389/1817 [14:09:31<51:53:25, 130.82s/it] {'loss': 1.9252, 'learning_rate': 2e-05, 'epoch': 0.21} 21%|██▏ | 389/1817 [14:09:30<51:53:25, 130.82s/it] 21%|██▏ | 390/1817 [14:11:43<51:59:45, 131.17s/it] {'loss': 1.9127, 'learning_rate': 2e-05, 'epoch': 0.21} 21%|██▏ | 390/1817 [14:11:42<51:59:13, 131.15s/it] 22%|██▏ | 391/1817 [14:13:52<51:49:42, 130.84s/it] {'loss': 1.9363, 'learning_rate': 2e-05, 'epoch': 0.22} 22%|██▏ | 391/1817 [14:13:52<51:49:42, 130.84s/it] 22%|██▏ | 392/1817 [14:16:05<51:56:35, 131.22s/it] {'loss': 1.9493, 'learning_rate': 2e-05, 'epoch': 0.22} 22%|██▏ | 392/1817 [14:16:05<51:56:31, 131.22s/it] 22%|██▏ | 393/1817 [14:18:17<52:04:33, 131.65s/it] {'loss': 1.8753, 'learning_rate': 2e-05, 'epoch': 0.22} 22%|██▏ | 393/1817 [14:18:17<52:04:33, 131.65s/it] 22%|██▏ | 394/1817 [14:20:28<51:56:03, 131.39s/it] {'loss': 1.9245, 'learning_rate': 2e-05, 'epoch': 0.22} 22%|██▏ | 394/1817 [14:20:28<51:56:03, 131.39s/it] 22%|██▏ | 395/1817 [14:22:39<51:47:49, 131.13s/it] {'loss': 1.9324, 'learning_rate': 2e-05, 'epoch': 0.22} 22%|██▏ | 395/1817 [14:22:39<51:47:49, 131.13s/it] 22%|██▏ | 396/1817 [14:24:49<51:42:26, 131.00s/it] {'loss': 1.9286, 'learning_rate': 2e-05, 'epoch': 0.22} 22%|██▏ | 396/1817 [14:24:49<51:42:26, 131.00s/it] 22%|██▏ | 397/1817 [14:26:59<51:27:00, 130.44s/it] {'loss': 1.9048, 'learning_rate': 2e-05, 'epoch': 0.22} 22%|██▏ | 397/1817 [14:26:58<51:27:01, 130.44s/it] 22%|██▏ | 398/1817 [14:29:10<51:31:40, 130.73s/it] {'loss': 1.8699, 'learning_rate': 2e-05, 'epoch': 0.22} 22%|██▏ | 398/1817 [14:29:10<51:31:40, 130.73s/it] 22%|██▏ | 399/1817 [14:31:19<51:13:36, 130.05s/it] {'loss': 1.9285, 'learning_rate': 2e-05, 'epoch': 0.22} 22%|██▏ | 399/1817 [14:31:18<51:13:37, 130.05s/it] 22%|██▏ | 400/1817 [14:33:26<50:52:31, 129.25s/it] {'loss': 1.8935, 'learning_rate': 2e-05, 'epoch': 0.22} 22%|██▏ | 400/1817 [14:33:26<50:52:31, 129.25s/it] 22%|██▏ | 401/1817 [14:35:33<50:33:21, 128.53s/it] {'loss': 1.9213, 'learning_rate': 2e-05, 'epoch': 0.22} 22%|██▏ | 401/1817 [14:35:33<50:33:21, 128.53s/it] 22%|██▏ | 402/1817 [14:37:45<50:59:54, 129.75s/it] {'loss': 1.9069, 'learning_rate': 2e-05, 'epoch': 0.22} 22%|██▏ | 402/1817 [14:37:45<50:59:54, 129.75s/it] 22%|██▏ | 403/1817 [14:39:55<50:56:24, 129.69s/it] {'loss': 1.9141, 'learning_rate': 2e-05, 'epoch': 0.22} 22%|██▏ | 403/1817 [14:39:55<50:56:24, 129.69s/it] 22%|██▏ | 404/1817 [14:42:08<51:19:58, 130.78s/it] {'loss': 1.8961, 'learning_rate': 2e-05, 'epoch': 0.22} 22%|██▏ | 404/1817 [14:42:08<51:19:58, 130.78s/it] 22%|██▏ | 405/1817 [14:44:16<50:56:15, 129.87s/it] {'loss': 1.9071, 'learning_rate': 2e-05, 'epoch': 0.22} 22%|██▏ | 405/1817 [14:44:16<50:56:16, 129.87s/it] 22%|██▏ | 406/1817 [14:46:25<50:47:18, 129.58s/it] {'loss': 1.9444, 'learning_rate': 2e-05, 'epoch': 0.22} 22%|██▏ | 406/1817 [14:46:25<50:47:18, 129.58s/it] 22%|██▏ | 407/1817 [14:48:35<50:50:27, 129.81s/it] {'loss': 1.9129, 'learning_rate': 2e-05, 'epoch': 0.22} 22%|██▏ | 407/1817 [14:48:35<50:50:27, 129.81s/it] 22%|██▏ | 408/1817 [14:50:44<50:43:01, 129.58s/it] {'loss': 1.9553, 'learning_rate': 2e-05, 'epoch': 0.22} 22%|██▏ | 408/1817 [14:50:44<50:43:01, 129.58s/it] 23%|██▎ | 409/1817 [14:52:55<50:47:33, 129.87s/it] {'loss': 1.897, 'learning_rate': 2e-05, 'epoch': 0.23} 23%|██▎ | 409/1817 [14:52:55<50:47:33, 129.87s/it] 23%|██▎ | 410/1817 [14:55:02<50:28:14, 129.14s/it] {'loss': 1.9329, 'learning_rate': 2e-05, 'epoch': 0.23} 23%|██▎ | 410/1817 [14:55:02<50:28:14, 129.14s/it] 23%|██▎ | 411/1817 [14:57:12<50:33:10, 129.44s/it] {'loss': 1.8805, 'learning_rate': 2e-05, 'epoch': 0.23} 23%|██▎ | 411/1817 [14:57:12<50:33:10, 129.44s/it] 23%|██▎ | 412/1817 [14:59:23<50:38:45, 129.77s/it] {'loss': 1.9062, 'learning_rate': 2e-05, 'epoch': 0.23} 23%|██▎ | 412/1817 [14:59:23<50:38:45, 129.77s/it] 23%|██▎ | 413/1817 [15:01:32<50:32:55, 129.61s/it] {'loss': 1.9025, 'learning_rate': 2e-05, 'epoch': 0.23} 23%|██▎ | 413/1817 [15:01:32<50:32:55, 129.61s/it] 23%|██▎ | 414/1817 [15:03:39<50:13:53, 128.89s/it] {'loss': 1.9154, 'learning_rate': 2e-05, 'epoch': 0.23} 23%|██▎ | 414/1817 [15:03:39<50:13:52, 128.89s/it] 23%|██▎ | 415/1817 [15:05:49<50:19:38, 129.23s/it] {'loss': 1.9275, 'learning_rate': 2e-05, 'epoch': 0.23} 23%|██▎ | 415/1817 [15:05:49<50:19:39, 129.23s/it] 23%|██▎ | 416/1817 [15:08:02<50:44:55, 130.40s/it] {'loss': 1.9033, 'learning_rate': 2e-05, 'epoch': 0.23} 23%|██▎ | 416/1817 [15:08:02<50:44:55, 130.40s/it] 23%|██▎ | 417/1817 [15:10:11<50:31:23, 129.92s/it] {'loss': 1.9096, 'learning_rate': 2e-05, 'epoch': 0.23} 23%|██▎ | 417/1817 [15:10:11<50:30:55, 129.90s/it] 23%|██▎ | 418/1817 [15:12:21<50:30:22, 129.97s/it] {'loss': 1.8482, 'learning_rate': 2e-05, 'epoch': 0.23} 23%|██▎ | 418/1817 [15:12:21<50:30:22, 129.97s/it] 23%|██▎ | 419/1817 [15:14:31<50:29:34, 130.02s/it] {'loss': 1.9078, 'learning_rate': 2e-05, 'epoch': 0.23} 23%|██▎ | 419/1817 [15:14:31<50:29:34, 130.02s/it] 23%|██▎ | 420/1817 [15:16:40<50:17:36, 129.60s/it] {'loss': 1.9346, 'learning_rate': 2e-05, 'epoch': 0.23} 23%|██▎ | 420/1817 [15:16:40<50:17:36, 129.60s/it] 23%|██▎ | 421/1817 [15:18:50<50:14:11, 129.55s/it] {'loss': 1.9012, 'learning_rate': 2e-05, 'epoch': 0.23} 23%|██▎ | 421/1817 [15:18:49<50:14:14, 129.55s/it] 23%|██▎ | 422/1817 [15:20:59<50:11:15, 129.52s/it] {'loss': 1.9104, 'learning_rate': 2e-05, 'epoch': 0.23} 23%|██▎ | 422/1817 [15:20:59<50:11:18, 129.52s/it] 23%|██▎ | 423/1817 [15:23:07<49:59:30, 129.10s/it] {'loss': 1.8855, 'learning_rate': 2e-05, 'epoch': 0.23} 23%|██▎ | 423/1817 [15:23:07<49:59:30, 129.10s/it] 23%|██▎ | 424/1817 [15:25:17<49:59:14, 129.18s/it] {'loss': 1.9186, 'learning_rate': 2e-05, 'epoch': 0.23} 23%|██▎ | 424/1817 [15:25:16<49:59:14, 129.19s/it] 23%|██▎ | 425/1817 [15:27:24<49:45:16, 128.68s/it] {'loss': 1.9778, 'learning_rate': 2e-05, 'epoch': 0.23} 23%|██▎ | 425/1817 [15:27:24<49:45:17, 128.68s/it] 23%|██▎ | 426/1817 [15:29:39<50:25:57, 130.52s/it] {'loss': 1.9105, 'learning_rate': 2e-05, 'epoch': 0.23} 23%|██▎ | 426/1817 [15:29:39<50:25:58, 130.52s/it] 24%|██▎ | 427/1817 [15:31:50<50:27:39, 130.69s/it] {'loss': 1.9029, 'learning_rate': 2e-05, 'epoch': 0.23} 24%|██▎ | 427/1817 [15:31:50<50:27:39, 130.69s/it] 24%|██▎ | 428/1817 [15:34:01<50:27:12, 130.76s/it] {'loss': 1.9306, 'learning_rate': 2e-05, 'epoch': 0.24} 24%|██▎ | 428/1817 [15:34:01<50:27:12, 130.76s/it] 24%|██▎ | 429/1817 [15:36:13<50:36:31, 131.26s/it] {'loss': 1.9749, 'learning_rate': 2e-05, 'epoch': 0.24} 24%|██▎ | 429/1817 [15:36:13<50:36:32, 131.26s/it] 24%|██▎ | 430/1817 [15:38:27<50:48:35, 131.88s/it] {'loss': 1.9165, 'learning_rate': 2e-05, 'epoch': 0.24} 24%|██▎ | 430/1817 [15:38:26<50:48:35, 131.88s/it] 24%|██▎ | 431/1817 [15:40:38<50:41:32, 131.67s/it] {'loss': 1.918, 'learning_rate': 2e-05, 'epoch': 0.24} 24%|██▎ | 431/1817 [15:40:38<50:41:32, 131.67s/it] 24%|██▍ | 432/1817 [15:42:46<50:17:16, 130.71s/it] {'loss': 1.927, 'learning_rate': 2e-05, 'epoch': 0.24} 24%|██▍ | 432/1817 [15:42:46<50:17:17, 130.71s/it] 24%|██▍ | 433/1817 [15:44:56<50:08:38, 130.43s/it] {'loss': 1.9326, 'learning_rate': 2e-05, 'epoch': 0.24} 24%|██▍ | 433/1817 [15:44:56<50:08:38, 130.43s/it] 24%|██▍ | 434/1817 [15:47:08<50:15:58, 130.84s/it] {'loss': 1.9189, 'learning_rate': 2e-05, 'epoch': 0.24} 24%|██▍ | 434/1817 [15:47:08<50:15:58, 130.85s/it] 24%|██▍ | 435/1817 [15:49:17<50:03:28, 130.40s/it] {'loss': 1.9344, 'learning_rate': 2e-05, 'epoch': 0.24} 24%|██▍ | 435/1817 [15:49:17<50:03:29, 130.40s/it] 24%|██▍ | 436/1817 [15:51:32<50:29:20, 131.62s/it] {'loss': 1.934, 'learning_rate': 2e-05, 'epoch': 0.24} 24%|██▍ | 436/1817 [15:51:31<50:29:21, 131.62s/it] 24%|██▍ | 437/1817 [15:53:48<51:01:08, 133.09s/it] {'loss': 1.9261, 'learning_rate': 2e-05, 'epoch': 0.24} 24%|██▍ | 437/1817 [15:53:48<51:01:08, 133.09s/it] 24%|██▍ | 438/1817 [15:55:58<50:36:19, 132.11s/it] {'loss': 1.9314, 'learning_rate': 2e-05, 'epoch': 0.24} 24%|██▍ | 438/1817 [15:55:58<50:36:19, 132.11s/it] 24%|██▍ | 439/1817 [15:58:06<50:05:54, 130.88s/it] {'loss': 1.9079, 'learning_rate': 2e-05, 'epoch': 0.24} 24%|██▍ | 439/1817 [15:58:06<50:05:55, 130.88s/it] 24%|██▍ | 440/1817 [16:00:16<50:00:34, 130.74s/it] {'loss': 1.9059, 'learning_rate': 2e-05, 'epoch': 0.24} 24%|██▍ | 440/1817 [16:00:16<50:00:34, 130.74s/it] 24%|██▍ | 441/1817 [16:02:30<50:16:20, 131.53s/it] {'loss': 1.9042, 'learning_rate': 2e-05, 'epoch': 0.24} 24%|██▍ | 441/1817 [16:02:30<50:16:19, 131.53s/it] 24%|██▍ | 442/1817 [16:04:38<49:50:32, 130.50s/it] {'loss': 1.9208, 'learning_rate': 2e-05, 'epoch': 0.24} 24%|██▍ | 442/1817 [16:04:38<49:50:32, 130.50s/it] 24%|██▍ | 443/1817 [16:06:48<49:45:27, 130.37s/it] {'loss': 1.8698, 'learning_rate': 2e-05, 'epoch': 0.24} 24%|██▍ | 443/1817 [16:06:48<49:45:27, 130.37s/it] 24%|██▍ | 444/1817 [16:09:01<50:00:45, 131.13s/it] {'loss': 1.9162, 'learning_rate': 2e-05, 'epoch': 0.24} 24%|██▍ | 444/1817 [16:09:01<50:00:46, 131.13s/it] 24%|██▍ | 445/1817 [16:11:09<49:39:20, 130.29s/it] {'loss': 1.8938, 'learning_rate': 2e-05, 'epoch': 0.24} 24%|██▍ | 445/1817 [16:11:09<49:39:18, 130.29s/it] 25%|██▍ | 446/1817 [16:13:20<49:40:19, 130.43s/it] {'loss': 1.9024, 'learning_rate': 2e-05, 'epoch': 0.25} 25%|██▍ | 446/1817 [16:13:20<49:40:14, 130.43s/it] 25%|██▍ | 447/1817 [16:15:32<49:46:16, 130.79s/it] {'loss': 1.8673, 'learning_rate': 2e-05, 'epoch': 0.25} 25%|██▍ | 447/1817 [16:15:31<49:46:13, 130.78s/it] 25%|██▍ | 448/1817 [16:17:43<49:47:52, 130.95s/it] {'loss': 1.9186, 'learning_rate': 2e-05, 'epoch': 0.25} 25%|██▍ | 448/1817 [16:17:43<49:47:52, 130.95s/it] 25%|██▍ | 449/1817 [16:19:56<50:00:25, 131.60s/it] {'loss': 1.8894, 'learning_rate': 2e-05, 'epoch': 0.25} 25%|██▍ | 449/1817 [16:19:56<50:00:25, 131.60s/it] 25%|██▍ | 450/1817 [16:22:08<50:05:05, 131.90s/it] {'loss': 1.934, 'learning_rate': 2e-05, 'epoch': 0.25} 25%|██▍ | 450/1817 [16:22:08<50:05:05, 131.90s/it] 25%|██▍ | 451/1817 [16:24:18<49:46:40, 131.19s/it] {'loss': 1.9223, 'learning_rate': 2e-05, 'epoch': 0.25} 25%|██▍ | 451/1817 [16:24:18<49:46:40, 131.19s/it] 25%|██▍ | 452/1817 [16:26:28<49:36:57, 130.86s/it] {'loss': 1.8805, 'learning_rate': 2e-05, 'epoch': 0.25} 25%|██▍ | 452/1817 [16:26:28<49:36:57, 130.86s/it] 25%|██▍ | 453/1817 [16:28:42<49:56:14, 131.80s/it] {'loss': 1.9136, 'learning_rate': 2e-05, 'epoch': 0.25} 25%|██▍ | 453/1817 [16:28:42<49:56:14, 131.80s/it] 25%|██▍ | 454/1817 [16:30:53<49:46:05, 131.45s/it] {'loss': 1.9089, 'learning_rate': 2e-05, 'epoch': 0.25} 25%|██▍ | 454/1817 [16:30:53<49:46:06, 131.45s/it] 25%|██▌ | 455/1817 [16:33:01<49:26:11, 130.67s/it] {'loss': 1.9165, 'learning_rate': 2e-05, 'epoch': 0.25} 25%|██▌ | 455/1817 [16:33:01<49:26:11, 130.67s/it] 25%|██▌ | 456/1817 [16:35:12<49:20:57, 130.53s/it] {'loss': 1.9145, 'learning_rate': 2e-05, 'epoch': 0.25} 25%|██▌ | 456/1817 [16:35:12<49:20:57, 130.53s/it] 25%|██▌ | 457/1817 [16:37:20<49:04:23, 129.90s/it] {'loss': 1.9333, 'learning_rate': 2e-05, 'epoch': 0.25} 25%|██▌ | 457/1817 [16:37:20<49:04:23, 129.90s/it] 25%|██▌ | 458/1817 [16:39:34<49:26:38, 130.98s/it] {'loss': 1.9171, 'learning_rate': 2e-05, 'epoch': 0.25} 25%|██▌ | 458/1817 [16:39:34<49:26:38, 130.98s/it] 25%|██▌ | 459/1817 [16:41:45<49:26:01, 131.05s/it] {'loss': 1.9254, 'learning_rate': 2e-05, 'epoch': 0.25} 25%|██▌ | 459/1817 [16:41:45<49:26:01, 131.05s/it] 25%|██▌ | 460/1817 [16:44:01<49:57:12, 132.52s/it] {'loss': 1.8968, 'learning_rate': 2e-05, 'epoch': 0.25} 25%|██▌ | 460/1817 [16:44:01<49:57:12, 132.52s/it] 25%|██▌ | 461/1817 [16:46:12<49:43:49, 132.03s/it] {'loss': 1.9143, 'learning_rate': 2e-05, 'epoch': 0.25} 25%|██▌ | 461/1817 [16:46:12<49:43:49, 132.03s/it] 25%|██▌ | 462/1817 [16:48:22<49:28:26, 131.44s/it] {'loss': 1.892, 'learning_rate': 2e-05, 'epoch': 0.25} 25%|██▌ | 462/1817 [16:48:22<49:28:27, 131.44s/it] 25%|██▌ | 463/1817 [16:50:32<49:14:53, 130.94s/it] {'loss': 1.9211, 'learning_rate': 2e-05, 'epoch': 0.25} 25%|██▌ | 463/1817 [16:50:32<49:14:56, 130.94s/it] 26%|██▌ | 464/1817 [16:52:43<49:15:42, 131.07s/it] {'loss': 1.9267, 'learning_rate': 2e-05, 'epoch': 0.26} 26%|██▌ | 464/1817 [16:52:43<49:15:45, 131.08s/it] 26%|██▌ | 465/1817 [16:54:54<49:13:33, 131.07s/it] {'loss': 1.8837, 'learning_rate': 2e-05, 'epoch': 0.26} 26%|██▌ | 465/1817 [16:54:54<49:13:33, 131.07s/it] 26%|██▌ | 466/1817 [16:57:03<48:53:52, 130.30s/it] {'loss': 1.9112, 'learning_rate': 2e-05, 'epoch': 0.26} 26%|██▌ | 466/1817 [16:57:02<48:53:53, 130.30s/it] 26%|██▌ | 467/1817 [16:59:11<48:37:18, 129.66s/it] {'loss': 1.9417, 'learning_rate': 2e-05, 'epoch': 0.26} 26%|██▌ | 467/1817 [16:59:11<48:37:18, 129.66s/it] 26%|██▌ | 468/1817 [17:01:19<48:27:33, 129.32s/it] {'loss': 1.9567, 'learning_rate': 2e-05, 'epoch': 0.26} 26%|██▌ | 468/1817 [17:01:19<48:27:34, 129.32s/it] 26%|██▌ | 469/1817 [17:03:33<48:53:20, 130.56s/it] {'loss': 1.9211, 'learning_rate': 2e-05, 'epoch': 0.26} 26%|██▌ | 469/1817 [17:03:33<48:53:19, 130.56s/it] 26%|██▌ | 470/1817 [17:05:43<48:48:55, 130.46s/it] {'loss': 1.9395, 'learning_rate': 2e-05, 'epoch': 0.26} 26%|██▌ | 470/1817 [17:05:43<48:48:55, 130.46s/it] 26%|██▌ | 471/1817 [17:07:57<49:08:42, 131.44s/it] {'loss': 1.9345, 'learning_rate': 2e-05, 'epoch': 0.26} 26%|██▌ | 471/1817 [17:07:57<49:08:41, 131.44s/it] 26%|██▌ | 472/1817 [17:10:05<48:47:45, 130.61s/it] {'loss': 1.9329, 'learning_rate': 2e-05, 'epoch': 0.26} 26%|██▌ | 472/1817 [17:10:05<48:47:45, 130.61s/it] 26%|██▌ | 473/1817 [17:12:14<48:35:57, 130.18s/it] {'loss': 1.9056, 'learning_rate': 2e-05, 'epoch': 0.26} 26%|██▌ | 473/1817 [17:12:14<48:35:57, 130.18s/it] 26%|██▌ | 474/1817 [17:14:24<48:26:50, 129.87s/it] {'loss': 1.9013, 'learning_rate': 2e-05, 'epoch': 0.26} 26%|██▌ | 474/1817 [17:14:24<48:26:50, 129.87s/it] 26%|██▌ | 475/1817 [17:16:36<48:44:37, 130.76s/it] {'loss': 1.9134, 'learning_rate': 2e-05, 'epoch': 0.26} 26%|██▌ | 475/1817 [17:16:36<48:44:37, 130.76s/it] 26%|██▌ | 476/1817 [17:18:48<48:46:32, 130.94s/it] {'loss': 1.867, 'learning_rate': 2e-05, 'epoch': 0.26} 26%|██▌ | 476/1817 [17:18:48<48:46:32, 130.94s/it] 26%|██▋ | 477/1817 [17:20:59<48:43:07, 130.89s/it] {'loss': 1.8881, 'learning_rate': 2e-05, 'epoch': 0.26} 26%|██▋ | 477/1817 [17:20:59<48:43:07, 130.89s/it] 26%|██▋ | 478/1817 [17:23:11<48:50:19, 131.31s/it] {'loss': 1.8783, 'learning_rate': 2e-05, 'epoch': 0.26} 26%|██▋ | 478/1817 [17:23:11<48:50:19, 131.31s/it] 26%|██▋ | 479/1817 [17:25:21<48:38:31, 130.88s/it] {'loss': 1.949, 'learning_rate': 2e-05, 'epoch': 0.26} 26%|██▋ | 479/1817 [17:25:21<48:38:31, 130.88s/it] 26%|██▋ | 480/1817 [17:27:31<48:32:27, 130.70s/it] {'loss': 1.9231, 'learning_rate': 2e-05, 'epoch': 0.26} 26%|██▋ | 480/1817 [17:27:31<48:32:27, 130.70s/it] 26%|██▋ | 481/1817 [17:29:38<48:04:29, 129.54s/it] {'loss': 1.9427, 'learning_rate': 2e-05, 'epoch': 0.26} 26%|██▋ | 481/1817 [17:29:38<48:04:29, 129.54s/it] 27%|██▋ | 482/1817 [17:31:47<47:58:08, 129.35s/it] {'loss': 1.8828, 'learning_rate': 2e-05, 'epoch': 0.27} 27%|██▋ | 482/1817 [17:31:47<47:58:09, 129.36s/it] 27%|██▋ | 483/1817 [17:33:59<48:14:59, 130.21s/it] {'loss': 1.9138, 'learning_rate': 2e-05, 'epoch': 0.27} 27%|██▋ | 483/1817 [17:33:59<48:14:59, 130.21s/it] 27%|██▋ | 484/1817 [17:36:09<48:10:06, 130.09s/it] {'loss': 1.895, 'learning_rate': 2e-05, 'epoch': 0.27} 27%|██▋ | 484/1817 [17:36:09<48:10:06, 130.09s/it] 27%|██▋ | 485/1817 [17:38:23<48:35:14, 131.32s/it] {'loss': 1.874, 'learning_rate': 2e-05, 'epoch': 0.27} 27%|██▋ | 485/1817 [17:38:23<48:35:17, 131.32s/it] 27%|██▋ | 486/1817 [17:40:34<48:28:18, 131.10s/it] {'loss': 1.8656, 'learning_rate': 2e-05, 'epoch': 0.27} 27%|██▋ | 486/1817 [17:40:34<48:28:22, 131.11s/it] 27%|██▋ | 487/1817 [17:42:44<48:18:21, 130.75s/it] {'loss': 1.8858, 'learning_rate': 2e-05, 'epoch': 0.27} 27%|██▋ | 487/1817 [17:42:43<48:18:24, 130.76s/it] 27%|██▋ | 488/1817 [17:44:54<48:15:40, 130.73s/it] {'loss': 1.9163, 'learning_rate': 2e-05, 'epoch': 0.27} 27%|██▋ | 488/1817 [17:44:54<48:15:40, 130.73s/it] 27%|██▋ | 489/1817 [17:47:04<48:08:21, 130.50s/it] {'loss': 1.8743, 'learning_rate': 2e-05, 'epoch': 0.27} 27%|██▋ | 489/1817 [17:47:04<48:08:21, 130.50s/it] 27%|██▋ | 490/1817 [17:49:10<47:35:42, 129.12s/it] {'loss': 1.9309, 'learning_rate': 2e-05, 'epoch': 0.27} 27%|██▋ | 490/1817 [17:49:10<47:35:45, 129.12s/it] 27%|██▋ | 491/1817 [17:51:20<47:38:11, 129.33s/it] {'loss': 1.9093, 'learning_rate': 2e-05, 'epoch': 0.27} 27%|██▋ | 491/1817 [17:51:20<47:38:14, 129.33s/it] 27%|██▋ | 492/1817 [17:53:30<47:37:28, 129.39s/it] {'loss': 1.929, 'learning_rate': 2e-05, 'epoch': 0.27} 27%|██▋ | 492/1817 [17:53:29<47:37:30, 129.40s/it] 27%|██▋ | 493/1817 [17:55:40<47:44:10, 129.80s/it] {'loss': 1.8746, 'learning_rate': 2e-05, 'epoch': 0.27} 27%|██▋ | 493/1817 [17:55:40<47:44:13, 129.80s/it] 27%|██▋ | 494/1817 [17:57:49<47:36:01, 129.52s/it] {'loss': 1.9342, 'learning_rate': 2e-05, 'epoch': 0.27} 27%|██▋ | 494/1817 [17:57:49<47:36:04, 129.53s/it] 27%|██▋ | 495/1817 [17:59:59<47:36:25, 129.64s/it] {'loss': 1.9433, 'learning_rate': 2e-05, 'epoch': 0.27} 27%|██▋ | 495/1817 [17:59:59<47:36:28, 129.64s/it] 27%|██▋ | 496/1817 [18:02:09<47:38:45, 129.84s/it] {'loss': 1.9712, 'learning_rate': 2e-05, 'epoch': 0.27} 27%|██▋ | 496/1817 [18:02:09<47:38:48, 129.85s/it] 27%|██▋ | 497/1817 [18:04:22<47:54:30, 130.66s/it] {'loss': 1.9166, 'learning_rate': 2e-05, 'epoch': 0.27} 27%|██▋ | 497/1817 [18:04:22<47:54:30, 130.66s/it] 27%|██▋ | 498/1817 [18:06:35<48:08:27, 131.39s/it] {'loss': 1.909, 'learning_rate': 2e-05, 'epoch': 0.27} 27%|██▋ | 498/1817 [18:06:35<48:08:27, 131.39s/it] 27%|██▋ | 499/1817 [18:08:49<48:23:16, 132.17s/it] {'loss': 1.9071, 'learning_rate': 2e-05, 'epoch': 0.27} 27%|██▋ | 499/1817 [18:08:49<48:23:05, 132.16s/it] 28%|██▊ | 500/1817 [18:10:57<47:53:11, 130.90s/it] {'loss': 1.8647, 'learning_rate': 2e-05, 'epoch': 0.28} 28%|██▊ | 500/1817 [18:10:57<47:52:41, 130.87s/it][INFO|trainer.py:2830] 2024-01-30 05:56:34,028 >> Saving model checkpoint to ./checkpoints_ct/ac/allm-ac-13b/checkpoint-500 [INFO|configuration_utils.py:457] 2024-01-30 05:56:34,033 >> Configuration saved in ./checkpoints_ct/ac/allm-ac-13b/checkpoint-500/config.json [INFO|configuration_utils.py:362] 2024-01-30 05:56:34,038 >> Configuration saved in ./checkpoints_ct/ac/allm-ac-13b/checkpoint-500/generation_config.json [INFO|modeling_utils.py:1759] 2024-01-30 05:56:34,074 >> Model weights saved in ./checkpoints_ct/ac/allm-ac-13b/checkpoint-500/pytorch_model.bin [INFO|tokenization_utils_base.py:2164] 2024-01-30 05:56:34,076 >> tokenizer config file saved in ./checkpoints_ct/ac/allm-ac-13b/checkpoint-500/tokenizer_config.json [INFO|tokenization_utils_base.py:2171] 2024-01-30 05:56:34,077 >> Special tokens file saved in ./checkpoints_ct/ac/allm-ac-13b/checkpoint-500/special_tokens_map.json [INFO|tokenization_utils_base.py:2221] 2024-01-30 05:56:34,077 >> added tokens file saved in ./checkpoints_ct/ac/allm-ac-13b/checkpoint-500/added_tokens.json [2024-01-30 05:56:48,864] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step500 is ready now! d! [2024-01-30 05:56:48,840] [INFO] [engine.py:3492:save_16bit_model] Saving model weights to ./checkpoints_ct/ac/allm-ac-13b/checkpoint-500/pytorch_model.bin, tag: global_step500 [2024-01-30 05:56:48,840] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoints_ct/ac/allm-ac-13b/checkpoint-500/pytorch_model.bin... /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. warnings.warn( /apdcephfs/share_733425/vinnylywang/jianhuipang/llama2_sft/envs/lib/python3.8/site-packages/torch/nn/modules/module.py:1432: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. warnings.warn( [2024-01-30 05:57:19,940] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoints_ct/ac/allm-ac-13b/checkpoint-500/global_step500/zero_pp_rank_24_mp_rank_00_model_states.pt...[2024-01-30 05:57:19,963] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./checkpoints_ct/ac/allm-ac-13b/checkpoint-500/global_step500/zero_pp_rank_8_mp_rank_00_model_states.pt. [2024-01-30 05:57:19,985] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoints_ct/ac/allm-ac-13b/checkpoint-500/global_step500/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt... .. ates.pt [2024-01-30 05:57:19,911] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoints_ct/ac/allm-ac-13b/checkpoint-500/global_step500/zero_pp_rank_0_mp_rank_00_model_states.pt... [2024-01-30 05:57:19,938] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./checkpoints_ct/ac/allm-ac-13b/checkpoint-500/global_step500/zero_pp_rank_0_mp_rank_00_model_states.pt. [2024-01-30 05:57:19,985] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoints_ct/ac/allm-ac-13b/checkpoint-500/global_step500/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt... [2024-01-30 05:57:34,220] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./checkpoints_ct/ac/allm-ac-13b/checkpoint-500/global_step500/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt. [2024-01-30 05:57:34,220] [INFO] [engine.py:3381:_save_zero_checkpoint] zero checkpoint saved ./checkpoints_ct/ac/allm-ac-13b/checkpoint-500/global_step500/bf16_zero_pp_rank_24_mp_rank_00_optim_states.p[2024-01-30 05:57:34,409] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step500 is ready now! ! 28%|██▊ | 501/1817 [18:14:10<54:40:48, 149.58s/it] {'loss': 1.8837, 'learning_rate': 2e-05, 'epoch': 0.28} 28%|██▊ | 501/1817 [18:14:10<54:40:29, 149.57s/it] 28%|██▊ | 502/1817 [18:16:19<52:25:08, 143.50s/it] {'loss': 1.8581, 'learning_rate': 2e-05, 'epoch': 0.28} 28%|██▊ | 502/1817 [18:16:19<52:25:08, 143.50s/it] 28%|██▊ | 503/1817 [18:18:29<50:53:31, 139.43s/it] {'loss': 1.8704, 'learning_rate': 2e-05, 'epoch': 0.28} 28%|██▊ | 503/1817 [18:18:29<50:53:31, 139.43s/it] 28%|██▊ | 504/1817 [18:20:37<49:35:41, 135.98s/it] {'loss': 1.916, 'learning_rate': 2e-05, 'epoch': 0.28} 28%|██▊ | 504/1817 [18:20:37<49:35:43, 135.98s/it] 28%|██▊ | 505/1817 [18:22:48<48:59:37, 134.43s/it] {'loss': 1.9374, 'learning_rate': 2e-05, 'epoch': 0.28} 28%|██▊ | 505/1817 [18:22:48<48:59:38, 134.44s/it] 28%|██▊ | 506/1817 [18:24:57<48:24:11, 132.91s/it] {'loss': 1.9092, 'learning_rate': 2e-05, 'epoch': 0.28} 28%|██▊ | 506/1817 [18:24:57<48:24:11, 132.91s/it] 28%|██▊ | 507/1817 [18:27:09<48:14:19, 132.56s/it] {'loss': 1.9155, 'learning_rate': 2e-05, 'epoch': 0.28} 28%|██▊ | 507/1817 [18:27:09<48:14:19, 132.56s/it] 28%|██▊ | 508/1817 [18:29:18<47:47:37, 131.44s/it] {'loss': 1.92, 'learning_rate': 2e-05, 'epoch': 0.28} 28%|██▊ | 508/1817 [18:29:18<47:47:37, 131.44s/it] 28%|██▊ | 509/1817 [18:31:27<47:28:35, 130.67s/it] {'loss': 1.897, 'learning_rate': 2e-05, 'epoch': 0.28} 28%|██▊ | 509/1817 [18:31:27<47:28:35, 130.67s/it] 28%|██▊ | 510/1817 [18:33:34<47:02:26, 129.57s/it] {'loss': 1.9057, 'learning_rate': 2e-05, 'epoch': 0.28} 28%|██▊ | 510/1817 [18:33:34<47:02:26, 129.57s/it] 28%|██▊ | 511/1817 [18:35:44<47:03:41, 129.73s/it] {'loss': 1.9007, 'learning_rate': 2e-05, 'epoch': 0.28} 28%|██▊ | 511/1817 [18:35:44<47:03:41, 129.73s/it] 28%|██▊ | 512/1817 [18:37:55<47:06:37, 129.96s/it] {'loss': 1.8729, 'learning_rate': 2e-05, 'epoch': 0.28} 28%|██▊ | 512/1817 [18:37:54<47:06:39, 129.96s/it] 28%|██▊ | 513/1817 [18:40:06<47:16:54, 130.53s/it] {'loss': 1.9174, 'learning_rate': 2e-05, 'epoch': 0.28} 28%|██▊ | 513/1817 [18:40:06<47:16:54, 130.53s/it] 28%|██▊ | 514/1817 [18:42:19<47:29:20, 131.21s/it] {'loss': 1.868, 'learning_rate': 2e-05, 'epoch': 0.28} 28%|██▊ | 514/1817 [18:42:19<47:29:20, 131.21s/it] 28%|██▊ | 515/1817 [18:44:30<47:26:48, 131.19s/it] {'loss': 1.8913, 'learning_rate': 2e-05, 'epoch': 0.28} 28%|██▊ | 515/1817 [18:44:30<47:26:47, 131.19s/it] 28%|██▊ | 516/1817 [18:46:40<47:17:25, 130.86s/it] {'loss': 1.9744, 'learning_rate': 2e-05, 'epoch': 0.28} 28%|██▊ | 516/1817 [18:46:40<47:17:25, 130.86s/it] 28%|██▊ | 517/1817 [18:48:49<47:02:52, 130.29s/it] {'loss': 1.9297, 'learning_rate': 2e-05, 'epoch': 0.28} 28%|██▊ | 517/1817 [18:48:49<47:02:52, 130.29s/it] 29%|██▊ | 518/1817 [18:50:58<46:53:22, 129.95s/it] {'loss': 1.9184, 'learning_rate': 2e-05, 'epoch': 0.29} 29%|██▊ | 518/1817 [18:50:58<46:53:22, 129.95s/it] 29%|██▊ | 519/1817 [18:53:06<46:37:13, 129.30s/it] {'loss': 1.9196, 'learning_rate': 2e-05, 'epoch': 0.29} 29%|██▊ | 519/1817 [18:53:06<46:37:13, 129.30s/it] 29%|██▊ | 520/1817 [18:55:18<46:50:57, 130.04s/it] {'loss': 1.9045, 'learning_rate': 2e-05, 'epoch': 0.29} 29%|██▊ | 520/1817 [18:55:18<46:51:00, 130.04s/it] 29%|██▊ | 521/1817 [18:57:29<46:55:01, 130.33s/it] {'loss': 1.9371, 'learning_rate': 2e-05, 'epoch': 0.29} 29%|██▊ | 521/1817 [18:57:29<46:55:01, 130.33s/it] 29%|██▊ | 522/1817 [18:59:41<47:01:11, 130.71s/it] {'loss': 1.8937, 'learning_rate': 2e-05, 'epoch': 0.29} 29%|██▊ | 522/1817 [18:59:41<47:01:11, 130.71s/it] 29%|██▉ | 523/1817 [19:01:55<47:21:46, 131.77s/it] {'loss': 1.8936, 'learning_rate': 2e-05, 'epoch': 0.29} 29%|██▉ | 523/1817 [19:01:55<47:21:46, 131.77s/it] 29%|██▉ | 524/1817 [19:04:06<47:14:44, 131.54s/it] {'loss': 1.8926, 'learning_rate': 2e-05, 'epoch': 0.29} 29%|██▉ | 524/1817 [19:04:06<47:14:44, 131.54s/it] 29%|██▉ | 525/1817 [19:06:15<46:55:47, 130.76s/it] {'loss': 1.9085, 'learning_rate': 2e-05, 'epoch': 0.29} 29%|██▉ | 525/1817 [19:06:15<46:55:49, 130.77s/it] 29%|██▉ | 526/1817 [19:08:30<47:21:52, 132.08s/it] {'loss': 1.8919, 'learning_rate': 2e-05, 'epoch': 0.29} 29%|██▉ | 526/1817 [19:08:30<47:21:44, 132.07s/it] 29%|██▉ | 527/1817 [19:10:41<47:10:23, 131.65s/it] {'loss': 1.9403, 'learning_rate': 2e-05, 'epoch': 0.29} 29%|██▉ | 527/1817 [19:10:40<47:10:22, 131.65s/it] 29%|██▉ | 528/1817 [19:12:53<47:15:42, 132.00s/it] {'loss': 1.9308, 'learning_rate': 2e-05, 'epoch': 0.29} 29%|██▉ | 528/1817 [19:12:53<47:15:42, 132.00s/it] 29%|██▉ | 529/1817 [19:15:02<46:53:31, 131.06s/it] {'loss': 1.9068, 'learning_rate': 2e-05, 'epoch': 0.29} 29%|██▉ | 529/1817 [19:15:02<46:53:31, 131.06s/it] 29%|██▉ | 530/1817 [19:17:15<47:04:54, 131.70s/it] {'loss': 1.8666, 'learning_rate': 2e-05, 'epoch': 0.29} 29%|██▉ | 530/1817 [19:17:15<47:04:54, 131.70s/it] 29%|██▉ | 531/1817 [19:19:25<46:45:24, 130.89s/it] {'loss': 1.8697, 'learning_rate': 2e-05, 'epoch': 0.29} 29%|██▉ | 531/1817 [19:19:24<46:45:28, 130.89s/it] 29%|██▉ | 532/1817 [19:21:38<46:56:14, 131.50s/it] {'loss': 1.9599, 'learning_rate': 2e-05, 'epoch': 0.29} 29%|██▉ | 532/1817 [19:21:37<46:56:18, 131.50s/it] 29%|██▉ | 533/1817 [19:23:48<46:49:24, 131.28s/it] {'loss': 1.9023, 'learning_rate': 2e-05, 'epoch': 0.29} 29%|██▉ | 533/1817 [19:23:48<46:49:24, 131.28s/it] 29%|██▉ | 534/1817 [19:26:01<47:00:20, 131.89s/it] {'loss': 1.854, 'learning_rate': 2e-05, 'epoch': 0.29} 29%|██▉ | 534/1817 [19:26:01<47:00:20, 131.89s/it] 29%|██▉ | 535/1817 [19:28:14<47:03:32, 132.15s/it] {'loss': 1.8952, 'learning_rate': 2e-05, 'epoch': 0.29} 29%|██▉ | 535/1817 [19:28:14<47:03:32, 132.15s/it] 29%|██▉ | 536/1817 [19:30:24<46:46:10, 131.44s/it] {'loss': 1.913, 'learning_rate': 2e-05, 'epoch': 0.29} 29%|██▉ | 536/1817 [19:30:24<46:46:10, 131.44s/it] 30%|██▉ | 537/1817 [19:32:34<46:37:32, 131.13s/it] {'loss': 1.9093, 'learning_rate': 2e-05, 'epoch': 0.3} 30%|██▉ | 537/1817 [19:32:34<46:37:32, 131.13s/it] 30%|██▉ | 538/1817 [19:34:48<46:49:50, 131.81s/it] {'loss': 1.872, 'learning_rate': 2e-05, 'epoch': 0.3} 30%|██▉ | 538/1817 [19:34:48<46:49:50, 131.81s/it] 30%|██▉ | 539/1817 [19:37:02<47:04:29, 132.61s/it] {'loss': 1.9208, 'learning_rate': 2e-05, 'epoch': 0.3} 30%|██▉ | 539/1817 [19:37:02<47:04:29, 132.61s/it] 30%|██▉ | 540/1817 [19:39:13<46:50:02, 132.03s/it] {'loss': 1.9059, 'learning_rate': 2e-05, 'epoch': 0.3} 30%|██▉ | 540/1817 [19:39:13<46:50:02, 132.03s/it] 30%|██▉ | 541/1817 [19:41:24<46:37:58, 131.57s/it] {'loss': 1.9071, 'learning_rate': 2e-05, 'epoch': 0.3} 30%|██▉ | 541/1817 [19:41:23<46:37:58, 131.57s/it] 30%|██▉ | 542/1817 [19:43:34<46:29:39, 131.28s/it] {'loss': 1.8868, 'learning_rate': 2e-05, 'epoch': 0.3} 30%|██▉ | 542/1817 [19:43:34<46:29:39, 131.28s/it] 30%|██▉ | 543/1817 [19:45:45<46:23:36, 131.10s/it] {'loss': 1.8977, 'learning_rate': 2e-05, 'epoch': 0.3} 30%|██▉ | 543/1817 [19:45:45<46:23:38, 131.10s/it] 30%|██▉ | 544/1817 [19:47:55<46:14:59, 130.79s/it] {'loss': 1.8979, 'learning_rate': 2e-05, 'epoch': 0.3} 30%|██▉ | 544/1817 [19:47:55<46:14:59, 130.79s/it] 30%|██▉ | 545/1817 [19:50:07<46:23:56, 131.32s/it] {'loss': 1.9372, 'learning_rate': 2e-05, 'epoch': 0.3} 30%|██▉ | 545/1817 [19:50:07<46:23:56, 131.32s/it] 30%|███ | 546/1817 [19:52:17<46:12:45, 130.89s/it] {'loss': 1.909, 'learning_rate': 2e-05, 'epoch': 0.3} 30%|███ | 546/1817 [19:52:17<46:12:45, 130.89s/it] 30%|███ | 547/1817 [19:54:26<45:59:01, 130.35s/it] {'loss': 1.8974, 'learning_rate': 2e-05, 'epoch': 0.3} 30%|███ | 547/1817 [19:54:26<45:59:00, 130.35s/it] 30%|███ | 548/1817 [19:56:39<46:13:17, 131.12s/it] {'loss': 1.9191, 'learning_rate': 2e-05, 'epoch': 0.3} 30%|███ | 548/1817 [19:56:39<46:13:17, 131.12s/it] 30%|███ | 549/1817 [19:58:50<46:06:49, 130.92s/it] {'loss': 1.9052, 'learning_rate': 2e-05, 'epoch': 0.3} 30%|███ | 549/1817 [19:58:50<46:06:49, 130.92s/it] 30%|███ | 550/1817 [20:00:58<45:44:25, 129.96s/it] {'loss': 1.8994, 'learning_rate': 2e-05, 'epoch': 0.3} 30%|███ | 550/1817 [20:00:57<45:44:25, 129.96s/it] 30%|███ | 551/1817 [20:03:09<45:48:43, 130.27s/it] {'loss': 1.9108, 'learning_rate': 2e-05, 'epoch': 0.3} 30%|███ | 551/1817 [20:03:08<45:48:43, 130.27s/it] 30%|███ | 552/1817 [20:05:20<45:53:08, 130.58s/it] {'loss': 1.9328, 'learning_rate': 2e-05, 'epoch': 0.3} 30%|███ | 552/1817 [20:05:20<45:53:11, 130.59s/it] 30%|███ | 553/1817 [20:07:32<46:03:25, 131.18s/it] {'loss': 1.916, 'learning_rate': 2e-05, 'epoch': 0.3} 30%|███ | 553/1817 [20:07:32<46:03:26, 131.18s/it] 30%|███ | 554/1817 [20:09:46<46:16:25, 131.90s/it] {'loss': 1.9225, 'learning_rate': 2e-05, 'epoch': 0.3} 30%|███ | 554/1817 [20:09:46<46:16:26, 131.90s/it] 31%|███ | 555/1817 [20:12:00<46:27:25, 132.52s/it] {'loss': 1.8679, 'learning_rate': 2e-05, 'epoch': 0.31} 31%|███ | 555/1817 [20:12:00<46:27:17, 132.52s/it] 31%|███ | 556/1817 [20:14:11<46:15:36, 132.07s/it] {'loss': 1.8855, 'learning_rate': 2e-05, 'epoch': 0.31} 31%|███ | 556/1817 [20:14:11<46:15:30, 132.06s/it] 31%|███ | 557/1817 [20:16:25<46:23:59, 132.57s/it] {'loss': 1.924, 'learning_rate': 2e-05, 'epoch': 0.31} 31%|███ | 557/1817 [20:16:25<46:24:00, 132.57s/it] 31%|███ | 558/1817 [20:18:34<46:01:53, 131.62s/it] {'loss': 1.9119, 'learning_rate': 2e-05, 'epoch': 0.31} 31%|███ | 558/1817 [20:18:34<46:01:54, 131.62s/it] 31%|███ | 559/1817 [20:20:48<46:13:10, 132.27s/it] {'loss': 1.8524, 'learning_rate': 2e-05, 'epoch': 0.31} 31%|███ | 559/1817 [20:20:48<46:13:10, 132.27s/it] 31%|███ | 560/1817 [20:22:58<45:57:52, 131.64s/it] {'loss': 1.9112, 'learning_rate': 2e-05, 'epoch': 0.31} 31%|███ | 560/1817 [20:22:58<45:57:52, 131.64s/it] 31%|███ | 561/1817 [20:25:06<45:30:55, 130.46s/it] {'loss': 1.904, 'learning_rate': 2e-05, 'epoch': 0.31} 31%|███ | 561/1817 [20:25:06<45:30:55, 130.46s/it] 31%|███ | 562/1817 [20:27:14<45:15:04, 129.80s/it] {'loss': 1.8523, 'learning_rate': 2e-05, 'epoch': 0.31} 31%|███ | 562/1817 [20:27:14<45:15:04, 129.80s/it] 31%|███ | 563/1817 [20:29:21<44:53:20, 128.87s/it] {'loss': 1.9265, 'learning_rate': 2e-05, 'epoch': 0.31} 31%|███ | 563/1817 [20:29:21<44:53:20, 128.87s/it] 31%|███ | 564/1817 [20:31:35<45:24:01, 130.44s/it] {'loss': 1.8735, 'learning_rate': 2e-05, 'epoch': 0.31} 31%|███ | 564/1817 [20:31:35<45:24:04, 130.44s/it] 31%|███ | 565/1817 [20:33:42<45:03:29, 129.56s/it] {'loss': 1.8906, 'learning_rate': 2e-05, 'epoch': 0.31} 31%|███ | 565/1817 [20:33:42<45:03:30, 129.56s/it] 31%|███ | 566/1817 [20:35:52<44:59:31, 129.47s/it] {'loss': 1.9265, 'learning_rate': 2e-05, 'epoch': 0.31} 31%|███ | 566/1817 [20:35:51<44:59:32, 129.47s/it] 31%|███ | 567/1817 [20:37:59<44:45:00, 128.88s/it] {'loss': 1.8794, 'learning_rate': 2e-05, 'epoch': 0.31} 31%|███ | 567/1817 [20:37:59<44:45:00, 128.88s/it] 31%|███▏ | 568/1817 [20:40:09<44:49:12, 129.19s/it] {'loss': 1.9257, 'learning_rate': 2e-05, 'epoch': 0.31} 31%|███▏ | 568/1817 [20:40:09<44:49:12, 129.19s/it] 31%|███▏ | 569/1817 [20:42:22<45:09:50, 130.28s/it] {'loss': 1.8942, 'learning_rate': 2e-05, 'epoch': 0.31} 31%|███▏ | 569/1817 [20:42:22<45:09:51, 130.28s/it] 31%|███▏ | 570/1817 [20:44:38<45:44:12, 132.04s/it] {'loss': 1.9291, 'learning_rate': 2e-05, 'epoch': 0.31} 31%|███▏ | 570/1817 [20:44:38<45:44:12, 132.04s/it] 31%|███▏ | 571/1817 [20:46:46<45:18:28, 130.91s/it] {'loss': 1.9193, 'learning_rate': 2e-05, 'epoch': 0.31} 31%|███▏ | 571/1817 [20:46:46<45:18:29, 130.91s/it] 31%|███▏ | 572/1817 [20:48:58<45:18:43, 131.02s/it] {'loss': 1.9128, 'learning_rate': 2e-05, 'epoch': 0.31} 31%|███▏ | 572/1817 [20:48:57<45:18:44, 131.02s/it] 32%|███▏ | 573/1817 [20:51:05<44:54:57, 129.98s/it] {'loss': 1.9513, 'learning_rate': 2e-05, 'epoch': 0.32} 32%|███▏ | 573/1817 [20:51:05<44:54:57, 129.98s/it] 32%|███▏ | 574/1817 [20:53:18<45:07:39, 130.70s/it] {'loss': 1.9417, 'learning_rate': 2e-05, 'epoch': 0.32} 32%|███▏ | 574/1817 [20:53:17<45:07:39, 130.70s/it] 32%|███▏ | 575/1817 [20:55:29<45:11:04, 130.97s/it] {'loss': 1.9282, 'learning_rate': 2e-05, 'epoch': 0.32} 32%|███▏ | 575/1817 [20:55:29<45:11:04, 130.97s/it] 32%|███▏ | 576/1817 [20:57:38<44:53:34, 130.23s/it] {'loss': 1.9063, 'learning_rate': 2e-05, 'epoch': 0.32} 32%|███▏ | 576/1817 [20:57:37<44:53:34, 130.23s/it] 32%|███▏ | 577/1817 [20:59:48<44:50:38, 130.19s/it] {'loss': 1.8884, 'learning_rate': 2e-05, 'epoch': 0.32} 32%|███▏ | 577/1817 [20:59:48<44:50:38, 130.19s/it] 32%|███▏ | 578/1817 [21:01:58<44:49:38, 130.25s/it] {'loss': 1.9453, 'learning_rate': 2e-05, 'epoch': 0.32} 32%|███▏ | 578/1817 [21:01:58<44:49:40, 130.25s/it] 32%|███▏ | 579/1817 [21:04:08<44:47:07, 130.23s/it] {'loss': 1.8888, 'learning_rate': 2e-05, 'epoch': 0.32} 32%|███▏ | 579/1817 [21:04:08<44:47:10, 130.23s/it] 32%|███▏ | 580/1817 [21:06:21<45:04:08, 131.16s/it] {'loss': 1.92, 'learning_rate': 2e-05, 'epoch': 0.32} 32%|███▏ | 580/1817 [21:06:22<45:04:08, 131.16s/it] {'loss': 1.92, 'learning_rate': 2e-05, 'epoch': 0.32} 32%|███▏ | 580/1817 [21:06:21<45:04:08, 131.16s/it] 32%|███▏ | 581/1817 [21:08:34<45:09:12, 131.51s/it] {'loss': 1.8859, 'learning_rate': 2e-05, 'epoch': 0.32} 32%|███▏ | 581/1817 [21:08:34<45:09:12, 131.52s/it] 32%|███▏ | 582/1817 [21:10:46<45:10:04, 131.66s/it] {'loss': 1.8833, 'learning_rate': 2e-05, 'epoch': 0.32} 32%|███▏ | 582/1817 [21:10:46<45:09:59, 131.66s/it] 32%|███▏ | 583/1817 [21:12:56<44:55:32, 131.06s/it] {'loss': 1.8938, 'learning_rate': 2e-05, 'epoch': 0.32} 32%|███▏ | 583/1817 [21:12:55<44:55:28, 131.06s/it] 32%|███▏ | 584/1817 [21:15:10<45:13:08, 132.03s/it] {'loss': 1.8982, 'learning_rate': 2e-05, 'epoch': 0.32} 32%|███▏ | 584/1817 [21:15:10<45:13:06, 132.02s/it] 32%|███▏ | 585/1817 [21:17:21<45:07:29, 131.86s/it] {'loss': 1.8773, 'learning_rate': 2e-05, 'epoch': 0.32} 32%|███▏ | 585/1817 [21:17:21<45:07:29, 131.86s/it] 32%|███▏ | 586/1817 [21:19:31<44:52:48, 131.25s/it] {'loss': 1.9153, 'learning_rate': 2e-05, 'epoch': 0.32} 32%|███▏ | 586/1817 [21:19:31<44:52:48, 131.25s/it] 32%|███▏ | 587/1817 [21:21:41<44:40:44, 130.77s/it] {'loss': 1.8638, 'learning_rate': 2e-05, 'epoch': 0.32} 32%|███▏ | 587/1817 [21:21:41<44:40:44, 130.77s/it] 32%|███▏ | 588/1817 [21:23:48<44:18:07, 129.77s/it] {'loss': 1.9309, 'learning_rate': 2e-05, 'epoch': 0.32} 32%|███▏ | 588/1817 [21:23:48<44:18:10, 129.77s/it] 32%|███▏ | 589/1817 [21:25:57<44:11:42, 129.56s/it] {'loss': 1.9395, 'learning_rate': 2e-05, 'epoch': 0.32} 32%|███▏ | 589/1817 [21:25:57<44:11:42, 129.56s/it] 32%|███▏ | 590/1817 [21:28:10<44:25:29, 130.34s/it] {'loss': 1.9212, 'learning_rate': 2e-05, 'epoch': 0.32} 32%|███▏ | 590/1817 [21:28:09<44:25:29, 130.34s/it] 33%|███▎ | 591/1817 [21:30:17<44:04:09, 129.40s/it] {'loss': 1.9118, 'learning_rate': 2e-05, 'epoch': 0.33} 33%|███▎ | 591/1817 [21:30:17<44:04:10, 129.40s/it] 33%|███▎ | 592/1817 [21:32:27<44:09:20, 129.76s/it] {'loss': 1.8964, 'learning_rate': 2e-05, 'epoch': 0.33} 33%|███▎ | 592/1817 [21:32:27<44:09:20, 129.76s/it] 33%|███▎ | 593/1817 [21:34:38<44:13:12, 130.06s/it] {'loss': 1.893, 'learning_rate': 2e-05, 'epoch': 0.33} 33%|███▎ | 593/1817 [21:34:38<44:13:13, 130.06s/it] 33%|███▎ | 594/1817 [21:36:48<44:08:16, 129.92s/it] {'loss': 1.8865, 'learning_rate': 2e-05, 'epoch': 0.33} 33%|███▎ | 594/1817 [21:36:47<44:08:17, 129.92s/it] 33%|███▎ | 595/1817 [21:38:59<44:17:11, 130.47s/it] {'loss': 1.8909, 'learning_rate': 2e-05, 'epoch': 0.33} 33%|███▎ | 595/1817 [21:38:59<44:17:11, 130.47s/it] 33%|███▎ | 596/1817 [21:41:08<44:00:17, 129.74s/it] {'loss': 1.9043, 'learning_rate': 2e-05, 'epoch': 0.33} 33%|███▎ | 596/1817 [21:41:07<44:00:17, 129.74s/it] 33%|███▎ | 597/1817 [21:43:17<43:59:38, 129.82s/it] {'loss': 1.9043, 'learning_rate': 2e-05, 'epoch': 0.33} 33%|███▎ | 597/1817 [21:43:17<43:59:38, 129.82s/it] 33%|███▎ | 598/1817 [21:45:26<43:50:28, 129.47s/it] {'loss': 1.9114, 'learning_rate': 2e-05, 'epoch': 0.33} 33%|███▎ | 598/1817 [21:45:26<43:50:29, 129.47s/it] 33%|███▎ | 599/1817 [21:47:37<43:57:37, 129.93s/it] {'loss': 1.919, 'learning_rate': 2e-05, 'epoch': 0.33} 33%|███▎ | 599/1817 [21:47:37<43:57:37, 129.93s/it] 33%|███▎ | 600/1817 [21:49:47<43:57:12, 130.02s/it] {'loss': 1.9009, 'learning_rate': 2e-05, 'epoch': 0.33} 33%|███▎ | 600/1817 [21:49:47<43:57:13, 130.02s/it] 33%|███▎ | 601/1817 [21:51:56<43:48:09, 129.68s/it] {'loss': 1.9104, 'learning_rate': 2e-05, 'epoch': 0.33} 33%|███▎ | 601/1817 [21:51:56<43:48:09, 129.68s/it] 33%|███▎ | 602/1817 [21:54:03<43:26:12, 128.70s/it] {'loss': 1.909, 'learning_rate': 2e-05, 'epoch': 0.33} 33%|███▎ | 602/1817 [21:54:02<43:26:12, 128.70s/it] 33%|███▎ | 603/1817 [21:56:14<43:38:27, 129.41s/it] {'loss': 1.892, 'learning_rate': 2e-05, 'epoch': 0.33} 33%|███▎ | 603/1817 [21:56:14<43:38:27, 129.41s/it] 33%|███▎ | 604/1817 [21:58:27<44:00:10, 130.59s/it] {'loss': 1.9114, 'learning_rate': 2e-05, 'epoch': 0.33} 33%|███▎ | 604/1817 [21:58:27<44:00:10, 130.59s/it] 33%|███▎ | 605/1817 [22:00:35<43:43:47, 129.89s/it] {'loss': 1.8872, 'learning_rate': 2e-05, 'epoch': 0.33} 33%|███▎ | 605/1817 [22:00:35<43:43:47, 129.89s/it] 33%|███▎ | 606/1817 [22:02:47<43:52:42, 130.44s/it] {'loss': 1.9763, 'learning_rate': 2e-05, 'epoch': 0.33} 33%|███▎ | 606/1817 [22:02:47<43:52:43, 130.44s/it] 33%|███▎ | 607/1817 [22:04:58<43:53:40, 130.60s/it] {'loss': 1.8846, 'learning_rate': 2e-05, 'epoch': 0.33} 33%|███▎ | 607/1817 [22:04:58<43:53:43, 130.60s/it] 33%|███▎ | 608/1817 [22:07:08<43:45:41, 130.31s/it] {'loss': 1.9013, 'learning_rate': 2e-05, 'epoch': 0.33} 33%|███▎ | 608/1817 [22:07:07<43:45:42, 130.31s/it] 34%|███▎ | 609/1817 [22:09:20<43:57:57, 131.02s/it] {'loss': 1.8632, 'learning_rate': 2e-05, 'epoch': 0.34} 34%|███▎ | 609/1817 [22:09:20<43:57:58, 131.03s/it] 34%|███▎ | 610/1817 [22:11:33<44:02:39, 131.37s/it] {'loss': 1.9203, 'learning_rate': 2e-05, 'epoch': 0.34} 34%|███▎ | 610/1817 [22:11:32<44:02:33, 131.36s/it] 34%|███▎ | 611/1817 [22:13:45<44:04:15, 131.55s/it] {'loss': 1.8815, 'learning_rate': 2e-05, 'epoch': 0.34} 34%|███▎ | 611/1817 [22:13:44<44:04:11, 131.55s/it] 34%|███▎ | 612/1817 [22:15:56<44:02:39, 131.58s/it] {'loss': 1.8842, 'learning_rate': 2e-05, 'epoch': 0.34} 34%|███▎ | 612/1817 [22:15:56<44:02:39, 131.58s/it] 34%|███▎ | 613/1817 [22:18:05<43:42:08, 130.67s/it] {'loss': 1.9255, 'learning_rate': 2e-05, 'epoch': 0.34} 34%|███▎ | 613/1817 [22:18:05<43:42:08, 130.67s/it] 34%|███▍ | 614/1817 [22:20:13<43:23:56, 129.87s/it] {'loss': 1.9214, 'learning_rate': 2e-05, 'epoch': 0.34} 34%|███▍ | 614/1817 [22:20:13<43:23:57, 129.87s/it] 34%|███▍ | 615/1817 [22:22:20<43:07:50, 129.18s/it] {'loss': 1.9206, 'learning_rate': 2e-05, 'epoch': 0.34} 34%|███▍ | 615/1817 [22:22:20<43:07:50, 129.18s/it] 34%|███▍ | 616/1817 [22:24:31<43:15:00, 129.64s/it] {'loss': 1.8912, 'learning_rate': 2e-05, 'epoch': 0.34} 34%|███▍ | 616/1817 [22:24:31<43:15:02, 129.64s/it] 34%|███▍ | 617/1817 [22:26:43<43:29:22, 130.47s/it] {'loss': 1.908, 'learning_rate': 2e-05, 'epoch': 0.34} 34%|███▍ | 617/1817 [22:26:43<43:29:21, 130.47s/it] 34%|███▍ | 618/1817 [22:28:56<43:41:11, 131.17s/it] {'loss': 1.8487, 'learning_rate': 2e-05, 'epoch': 0.34} 34%|███▍ | 618/1817 [22:28:56<43:41:11, 131.17s/it] 34%|███▍ | 619/1817 [22:31:06<43:30:51, 130.76s/it] {'loss': 1.9104, 'learning_rate': 2e-05, 'epoch': 0.34} 34%|███▍ | 619/1817 [22:31:06<43:30:50, 130.76s/it] 34%|███▍ | 620/1817 [22:33:18<43:34:41, 131.06s/it] {'loss': 1.9011, 'learning_rate': 2e-05, 'epoch': 0.34} 34%|███▍ | 620/1817 [22:33:18<43:34:41, 131.06s/it] 34%|███▍ | 621/1817 [22:35:29<43:35:43, 131.22s/it] {'loss': 1.8984, 'learning_rate': 2e-05, 'epoch': 0.34} 34%|███▍ | 621/1817 [22:35:29<43:35:46, 131.23s/it] 34%|███▍ | 622/1817 [22:37:42<43:39:20, 131.52s/it] {'loss': 1.9325, 'learning_rate': 2e-05, 'epoch': 0.34} 34%|███▍ | 622/1817 [22:37:41<43:39:20, 131.51s/it] 34%|███▍ | 623/1817 [22:39:50<43:18:33, 130.58s/it] {'loss': 1.853, 'learning_rate': 2e-05, 'epoch': 0.34} 34%|███▍ | 623/1817 [22:39:50<43:18:33, 130.58s/it] 34%|███▍ | 624/1817 [22:41:56<42:53:24, 129.43s/it] {'loss': 1.8913, 'learning_rate': 2e-05, 'epoch': 0.34} 34%|███▍ | 624/1817 [22:41:57<42:53:24, 129.43s/it] 34%|███▍ | 625/1817 [22:44:09<43:11:34, 130.45s/it] {'loss': 1.8883, 'learning_rate': 2e-05, 'epoch': 0.34} 34%|███▍ | 625/1817 [22:44:09<43:11:34, 130.45s/it] 34%|███▍ | 626/1817 [22:46:20<43:07:53, 130.37s/it] {'loss': 1.9384, 'learning_rate': 2e-05, 'epoch': 0.34} 34%|███▍ | 626/1817 [22:46:20<43:07:53, 130.37s/it] 35%|███▍ | 627/1817 [22:48:30<43:03:00, 130.24s/it] {'loss': 1.896, 'learning_rate': 2e-05, 'epoch': 0.34} 35%|███▍ | 627/1817 [22:48:29<43:03:03, 130.24s/it] 35%|███▍ | 628/1817 [22:50:43<43:17:41, 131.09s/it] {'loss': 1.902, 'learning_rate': 2e-05, 'epoch': 0.35} 35%|███▍ | 628/1817 [22:50:43<43:17:41, 131.09s/it] 35%|███▍ | 629/1817 [22:52:56<43:29:46, 131.81s/it] {'loss': 1.9063, 'learning_rate': 2e-05, 'epoch': 0.35} 35%|███▍ | 629/1817 [22:52:56<43:29:46, 131.81s/it] 35%|███▍ | 630/1817 [22:55:08<43:30:59, 131.98s/it] {'loss': 1.9431, 'learning_rate': 2e-05, 'epoch': 0.35} 35%|███▍ | 630/1817 [22:55:08<43:30:59, 131.98s/it] 35%|███▍ | 631/1817 [22:57:22<43:37:07, 132.40s/it] {'loss': 1.8958, 'learning_rate': 2e-05, 'epoch': 0.35} 35%|███▍ | 631/1817 [22:57:22<43:37:07, 132.40s/it] 35%|███▍ | 632/1817 [22:59:30<43:09:44, 131.13s/it] {'loss': 1.8866, 'learning_rate': 2e-05, 'epoch': 0.35} 35%|███▍ | 632/1817 [22:59:30<43:09:44, 131.13s/it] 35%|███▍ | 633/1817 [23:01:42<43:13:12, 131.41s/it] {'loss': 1.9167, 'learning_rate': 2e-05, 'epoch': 0.35} 35%|███▍ | 633/1817 [23:01:42<43:13:12, 131.41s/it] 35%|███▍ | 634/1817 [23:03:57<43:28:09, 132.28s/it] {'loss': 1.8514, 'learning_rate': 2e-05, 'epoch': 0.35} 35%|███▍ | 634/1817 [23:03:56<43:28:09, 132.28s/it] 35%|███▍ | 635/1817 [23:06:07<43:14:50, 131.72s/it] {'loss': 1.9018, 'learning_rate': 2e-05, 'epoch': 0.35} 35%|███▍ | 635/1817 [23:06:07<43:14:50, 131.72s/it] 35%|███▌ | 636/1817 [23:08:18<43:12:32, 131.71s/it] {'loss': 1.8664, 'learning_rate': 2e-05, 'epoch': 0.35} 35%|███▌ | 636/1817 [23:08:18<43:12:32, 131.71s/it] 35%|███▌ | 637/1817 [23:10:29<43:03:12, 131.35s/it] {'loss': 1.8894, 'learning_rate': 2e-05, 'epoch': 0.35} 35%|███▌ | 637/1817 [23:10:29<43:02:45, 131.33s/it] 35%|███▌ | 638/1817 [23:12:38<42:47:48, 130.68s/it] {'loss': 1.8827, 'learning_rate': 2e-05, 'epoch': 0.35} 35%|███▌ | 638/1817 [23:12:38<42:47:29, 130.66s/it] 35%|███▌ | 639/1817 [23:14:51<42:55:12, 131.16s/it] {'loss': 1.9414, 'learning_rate': 2e-05, 'epoch': 0.35} 35%|███▌ | 639/1817 [23:14:50<42:54:59, 131.15s/it] 35%|███▌ | 640/1817 [23:17:00<42:43:30, 130.68s/it] {'loss': 1.9006, 'learning_rate': 2e-05, 'epoch': 0.35} 35%|███▌ | 640/1817 [23:17:00<42:43:30, 130.68s/it] 35%|███▌ | 641/1817 [23:19:14<43:02:45, 131.77s/it] {'loss': 1.9205, 'learning_rate': 2e-05, 'epoch': 0.35} 35%|███▌ | 641/1817 [23:19:14<43:02:45, 131.77s/it] 35%|███▌ | 642/1817 [23:21:23<42:45:04, 130.98s/it] {'loss': 1.9106, 'learning_rate': 2e-05, 'epoch': 0.35} 35%|███▌ | 642/1817 [23:21:23<42:45:04, 130.98s/it] 35%|███▌ | 643/1817 [23:23:33<42:34:00, 130.53s/it] {'loss': 1.8897, 'learning_rate': 2e-05, 'epoch': 0.35} 35%|███▌ | 643/1817 [23:23:33<42:34:00, 130.53s/it] 35%|███▌ | 644/1817 [23:25:47<42:52:52, 131.60s/it] {'loss': 1.8741, 'learning_rate': 2e-05, 'epoch': 0.35} 35%|███▌ | 644/1817 [23:25:47<42:52:52, 131.60s/it] 35%|███▌ | 645/1817 [23:28:01<43:05:38, 132.37s/it] {'loss': 1.9279, 'learning_rate': 2e-05, 'epoch': 0.35} 35%|███▌ | 645/1817 [23:28:01<43:05:38, 132.37s/it] 36%|███▌ | 646/1817 [23:30:12<42:53:38, 131.87s/it] {'loss': 1.8958, 'learning_rate': 2e-05, 'epoch': 0.36} 36%|███▌ | 646/1817 [23:30:12<42:53:38, 131.87s/it] 36%|███▌ | 647/1817 [23:32:27<43:08:40, 132.75s/it] {'loss': 1.8758, 'learning_rate': 2e-05, 'epoch': 0.36} 36%|███▌ | 647/1817 [23:32:27<43:08:40, 132.75s/it] 36%|███▌ | 648/1817 [23:34:37<42:50:26, 131.93s/it] {'loss': 1.903, 'learning_rate': 2e-05, 'epoch': 0.36} 36%|███▌ | 648/1817 [23:34:37<42:50:26, 131.93s/it] 36%|███▌ | 649/1817 [23:36:46<42:31:07, 131.05s/it] {'loss': 1.8971, 'learning_rate': 2e-05, 'epoch': 0.36} 36%|███▌ | 649/1817 [23:36:46<42:31:07, 131.05s/it] 36%|███▌ | 650/1817 [23:38:57<42:30:50, 131.15s/it] {'loss': 1.9216, 'learning_rate': 2e-05, 'epoch': 0.36} 36%|███▌ | 650/1817 [23:38:57<42:30:50, 131.15s/it] 36%|███▌ | 651/1817 [23:41:09<42:32:35, 131.35s/it] {'loss': 1.8396, 'learning_rate': 2e-05, 'epoch': 0.36} 36%|███▌ | 651/1817 [23:41:09<42:32:35, 131.35s/it] 36%|███▌ | 652/1817 [23:43:20<42:30:28, 131.35s/it] {'loss': 1.8881, 'learning_rate': 2e-05, 'epoch': 0.36} 36%|███▌ | 652/1817 [23:43:20<42:30:30, 131.36s/it] 36%|███▌ | 653/1817 [23:45:29<42:13:45, 130.61s/it] {'loss': 1.886, 'learning_rate': 2e-05, 'epoch': 0.36} 36%|███▌ | 653/1817 [23:45:29<42:13:45, 130.61s/it] 36%|███▌ | 654/1817 [23:47:39<42:09:51, 130.52s/it] {'loss': 1.905, 'learning_rate': 2e-05, 'epoch': 0.36} 36%|███▌ | 654/1817 [23:47:39<42:09:51, 130.52s/it] 36%|███▌ | 655/1817 [23:49:49<42:02:38, 130.26s/it] {'loss': 1.8537, 'learning_rate': 2e-05, 'epoch': 0.36} 36%|███▌ | 655/1817 [23:49:49<42:02:38, 130.26s/it] 36%|███▌ | 656/1817 [23:52:02<42:13:20, 130.92s/it] {'loss': 1.9187, 'learning_rate': 2e-05, 'epoch': 0.36} 36%|███▌ | 656/1817 [23:52:01<42:13:22, 130.92s/it] 36%|███▌ | 657/1817 [23:54:11<42:00:08, 130.35s/it] {'loss': 1.9301, 'learning_rate': 2e-05, 'epoch': 0.36} 36%|███▌ | 657/1817 [23:54:10<42:00:11, 130.35s/it] 36%|███▌ | 658/1817 [23:56:22<42:04:24, 130.69s/it] {'loss': 1.8708, 'learning_rate': 2e-05, 'epoch': 0.36} 36%|███▌ | 658/1817 [23:56:22<42:04:24, 130.69s/it] 36%|███▋ | 659/1817 [23:58:34<42:11:31, 131.17s/it] {'loss': 1.8687, 'learning_rate': 2e-05, 'epoch': 0.36} 36%|███▋ | 659/1817 [23:58:34<42:11:33, 131.17s/it] 36%|███▋ | 660/1817 [24:00:47<42:18:37, 131.65s/it] {'loss': 1.9504, 'learning_rate': 2e-05, 'epoch': 0.36} 36%|███▋ | 660/1817 [24:00:47<42:18:39, 131.65s/it] 36%|███▋ | 661/1817 [24:02:59<42:14:36, 131.55s/it] {'loss': 1.9243, 'learning_rate': 2e-05, 'epoch': 0.36} 36%|███▋ | 661/1817 [24:02:58<42:14:39, 131.56s/it] 36%|███▋ | 662/1817 [24:05:12<42:22:10, 132.06s/it] {'loss': 1.8726, 'learning_rate': 2e-05, 'epoch': 0.36} 36%|███▋ | 662/1817 [24:05:12<42:22:10, 132.06s/it] 36%|███▋ | 663/1817 [24:07:23<42:14:12, 131.76s/it] {'loss': 1.9457, 'learning_rate': 2e-05, 'epoch': 0.36} 36%|███▋ | 663/1817 [24:07:23<42:14:12, 131.76s/it] 37%|███▋ | 664/1817 [24:09:34<42:11:02, 131.71s/it] {'loss': 1.8824, 'learning_rate': 2e-05, 'epoch': 0.37} 37%|███▋ | 664/1817 [24:09:34<42:10:43, 131.69s/it] 37%|███▋ | 665/1817 [24:11:47<42:14:52, 132.02s/it] {'loss': 1.8867, 'learning_rate': 2e-05, 'epoch': 0.37} 37%|███▋ | 665/1817 [24:11:47<42:14:29, 132.00s/it] 37%|███▋ | 666/1817 [24:13:58<42:07:06, 131.73s/it] {'loss': 1.9077, 'learning_rate': 2e-05, 'epoch': 0.37} 37%|███▋ | 666/1817 [24:13:58<42:06:50, 131.72s/it] 37%|███▋ | 667/1817 [24:16:10<42:03:14, 131.65s/it] {'loss': 1.9164, 'learning_rate': 2e-05, 'epoch': 0.37} 37%|███▋ | 667/1817 [24:16:09<42:03:13, 131.65s/it] 37%|███▋ | 668/1817 [24:18:24<42:14:53, 132.37s/it] {'loss': 1.8599, 'learning_rate': 2e-05, 'epoch': 0.37} 37%|███▋ | 668/1817 [24:18:23<42:14:55, 132.37s/it] 37%|███▋ | 669/1817 [24:20:34<42:00:08, 131.71s/it] {'loss': 1.9518, 'learning_rate': 2e-05, 'epoch': 0.37} 37%|███▋ | 669/1817 [24:20:34<42:00:10, 131.72s/it] 37%|███▋ | 670/1817 [24:22:49<42:15:33, 132.64s/it] {'loss': 1.8592, 'learning_rate': 2e-05, 'epoch': 0.37} 37%|███▋ | 670/1817 [24:22:48<42:15:35, 132.64s/it] 37%|███▋ | 671/1817 [24:24:58<41:53:46, 131.61s/it] {'loss': 1.9047, 'learning_rate': 2e-05, 'epoch': 0.37} 37%|███▋ | 671/1817 [24:24:58<41:53:49, 131.61s/it] 37%|███▋ | 672/1817 [24:27:09<41:46:01, 131.32s/it] {'loss': 1.8841, 'learning_rate': 2e-05, 'epoch': 0.37} 37%|███▋ | 672/1817 [24:27:08<41:46:03, 131.32s/it] 37%|███▋ | 673/1817 [24:29:17<41:29:37, 130.58s/it] {'loss': 1.8835, 'learning_rate': 2e-05, 'epoch': 0.37} 37%|███▋ | 673/1817 [24:29:17<41:29:38, 130.58s/it] 37%|███▋ | 674/1817 [24:31:28<41:24:48, 130.44s/it] {'loss': 1.8647, 'learning_rate': 2e-05, 'epoch': 0.37} 37%|███▋ | 674/1817 [24:31:27<41:24:50, 130.44s/it] 37%|███▋ | 675/1817 [24:33:38<41:20:43, 130.34s/it] {'loss': 1.8704, 'learning_rate': 2e-05, 'epoch': 0.37} 37%|███▋ | 675/1817 [24:33:37<41:20:46, 130.34s/it] 37%|███▋ | 676/1817 [24:35:49<41:27:16, 130.79s/it] {'loss': 1.8766, 'learning_rate': 2e-05, 'epoch': 0.37} 37%|███▋ | 676/1817 [24:35:49<41:27:16, 130.79s/it] 37%|███▋ | 677/1817 [24:38:01<41:27:38, 130.93s/it] {'loss': 1.9261, 'learning_rate': 2e-05, 'epoch': 0.37} 37%|███▋ | 677/1817 [24:38:00<41:27:37, 130.93s/it] 37%|███▋ | 678/1817 [24:40:11<41:20:46, 130.68s/it] {'loss': 1.8876, 'learning_rate': 2e-05, 'epoch': 0.37} 37%|███▋ | 678/1817 [24:40:11<41:20:46, 130.68s/it] 37%|███▋ | 679/1817 [24:42:24<41:34:22, 131.51s/it] {'loss': 1.8998, 'learning_rate': 2e-05, 'epoch': 0.37} 37%|███▋ | 679/1817 [24:42:24<41:34:22, 131.51s/it] 37%|███▋ | 680/1817 [24:44:36<41:32:09, 131.51s/it] {'loss': 1.8974, 'learning_rate': 2e-05, 'epoch': 0.37} 37%|███▋ | 680/1817 [24:44:36<41:32:09, 131.51s/it] 37%|███▋ | 681/1817 [24:46:44<41:08:44, 130.39s/it] {'loss': 1.9191, 'learning_rate': 2e-05, 'epoch': 0.37} 37%|███▋ | 681/1817 [24:46:43<41:08:44, 130.39s/it] 38%|███▊ | 682/1817 [24:48:54<41:08:58, 130.52s/it] {'loss': 1.8954, 'learning_rate': 2e-05, 'epoch': 0.38} 38%|███▊ | 682/1817 [24:48:54<41:08:58, 130.52s/it] 38%|███▊ | 683/1817 [24:51:08<41:25:00, 131.48s/it] {'loss': 1.9222, 'learning_rate': 2e-05, 'epoch': 0.38} 38%|███▊ | 683/1817 [24:51:08<41:25:01, 131.48s/it] 38%|███▊ | 684/1817 [24:53:21<41:30:15, 131.88s/it] {'loss': 1.9267, 'learning_rate': 2e-05, 'epoch': 0.38} 38%|███▊ | 684/1817 [24:53:21<41:30:15, 131.88s/it] 38%|███▊ | 685/1817 [24:55:34<41:34:05, 132.20s/it] {'loss': 1.8916, 'learning_rate': 2e-05, 'epoch': 0.38} 38%|███▊ | 685/1817 [24:55:34<41:34:05, 132.20s/it] 38%|███▊ | 686/1817 [24:57:47<41:36:48, 132.46s/it] {'loss': 1.9113, 'learning_rate': 2e-05, 'epoch': 0.38} 38%|███▊ | 686/1817 [24:57:47<41:36:48, 132.46s/it] 38%|███▊ | 687/1817 [24:59:57<41:23:04, 131.84s/it] {'loss': 1.8485, 'learning_rate': 2e-05, 'epoch': 0.38} 38%|███▊ | 687/1817 [24:59:57<41:23:04, 131.84s/it] 38%|███▊ | 688/1817 [25:02:08<41:17:18, 131.66s/it] {'loss': 1.9294, 'learning_rate': 2e-05, 'epoch': 0.38} 38%|███▊ | 688/1817 [25:02:08<41:17:18, 131.66s/it] 38%|███▊ | 689/1817 [25:04:19<41:10:42, 131.42s/it] {'loss': 1.8951, 'learning_rate': 2e-05, 'epoch': 0.38} 38%|███▊ | 689/1817 [25:04:19<41:10:42, 131.42s/it] 38%|███▊ | 690/1817 [25:06:31<41:09:55, 131.50s/it] {'loss': 1.868, 'learning_rate': 2e-05, 'epoch': 0.38} 38%|███▊ | 690/1817 [25:06:31<41:09:56, 131.50s/it] 38%|███▊ | 691/1817 [25:08:46<41:26:12, 132.48s/it] {'loss': 1.8922, 'learning_rate': 2e-05, 'epoch': 0.38} 38%|███▊ | 691/1817 [25:08:46<41:26:13, 132.48s/it] 38%|███▊ | 692/1817 [25:10:57<41:17:07, 132.11s/it] {'loss': 1.8933, 'learning_rate': 2e-05, 'epoch': 0.38} 38%|███▊ | 692/1817 [25:10:57<41:17:01, 132.11s/it] 38%|███▊ | 693/1817 [25:13:07<41:02:10, 131.43s/it] {'loss': 1.9372, 'learning_rate': 2e-05, 'epoch': 0.38} 38%|███▊ | 693/1817 [25:13:07<41:02:06, 131.43s/it] 38%|███▊ | 694/1817 [25:15:15<40:43:16, 130.54s/it] {'loss': 1.8725, 'learning_rate': 2e-05, 'epoch': 0.38} 38%|███▊ | 694/1817 [25:15:15<40:43:14, 130.54s/it] 38%|███▊ | 695/1817 [25:17:26<40:38:47, 130.42s/it] {'loss': 1.8851, 'learning_rate': 2e-05, 'epoch': 0.38} 38%|███▊ | 695/1817 [25:17:26<40:38:47, 130.42s/it] 38%|███▊ | 696/1817 [25:19:51<41:58:53, 134.82s/it] {'loss': 1.8667, 'learning_rate': 2e-05, 'epoch': 0.38} 38%|███▊ | 696/1817 [25:19:50<41:58:54, 134.82s/it] 38%|███▊ | 697/1817 [25:22:02<41:34:30, 133.63s/it] {'loss': 1.8619, 'learning_rate': 2e-05, 'epoch': 0.38} 38%|███▊ | 697/1817 [25:22:01<41:34:31, 133.63s/it] 38%|███▊ | 698/1817 [25:24:14<41:25:08, 133.25s/it] {'loss': 1.8821, 'learning_rate': 2e-05, 'epoch': 0.38} 38%|███▊ | 698/1817 [25:24:14<41:25:08, 133.25s/it] 38%|███▊ | 699/1817 [25:26:24<41:04:30, 132.26s/it] {'loss': 1.894, 'learning_rate': 2e-05, 'epoch': 0.38} 38%|███▊ | 699/1817 [25:26:24<41:04:31, 132.26s/it] 39%|███▊ | 700/1817 [25:28:34<40:51:08, 131.66s/it] {'loss': 1.8536, 'learning_rate': 2e-05, 'epoch': 0.39} 39%|███▊ | 700/1817 [25:28:34<40:51:09, 131.66s/it] 39%|███▊ | 701/1817 [25:30:43<40:35:39, 130.95s/it] {'loss': 1.8619, 'learning_rate': 2e-05, 'epoch': 0.39} 39%|███▊ | 701/1817 [25:30:43<40:35:40, 130.95s/it] 39%|███▊ | 702/1817 [25:32:56<40:44:54, 131.56s/it] {'loss': 1.8883, 'learning_rate': 2e-05, 'epoch': 0.39} 39%|███▊ | 702/1817 [25:32:56<40:44:55, 131.57s/it] 39%|███▊ | 703/1817 [25:35:10<40:55:24, 132.25s/it] {'loss': 1.897, 'learning_rate': 2e-05, 'epoch': 0.39} 39%|███▊ | 703/1817 [25:35:10<40:55:24, 132.25s/it] 39%|███▊ | 704/1817 [25:37:21<40:45:31, 131.83s/it] {'loss': 1.9163, 'learning_rate': 2e-05, 'epoch': 0.39} 39%|███▊ | 704/1817 [25:37:21<40:45:31, 131.83s/it] 39%|███▉ | 705/1817 [25:39:32<40:38:19, 131.56s/it] {'loss': 1.8644, 'learning_rate': 2e-05, 'epoch': 0.39} 39%|███▉ | 705/1817 [25:39:32<40:38:20, 131.57s/it] 39%|███▉ | 706/1817 [25:41:42<40:26:02, 131.02s/it] {'loss': 1.9272, 'learning_rate': 2e-05, 'epoch': 0.39} 39%|███▉ | 706/1817 [25:41:42<40:26:02, 131.02s/it] 39%|███▉ | 707/1817 [25:43:54<40:29:24, 131.32s/it] {'loss': 1.9084, 'learning_rate': 2e-05, 'epoch': 0.39} 39%|███▉ | 707/1817 [25:43:54<40:29:24, 131.32s/it] 39%|███▉ | 708/1817 [25:46:00<39:59:59, 129.85s/it] {'loss': 1.9199, 'learning_rate': 2e-05, 'epoch': 0.39} 39%|███▉ | 708/1817 [25:46:00<39:59:59, 129.85s/it] 39%|███▉ | 709/1817 [25:48:10<39:58:31, 129.88s/it] {'loss': 1.8982, 'learning_rate': 2e-05, 'epoch': 0.39} 39%|███▉ | 709/1817 [25:48:10<39:58:32, 129.88s/it] 39%|███▉ | 710/1817 [25:50:22<40:06:34, 130.44s/it] {'loss': 1.8985, 'learning_rate': 2e-05, 'epoch': 0.39} 39%|███▉ | 710/1817 [25:50:22<40:06:35, 130.44s/it] 39%|███▉ | 711/1817 [25:52:31<39:55:14, 129.94s/it] {'loss': 1.9065, 'learning_rate': 2e-05, 'epoch': 0.39} 39%|███▉ | 711/1817 [25:52:30<39:55:14, 129.94s/it] 39%|███▉ | 712/1817 [25:54:45<40:15:18, 131.15s/it] {'loss': 1.8699, 'learning_rate': 2e-05, 'epoch': 0.39} 39%|███▉ | 712/1817 [25:54:44<40:15:18, 131.15s/it] 39%|███▉ | 713/1817 [25:56:55<40:07:50, 130.86s/it] {'loss': 1.9024, 'learning_rate': 2e-05, 'epoch': 0.39} 39%|███▉ | 713/1817 [25:56:55<40:07:50, 130.86s/it] 39%|███▉ | 714/1817 [25:59:04<39:58:08, 130.45s/it] {'loss': 1.8974, 'learning_rate': 2e-05, 'epoch': 0.39} 39%|███▉ | 714/1817 [25:59:04<39:58:08, 130.45s/it] 39%|███▉ | 715/1817 [26:01:16<40:03:34, 130.87s/it] {'loss': 1.9168, 'learning_rate': 2e-05, 'epoch': 0.39} 39%|███▉ | 715/1817 [26:01:16<40:03:34, 130.87s/it] 39%|███▉ | 716/1817 [26:03:32<40:28:41, 132.35s/it] {'loss': 1.9447, 'learning_rate': 2e-05, 'epoch': 0.39} 39%|███▉ | 716/1817 [26:03:32<40:28:41, 132.35s/it] 39%|███▉ | 717/1817 [26:05:42<40:15:50, 131.77s/it] {'loss': 1.9064, 'learning_rate': 2e-05, 'epoch': 0.39} 39%|███▉ | 717/1817 [26:05:42<40:15:50, 131.77s/it] 40%|███▉ | 718/1817 [26:07:53<40:09:13, 131.53s/it] {'loss': 1.7958, 'learning_rate': 2e-05, 'epoch': 0.4} 40%|███▉ | 718/1817 [26:07:53<40:09:13, 131.53s/it] 40%|███▉ | 719/1817 [26:10:03<39:57:01, 130.99s/it] {'loss': 1.9244, 'learning_rate': 2e-05, 'epoch': 0.4} 40%|███▉ | 719/1817 [26:10:03<39:57:01, 130.98s/it] 40%|███▉ | 720/1817 [26:12:12<39:46:04, 130.51s/it] {'loss': 1.9347, 'learning_rate': 2e-05, 'epoch': 0.4} 40%|███▉ | 720/1817 [26:12:12<39:46:04, 130.51s/it] 40%|███▉ | 721/1817 [26:14:20<39:30:11, 129.75s/it] {'loss': 1.8959, 'learning_rate': 2e-05, 'epoch': 0.4} 40%|███▉ | 721/1817 [26:14:20<39:30:11, 129.75s/it] 40%|███▉ | 722/1817 [26:16:33<39:45:35, 130.72s/it] {'loss': 1.8884, 'learning_rate': 2e-05, 'epoch': 0.4} 40%|███▉ | 722/1817 [26:16:33<39:45:35, 130.72s/it] 40%|███▉ | 723/1817 [26:18:43<39:38:04, 130.42s/it] {'loss': 1.904, 'learning_rate': 2e-05, 'epoch': 0.4} 40%|███▉ | 723/1817 [26:18:43<39:38:04, 130.42s/it] 40%|███▉ | 724/1817 [26:20:55<39:44:47, 130.91s/it] {'loss': 1.9052, 'learning_rate': 2e-05, 'epoch': 0.4} 40%|███▉ | 724/1817 [26:20:55<39:44:47, 130.91s/it] 40%|███▉ | 725/1817 [26:23:09<39:57:30, 131.73s/it] {'loss': 1.9032, 'learning_rate': 2e-05, 'epoch': 0.4} 40%|███▉ | 725/1817 [26:23:09<39:57:32, 131.73s/it] 40%|███▉ | 726/1817 [26:25:22<40:01:32, 132.07s/it] {'loss': 1.8981, 'learning_rate': 2e-05, 'epoch': 0.4} 40%|███▉ | 726/1817 [26:25:22<40:01:32, 132.07s/it] 40%|████ | 727/1817 [26:27:34<40:01:33, 132.20s/it] {'loss': 1.8575, 'learning_rate': 2e-05, 'epoch': 0.4} 40%|████ | 727/1817 [26:27:34<40:01:33, 132.20s/it] 40%|████ | 728/1817 [26:29:46<39:58:40, 132.16s/it] {'loss': 1.88, 'learning_rate': 2e-05, 'epoch': 0.4} 40%|████ | 728/1817 [26:29:46<39:58:40, 132.16s/it] 40%|████ | 729/1817 [26:32:01<40:07:40, 132.78s/it] {'loss': 1.8633, 'learning_rate': 2e-05, 'epoch': 0.4} 40%|████ | 729/1817 [26:32:00<40:07:43, 132.78s/it] 40%|████ | 730/1817 [26:34:10<39:45:13, 131.66s/it] {'loss': 1.9089, 'learning_rate': 2e-05, 'epoch': 0.4} 40%|████ | 730/1817 [26:34:09<39:45:15, 131.66s/it] 40%|████ | 731/1817 [26:36:25<40:01:08, 132.66s/it] {'loss': 1.8873, 'learning_rate': 2e-05, 'epoch': 0.4} 40%|████ | 731/1817 [26:36:24<40:01:10, 132.66s/it] 40%|████ | 732/1817 [26:38:35<39:47:50, 132.05s/it] {'loss': 1.8974, 'learning_rate': 2e-05, 'epoch': 0.4} 40%|████ | 732/1817 [26:38:35<39:47:52, 132.05s/it] 40%|████ | 733/1817 [26:40:45<39:32:09, 131.30s/it] {'loss': 1.913, 'learning_rate': 2e-05, 'epoch': 0.4} 40%|████ | 733/1817 [26:40:45<39:32:12, 131.30s/it] 40%|████ | 734/1817 [26:42:55<39:21:30, 130.83s/it] {'loss': 1.8952, 'learning_rate': 2e-05, 'epoch': 0.4} 40%|████ | 734/1817 [26:42:54<39:21:33, 130.83s/it] 40%|████ | 735/1817 [26:45:03<39:08:52, 130.25s/it] {'loss': 1.8674, 'learning_rate': 2e-05, 'epoch': 0.4} 40%|████ | 735/1817 [26:45:03<39:08:54, 130.25s/it] 41%|████ | 736/1817 [26:47:13<39:01:52, 129.98s/it] {'loss': 1.8533, 'learning_rate': 2e-05, 'epoch': 0.4} 41%|████ | 736/1817 [26:47:13<39:01:55, 129.99s/it] 41%|████ | 737/1817 [26:49:26<39:19:30, 131.08s/it] {'loss': 1.914, 'learning_rate': 2e-05, 'epoch': 0.41} 41%|████ | 737/1817 [26:49:26<39:19:33, 131.09s/it] 41%|████ | 738/1817 [26:51:42<39:39:49, 132.33s/it] {'loss': 1.9116, 'learning_rate': 2e-05, 'epoch': 0.41} 41%|████ | 738/1817 [26:51:41<39:39:51, 132.34s/it] 41%|████ | 739/1817 [26:53:53<39:30:42, 131.95s/it] {'loss': 1.9141, 'learning_rate': 2e-05, 'epoch': 0.41} 41%|████ | 739/1817 [26:53:52<39:30:42, 131.95s/it] 41%|████ | 740/1817 [26:56:03<39:17:35, 131.34s/it] {'loss': 1.9161, 'learning_rate': 2e-05, 'epoch': 0.41} 41%|████ | 740/1817 [26:56:02<39:17:34, 131.34s/it] 41%|████ | 741/1817 [26:58:13<39:09:05, 130.99s/it] {'loss': 1.9456, 'learning_rate': 2e-05, 'epoch': 0.41} 41%|████ | 741/1817 [26:58:13<39:09:05, 130.99s/it] 41%|████ | 742/1817 [27:00:23<39:05:00, 130.88s/it] {'loss': 1.8975, 'learning_rate': 2e-05, 'epoch': 0.41} 41%|████ | 742/1817 [27:00:23<39:05:00, 130.88s/it] 41%|████ | 743/1817 [27:02:38<39:21:15, 131.91s/it] {'loss': 1.8891, 'learning_rate': 2e-05, 'epoch': 0.41} 41%|████ | 743/1817 [27:02:38<39:21:17, 131.92s/it] 41%|████ | 744/1817 [27:04:48<39:11:50, 131.51s/it] {'loss': 1.8868, 'learning_rate': 2e-05, 'epoch': 0.41} 41%|████ | 744/1817 [27:04:48<39:11:50, 131.51s/it] 41%|████ | 745/1817 [27:07:01<39:15:19, 131.83s/it] {'loss': 1.8637, 'learning_rate': 2e-05, 'epoch': 0.41} 41%|████ | 745/1817 [27:07:01<39:15:18, 131.83s/it] 41%|████ | 746/1817 [27:09:08<38:46:52, 130.36s/it] {'loss': 1.9557, 'learning_rate': 2e-05, 'epoch': 0.41} 41%|████ | 746/1817 [27:09:08<38:46:52, 130.36s/it] 41%|████ | 747/1817 [27:11:19<38:49:55, 130.65s/it] {'loss': 1.9402, 'learning_rate': 2e-05, 'epoch': 0.41} 41%|████ | 747/1817 [27:11:19<38:49:55, 130.65s/it] 41%|████ | 748/1817 [27:13:29<38:43:36, 130.42s/it] {'loss': 1.889, 'learning_rate': 2e-05, 'epoch': 0.41} 41%|████ | 748/1817 [27:13:29<38:43:32, 130.41s/it] 41%|████ | 749/1817 [27:15:39<38:41:22, 130.41s/it] {'loss': 1.9223, 'learning_rate': 2e-05, 'epoch': 0.41} 41%|████ | 749/1817 [27:15:39<38:41:22, 130.41s/it] 41%|████▏ | 750/1817 [27:17:48<38:27:46, 129.77s/it] {'loss': 1.9007, 'learning_rate': 2e-05, 'epoch': 0.41} 41%|████▏ | 750/1817 [27:17:47<38:27:47, 129.77s/it] 41%|████▏ | 751/1817 [27:19:58<38:26:40, 129.83s/it] {'loss': 1.9094, 'learning_rate': 2e-05, 'epoch': 0.41} 41%|████▏ | 751/1817 [27:19:57<38:26:40, 129.83s/it] 41%|████▏ | 752/1817 [27:22:06<38:16:49, 129.40s/it] {'loss': 1.8861, 'learning_rate': 2e-05, 'epoch': 0.41} 41%|████▏ | 752/1817 [27:22:06<38:16:49, 129.40s/it] 41%|████▏ | 753/1817 [27:24:12<37:56:41, 128.38s/it] {'loss': 1.8757, 'learning_rate': 2e-05, 'epoch': 0.41} 41%|████▏ | 753/1817 [27:24:12<37:56:41, 128.38s/it] 41%|████▏ | 754/1817 [27:26:21<37:54:55, 128.41s/it] {'loss': 1.9088, 'learning_rate': 2e-05, 'epoch': 0.41} 41%|████▏ | 754/1817 [27:26:20<37:54:55, 128.41s/it] 42%|████▏ | 755/1817 [27:28:29<37:52:33, 128.39s/it] {'loss': 1.9726, 'learning_rate': 2e-05, 'epoch': 0.42} 42%|████▏ | 755/1817 [27:28:29<37:52:33, 128.39s/it] 42%|████▏ | 756/1817 [27:30:40<38:03:41, 129.14s/it] {'loss': 1.8578, 'learning_rate': 2e-05, 'epoch': 0.42} 42%|████▏ | 756/1817 [27:30:40<38:03:41, 129.14s/it] 42%|████▏ | 757/1817 [27:32:50<38:08:48, 129.56s/it] {'loss': 1.9242, 'learning_rate': 2e-05, 'epoch': 0.42} 42%|████▏ | 757/1817 [27:32:50<38:08:48, 129.56s/it] 42%|████▏ | 758/1817 [27:35:01<38:12:39, 129.90s/it] {'loss': 1.9324, 'learning_rate': 2e-05, 'epoch': 0.42} 42%|████▏ | 758/1817 [27:35:01<38:12:39, 129.90s/it] 42%|████▏ | 759/1817 [27:37:13<38:23:08, 130.61s/it] {'loss': 1.8984, 'learning_rate': 2e-05, 'epoch': 0.42} 42%|████▏ | 759/1817 [27:37:13<38:23:08, 130.61s/it] 42%|████▏ | 760/1817 [27:39:22<38:10:52, 130.04s/it] {'loss': 1.9099, 'learning_rate': 2e-05, 'epoch': 0.42} 42%|████▏ | 760/1817 [27:39:22<38:10:52, 130.04s/it] 42%|████▏ | 761/1817 [27:41:34<38:18:30, 130.60s/it] {'loss': 1.9125, 'learning_rate': 2e-05, 'epoch': 0.42} 42%|████▏ | 761/1817 [27:41:34<38:18:30, 130.60s/it] 42%|████▏ | 762/1817 [27:43:47<38:32:09, 131.50s/it] {'loss': 1.8791, 'learning_rate': 2e-05, 'epoch': 0.42} 42%|████▏ | 762/1817 [27:43:47<38:32:09, 131.50s/it] 42%|████▏ | 763/1817 [27:46:01<38:40:59, 132.12s/it] {'loss': 1.8705, 'learning_rate': 2e-05, 'epoch': 0.42} 42%|████▏ | 763/1817 [27:46:01<38:40:59, 132.12s/it] 42%|████▏ | 764/1817 [27:48:12<38:34:37, 131.89s/it] {'loss': 1.9098, 'learning_rate': 2e-05, 'epoch': 0.42} 42%|████▏ | 764/1817 [27:48:12<38:34:37, 131.89s/it] 42%|████▏ | 765/1817 [27:50:21<38:14:04, 130.84s/it] {'loss': 1.8933, 'learning_rate': 2e-05, 'epoch': 0.42} 42%|████▏ | 765/1817 [27:50:21<38:14:04, 130.84s/it] 42%|████▏ | 766/1817 [27:52:28<37:55:02, 129.88s/it] {'loss': 1.8978, 'learning_rate': 2e-05, 'epoch': 0.42} 42%|████▏ | 766/1817 [27:52:28<37:55:02, 129.88s/it] 42%|████▏ | 767/1817 [27:54:41<38:05:16, 130.59s/it] {'loss': 1.9248, 'learning_rate': 2e-05, 'epoch': 0.42} 42%|████▏ | 767/1817 [27:54:40<38:05:15, 130.59s/it] 42%|████▏ | 768/1817 [27:56:50<37:54:07, 130.07s/it] {'loss': 1.8808, 'learning_rate': 2e-05, 'epoch': 0.42} 42%|████▏ | 768/1817 [27:56:49<37:54:09, 130.08s/it] 42%|████▏ | 769/1817 [27:59:01<37:58:23, 130.44s/it] {'loss': 1.9614, 'learning_rate': 2e-05, 'epoch': 0.42} 42%|████▏ | 769/1817 [27:59:01<37:58:23, 130.44s/it] 42%|████▏ | 770/1817 [28:01:10<37:51:38, 130.18s/it] {'loss': 1.906, 'learning_rate': 2e-05, 'epoch': 0.42} 42%|████▏ | 770/1817 [28:01:10<37:51:38, 130.18s/it] 42%|████▏ | 771/1817 [28:03:18<37:35:17, 129.37s/it] {'loss': 1.915, 'learning_rate': 2e-05, 'epoch': 0.42} 42%|████▏ | 771/1817 [28:03:18<37:35:17, 129.37s/it] 42%|████▏ | 772/1817 [28:05:25<37:22:14, 128.74s/it] {'loss': 1.9583, 'learning_rate': 2e-05, 'epoch': 0.42} 42%|████▏ | 772/1817 [28:05:25<37:22:14, 128.74s/it] 43%|████▎ | 773/1817 [28:07:34<37:17:55, 128.62s/it] {'loss': 1.866, 'learning_rate': 2e-05, 'epoch': 0.43} 43%|████▎ | 773/1817 [28:07:33<37:17:55, 128.62s/it] 43%|████▎ | 774/1817 [28:09:42<37:17:29, 128.72s/it] {'loss': 1.8867, 'learning_rate': 2e-05, 'epoch': 0.43} 43%|████▎ | 774/1817 [28:09:42<37:17:29, 128.72s/it] 43%|████▎ | 775/1817 [28:11:54<37:30:06, 129.57s/it] {'loss': 1.8819, 'learning_rate': 2e-05, 'epoch': 0.43} 43%|████▎ | 775/1817 [28:11:54<37:30:02, 129.56s/it] 43%|████▎ | 776/1817 [28:14:08<37:51:06, 130.90s/it] {'loss': 1.8696, 'learning_rate': 2e-05, 'epoch': 0.43} 43%|████▎ | 776/1817 [28:14:08<37:51:06, 130.90s/it] 43%|████▎ | 777/1817 [28:16:18<37:46:23, 130.75s/it] {'loss': 1.8715, 'learning_rate': 2e-05, 'epoch': 0.43} 43%|████▎ | 777/1817 [28:16:18<37:46:23, 130.75s/it] 43%|████▎ | 778/1817 [28:18:26<37:28:03, 129.82s/it] {'loss': 1.8999, 'learning_rate': 2e-05, 'epoch': 0.43} 43%|████▎ | 778/1817 [28:18:26<37:28:03, 129.82s/it] 43%|████▎ | 779/1817 [28:20:36<37:30:19, 130.08s/it] {'loss': 1.8804, 'learning_rate': 2e-05, 'epoch': 0.43} 43%|████▎ | 779/1817 [28:20:37<37:30:19, 130.08s/it] 43%|████▎ | 780/1817 [28:22:48<37:35:28, 130.50s/it] {'loss': 1.8567, 'learning_rate': 2e-05, 'epoch': 0.43} 43%|████▎ | 780/1817 [28:22:48<37:35:28, 130.50s/it] 43%|████▎ | 781/1817 [28:25:01<37:45:59, 131.23s/it] {'loss': 1.9198, 'learning_rate': 2e-05, 'epoch': 0.43} 43%|████▎ | 781/1817 [28:25:01<37:45:58, 131.23s/it] 43%|████▎ | 782/1817 [28:27:13<37:49:35, 131.57s/it] {'loss': 1.8899, 'learning_rate': 2e-05, 'epoch': 0.43} 43%|████▎ | 782/1817 [28:27:13<37:49:35, 131.57s/it] 43%|████▎ | 783/1817 [28:29:26<37:51:40, 131.82s/it] {'loss': 1.8726, 'learning_rate': 2e-05, 'epoch': 0.43} 43%|████▎ | 783/1817 [28:29:26<37:51:42, 131.82s/it] 43%|████▎ | 784/1817 [28:31:39<37:57:44, 132.30s/it] {'loss': 1.9098, 'learning_rate': 2e-05, 'epoch': 0.43} 43%|████▎ | 784/1817 [28:31:39<37:57:46, 132.30s/it] 43%|████▎ | 785/1817 [28:33:50<37:47:17, 131.82s/it] {'loss': 1.908, 'learning_rate': 2e-05, 'epoch': 0.43} 43%|████▎ | 785/1817 [28:33:50<37:47:19, 131.82s/it] 43%|████▎ | 786/1817 [28:36:06<38:03:44, 132.90s/it] {'loss': 1.8889, 'learning_rate': 2e-05, 'epoch': 0.43} 43%|████▎ | 786/1817 [28:36:05<38:03:46, 132.91s/it] 43%|████▎ | 787/1817 [28:38:16<37:47:38, 132.10s/it] {'loss': 1.9478, 'learning_rate': 2e-05, 'epoch': 0.43} 43%|████▎ | 787/1817 [28:38:15<37:47:41, 132.10s/it] 43%|████▎ | 788/1817 [28:40:25<37:33:20, 131.39s/it] {'loss': 1.9056, 'learning_rate': 2e-05, 'epoch': 0.43} 43%|████▎ | 788/1817 [28:40:25<37:33:22, 131.39s/it] 43%|████▎ | 789/1817 [28:42:37<37:30:01, 131.32s/it] {'loss': 1.9033, 'learning_rate': 2e-05, 'epoch': 0.43} 43%|████▎ | 789/1817 [28:42:36<37:30:03, 131.33s/it] 43%|████▎ | 790/1817 [28:44:44<37:06:06, 130.05s/it] {'loss': 1.8792, 'learning_rate': 2e-05, 'epoch': 0.43} 43%|████▎ | 790/1817 [28:44:43<37:06:08, 130.06s/it] 44%|████▎ | 791/1817 [28:46:54<37:03:38, 130.04s/it] {'loss': 1.8792, 'learning_rate': 2e-05, 'epoch': 0.44} 44%|████▎ | 791/1817 [28:46:53<37:03:38, 130.04s/it] 44%|████▎ | 792/1817 [28:49:09<37:30:02, 131.71s/it] {'loss': 1.8711, 'learning_rate': 2e-05, 'epoch': 0.44} 44%|████▎ | 792/1817 [28:49:09<37:30:02, 131.71s/it] 44%|████▎ | 793/1817 [28:51:18<37:10:24, 130.69s/it] {'loss': 1.8923, 'learning_rate': 2e-05, 'epoch': 0.44} 44%|████▎ | 793/1817 [28:51:17<37:10:24, 130.69s/it] 44%|████▎ | 794/1817 [28:53:28<37:08:49, 130.72s/it] {'loss': 1.9119, 'learning_rate': 2e-05, 'epoch': 0.44} 44%|████▎ | 794/1817 [28:53:28<37:08:51, 130.72s/it] 44%|████▍ | 795/1817 [28:55:39<37:03:30, 130.54s/it] {'loss': 1.8587, 'learning_rate': 2e-05, 'epoch': 0.44} 44%|████▍ | 795/1817 [28:55:38<37:03:32, 130.54s/it] 44%|████▍ | 796/1817 [28:57:47<36:51:54, 129.99s/it] {'loss': 1.8505, 'learning_rate': 2e-05, 'epoch': 0.44} 44%|████▍ | 796/1817 [28:57:47<36:51:57, 129.99s/it] 44%|████▍ | 797/1817 [28:59:59<37:02:06, 130.71s/it] {'loss': 1.9259, 'learning_rate': 2e-05, 'epoch': 0.44} 44%|████▍ | 797/1817 [28:59:59<37:02:06, 130.71s/it] 44%|████▍ | 798/1817 [29:02:12<37:10:00, 131.31s/it] {'loss': 1.8981, 'learning_rate': 2e-05, 'epoch': 0.44} 44%|████▍ | 798/1817 [29:02:12<37:10:03, 131.31s/it] 44%|████▍ | 799/1817 [29:04:23<37:06:46, 131.24s/it] {'loss': 1.8996, 'learning_rate': 2e-05, 'epoch': 0.44} 44%|████▍ | 799/1817 [29:04:23<37:06:46, 131.24s/it] 44%|████▍ | 800/1817 [29:06:35<37:04:11, 131.22s/it] {'loss': 1.8375, 'learning_rate': 2e-05, 'epoch': 0.44} 44%|████▍ | 800/1817 [29:06:34<37:04:13, 131.22s/it] 44%|████▍ | 801/1817 [29:08:44<36:53:01, 130.69s/it] {'loss': 1.9046, 'learning_rate': 2e-05, 'epoch': 0.44} 44%|████▍ | 801/1817 [29:08:44<36:53:01, 130.69s/it] 44%|████▍ | 802/1817 [29:10:55<36:51:34, 130.73s/it] {'loss': 1.9244, 'learning_rate': 2e-05, 'epoch': 0.44} 44%|████▍ | 802/1817 [29:10:55<36:51:36, 130.74s/it] 44%|████▍ | 803/1817 [29:13:08<37:03:11, 131.55s/it] {'loss': 1.9387, 'learning_rate': 2e-05, 'epoch': 0.44} 44%|████▍ | 803/1817 [29:13:08<37:03:12, 131.55s/it] 44%|████▍ | 804/1817 [29:15:20<37:00:05, 131.50s/it] {'loss': 1.8634, 'learning_rate': 2e-05, 'epoch': 0.44} 44%|████▍ | 804/1817 [29:15:19<36:59:56, 131.49s/it] 44%|████▍ | 805/1817 [29:17:32<37:03:22, 131.82s/it] {'loss': 1.9196, 'learning_rate': 2e-05, 'epoch': 0.44} 44%|████▍ | 805/1817 [29:17:32<37:03:21, 131.82s/it] 44%|████▍ | 806/1817 [29:19:44<36:59:52, 131.74s/it] {'loss': 1.8731, 'learning_rate': 2e-05, 'epoch': 0.44} 44%|████▍ | 806/1817 [29:19:44<36:59:52, 131.74s/it] 44%|████▍ | 807/1817 [29:21:53<36:43:47, 130.92s/it] {'loss': 1.8971, 'learning_rate': 2e-05, 'epoch': 0.44} 44%|████▍ | 807/1817 [29:21:53<36:43:47, 130.92s/it] 44%|████▍ | 808/1817 [29:24:01<36:27:38, 130.09s/it] {'loss': 1.9047, 'learning_rate': 2e-05, 'epoch': 0.44} 44%|████▍ | 808/1817 [29:24:01<36:27:37, 130.09s/it] 45%|████▍ | 809/1817 [29:26:20<37:09:21, 132.70s/it] {'loss': 1.8673, 'learning_rate': 2e-05, 'epoch': 0.45} 45%|████▍ | 809/1817 [29:26:20<37:09:21, 132.70s/it] 45%|████▍ | 810/1817 [29:28:32<37:05:13, 132.59s/it] {'loss': 1.8749, 'learning_rate': 2e-05, 'epoch': 0.45} 45%|████▍ | 810/1817 [29:28:32<37:05:13, 132.59s/it] 45%|████▍ | 811/1817 [29:30:46<37:10:47, 133.05s/it] {'loss': 1.8865, 'learning_rate': 2e-05, 'epoch': 0.45} 45%|████▍ | 811/1817 [29:30:46<37:10:46, 133.05s/it] 45%|████▍ | 812/1817 [29:32:54<36:42:09, 131.47s/it] {'loss': 1.9296, 'learning_rate': 2e-05, 'epoch': 0.45} 45%|████▍ | 812/1817 [29:32:54<36:42:08, 131.47s/it] 45%|████▍ | 813/1817 [29:35:03<36:26:32, 130.67s/it] {'loss': 1.8561, 'learning_rate': 2e-05, 'epoch': 0.45} 45%|████▍ | 813/1817 [29:35:03<36:26:32, 130.67s/it] 45%|████▍ | 814/1817 [29:37:15<36:32:31, 131.16s/it] {'loss': 1.9185, 'learning_rate': 2e-05, 'epoch': 0.45} 45%|████▍ | 814/1817 [29:37:15<36:32:31, 131.16s/it] 45%|████▍ | 815/1817 [29:39:24<36:19:53, 130.53s/it] {'loss': 1.9147, 'learning_rate': 2e-05, 'epoch': 0.45} 45%|████▍ | 815/1817 [29:39:24<36:19:53, 130.53s/it] 45%|████▍ | 816/1817 [29:41:38<36:32:34, 131.42s/it] {'loss': 1.8756, 'learning_rate': 2e-05, 'epoch': 0.45} 45%|████▍ | 816/1817 [29:41:37<36:32:34, 131.42s/it] 45%|████▍ | 817/1817 [29:43:46<36:15:37, 130.54s/it] {'loss': 1.9365, 'learning_rate': 2e-05, 'epoch': 0.45} 45%|████▍ | 817/1817 [29:43:46<36:15:37, 130.54s/it] 45%|████▌ | 818/1817 [29:45:58<36:21:44, 131.04s/it] {'loss': 1.8882, 'learning_rate': 2e-05, 'epoch': 0.45} 45%|████▌ | 818/1817 [29:45:58<36:21:44, 131.04s/it] 45%|████▌ | 819/1817 [29:48:09<36:19:26, 131.03s/it] {'loss': 1.8844, 'learning_rate': 2e-05, 'epoch': 0.45} 45%|████▌ | 819/1817 [29:48:09<36:19:26, 131.03s/it] 45%|████▌ | 820/1817 [29:50:20<36:13:06, 130.78s/it] {'loss': 1.8381, 'learning_rate': 2e-05, 'epoch': 0.45} 45%|████▌ | 820/1817 [29:50:19<36:13:06, 130.78s/it] 45%|████▌ | 821/1817 [29:52:31<36:13:01, 130.90s/it] {'loss': 1.8872, 'learning_rate': 2e-05, 'epoch': 0.45} 45%|████▌ | 821/1817 [29:52:31<36:13:01, 130.90s/it] 45%|████▌ | 822/1817 [29:54:42<36:13:14, 131.05s/it] {'loss': 1.9073, 'learning_rate': 2e-05, 'epoch': 0.45} 45%|████▌ | 822/1817 [29:54:42<36:13:14, 131.05s/it] 45%|████▌ | 823/1817 [29:56:55<36:20:04, 131.59s/it] {'loss': 1.8465, 'learning_rate': 2e-05, 'epoch': 0.45} 45%|████▌ | 823/1817 [29:56:55<36:20:04, 131.59s/it] 45%|████▌ | 824/1817 [29:59:10<36:34:49, 132.62s/it] {'loss': 1.8778, 'learning_rate': 2e-05, 'epoch': 0.45} 45%|████▌ | 824/1817 [29:59:10<36:34:51, 132.62s/it] 45%|████▌ | 825/1817 [30:01:20<36:18:54, 131.79s/it] {'loss': 1.9069, 'learning_rate': 2e-05, 'epoch': 0.45} 45%|████▌ | 825/1817 [30:01:20<36:18:56, 131.79s/it] 45%|████▌ | 826/1817 [30:03:32<36:16:29, 131.78s/it] {'loss': 1.8739, 'learning_rate': 2e-05, 'epoch': 0.45} 45%|████▌ | 826/1817 [30:03:31<36:16:31, 131.78s/it] 46%|████▌ | 827/1817 [30:05:41<36:00:41, 130.95s/it] {'loss': 1.9229, 'learning_rate': 2e-05, 'epoch': 0.46} 46%|████▌ | 827/1817 [30:05:40<36:00:41, 130.95s/it] 46%|████▌ | 828/1817 [30:07:51<35:55:06, 130.74s/it] {'loss': 1.9003, 'learning_rate': 2e-05, 'epoch': 0.46} 46%|████▌ | 828/1817 [30:07:51<35:55:06, 130.74s/it] 46%|████▌ | 829/1817 [30:10:00<35:46:39, 130.36s/it] {'loss': 1.8803, 'learning_rate': 2e-05, 'epoch': 0.46} 46%|████▌ | 829/1817 [30:10:00<35:46:39, 130.36s/it] 46%|████▌ | 830/1817 [30:12:09<35:34:54, 129.78s/it] {'loss': 1.8835, 'learning_rate': 2e-05, 'epoch': 0.46} 46%|████▌ | 830/1817 [30:12:09<35:34:55, 129.78s/it] 46%|████▌ | 831/1817 [30:14:19<35:36:15, 130.00s/it] {'loss': 1.8935, 'learning_rate': 2e-05, 'epoch': 0.46} 46%|████▌ | 831/1817 [30:14:19<35:36:16, 130.00s/it] 46%|████▌ | 832/1817 [30:16:30<35:39:00, 130.30s/it] {'loss': 1.8972, 'learning_rate': 2e-05, 'epoch': 0.46} 46%|████▌ | 832/1817 [30:16:30<35:39:00, 130.30s/it] 46%|████▌ | 833/1817 [30:18:48<36:13:25, 132.53s/it] {'loss': 1.853, 'learning_rate': 2e-05, 'epoch': 0.46} 46%|████▌ | 833/1817 [30:18:48<36:13:25, 132.53s/it] 46%|████▌ | 834/1817 [30:21:03<36:23:17, 133.26s/it] {'loss': 1.8812, 'learning_rate': 2e-05, 'epoch': 0.46} 46%|████▌ | 834/1817 [30:21:03<36:23:17, 133.26s/it] 46%|████▌ | 835/1817 [30:23:15<36:15:16, 132.91s/it] {'loss': 1.9219, 'learning_rate': 2e-05, 'epoch': 0.46} 46%|████▌ | 835/1817 [30:23:15<36:15:16, 132.91s/it] 46%|████▌ | 836/1817 [30:25:29<36:15:40, 133.07s/it] {'loss': 1.8873, 'learning_rate': 2e-05, 'epoch': 0.46} 46%|████▌ | 836/1817 [30:25:28<36:15:41, 133.07s/it] 46%|████▌ | 837/1817 [30:27:42<36:15:14, 133.18s/it] {'loss': 1.8896, 'learning_rate': 2e-05, 'epoch': 0.46} 46%|████▌ | 837/1817 [30:27:42<36:15:14, 133.18s/it] 46%|████▌ | 838/1817 [30:29:56<36:19:26, 133.57s/it] {'loss': 1.8982, 'learning_rate': 2e-05, 'epoch': 0.46} 46%|████▌ | 838/1817 [30:29:56<36:19:26, 133.57s/it] 46%|████▌ | 839/1817 [30:32:05<35:53:25, 132.11s/it] {'loss': 1.8872, 'learning_rate': 2e-05, 'epoch': 0.46} 46%|████▌ | 839/1817 [30:32:05<35:53:25, 132.11s/it] 46%|████▌ | 840/1817 [30:34:19<35:58:06, 132.53s/it] {'loss': 1.9196, 'learning_rate': 2e-05, 'epoch': 0.46} 46%|████▌ | 840/1817 [30:34:18<35:58:06, 132.53s/it] 46%|████▋ | 841/1817 [30:36:30<35:50:43, 132.22s/it] {'loss': 1.8997, 'learning_rate': 2e-05, 'epoch': 0.46} 46%|████▋ | 841/1817 [30:36:30<35:50:43, 132.22s/it] 46%|████▋ | 842/1817 [30:38:41<35:43:50, 131.93s/it] {'loss': 1.8841, 'learning_rate': 2e-05, 'epoch': 0.46} 46%|████▋ | 842/1817 [30:38:41<35:43:50, 131.93s/it] 46%|████▋ | 843/1817 [30:40:51<35:29:52, 131.20s/it] {'loss': 1.8729, 'learning_rate': 2e-05, 'epoch': 0.46} 46%|████▋ | 843/1817 [30:40:51<35:29:52, 131.20s/it] 46%|████▋ | 844/1817 [30:43:06<35:44:30, 132.24s/it] {'loss': 1.925, 'learning_rate': 2e-05, 'epoch': 0.46} 46%|████▋ | 844/1817 [30:43:05<35:44:30, 132.24s/it] 47%|████▋ | 845/1817 [30:45:19<35:48:57, 132.65s/it] {'loss': 1.8937, 'learning_rate': 2e-05, 'epoch': 0.46} 47%|████▋ | 845/1817 [30:45:19<35:48:57, 132.65s/it] 47%|████▋ | 846/1817 [30:47:29<35:32:12, 131.75s/it] {'loss': 1.9137, 'learning_rate': 2e-05, 'epoch': 0.47} 47%|████▋ | 846/1817 [30:47:29<35:32:12, 131.75s/it] 47%|████▋ | 847/1817 [30:49:38<35:18:11, 131.02s/it] {'loss': 1.9146, 'learning_rate': 2e-05, 'epoch': 0.47} 47%|████▋ | 847/1817 [30:49:38<35:18:11, 131.02s/it] 47%|████▋ | 848/1817 [30:51:52<35:29:24, 131.85s/it] {'loss': 1.945, 'learning_rate': 2e-05, 'epoch': 0.47} 47%|████▋ | 848/1817 [30:51:52<35:29:24, 131.85s/it] 47%|████▋ | 849/1817 [30:54:04<35:28:11, 131.91s/it] {'loss': 1.8766, 'learning_rate': 2e-05, 'epoch': 0.47} 47%|████▋ | 849/1817 [30:54:04<35:28:11, 131.91s/it] 47%|████▋ | 850/1817 [30:56:15<35:22:34, 131.70s/it] {'loss': 1.8764, 'learning_rate': 2e-05, 'epoch': 0.47} 47%|████▋ | 850/1817 [30:56:15<35:22:34, 131.70s/it] 47%|████▋ | 851/1817 [30:58:27<35:21:47, 131.79s/it] {'loss': 1.8517, 'learning_rate': 2e-05, 'epoch': 0.47} 47%|████▋ | 851/1817 [30:58:27<35:21:48, 131.79s/it] 47%|████▋ | 852/1817 [31:00:37<35:11:12, 131.27s/it] {'loss': 1.8898, 'learning_rate': 2e-05, 'epoch': 0.47} 47%|████▋ | 852/1817 [31:00:37<35:11:12, 131.27s/it] 47%|████▋ | 853/1817 [31:02:45<34:53:57, 130.33s/it] {'loss': 1.8935, 'learning_rate': 2e-05, 'epoch': 0.47} 47%|████▋ | 853/1817 [31:02:45<34:53:57, 130.33s/it] 47%|████▋ | 854/1817 [31:04:55<34:51:03, 130.28s/it] {'loss': 1.8709, 'learning_rate': 2e-05, 'epoch': 0.47} 47%|████▋ | 854/1817 [31:04:55<34:51:03, 130.28s/it] 47%|████▋ | 855/1817 [31:07:08<34:58:38, 130.89s/it] {'loss': 1.8775, 'learning_rate': 2e-05, 'epoch': 0.47} 47%|████▋ | 855/1817 [31:07:08<34:58:38, 130.89s/it] 47%|████▋ | 856/1817 [31:09:18<34:52:23, 130.64s/it] {'loss': 1.9107, 'learning_rate': 2e-05, 'epoch': 0.47} 47%|████▋ | 856/1817 [31:09:18<34:52:23, 130.64s/it] 47%|████▋ | 857/1817 [31:11:29<34:53:45, 130.86s/it] {'loss': 1.8852, 'learning_rate': 2e-05, 'epoch': 0.47} 47%|████▋ | 857/1817 [31:11:29<34:53:45, 130.86s/it] 47%|████▋ | 858/1817 [31:13:36<34:30:23, 129.53s/it] {'loss': 1.9087, 'learning_rate': 2e-05, 'epoch': 0.47} 47%|████▋ | 858/1817 [31:13:36<34:30:23, 129.53s/it] 47%|████▋ | 859/1817 [31:15:49<34:43:31, 130.49s/it] {'loss': 1.8538, 'learning_rate': 2e-05, 'epoch': 0.47} 47%|████▋ | 859/1817 [31:15:48<34:43:28, 130.49s/it] 47%|████▋ | 860/1817 [31:18:00<34:45:52, 130.78s/it] {'loss': 1.9032, 'learning_rate': 2e-05, 'epoch': 0.47} 47%|████▋ | 860/1817 [31:18:00<34:45:52, 130.78s/it] 47%|████▋ | 861/1817 [31:20:10<34:43:35, 130.77s/it] {'loss': 1.8919, 'learning_rate': 2e-05, 'epoch': 0.47} 47%|████▋ | 861/1817 [31:20:10<34:43:35, 130.77s/it] 47%|████▋ | 862/1817 [31:22:21<34:36:55, 130.49s/it] {'loss': 1.8806, 'learning_rate': 2e-05, 'epoch': 0.47} 47%|████▋ | 862/1817 [31:22:20<34:36:56, 130.49s/it] 47%|████▋ | 863/1817 [31:24:33<34:46:10, 131.21s/it] {'loss': 1.8857, 'learning_rate': 2e-05, 'epoch': 0.47} 47%|████▋ | 863/1817 [31:24:33<34:46:10, 131.21s/it] 48%|████▊ | 864/1817 [31:26:43<34:35:55, 130.70s/it] {'loss': 1.8794, 'learning_rate': 2e-05, 'epoch': 0.48} 48%|████▊ | 864/1817 [31:26:43<34:35:56, 130.70s/it] 48%|████▊ | 865/1817 [31:28:57<34:51:16, 131.80s/it] {'loss': 1.8765, 'learning_rate': 2e-05, 'epoch': 0.48} 48%|████▊ | 865/1817 [31:28:57<34:51:16, 131.80s/it] 48%|████▊ | 866/1817 [31:31:10<34:53:28, 132.08s/it] {'loss': 1.927, 'learning_rate': 2e-05, 'epoch': 0.48} 48%|████▊ | 866/1817 [31:31:10<34:53:28, 132.08s/it] 48%|████▊ | 867/1817 [31:33:20<34:41:28, 131.46s/it] {'loss': 1.909, 'learning_rate': 2e-05, 'epoch': 0.48} 48%|████▊ | 867/1817 [31:33:20<34:41:29, 131.46s/it] 48%|████▊ | 868/1817 [31:35:30<34:31:48, 130.99s/it] {'loss': 1.8908, 'learning_rate': 2e-05, 'epoch': 0.48} 48%|████▊ | 868/1817 [31:35:30<34:31:48, 130.99s/it] 48%|████▊ | 869/1817 [31:37:39<34:19:00, 130.32s/it] {'loss': 1.9074, 'learning_rate': 2e-05, 'epoch': 0.48} 48%|████▊ | 869/1817 [31:37:38<34:19:01, 130.32s/it] 48%|████▊ | 870/1817 [31:39:48<34:11:28, 129.98s/it] {'loss': 1.8701, 'learning_rate': 2e-05, 'epoch': 0.48} 48%|████▊ | 870/1817 [31:39:48<34:11:29, 129.98s/it] 48%|████▊ | 871/1817 [31:42:00<34:20:32, 130.69s/it] {'loss': 1.8974, 'learning_rate': 2e-05, 'epoch': 0.48} 48%|████▊ | 871/1817 [31:42:00<34:20:32, 130.69s/it] 48%|████▊ | 872/1817 [31:44:10<34:13:12, 130.36s/it] {'loss': 1.8787, 'learning_rate': 2e-05, 'epoch': 0.48} 48%|████▊ | 872/1817 [31:44:10<34:13:12, 130.36s/it] 48%|████▊ | 873/1817 [31:46:22<34:20:20, 130.95s/it] {'loss': 1.86, 'learning_rate': 2e-05, 'epoch': 0.48} 48%|████▊ | 873/1817 [31:46:22<34:20:20, 130.95s/it] 48%|████▊ | 874/1817 [31:48:30<34:04:35, 130.09s/it] {'loss': 1.9104, 'learning_rate': 2e-05, 'epoch': 0.48} 48%|████▊ | 874/1817 [31:48:30<34:04:35, 130.09s/it] 48%|████▊ | 875/1817 [31:50:41<34:06:55, 130.38s/it] {'loss': 1.8769, 'learning_rate': 2e-05, 'epoch': 0.48} 48%|████▊ | 875/1817 [31:50:41<34:06:55, 130.38s/it] 48%|████▊ | 876/1817 [31:52:53<34:10:49, 130.76s/it] {'loss': 1.8727, 'learning_rate': 2e-05, 'epoch': 0.48} 48%|████▊ | 876/1817 [31:52:53<34:10:51, 130.77s/it] 48%|████▊ | 877/1817 [31:55:07<34:23:05, 131.69s/it] {'loss': 1.9316, 'learning_rate': 2e-05, 'epoch': 0.48} 48%|████▊ | 877/1817 [31:55:07<34:23:05, 131.69s/it] 48%|████▊ | 878/1817 [31:57:19<34:23:40, 131.86s/it] {'loss': 1.8887, 'learning_rate': 2e-05, 'epoch': 0.48} 48%|████▊ | 878/1817 [31:57:19<34:23:40, 131.86s/it] 48%|████▊ | 879/1817 [31:59:31<34:21:34, 131.87s/it] {'loss': 1.9023, 'learning_rate': 2e-05, 'epoch': 0.48} 48%|████▊ | 879/1817 [31:59:31<34:21:34, 131.87s/it] 48%|████▊ | 880/1817 [32:01:44<34:26:27, 132.32s/it] {'loss': 1.8771, 'learning_rate': 2e-05, 'epoch': 0.48} 48%|████▊ | 880/1817 [32:01:44<34:26:27, 132.32s/it] 48%|████▊ | 881/1817 [32:03:55<34:17:45, 131.91s/it] {'loss': 1.9207, 'learning_rate': 2e-05, 'epoch': 0.48} 48%|████▊ | 881/1817 [32:03:55<34:17:45, 131.91s/it] 49%|████▊ | 882/1817 [32:06:04<33:59:08, 130.85s/it] {'loss': 1.8718, 'learning_rate': 2e-05, 'epoch': 0.49} 49%|████▊ | 882/1817 [32:06:03<33:59:08, 130.85s/it] 49%|████▊ | 883/1817 [32:08:17<34:07:47, 131.55s/it] {'loss': 1.8992, 'learning_rate': 2e-05, 'epoch': 0.49} 49%|████▊ | 883/1817 [32:08:17<34:07:47, 131.55s/it] 49%|████▊ | 884/1817 [32:10:29<34:06:12, 131.59s/it] {'loss': 1.8982, 'learning_rate': 2e-05, 'epoch': 0.49} 49%|████▊ | 884/1817 [32:10:28<34:06:09, 131.59s/it] 49%|████▊ | 885/1817 [32:12:38<33:53:48, 130.93s/it] {'loss': 1.8542, 'learning_rate': 2e-05, 'epoch': 0.49} 49%|████▊ | 885/1817 [32:12:38<33:53:48, 130.93s/it] 49%|████▉ | 886/1817 [32:14:45<33:35:14, 129.88s/it] {'loss': 1.9203, 'learning_rate': 2e-05, 'epoch': 0.49} 49%|████▉ | 886/1817 [32:14:45<33:35:11, 129.87s/it] 49%|████▉ | 887/1817 [32:16:55<33:34:19, 129.96s/it] {'loss': 1.8858, 'learning_rate': 2e-05, 'epoch': 0.49} 49%|████▉ | 887/1817 [32:16:55<33:34:19, 129.96s/it] 49%|████▉ | 888/1817 [32:19:05<33:30:24, 129.84s/it] {'loss': 1.89, 'learning_rate': 2e-05, 'epoch': 0.49} 49%|████▉ | 888/1817 [32:19:05<33:30:24, 129.84s/it] 49%|████▉ | 889/1817 [32:21:15<33:30:29, 129.99s/it] {'loss': 1.8728, 'learning_rate': 2e-05, 'epoch': 0.49} 49%|████▉ | 889/1817 [32:21:15<33:30:29, 129.99s/it] 49%|████▉ | 890/1817 [32:23:24<33:23:30, 129.68s/it] {'loss': 1.8699, 'learning_rate': 2e-05, 'epoch': 0.49} 49%|████▉ | 890/1817 [32:23:24<33:23:30, 129.68s/it] 49%|████▉ | 891/1817 [32:25:36<33:31:02, 130.31s/it] {'loss': 1.8965, 'learning_rate': 2e-05, 'epoch': 0.49} 49%|████▉ | 891/1817 [32:25:36<33:31:02, 130.31s/it] 49%|████▉ | 892/1817 [32:27:44<33:18:50, 129.65s/it] {'loss': 1.9073, 'learning_rate': 2e-05, 'epoch': 0.49} 49%|████▉ | 892/1817 [32:27:44<33:18:50, 129.65s/it] 49%|████▉ | 893/1817 [32:29:52<33:07:47, 129.08s/it] {'loss': 1.8919, 'learning_rate': 2e-05, 'epoch': 0.49} 49%|████▉ | 893/1817 [32:29:52<33:07:47, 129.08s/it] 49%|████▉ | 894/1817 [32:32:06<33:30:39, 130.70s/it] {'loss': 1.8608, 'learning_rate': 2e-05, 'epoch': 0.49} 49%|████▉ | 894/1817 [32:32:06<33:30:39, 130.70s/it] 49%|████▉ | 895/1817 [32:34:17<33:30:14, 130.82s/it] {'loss': 1.8717, 'learning_rate': 2e-05, 'epoch': 0.49} 49%|████▉ | 895/1817 [32:34:17<33:30:14, 130.82s/it] 49%|████▉ | 896/1817 [32:36:33<33:49:27, 132.21s/it] {'loss': 1.9547, 'learning_rate': 2e-05, 'epoch': 0.49} 49%|████▉ | 896/1817 [32:36:33<33:49:29, 132.21s/it] 49%|████▉ | 897/1817 [32:38:47<33:54:59, 132.72s/it] {'loss': 1.9197, 'learning_rate': 2e-05, 'epoch': 0.49} 49%|████▉ | 897/1817 [32:38:47<33:54:59, 132.72s/it] 49%|████▉ | 898/1817 [32:41:02<34:05:32, 133.55s/it] {'loss': 1.847, 'learning_rate': 2e-05, 'epoch': 0.49} 49%|████▉ | 898/1817 [32:41:02<34:05:32, 133.55s/it] 49%|████▉ | 899/1817 [32:43:17<34:06:51, 133.78s/it] {'loss': 1.8411, 'learning_rate': 2e-05, 'epoch': 0.49} 49%|████▉ | 899/1817 [32:43:17<34:06:52, 133.78s/it] 50%|████▉ | 900/1817 [32:45:29<33:57:56, 133.34s/it] {'loss': 1.8816, 'learning_rate': 2e-05, 'epoch': 0.5} 50%|████▉ | 900/1817 [32:45:29<33:57:56, 133.34s/it] 50%|████▉ | 901/1817 [32:47:36<33:27:07, 131.47s/it] {'loss': 1.912, 'learning_rate': 2e-05, 'epoch': 0.5} 50%|████▉ | 901/1817 [32:47:36<33:27:09, 131.47s/it] 50%|████▉ | 902/1817 [32:49:52<33:46:13, 132.87s/it] {'loss': 1.8565, 'learning_rate': 2e-05, 'epoch': 0.5} 50%|████▉ | 902/1817 [32:49:52<33:46:13, 132.87s/it] 50%|████▉ | 903/1817 [32:52:04<33:37:31, 132.44s/it] {'loss': 1.881, 'learning_rate': 2e-05, 'epoch': 0.5} 50%|████▉ | 903/1817 [32:52:04<33:37:31, 132.44s/it] 50%|████▉ | 904/1817 [32:54:16<33:35:44, 132.47s/it] {'loss': 1.8722, 'learning_rate': 2e-05, 'epoch': 0.5} 50%|████▉ | 904/1817 [32:54:16<33:35:44, 132.47s/it] 50%|████▉ | 905/1817 [32:56:24<33:14:00, 131.19s/it] {'loss': 1.8967, 'learning_rate': 2e-05, 'epoch': 0.5} 50%|████▉ | 905/1817 [32:56:24<33:14:00, 131.19s/it] 50%|████▉ | 906/1817 [32:58:36<33:11:38, 131.17s/it] {'loss': 1.9057, 'learning_rate': 2e-05, 'epoch': 0.5} 50%|████▉ | 906/1817 [32:58:35<33:11:39, 131.17s/it] 50%|████▉ | 907/1817 [33:00:52<33:33:24, 132.75s/it] {'loss': 1.8811, 'learning_rate': 2e-05, 'epoch': 0.5} 50%|████▉ | 907/1817 [33:00:52<33:33:24, 132.75s/it] 50%|████▉ | 908/1817 [33:03:01<33:15:45, 131.73s/it] {'loss': 1.8755, 'learning_rate': 2e-05, 'epoch': 0.5} 50%|████▉ | 908/1817 [33:03:01<33:15:47, 131.74s/it] 50%|█████ | 909/1817 [33:05:13<33:12:20, 131.65s/it] {'loss': 1.8492, 'learning_rate': 2e-05, 'epoch': 0.5} 50%|█████ | 909/1817 [33:05:13<33:12:21, 131.65s/it] 50%|█████ | 910/1817 [33:07:25<33:10:50, 131.70s/it] {'loss': 1.8842, 'learning_rate': 2e-05, 'epoch': 0.5} 50%|█████ | 910/1817 [33:07:24<33:10:50, 131.70s/it] 50%|█████ | 911/1817 [33:09:34<32:57:55, 130.99s/it] {'loss': 1.883, 'learning_rate': 2e-05, 'epoch': 0.5} 50%|█████ | 911/1817 [33:09:34<32:57:55, 130.99s/it] 50%|█████ | 912/1817 [33:11:47<33:06:59, 131.73s/it] {'loss': 1.9037, 'learning_rate': 2e-05, 'epoch': 0.5} 50%|█████ | 912/1817 [33:11:47<33:06:59, 131.73s/it] 50%|█████ | 913/1817 [33:13:56<32:51:35, 130.86s/it] {'loss': 1.92, 'learning_rate': 2e-05, 'epoch': 0.5} 50%|█████ | 913/1817 [33:13:56<32:51:35, 130.86s/it] 50%|█████ | 914/1817 [33:16:04<32:37:16, 130.05s/it] {'loss': 1.8821, 'learning_rate': 2e-05, 'epoch': 0.5} 50%|█████ | 914/1817 [33:16:04<32:37:16, 130.05s/it] 50%|█████ | 915/1817 [33:18:15<32:35:18, 130.06s/it] {'loss': 1.8586, 'learning_rate': 2e-05, 'epoch': 0.5} 50%|█████ | 915/1817 [33:18:14<32:35:18, 130.06s/it] 50%|█████ | 916/1817 [33:20:27<32:43:31, 130.76s/it] {'loss': 1.9153, 'learning_rate': 2e-05, 'epoch': 0.5} 50%|█████ | 916/1817 [33:20:27<32:43:31, 130.76s/it] 50%|█████ | 917/1817 [33:22:43<33:05:18, 132.35s/it] {'loss': 1.8717, 'learning_rate': 2e-05, 'epoch': 0.5} 50%|█████ | 917/1817 [33:22:43<33:05:18, 132.35s/it] 51%|█████ | 918/1817 [33:24:52<32:46:44, 131.26s/it] {'loss': 1.8793, 'learning_rate': 2e-05, 'epoch': 0.51} 51%|█████ | 918/1817 [33:24:51<32:46:44, 131.26s/it] 51%|█████ | 919/1817 [33:27:03<32:45:13, 131.31s/it] {'loss': 1.8873, 'learning_rate': 2e-05, 'epoch': 0.51} 51%|█████ | 919/1817 [33:27:03<32:45:13, 131.31s/it] 51%|█████ | 920/1817 [33:29:18<33:00:53, 132.50s/it] {'loss': 1.9426, 'learning_rate': 2e-05, 'epoch': 0.51} 51%|█████ | 920/1817 [33:29:18<33:00:54, 132.50s/it] 51%|█████ | 921/1817 [33:31:28<32:47:49, 131.77s/it] {'loss': 1.9002, 'learning_rate': 2e-05, 'epoch': 0.51} 51%|█████ | 921/1817 [33:31:28<32:47:50, 131.77s/it] 51%|█████ | 922/1817 [33:33:47<33:18:39, 133.99s/it] {'loss': 1.8992, 'learning_rate': 2e-05, 'epoch': 0.51} 51%|█████ | 922/1817 [33:33:47<33:18:39, 133.99s/it] 51%|█████ | 923/1817 [33:35:55<32:47:25, 132.04s/it] {'loss': 1.8753, 'learning_rate': 2e-05, 'epoch': 0.51} 51%|█████ | 923/1817 [33:35:55<32:47:25, 132.04s/it] 51%|█████ | 924/1817 [33:38:08<32:48:45, 132.28s/it] {'loss': 1.8934, 'learning_rate': 2e-05, 'epoch': 0.51} 51%|█████ | 924/1817 [33:38:08<32:48:45, 132.28s/it] 51%|█████ | 925/1817 [33:40:18<32:37:20, 131.66s/it] {'loss': 1.9158, 'learning_rate': 2e-05, 'epoch': 0.51} 51%|█████ | 925/1817 [33:40:18<32:37:20, 131.66s/it] 51%|█████ | 926/1817 [33:42:32<32:43:53, 132.25s/it] {'loss': 1.8315, 'learning_rate': 2e-05, 'epoch': 0.51} 51%|█████ | 926/1817 [33:42:32<32:43:54, 132.25s/it] 51%|█████ | 927/1817 [33:44:43<32:40:04, 132.14s/it] {'loss': 1.9026, 'learning_rate': 2e-05, 'epoch': 0.51} 51%|█████ | 927/1817 [33:44:43<32:40:04, 132.14s/it] 51%|█████ | 928/1817 [33:46:54<32:31:31, 131.71s/it] {'loss': 1.8585, 'learning_rate': 2e-05, 'epoch': 0.51} 51%|█████ | 928/1817 [33:46:54<32:31:31, 131.71s/it] 51%|█████ | 929/1817 [33:49:07<32:35:28, 132.13s/it] {'loss': 1.8862, 'learning_rate': 2e-05, 'epoch': 0.51} 51%|█████ | 929/1817 [33:49:07<32:35:28, 132.13s/it] 51%|█████ | 930/1817 [33:51:19<32:28:36, 131.81s/it] {'loss': 1.9132, 'learning_rate': 2e-05, 'epoch': 0.51} 51%|█████ | 930/1817 [33:51:18<32:28:37, 131.81s/it] 51%|█████ | 931/1817 [33:53:31<32:30:41, 132.10s/it] {'loss': 1.8828, 'learning_rate': 2e-05, 'epoch': 0.51} 51%|█████ | 931/1817 [33:53:31<32:30:41, 132.10s/it] 51%|█████▏ | 932/1817 [33:55:41<32:16:09, 131.26s/it] {'loss': 1.9186, 'learning_rate': 2e-05, 'epoch': 0.51} 51%|█████▏ | 932/1817 [33:55:40<32:16:09, 131.27s/it] 51%|█████▏ | 933/1817 [33:57:51<32:11:02, 131.07s/it] {'loss': 1.9085, 'learning_rate': 2e-05, 'epoch': 0.51} 51%|█████▏ | 933/1817 [33:57:51<32:11:02, 131.07s/it] 51%|█████▏ | 934/1817 [34:00:04<32:17:41, 131.67s/it] {'loss': 1.8993, 'learning_rate': 2e-05, 'epoch': 0.51} 51%|█████▏ | 934/1817 [34:00:04<32:17:42, 131.67s/it] 51%|█████▏ | 935/1817 [34:02:14<32:06:27, 131.05s/it] {'loss': 1.9558, 'learning_rate': 2e-05, 'epoch': 0.51} 51%|█████▏ | 935/1817 [34:02:14<32:06:27, 131.05s/it] 52%|█████▏ | 936/1817 [34:04:24<31:59:20, 130.72s/it] {'loss': 1.882, 'learning_rate': 2e-05, 'epoch': 0.52} 52%|█████▏ | 936/1817 [34:04:24<31:59:21, 130.72s/it] 52%|█████▏ | 937/1817 [34:06:35<31:58:28, 130.80s/it] {'loss': 1.8883, 'learning_rate': 2e-05, 'epoch': 0.52} 52%|█████▏ | 937/1817 [34:06:35<31:58:28, 130.80s/it] 52%|█████▏ | 938/1817 [34:08:46<31:56:06, 130.79s/it] {'loss': 1.9215, 'learning_rate': 2e-05, 'epoch': 0.52} 52%|█████▏ | 938/1817 [34:08:45<31:56:06, 130.79s/it] 52%|█████▏ | 939/1817 [34:10:55<31:47:31, 130.35s/it] {'loss': 1.9307, 'learning_rate': 2e-05, 'epoch': 0.52} 52%|█████▏ | 939/1817 [34:10:55<31:47:27, 130.35s/it] 52%|█████▏ | 940/1817 [34:13:02<31:29:53, 129.30s/it] {'loss': 1.9172, 'learning_rate': 2e-05, 'epoch': 0.52} 52%|█████▏ | 940/1817 [34:13:02<31:29:50, 129.29s/it] 52%|█████▏ | 941/1817 [34:15:16<31:48:23, 130.71s/it] {'loss': 1.9105, 'learning_rate': 2e-05, 'epoch': 0.52} 52%|█████▏ | 941/1817 [34:15:16<31:48:23, 130.71s/it] 52%|█████▏ | 942/1817 [34:17:32<32:12:05, 132.49s/it] {'loss': 1.9054, 'learning_rate': 2e-05, 'epoch': 0.52} 52%|█████▏ | 942/1817 [34:17:32<32:12:05, 132.49s/it] 52%|█████▏ | 943/1817 [34:19:44<32:06:28, 132.25s/it] {'loss': 1.9253, 'learning_rate': 2e-05, 'epoch': 0.52} 52%|█████▏ | 943/1817 [34:19:44<32:06:28, 132.25s/it] 52%|█████▏ | 944/1817 [34:21:55<31:58:34, 131.86s/it] {'loss': 1.915, 'learning_rate': 2e-05, 'epoch': 0.52} 52%|█████▏ | 944/1817 [34:21:55<31:58:34, 131.86s/it] 52%|█████▏ | 945/1817 [34:24:06<31:51:26, 131.52s/it] {'loss': 1.8788, 'learning_rate': 2e-05, 'epoch': 0.52} 52%|█████▏ | 945/1817 [34:24:06<31:51:26, 131.52s/it] 52%|█████▏ | 946/1817 [34:26:14<31:35:25, 130.57s/it] {'loss': 1.9096, 'learning_rate': 2e-05, 'epoch': 0.52} 52%|█████▏ | 946/1817 [34:26:14<31:35:25, 130.57s/it] 52%|█████▏ | 947/1817 [34:28:23<31:26:38, 130.11s/it] {'loss': 1.8688, 'learning_rate': 2e-05, 'epoch': 0.52} 52%|█████▏ | 947/1817 [34:28:23<31:26:38, 130.11s/it] 52%|█████▏ | 948/1817 [34:30:34<31:27:31, 130.32s/it] {'loss': 1.8682, 'learning_rate': 2e-05, 'epoch': 0.52} 52%|█████▏ | 948/1817 [34:30:34<31:27:31, 130.32s/it] 52%|█████▏ | 949/1817 [34:32:44<31:24:28, 130.26s/it] {'loss': 1.8611, 'learning_rate': 2e-05, 'epoch': 0.52} 52%|█████▏ | 949/1817 [34:32:44<31:24:28, 130.26s/it] 52%|█████▏ | 950/1817 [34:34:57<31:35:38, 131.19s/it] {'loss': 1.8843, 'learning_rate': 2e-05, 'epoch': 0.52} 52%|█████▏ | 950/1817 [34:34:58<31:35:37, 131.19s/it] 52%|█████▏ | 951/1817 [34:37:09<31:35:30, 131.33s/it] {'loss': 1.9371, 'learning_rate': 2e-05, 'epoch': 0.52} 52%|█████▏ | 951/1817 [34:37:09<31:35:30, 131.33s/it] 52%|█████▏ | 952/1817 [34:39:20<31:31:13, 131.18s/it] {'loss': 1.884, 'learning_rate': 2e-05, 'epoch': 0.52} 52%|█████▏ | 952/1817 [34:39:20<31:31:13, 131.18s/it] 52%|█████▏ | 953/1817 [34:41:29<31:20:07, 130.56s/it] {'loss': 1.8896, 'learning_rate': 2e-05, 'epoch': 0.52} 52%|█████▏ | 953/1817 [34:41:29<31:20:07, 130.56s/it] 53%|█████▎ | 954/1817 [34:43:45<31:42:42, 132.29s/it] {'loss': 1.915, 'learning_rate': 2e-05, 'epoch': 0.52} 53%|█████▎ | 954/1817 [34:43:45<31:42:42, 132.29s/it] 53%|█████▎ | 955/1817 [34:45:52<31:16:55, 130.64s/it] {'loss': 1.8892, 'learning_rate': 2e-05, 'epoch': 0.53} 53%|█████▎ | 955/1817 [34:45:52<31:16:55, 130.64s/it] 53%|█████▎ | 956/1817 [34:48:03<31:16:07, 130.74s/it] {'loss': 1.8942, 'learning_rate': 2e-05, 'epoch': 0.53} 53%|█████▎ | 956/1817 [34:48:03<31:16:07, 130.74s/it] 53%|█████▎ | 957/1817 [34:50:13<31:09:02, 130.40s/it] {'loss': 1.9085, 'learning_rate': 2e-05, 'epoch': 0.53} 53%|█████▎ | 957/1817 [34:50:13<31:09:02, 130.40s/it] 53%|█████▎ | 958/1817 [34:52:25<31:14:12, 130.91s/it] {'loss': 1.9041, 'learning_rate': 2e-05, 'epoch': 0.53} 53%|█████▎ | 958/1817 [34:52:25<31:14:12, 130.91s/it] 53%|█████▎ | 959/1817 [34:54:36<31:15:10, 131.13s/it] {'loss': 1.9204, 'learning_rate': 2e-05, 'epoch': 0.53} 53%|█████▎ | 959/1817 [34:54:36<31:15:10, 131.13s/it] 53%|█████▎ | 960/1817 [34:56:48<31:12:58, 131.13s/it] {'loss': 1.897, 'learning_rate': 2e-05, 'epoch': 0.53} 53%|█████▎ | 960/1817 [34:56:47<31:12:59, 131.13s/it] 53%|█████▎ | 961/1817 [34:58:57<31:04:50, 130.71s/it] {'loss': 1.912, 'learning_rate': 2e-05, 'epoch': 0.53} 53%|█████▎ | 961/1817 [34:58:57<31:04:50, 130.71s/it] 53%|█████▎ | 962/1817 [35:01:09<31:08:18, 131.11s/it] {'loss': 1.9181, 'learning_rate': 2e-05, 'epoch': 0.53} 53%|█████▎ | 962/1817 [35:01:09<31:08:18, 131.11s/it] 53%|█████▎ | 963/1817 [35:03:23<31:18:44, 132.00s/it] {'loss': 1.8813, 'learning_rate': 2e-05, 'epoch': 0.53} 53%|█████▎ | 963/1817 [35:03:23<31:18:44, 132.00s/it] 53%|█████▎ | 964/1817 [35:05:34<31:10:51, 131.60s/it] {'loss': 1.9225, 'learning_rate': 2e-05, 'epoch': 0.53} 53%|█████▎ | 964/1817 [35:05:34<31:10:51, 131.60s/it] 53%|█████▎ | 965/1817 [35:07:44<31:03:17, 131.22s/it] {'loss': 1.916, 'learning_rate': 2e-05, 'epoch': 0.53} 53%|█████▎ | 965/1817 [35:07:44<31:03:17, 131.22s/it] 53%|█████▎ | 966/1817 [35:09:56<31:04:32, 131.46s/it] {'loss': 1.8758, 'learning_rate': 2e-05, 'epoch': 0.53} 53%|█████▎ | 966/1817 [35:09:56<31:04:32, 131.46s/it] 53%|█████▎ | 967/1817 [35:12:06<30:53:27, 130.83s/it] {'loss': 1.8892, 'learning_rate': 2e-05, 'epoch': 0.53} 53%|█████▎ | 967/1817 [35:12:06<30:53:23, 130.83s/it] 53%|█████▎ | 968/1817 [35:14:15<30:45:17, 130.41s/it] {'loss': 1.8656, 'learning_rate': 2e-05, 'epoch': 0.53} 53%|█████▎ | 968/1817 [35:14:15<30:45:17, 130.41s/it] 53%|█████▎ | 969/1817 [35:16:26<30:42:50, 130.39s/it] {'loss': 1.9366, 'learning_rate': 2e-05, 'epoch': 0.53} 53%|█████▎ | 969/1817 [35:16:25<30:42:50, 130.39s/it] 53%|█████▎ | 970/1817 [35:18:35<30:34:39, 129.96s/it] {'loss': 1.8915, 'learning_rate': 2e-05, 'epoch': 0.53} 53%|█████▎ | 970/1817 [35:18:34<30:34:38, 129.96s/it] 53%|█████▎ | 971/1817 [35:20:42<30:19:21, 129.03s/it] {'loss': 1.9064, 'learning_rate': 2e-05, 'epoch': 0.53} 53%|█████▎ | 971/1817 [35:20:41<30:19:21, 129.03s/it] 53%|█████▎ | 972/1817 [35:22:50<30:16:16, 128.97s/it] {'loss': 1.9157, 'learning_rate': 2e-05, 'epoch': 0.53} 53%|█████▎ | 972/1817 [35:22:50<30:16:16, 128.97s/it] 54%|█████▎ | 973/1817 [35:25:04<30:33:44, 130.36s/it] {'loss': 1.9197, 'learning_rate': 2e-05, 'epoch': 0.54} 54%|█████▎ | 973/1817 [35:25:04<30:33:43, 130.36s/it] 54%|█████▎ | 974/1817 [35:27:15<30:35:39, 130.65s/it] {'loss': 1.8884, 'learning_rate': 2e-05, 'epoch': 0.54} 54%|█████▎ | 974/1817 [35:27:15<30:35:39, 130.65s/it] 54%|█████▎ | 975/1817 [35:29:22<30:16:08, 129.42s/it] {'loss': 1.8942, 'learning_rate': 2e-05, 'epoch': 0.54} 54%|█████▎ | 975/1817 [35:29:22<30:16:10, 129.42s/it] 54%|█████▎ | 976/1817 [35:31:31<30:11:55, 129.27s/it] {'loss': 1.9127, 'learning_rate': 2e-05, 'epoch': 0.54} 54%|█████▎ | 976/1817 [35:31:31<30:11:55, 129.27s/it] 54%|█████▍ | 977/1817 [35:33:39<30:07:50, 129.13s/it] {'loss': 1.8907, 'learning_rate': 2e-05, 'epoch': 0.54} 54%|█████▍ | 977/1817 [35:33:39<30:07:50, 129.13s/it] 54%|█████▍ | 978/1817 [35:35:47<29:59:27, 128.69s/it] {'loss': 1.894, 'learning_rate': 2e-05, 'epoch': 0.54} 54%|█████▍ | 978/1817 [35:35:47<29:59:27, 128.69s/it] 54%|█████▍ | 979/1817 [35:37:58<30:07:40, 129.43s/it] {'loss': 1.9029, 'learning_rate': 2e-05, 'epoch': 0.54} 54%|█████▍ | 979/1817 [35:37:58<30:07:42, 129.43s/it] 54%|█████▍ | 980/1817 [35:40:11<30:20:01, 130.47s/it] {'loss': 1.8955, 'learning_rate': 2e-05, 'epoch': 0.54} 54%|█████▍ | 980/1817 [35:40:11<30:20:01, 130.47s/it] 54%|█████▍ | 981/1817 [35:42:26<30:36:13, 131.79s/it] {'loss': 1.8608, 'learning_rate': 2e-05, 'epoch': 0.54} 54%|█████▍ | 981/1817 [35:42:26<30:36:15, 131.79s/it] 54%|█████▍ | 982/1817 [35:44:36<30:24:23, 131.09s/it] {'loss': 1.8831, 'learning_rate': 2e-05, 'epoch': 0.54} 54%|█████▍ | 982/1817 [35:44:35<30:24:25, 131.10s/it] 54%|█████▍ | 983/1817 [35:46:51<30:40:33, 132.41s/it] {'loss': 1.8903, 'learning_rate': 2e-05, 'epoch': 0.54} 54%|█████▍ | 983/1817 [35:46:51<30:40:35, 132.42s/it] 54%|█████▍ | 984/1817 [35:48:59<30:18:23, 130.98s/it] {'loss': 1.915, 'learning_rate': 2e-05, 'epoch': 0.54} 54%|█████▍ | 984/1817 [35:48:59<30:18:25, 130.98s/it] 54%|█████▍ | 985/1817 [35:51:09<30:12:28, 130.71s/it] {'loss': 1.8776, 'learning_rate': 2e-05, 'epoch': 0.54} 54%|█████▍ | 985/1817 [35:51:09<30:12:29, 130.71s/it] 54%|█████▍ | 986/1817 [35:53:22<30:21:48, 131.54s/it] {'loss': 1.9088, 'learning_rate': 2e-05, 'epoch': 0.54} 54%|█████▍ | 986/1817 [35:53:22<30:21:47, 131.54s/it] 54%|█████▍ | 987/1817 [35:55:35<30:24:20, 131.88s/it] {'loss': 1.8839, 'learning_rate': 2e-05, 'epoch': 0.54} 54%|█████▍ | 987/1817 [35:55:35<30:24:22, 131.88s/it] 54%|█████▍ | 988/1817 [35:57:47<30:21:48, 131.86s/it] {'loss': 1.909, 'learning_rate': 2e-05, 'epoch': 0.54} 54%|█████▍ | 988/1817 [35:57:47<30:21:50, 131.86s/it] 54%|█████▍ | 989/1817 [35:59:58<30:17:53, 131.73s/it] {'loss': 1.8803, 'learning_rate': 2e-05, 'epoch': 0.54} 54%|█████▍ | 989/1817 [35:59:58<30:17:53, 131.73s/it] 54%|█████▍ | 990/1817 [36:02:10<30:13:49, 131.60s/it] {'loss': 1.8862, 'learning_rate': 2e-05, 'epoch': 0.54} 54%|█████▍ | 990/1817 [36:02:09<30:13:51, 131.60s/it] 55%|█████▍ | 991/1817 [36:04:23<30:18:50, 132.12s/it] {'loss': 1.896, 'learning_rate': 2e-05, 'epoch': 0.55} 55%|█████▍ | 991/1817 [36:04:23<30:18:50, 132.12s/it] 55%|█████▍ | 992/1817 [36:06:35<30:16:24, 132.10s/it] {'loss': 1.9077, 'learning_rate': 2e-05, 'epoch': 0.55} 55%|█████▍ | 992/1817 [36:06:35<30:16:24, 132.10s/it] 55%|█████▍ | 993/1817 [36:08:46<30:08:39, 131.70s/it] {'loss': 1.9186, 'learning_rate': 2e-05, 'epoch': 0.55} 55%|█████▍ | 993/1817 [36:08:45<30:08:39, 131.70s/it] 55%|█████▍ | 994/1817 [36:10:57<30:04:30, 131.56s/it] {'loss': 1.8761, 'learning_rate': 2e-05, 'epoch': 0.55} 55%|█████▍ | 994/1817 [36:10:57<30:04:31, 131.56s/it] 55%|█████▍ | 995/1817 [36:13:10<30:07:45, 131.95s/it] {'loss': 1.9032, 'learning_rate': 2e-05, 'epoch': 0.55} 55%|█████▍ | 995/1817 [36:13:10<30:07:45, 131.95s/it] 55%|█████▍ | 996/1817 [36:15:19<29:55:56, 131.25s/it] {'loss': 1.8955, 'learning_rate': 2e-05, 'epoch': 0.55} 55%|█████▍ | 996/1817 [36:15:19<29:55:56, 131.25s/it] 55%|█████▍ | 997/1817 [36:17:31<29:56:11, 131.43s/it] {'loss': 1.8965, 'learning_rate': 2e-05, 'epoch': 0.55} 55%|█████▍ | 997/1817 [36:17:31<29:56:10, 131.43s/it] 55%|█████▍ | 998/1817 [36:19:38<29:34:39, 130.01s/it] {'loss': 1.8675, 'learning_rate': 2e-05, 'epoch': 0.55} 55%|█████▍ | 998/1817 [36:19:38<29:34:38, 130.01s/it] 55%|█████▍ | 999/1817 [36:21:52<29:48:23, 131.18s/it] {'loss': 1.8615, 'learning_rate': 2e-05, 'epoch': 0.55} 55%|█████▍ | 999/1817 [36:21:52<29:48:23, 131.18s/it][2024-01-31 00:09:40,385] [INFO] [logging.py:96:log_dist] [Rank 0] step [2024-01-3 {'loss': 1.9683, 'learning_rate': 2e-05, 'epoch': 0.55} 55%|█████▌ | 1000/1817 [36:24:03<29:47:43, 131.29s/it], CurrSamplesPerSec=3.8947130142313986, MemAllocated=0.13GB, MaxMemAllocated=15.7GB 55%|█████▌ | 1000/1817 [36:24:03<29:47:44, 131.29s/it] {'loss': 1.9683, 'learning_rate': 2e-05, 'epoch': 0.55} 55%|█████▌ | 1000/1817 [36:24:03<29:47:44, 131.29s/it][INFO|trainer.py:2830] 2024-01-31 00:09:40,441 >> Saving model checkpoint to ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1000 [INFO|configuration_utils.py:457] 2024-01-31 00:09:40,447 >> Configuration saved in ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1000/config.json [INFO|configuration_utils.py:362] 2024-01-31 00:09:40,451 >> Configuration saved in ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1000/generation_config.json [INFO|modeling_utils.py:1759] 2024-01-31 00:09:40,487 >> Model weights saved in ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1000/pytorch_model.bin [INFO|tokenization_utils_base.py:2164] 2024-01-31 00:09:40,489 >> tokenizer config file saved in ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1000/tokenizer_config.json [INFO|tokenization_utils_base.py:2171] 2024-01-31 00:09:40,490 >> Special tokens file saved in ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1000/special_tokens_map.json [INFO|tokenization_utils_base.py:2221] 2024-01-31 00:09:40,491 >> added tokens file saved in ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1000/added_tokens.json [2024-01-31 00:09:55,274] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step1000 is about to be saved! [2024-01-31 00:09:55,275] [INFO] [engine.py:3492:save_16bit_model] Saving model weights to ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1000/pytorch_model.bin, tag: global_step1000 [2024-01-31 00:09:55,275] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1000/pytorch_model.bin... [2024-01-31 00:10:24,020] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1000/global_step1000/zero_pp_rank_24_mp_rank_00_model_states.pt...[[2024-01-31 00:10:24,067] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1000/global_step1000/zero_pp_rank_24_mp_rank_00_model_states.pt[2[2024-01-31 00:10:24,091] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1000/global_step1000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.[20[2024-01-31 00:10:37,867] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1000/global_step1000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt. [2024-01-31 00:10:37,867] [INFO] [engine.py:3381:_save_zero_checkpoint] zero checkpoint saved ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1000/global_step1000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt [2024-01-31 00:10:38,507] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step1000 is ready now! /global_step1000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt... [2024-01-31 00:10:38,513] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step1000 is ready now! point-1000/global_step1000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt. [2024-01-31 00:10:38,459] [INFO] [engine.py:3381:_save_zero_checkpoint] zero checkpoint saved ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1000/global_step1000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt [2[2024-01-31 00:10:38,486] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step1000 is ready now[INFO|trainer.py:2908] 2024-01-31 00:10:38,553 >> Deleting older checkpoint [checkpoints_ct/ac/allm-ac-13b/checkpoint-500] due to args.save_total_limit 55%|█████▌ | 1001/1817 [36:27:13<33:42:23, 148.71s/it] {'loss': 1.8944, 'learning_rate': 2e-05, 'epoch': 0.55} 55%|█████▌ | 1001/1817 [36:27:12<33:42:23, 148.70s/it] 55%|█████▌ | 1002/1817 [36:29:23<32:26:26, 143.30s/it] {'loss': 1.8843, 'learning_rate': 2e-05, 'epoch': 0.55} 55%|█████▌ | 1002/1817 [36:29:23<32:26:26, 143.30s/it] 55%|█████▌ | 1003/1817 [36:31:32<31:25:55, 139.01s/it] {'loss': 1.9064, 'learning_rate': 2e-05, 'epoch': 0.55} 55%|█████▌ | 1003/1817 [36:31:32<31:25:54, 139.01s/it] 55%|█████▌ | 1004/1817 [36:33:40<30:38:59, 135.72s/it] {'loss': 1.9063, 'learning_rate': 2e-05, 'epoch': 0.55} 55%|█████▌ | 1004/1817 [36:33:40<30:38:59, 135.72s/it] 55%|█████▌ | 1005/1817 [36:35:48<30:04:02, 133.30s/it] {'loss': 1.9127, 'learning_rate': 2e-05, 'epoch': 0.55} 55%|█████▌ | 1005/1817 [36:35:48<30:04:01, 133.30s/it] 55%|█████▌ | 1006/1817 [36:38:00<29:55:14, 132.82s/it] {'loss': 1.906, 'learning_rate': 2e-05, 'epoch': 0.55} 55%|█████▌ | 1006/1817 [36:38:00<29:55:13, 132.82s/it] 55%|█████▌ | 1007/1817 [36:40:12<29:48:51, 132.51s/it] {'loss': 1.9056, 'learning_rate': 2e-05, 'epoch': 0.55} 55%|█████▌ | 1007/1817 [36:40:11<29:48:51, 132.51s/it] 55%|█████▌ | 1008/1817 [36:42:20<29:31:54, 131.41s/it] {'loss': 1.9235, 'learning_rate': 2e-05, 'epoch': 0.55} 55%|█████▌ | 1008/1817 [36:42:20<29:31:54, 131.41s/it] 56%|█████▌ | 1009/1817 [36:44:31<29:24:38, 131.04s/it] {'loss': 1.8807, 'learning_rate': 2e-05, 'epoch': 0.56} 56%|█████▌ | 1009/1817 [36:44:30<29:24:40, 131.04s/it] 56%|█████▌ | 1010/1817 [36:46:38<29:06:50, 129.88s/it] {'loss': 1.9605, 'learning_rate': 2e-05, 'epoch': 0.56} 56%|█████▌ | 1010/1817 [36:46:38<29:06:50, 129.88s/it] 56%|█████▌ | 1011/1817 [36:48:46<28:58:32, 129.42s/it] {'loss': 1.9487, 'learning_rate': 2e-05, 'epoch': 0.56} 56%|█████▌ | 1011/1817 [36:48:46<28:58:32, 129.42s/it] 56%|█████▌ | 1012/1817 [36:50:54<28:51:44, 129.07s/it] {'loss': 2.0293, 'learning_rate': 2e-05, 'epoch': 0.56} 56%|█████▌ | 1012/1817 [36:50:54<28:51:46, 129.08s/it] 56%|█████▌ | 1013/1817 [36:53:04<28:53:25, 129.36s/it] {'loss': 1.9752, 'learning_rate': 2e-05, 'epoch': 0.56} 56%|█████▌ | 1013/1817 [36:53:04<28:53:26, 129.36s/it] 56%|█████▌ | 1014/1817 [36:55:11<28:40:52, 128.58s/it] {'loss': 2.0936, 'learning_rate': 2e-05, 'epoch': 0.56} 56%|█████▌ | 1014/1817 [36:55:11<28:40:52, 128.58s/it] 56%|█████▌ | 1015/1817 [36:57:21<28:45:53, 129.12s/it] {'loss': 2.0737, 'learning_rate': 2e-05, 'epoch': 0.56} 56%|█████▌ | 1015/1817 [36:57:21<28:45:53, 129.12s/it] 56%|█████▌ | 1016/1817 [36:59:32<28:50:48, 129.65s/it] {'loss': 2.0234, 'learning_rate': 2e-05, 'epoch': 0.56} 56%|█████▌ | 1016/1817 [36:59:32<28:50:48, 129.65s/it] 56%|█████▌ | 1017/1817 [37:01:40<28:42:12, 129.17s/it] {'loss': 2.0537, 'learning_rate': 2e-05, 'epoch': 0.56} 56%|█████▌ | 1017/1817 [37:01:40<28:42:12, 129.17s/it] 56%|█████▌ | 1018/1817 [37:03:53<28:55:57, 130.36s/it] {'loss': 2.0795, 'learning_rate': 2e-05, 'epoch': 0.56} 56%|█████▌ | 1018/1817 [37:03:53<28:55:57, 130.36s/it] 56%|█████▌ | 1019/1817 [37:06:07<29:05:17, 131.23s/it] {'loss': 2.0374, 'learning_rate': 2e-05, 'epoch': 0.56} 56%|█████▌ | 1019/1817 [37:06:07<29:05:17, 131.23s/it] 56%|█████▌ | 1020/1817 [37:08:21<29:14:06, 132.05s/it] {'loss': 1.9704, 'learning_rate': 2e-05, 'epoch': 0.56} 56%|█████▌ | 1020/1817 [37:08:21<29:14:06, 132.05s/it] 56%|█████▌ | 1021/1817 [37:10:30<29:01:19, 131.26s/it] {'loss': 2.0487, 'learning_rate': 2e-05, 'epoch': 0.56} 56%|█████▌ | 1021/1817 [37:10:30<29:01:17, 131.25s/it] 56%|█████▌ | 1022/1817 [37:12:40<28:55:52, 131.01s/it] {'loss': 1.9866, 'learning_rate': 2e-05, 'epoch': 0.56} 56%|█████▌ | 1022/1817 [37:12:40<28:55:52, 131.01s/it] 56%|█████▋ | 1023/1817 [37:14:52<28:56:44, 131.24s/it] {'loss': 2.0211, 'learning_rate': 2e-05, 'epoch': 0.56} 56%|█████▋ | 1023/1817 [37:14:52<28:56:44, 131.24s/it] 56%|█████▋ | 1024/1817 [37:17:02<28:48:08, 130.75s/it] {'loss': 2.016, 'learning_rate': 2e-05, 'epoch': 0.56} 56%|█████▋ | 1024/1817 [37:17:02<28:48:08, 130.75s/it] 56%|█████▋ | 1025/1817 [37:19:14<28:49:57, 131.06s/it] {'loss': 2.0208, 'learning_rate': 2e-05, 'epoch': 0.56} 56%|█████▋ | 1025/1817 [37:19:14<28:49:57, 131.06s/it] 56%|█████▋ | 1026/1817 [37:21:22<28:36:53, 130.23s/it] {'loss': 2.0739, 'learning_rate': 2e-05, 'epoch': 0.56} 56%|█████▋ | 1026/1817 [37:21:22<28:36:53, 130.23s/it] 57%|█████▋ | 1027/1817 [37:23:37<28:54:16, 131.72s/it] {'loss': 2.0705, 'learning_rate': 2e-05, 'epoch': 0.57} 57%|█████▋ | 1027/1817 [37:23:37<28:54:16, 131.72s/it] 57%|█████▋ | 1028/1817 [37:25:44<28:30:16, 130.06s/it] {'loss': 2.0603, 'learning_rate': 2e-05, 'epoch': 0.57} 57%|█████▋ | 1028/1817 [37:25:43<28:30:16, 130.06s/it] 57%|█████▋ | 1029/1817 [37:27:57<28:41:38, 131.09s/it] {'loss': 2.0256, 'learning_rate': 2e-05, 'epoch': 0.57} 57%|█████▋ | 1029/1817 [37:27:57<28:41:38, 131.09s/it] 57%|█████▋ | 1030/1817 [37:30:07<28:35:56, 130.82s/it] {'loss': 2.0386, 'learning_rate': 2e-05, 'epoch': 0.57} 57%|█████▋ | 1030/1817 [37:30:07<28:35:56, 130.82s/it] 57%|█████▋ | 1031/1817 [37:32:19<28:36:09, 131.00s/it] {'loss': 2.0059, 'learning_rate': 2e-05, 'epoch': 0.57} 57%|█████▋ | 1031/1817 [37:32:18<28:36:09, 131.00s/it] 57%|█████▋ | 1032/1817 [37:34:27<28:24:43, 130.30s/it] {'loss': 1.9875, 'learning_rate': 2e-05, 'epoch': 0.57} 57%|█████▋ | 1032/1817 [37:34:27<28:24:45, 130.30s/it] 57%|█████▋ | 1033/1817 [37:36:38<28:25:54, 130.55s/it] {'loss': 1.963, 'learning_rate': 2e-05, 'epoch': 0.57} 57%|█████▋ | 1033/1817 [37:36:38<28:25:54, 130.55s/it] 57%|█████▋ | 1034/1817 [37:38:50<28:27:04, 130.81s/it] {'loss': 1.9372, 'learning_rate': 2e-05, 'epoch': 0.57} 57%|█████▋ | 1034/1817 [37:38:50<28:27:04, 130.81s/it] 57%|█████▋ | 1035/1817 [37:40:59<28:20:44, 130.49s/it] {'loss': 1.8994, 'learning_rate': 2e-05, 'epoch': 0.57} 57%|█████▋ | 1035/1817 [37:40:59<28:20:44, 130.49s/it] 57%|█████▋ | 1036/1817 [37:43:10<28:18:03, 130.45s/it] {'loss': 1.9196, 'learning_rate': 2e-05, 'epoch': 0.57} 57%|█████▋ | 1036/1817 [37:43:10<28:18:03, 130.45s/it] 57%|█████▋ | 1037/1817 [37:45:20<28:13:24, 130.26s/it] {'loss': 1.9083, 'learning_rate': 2e-05, 'epoch': 0.57} 57%|█████▋ | 1037/1817 [37:45:20<28:13:24, 130.26s/it] 57%|█████▋ | 1038/1817 [37:47:28<28:05:55, 129.85s/it] {'loss': 1.9411, 'learning_rate': 2e-05, 'epoch': 0.57} 57%|█████▋ | 1038/1817 [37:47:28<28:05:55, 129.85s/it] 57%|█████▋ | 1039/1817 [37:49:40<28:10:42, 130.39s/it] {'loss': 1.9164, 'learning_rate': 2e-05, 'epoch': 0.57} 57%|█████▋ | 1039/1817 [37:49:40<28:10:42, 130.39s/it] 57%|█████▋ | 1040/1817 [37:51:52<28:13:15, 130.75s/it] {'loss': 1.8869, 'learning_rate': 2e-05, 'epoch': 0.57} 57%|█████▋ | 1040/1817 [37:51:52<28:13:15, 130.75s/it] 57%|█████▋ | 1041/1817 [37:54:01<28:03:13, 130.15s/it] {'loss': 1.8765, 'learning_rate': 2e-05, 'epoch': 0.57} 57%|█████▋ | 1041/1817 [37:54:00<28:03:14, 130.15s/it] 57%|█████▋ | 1042/1817 [37:56:11<28:03:17, 130.32s/it] {'loss': 1.9326, 'learning_rate': 2e-05, 'epoch': 0.57} 57%|█████▋ | 1042/1817 [37:56:11<28:03:17, 130.32s/it] 57%|█████▋ | 1043/1817 [37:58:19<27:50:20, 129.48s/it] {'loss': 1.9051, 'learning_rate': 2e-05, 'epoch': 0.57} 57%|█████▋ | 1043/1817 [37:58:19<27:50:21, 129.48s/it] 57%|█████▋ | 1044/1817 [38:00:30<27:54:31, 129.98s/it] {'loss': 1.9671, 'learning_rate': 2e-05, 'epoch': 0.57} 57%|█████▋ | 1044/1817 [38:00:30<27:54:31, 129.98s/it] 58%|█████▊ | 1045/1817 [38:02:42<28:01:09, 130.66s/it] {'loss': 1.8995, 'learning_rate': 2e-05, 'epoch': 0.57} 58%|█████▊ | 1045/1817 [38:02:42<28:01:09, 130.66s/it] 58%|█████▊ | 1046/1817 [38:04:52<27:56:01, 130.43s/it] {'loss': 1.8705, 'learning_rate': 2e-05, 'epoch': 0.58} 58%|█████▊ | 1046/1817 [38:04:52<27:56:01, 130.43s/it] 58%|█████▊ | 1047/1817 [38:07:01<27:46:28, 129.85s/it] {'loss': 1.9161, 'learning_rate': 2e-05, 'epoch': 0.58} 58%|█████▊ | 1047/1817 [38:07:00<27:46:28, 129.86s/it] 58%|█████▊ | 1048/1817 [38:09:10<27:43:43, 129.81s/it] {'loss': 1.879, 'learning_rate': 2e-05, 'epoch': 0.58} 58%|█████▊ | 1048/1817 [38:09:10<27:43:43, 129.81s/it] 58%|█████▊ | 1049/1817 [38:11:20<27:40:04, 129.69s/it] {'loss': 1.8959, 'learning_rate': 2e-05, 'epoch': 0.58} 58%|█████▊ | 1049/1817 [38:11:20<27:40:00, 129.69s/it] 58%|█████▊ | 1050/1817 [38:13:33<27:50:29, 130.68s/it] {'loss': 1.8776, 'learning_rate': 2e-05, 'epoch': 0.58} 58%|█████▊ | 1050/1817 [38:13:33<27:50:29, 130.68s/it] 58%|█████▊ | 1051/1817 [38:15:46<28:00:05, 131.60s/it] {'loss': 1.898, 'learning_rate': 2e-05, 'epoch': 0.58} 58%|█████▊ | 1051/1817 [38:15:46<28:00:05, 131.60s/it] 58%|█████▊ | 1052/1817 [38:17:59<28:01:51, 131.91s/it] {'loss': 1.8915, 'learning_rate': 2e-05, 'epoch': 0.58} 58%|█████▊ | 1052/1817 [38:17:59<28:01:51, 131.91s/it] 58%|█████▊ | 1053/1817 [38:20:10<27:56:26, 131.66s/it] {'loss': 1.9079, 'learning_rate': 2e-05, 'epoch': 0.58} 58%|█████▊ | 1053/1817 [38:20:10<27:56:28, 131.66s/it] 58%|█████▊ | 1054/1817 [38:22:21<27:49:57, 131.32s/it] {'loss': 1.8977, 'learning_rate': 2e-05, 'epoch': 0.58} 58%|█████▊ | 1054/1817 [38:22:21<27:49:57, 131.32s/it] 58%|█████▊ | 1055/1817 [38:24:31<27:44:42, 131.08s/it] {'loss': 1.9186, 'learning_rate': 2e-05, 'epoch': 0.58} 58%|█████▊ | 1055/1817 [38:24:31<27:44:42, 131.08s/it] 58%|█████▊ | 1056/1817 [38:26:43<27:44:39, 131.25s/it] {'loss': 1.93, 'learning_rate': 2e-05, 'epoch': 0.58} 58%|█████▊ | 1056/1817 [38:26:43<27:44:39, 131.25s/it] 58%|█████▊ | 1057/1817 [38:28:52<27:33:53, 130.57s/it] {'loss': 1.8862, 'learning_rate': 2e-05, 'epoch': 0.58} 58%|█████▊ | 1057/1817 [38:28:52<27:33:53, 130.57s/it] 58%|█████▊ | 1058/1817 [38:31:03<27:33:33, 130.72s/it] {'loss': 1.9239, 'learning_rate': 2e-05, 'epoch': 0.58} 58%|█████▊ | 1058/1817 [38:31:03<27:33:33, 130.72s/it] 58%|█████▊ | 1059/1817 [38:33:11<27:22:38, 130.02s/it] {'loss': 1.9656, 'learning_rate': 2e-05, 'epoch': 0.58} 58%|█████▊ | 1059/1817 [38:33:11<27:22:39, 130.03s/it] 58%|█████▊ | 1060/1817 [38:35:19<27:12:55, 129.43s/it] {'loss': 1.8738, 'learning_rate': 2e-05, 'epoch': 0.58} 58%|█████▊ | 1060/1817 [38:35:19<27:12:56, 129.43s/it] 58%|█████▊ | 1061/1817 [38:37:28<27:07:12, 129.14s/it] {'loss': 1.871, 'learning_rate': 2e-05, 'epoch': 0.58} 58%|█████▊ | 1061/1817 [38:37:28<27:07:13, 129.14s/it] 58%|█████▊ | 1062/1817 [38:39:36<27:01:13, 128.84s/it] {'loss': 1.9152, 'learning_rate': 2e-05, 'epoch': 0.58} 58%|█████▊ | 1062/1817 [38:39:36<27:01:13, 128.84s/it] 59%|█████▊ | 1063/1817 [38:41:42<26:49:07, 128.05s/it] {'loss': 1.9247, 'learning_rate': 2e-05, 'epoch': 0.58} 59%|█████▊ | 1063/1817 [38:41:42<26:49:07, 128.05s/it] 59%|█████▊ | 1064/1817 [38:43:51<26:49:00, 128.21s/it] {'loss': 1.921, 'learning_rate': 2e-05, 'epoch': 0.59} 59%|█████▊ | 1064/1817 [38:43:51<26:49:01, 128.21s/it] 59%|█████▊ | 1065/1817 [38:46:01<26:54:59, 128.86s/it] {'loss': 1.9086, 'learning_rate': 2e-05, 'epoch': 0.59} 59%|█████▊ | 1065/1817 [38:46:01<26:54:59, 128.86s/it] 59%|█████▊ | 1066/1817 [38:48:16<27:17:35, 130.83s/it] {'loss': 1.8795, 'learning_rate': 2e-05, 'epoch': 0.59} 59%|█████▊ | 1066/1817 [38:48:16<27:17:35, 130.83s/it] 59%|█████▊ | 1067/1817 [38:50:27<27:12:58, 130.64s/it] {'loss': 1.9273, 'learning_rate': 2e-05, 'epoch': 0.59} 59%|█████▊ | 1067/1817 [38:50:27<27:12:58, 130.64s/it] 59%|█████▉ | 1068/1817 [38:52:38<27:15:22, 131.01s/it] {'loss': 1.8944, 'learning_rate': 2e-05, 'epoch': 0.59} 59%|█████▉ | 1068/1817 [38:52:38<27:15:22, 131.01s/it] 59%|█████▉ | 1069/1817 [38:54:53<27:24:38, 131.92s/it] {'loss': 1.9217, 'learning_rate': 2e-05, 'epoch': 0.59} 59%|█████▉ | 1069/1817 [38:54:53<27:24:38, 131.92s/it] 59%|█████▉ | 1070/1817 [38:57:03<27:15:03, 131.33s/it] {'loss': 1.9379, 'learning_rate': 2e-05, 'epoch': 0.59} 59%|█████▉ | 1070/1817 [38:57:02<27:15:03, 131.33s/it] 59%|█████▉ | 1071/1817 [38:59:09<26:55:01, 129.89s/it] {'loss': 1.9146, 'learning_rate': 2e-05, 'epoch': 0.59} 59%|█████▉ | 1071/1817 [38:59:09<26:55:01, 129.89s/it] 59%|█████▉ | 1072/1817 [39:01:21<26:59:10, 130.40s/it] {'loss': 1.8946, 'learning_rate': 2e-05, 'epoch': 0.59} 59%|█████▉ | 1072/1817 [39:01:21<26:59:10, 130.40s/it] 59%|█████▉ | 1073/1817 [39:03:33<27:04:01, 130.97s/it] {'loss': 1.8986, 'learning_rate': 2e-05, 'epoch': 0.59} 59%|█████▉ | 1073/1817 [39:03:33<27:04:01, 130.97s/it] 59%|█████▉ | 1074/1817 [39:05:43<26:58:39, 130.71s/it] {'loss': 1.9467, 'learning_rate': 2e-05, 'epoch': 0.59} 59%|█████▉ | 1074/1817 [39:05:43<26:58:39, 130.71s/it] 59%|█████▉ | 1075/1817 [39:07:53<26:52:31, 130.39s/it] {'loss': 1.9213, 'learning_rate': 2e-05, 'epoch': 0.59} 59%|█████▉ | 1075/1817 [39:07:53<26:52:31, 130.39s/it] 59%|█████▉ | 1076/1817 [39:10:08<27:08:02, 131.82s/it] {'loss': 1.9043, 'learning_rate': 2e-05, 'epoch': 0.59} 59%|█████▉ | 1076/1817 [39:10:08<27:08:02, 131.82s/it] 59%|█████▉ | 1077/1817 [39:12:22<27:14:01, 132.49s/it] {'loss': 1.9125, 'learning_rate': 2e-05, 'epoch': 0.59} 59%|█████▉ | 1077/1817 [39:12:22<27:13:57, 132.48s/it] 59%|█████▉ | 1078/1817 [39:14:32<27:02:03, 131.70s/it] {'loss': 1.875, 'learning_rate': 2e-05, 'epoch': 0.59} 59%|█████▉ | 1078/1817 [39:14:32<27:02:01, 131.69s/it] 59%|█████▉ | 1079/1817 [39:16:42<26:53:37, 131.19s/it] {'loss': 1.8515, 'learning_rate': 2e-05, 'epoch': 0.59} 59%|█████▉ | 1079/1817 [39:16:42<26:53:37, 131.19s/it] 59%|█████▉ | 1080/1817 [39:18:54<26:55:53, 131.55s/it] {'loss': 1.9106, 'learning_rate': 2e-05, 'epoch': 0.59} 59%|█████▉ | 1080/1817 [39:18:54<26:55:53, 131.55s/it] 59%|█████▉ | 1081/1817 [39:21:04<26:45:22, 130.87s/it] {'loss': 1.9164, 'learning_rate': 2e-05, 'epoch': 0.59} 59%|█████▉ | 1081/1817 [39:21:03<26:45:22, 130.87s/it] 60%|█████▉ | 1082/1817 [39:23:14<26:42:30, 130.82s/it] {'loss': 1.8857, 'learning_rate': 2e-05, 'epoch': 0.6} 60%|█████▉ | 1082/1817 [39:23:14<26:42:30, 130.82s/it] 60%|█████▉ | 1083/1817 [39:25:25<26:38:43, 130.69s/it] {'loss': 1.8584, 'learning_rate': 2e-05, 'epoch': 0.6} 60%|█████▉ | 1083/1817 [39:25:24<26:38:43, 130.69s/it] 60%|█████▉ | 1084/1817 [39:27:34<26:31:21, 130.26s/it] {'loss': 1.883, 'learning_rate': 2e-05, 'epoch': 0.6} 60%|█████▉ | 1084/1817 [39:27:34<26:31:21, 130.26s/it] 60%|█████▉ | 1085/1817 [39:29:50<26:50:41, 132.02s/it] {'loss': 1.8684, 'learning_rate': 2e-05, 'epoch': 0.6} 60%|█████▉ | 1085/1817 [39:29:50<26:50:41, 132.02s/it] 60%|█████▉ | 1086/1817 [39:31:58<26:33:46, 130.82s/it] {'loss': 1.9209, 'learning_rate': 2e-05, 'epoch': 0.6} 60%|█████▉ | 1086/1817 [39:31:58<26:33:46, 130.82s/it] 60%|█████▉ | 1087/1817 [39:34:07<26:24:46, 130.26s/it] {'loss': 1.8502, 'learning_rate': 2e-05, 'epoch': 0.6} 60%|█████▉ | 1087/1817 [39:34:07<26:24:45, 130.25s/it] 60%|█████▉ | 1088/1817 [39:36:21<26:34:59, 131.27s/it] {'loss': 1.9311, 'learning_rate': 2e-05, 'epoch': 0.6} 60%|█████▉ | 1088/1817 [39:36:20<26:34:59, 131.28s/it] 60%|█████▉ | 1089/1817 [39:38:29<26:22:34, 130.43s/it] {'loss': 1.8962, 'learning_rate': 2e-05, 'epoch': 0.6} 60%|█████▉ | 1089/1817 [39:38:29<26:22:34, 130.43s/it] 60%|█████▉ | 1090/1817 [39:40:39<26:18:21, 130.26s/it] {'loss': 1.9103, 'learning_rate': 2e-05, 'epoch': 0.6} 60%|█████▉ | 1090/1817 [39:40:39<26:18:21, 130.26s/it] 60%|██████ | 1091/1817 [39:42:49<26:16:14, 130.27s/it] {'loss': 1.8802, 'learning_rate': 2e-05, 'epoch': 0.6} 60%|██████ | 1091/1817 [39:42:49<26:16:15, 130.27s/it] 60%|██████ | 1092/1817 [39:45:01<26:18:04, 130.60s/it] {'loss': 1.9064, 'learning_rate': 2e-05, 'epoch': 0.6} 60%|██████ | 1092/1817 [39:45:00<26:18:04, 130.60s/it] 60%|██████ | 1093/1817 [39:47:11<26:13:52, 130.43s/it] {'loss': 1.8634, 'learning_rate': 2e-05, 'epoch': 0.6} 60%|██████ | 1093/1817 [39:47:10<26:13:52, 130.43s/it] 60%|██████ | 1094/1817 [39:49:23<26:18:29, 131.00s/it] {'loss': 1.8749, 'learning_rate': 2e-05, 'epoch': 0.6} 60%|██████ | 1094/1817 [39:49:23<26:18:31, 131.00s/it] 60%|██████ | 1095/1817 [39:51:30<26:00:57, 129.72s/it] {'loss': 1.9098, 'learning_rate': 2e-05, 'epoch': 0.6} 60%|██████ | 1095/1817 [39:51:30<26:00:59, 129.72s/it] 60%|██████ | 1096/1817 [39:53:35<25:42:28, 128.36s/it] {'loss': 1.8737, 'learning_rate': 2e-05, 'epoch': 0.6} 60%|██████ | 1096/1817 [39:53:35<25:42:29, 128.36s/it] 60%|██████ | 1097/1817 [39:55:44<25:40:58, 128.42s/it] {'loss': 1.8678, 'learning_rate': 2e-05, 'epoch': 0.6} 60%|██████ | 1097/1817 [39:55:43<25:40:59, 128.42s/it] 60%|██████ | 1098/1817 [39:57:53<25:43:31, 128.81s/it] {'loss': 1.8435, 'learning_rate': 2e-05, 'epoch': 0.6} 60%|██████ | 1098/1817 [39:57:53<25:43:31, 128.81s/it] 60%|██████ | 1099/1817 [40:00:02<25:42:19, 128.89s/it] {'loss': 1.8708, 'learning_rate': 2e-05, 'epoch': 0.6} 60%|██████ | 1099/1817 [40:00:02<25:42:20, 128.89s/it] 61%|██████ | 1100/1817 [40:02:15<25:54:03, 130.05s/it] {'loss': 1.8936, 'learning_rate': 2e-05, 'epoch': 0.61} 61%|██████ | 1100/1817 [40:02:15<25:54:03, 130.05s/it] 61%|██████ | 1101/1817 [40:04:27<25:57:09, 130.49s/it] {'loss': 1.9381, 'learning_rate': 2e-05, 'epoch': 0.61} 61%|██████ | 1101/1817 [40:04:26<25:57:10, 130.49s/it] 61%|██████ | 1102/1817 [40:06:40<26:06:25, 131.45s/it] {'loss': 1.9195, 'learning_rate': 2e-05, 'epoch': 0.61} 61%|██████ | 1102/1817 [40:06:40<26:06:25, 131.45s/it] 61%|██████ | 1103/1817 [40:08:49<25:54:46, 130.65s/it] {'loss': 1.9138, 'learning_rate': 2e-05, 'epoch': 0.61} 61%|██████ | 1103/1817 [40:08:49<25:54:47, 130.65s/it] 61%|██████ | 1104/1817 [40:10:58<25:46:46, 130.16s/it] {'loss': 1.9088, 'learning_rate': 2e-05, 'epoch': 0.61} 61%|██████ | 1104/1817 [40:10:58<25:46:46, 130.16s/it] 61%|██████ | 1105/1817 [40:13:08<25:44:44, 130.18s/it] {'loss': 1.8879, 'learning_rate': 2e-05, 'epoch': 0.61} 61%|██████ | 1105/1817 [40:13:08<25:44:44, 130.18s/it] 61%|██████ | 1106/1817 [40:15:17<25:37:43, 129.77s/it] {'loss': 1.9377, 'learning_rate': 2e-05, 'epoch': 0.61} 61%|██████ | 1106/1817 [40:15:17<25:37:43, 129.77s/it] 61%|██████ | 1107/1817 [40:17:27<25:36:17, 129.83s/it] {'loss': 1.8791, 'learning_rate': 2e-05, 'epoch': 0.61} 61%|██████ | 1107/1817 [40:17:27<25:36:17, 129.83s/it] 61%|██████ | 1108/1817 [40:19:42<25:50:55, 131.25s/it] {'loss': 1.8779, 'learning_rate': 2e-05, 'epoch': 0.61} 61%|██████ | 1108/1817 [40:19:41<25:50:55, 131.25s/it] 61%|██████ | 1109/1817 [40:21:52<25:45:01, 130.93s/it] {'loss': 1.8317, 'learning_rate': 2e-05, 'epoch': 0.61} 61%|██████ | 1109/1817 [40:21:52<25:45:00, 130.93s/it] 61%|██████ | 1110/1817 [40:24:00<25:34:32, 130.23s/it] {'loss': 1.8704, 'learning_rate': 2e-05, 'epoch': 0.61} 61%|██████ | 1110/1817 [40:24:00<25:34:32, 130.23s/it] 61%|██████ | 1111/1817 [40:26:09<25:26:49, 129.76s/it] {'loss': 1.9418, 'learning_rate': 2e-05, 'epoch': 0.61} 61%|██████ | 1111/1817 [40:26:09<25:26:49, 129.76s/it] 61%|██████ | 1112/1817 [40:28:21<25:31:37, 130.35s/it] {'loss': 1.8674, 'learning_rate': 2e-05, 'epoch': 0.61} 61%|██████ | 1112/1817 [40:28:21<25:31:37, 130.35s/it] 61%|██████▏ | 1113/1817 [40:30:28<25:17:27, 129.33s/it] {'loss': 1.8896, 'learning_rate': 2e-05, 'epoch': 0.61} 61%|██████▏ | 1113/1817 [40:30:28<25:17:27, 129.33s/it] 61%|██████▏ | 1114/1817 [40:32:38<25:17:23, 129.51s/it] {'loss': 1.9373, 'learning_rate': 2e-05, 'epoch': 0.61} 61%|██████▏ | 1114/1817 [40:32:37<25:17:23, 129.51s/it] 61%|██████▏ | 1115/1817 [40:34:48<25:18:23, 129.78s/it] {'loss': 1.8777, 'learning_rate': 2e-05, 'epoch': 0.61} 61%|██████▏ | 1115/1817 [40:34:48<25:18:24, 129.78s/it] 61%|██████▏ | 1116/1817 [40:36:59<25:19:17, 130.04s/it] {'loss': 1.9635, 'learning_rate': 2e-05, 'epoch': 0.61} 61%|██████▏ | 1116/1817 [40:36:59<25:19:17, 130.04s/it] 61%|██████▏ | 1117/1817 [40:39:10<25:22:45, 130.52s/it] {'loss': 1.9205, 'learning_rate': 2e-05, 'epoch': 0.61} 61%|██████▏ | 1117/1817 [40:39:10<25:22:46, 130.52s/it] 62%|██████▏ | 1118/1817 [40:41:19<25:14:08, 129.97s/it] {'loss': 1.9386, 'learning_rate': 2e-05, 'epoch': 0.62} 62%|██████▏ | 1118/1817 [40:41:19<25:14:08, 129.97s/it] 62%|██████▏ | 1119/1817 [40:43:33<25:25:34, 131.14s/it] {'loss': 1.8862, 'learning_rate': 2e-05, 'epoch': 0.62} 62%|██████▏ | 1119/1817 [40:43:33<25:25:34, 131.14s/it] 62%|██████▏ | 1120/1817 [40:45:44<25:21:44, 131.00s/it] {'loss': 1.8921, 'learning_rate': 2e-05, 'epoch': 0.62} 62%|██████▏ | 1120/1817 [40:45:43<25:21:44, 131.00s/it] 62%|██████▏ | 1121/1817 [40:47:54<25:17:06, 130.79s/it] {'loss': 1.904, 'learning_rate': 2e-05, 'epoch': 0.62} 62%|██████▏ | 1121/1817 [40:47:54<25:17:07, 130.79s/it] 62%|██████▏ | 1122/1817 [40:50:04<25:13:56, 130.70s/it] {'loss': 1.9216, 'learning_rate': 2e-05, 'epoch': 0.62} 62%|██████▏ | 1122/1817 [40:50:04<25:13:57, 130.70s/it] 62%|██████▏ | 1123/1817 [40:52:15<25:11:30, 130.68s/it] {'loss': 1.91, 'learning_rate': 2e-05, 'epoch': 0.62} 62%|██████▏ | 1123/1817 [40:52:15<25:11:30, 130.68s/it] 62%|██████▏ | 1124/1817 [40:54:22<24:57:59, 129.70s/it] {'loss': 1.8963, 'learning_rate': 2e-05, 'epoch': 0.62} 62%|██████▏ | 1124/1817 [40:54:22<24:57:59, 129.70s/it] 62%|██████▏ | 1125/1817 [40:56:33<24:59:28, 130.01s/it] {'loss': 1.9258, 'learning_rate': 2e-05, 'epoch': 0.62} 62%|██████▏ | 1125/1817 [40:56:33<24:59:29, 130.01s/it] 62%|██████▏ | 1126/1817 [40:58:44<24:58:59, 130.16s/it] {'loss': 1.8981, 'learning_rate': 2e-05, 'epoch': 0.62} 62%|██████▏ | 1126/1817 [40:58:43<24:58:59, 130.16s/it] 62%|██████▏ | 1127/1817 [41:00:56<25:03:16, 130.72s/it] {'loss': 1.9042, 'learning_rate': 2e-05, 'epoch': 0.62} 62%|██████▏ | 1127/1817 [41:00:55<25:03:16, 130.72s/it] 62%|██████▏ | 1128/1817 [41:03:11<25:18:12, 132.21s/it] {'loss': 1.9012, 'learning_rate': 2e-05, 'epoch': 0.62} 62%|██████▏ | 1128/1817 [41:03:11<25:18:12, 132.21s/it] 62%|██████▏ | 1129/1817 [41:05:22<25:08:58, 131.60s/it] {'loss': 1.9259, 'learning_rate': 2e-05, 'epoch': 0.62} 62%|██████▏ | 1129/1817 [41:05:21<25:08:58, 131.60s/it] 62%|██████▏ | 1130/1817 [41:07:30<24:56:03, 130.66s/it] {'loss': 1.9072, 'learning_rate': 2e-05, 'epoch': 0.62} 62%|██████▏ | 1130/1817 [41:07:30<24:56:03, 130.66s/it] 62%|██████▏ | 1131/1817 [41:09:42<24:56:54, 130.92s/it] {'loss': 1.9223, 'learning_rate': 2e-05, 'epoch': 0.62} 62%|██████▏ | 1131/1817 [41:09:41<24:56:54, 130.92s/it] 62%|██████▏ | 1132/1817 [41:11:52<24:53:15, 130.80s/it] {'loss': 1.9085, 'learning_rate': 2e-05, 'epoch': 0.62} 62%|██████▏ | 1132/1817 [41:11:52<24:53:13, 130.79s/it] 62%|██████▏ | 1133/1817 [41:14:01<24:44:53, 130.25s/it] {'loss': 1.8689, 'learning_rate': 2e-05, 'epoch': 0.62} 62%|██████▏ | 1133/1817 [41:14:01<24:44:52, 130.25s/it] 62%|██████▏ | 1134/1817 [41:16:10<24:37:22, 129.78s/it] {'loss': 1.8966, 'learning_rate': 2e-05, 'epoch': 0.62} 62%|██████▏ | 1134/1817 [41:16:10<24:37:21, 129.78s/it] 62%|██████▏ | 1135/1817 [41:18:19<24:32:33, 129.55s/it] {'loss': 1.905, 'learning_rate': 2e-05, 'epoch': 0.62} 62%|██████▏ | 1135/1817 [41:18:19<24:32:33, 129.55s/it] 63%|██████▎ | 1136/1817 [41:20:33<24:47:26, 131.05s/it] {'loss': 1.8673, 'learning_rate': 2e-05, 'epoch': 0.63} 63%|██████▎ | 1136/1817 [41:20:33<24:47:26, 131.05s/it] 63%|██████▎ | 1137/1817 [41:22:41<24:34:29, 130.10s/it] {'loss': 1.907, 'learning_rate': 2e-05, 'epoch': 0.63} 63%|██████▎ | 1137/1817 [41:22:41<24:34:29, 130.10s/it] 63%|██████▎ | 1138/1817 [41:24:50<24:28:35, 129.77s/it] {'loss': 1.8588, 'learning_rate': 2e-05, 'epoch': 0.63} 63%|██████▎ | 1138/1817 [41:24:50<24:28:35, 129.77s/it] 63%|██████▎ | 1139/1817 [41:27:04<24:38:21, 130.83s/it] {'loss': 1.8956, 'learning_rate': 2e-05, 'epoch': 0.63} 63%|██████▎ | 1139/1817 [41:27:03<24:38:21, 130.83s/it] 63%|██████▎ | 1140/1817 [41:29:11<24:24:03, 129.75s/it] {'loss': 1.8785, 'learning_rate': 2e-05, 'epoch': 0.63} 63%|██████▎ | 1140/1817 [41:29:11<24:24:04, 129.75s/it] 63%|██████▎ | 1141/1817 [41:31:19<24:16:18, 129.26s/it] {'loss': 1.8361, 'learning_rate': 2e-05, 'epoch': 0.63} 63%|██████▎ | 1141/1817 [41:31:19<24:16:18, 129.26s/it] 63%|██████▎ | 1142/1817 [41:33:27<24:11:58, 129.06s/it] {'loss': 1.9203, 'learning_rate': 2e-05, 'epoch': 0.63} 63%|██████▎ | 1142/1817 [41:33:27<24:11:58, 129.06s/it] 63%|██████▎ | 1143/1817 [41:35:37<24:11:45, 129.24s/it] {'loss': 1.9283, 'learning_rate': 2e-05, 'epoch': 0.63} 63%|██████▎ | 1143/1817 [41:35:37<24:11:45, 129.24s/it] 63%|██████▎ | 1144/1817 [41:37:50<24:24:01, 130.52s/it] {'loss': 1.8737, 'learning_rate': 2e-05, 'epoch': 0.63} 63%|██████▎ | 1144/1817 [41:37:50<24:24:01, 130.52s/it] 63%|██████▎ | 1145/1817 [41:40:00<24:17:35, 130.14s/it] {'loss': 1.8577, 'learning_rate': 2e-05, 'epoch': 0.63} 63%|██████▎ | 1145/1817 [41:40:00<24:17:35, 130.14s/it] 63%|██████▎ | 1146/1817 [41:42:11<24:19:28, 130.50s/it] {'loss': 1.8932, 'learning_rate': 2e-05, 'epoch': 0.63} 63%|██████▎ | 1146/1817 [41:42:11<24:19:28, 130.50s/it] 63%|██████▎ | 1147/1817 [41:44:26<24:30:58, 131.73s/it] {'loss': 1.8689, 'learning_rate': 2e-05, 'epoch': 0.63} 63%|██████▎ | 1147/1817 [41:44:26<24:30:58, 131.73s/it] 63%|██████▎ | 1148/1817 [41:46:36<24:24:21, 131.33s/it] {'loss': 1.8874, 'learning_rate': 2e-05, 'epoch': 0.63} 63%|██████▎ | 1148/1817 [41:46:36<24:24:21, 131.33s/it] 63%|██████▎ | 1149/1817 [41:48:46<24:17:10, 130.88s/it] {'loss': 1.8937, 'learning_rate': 2e-05, 'epoch': 0.63} 63%|██████▎ | 1149/1817 [41:48:46<24:17:10, 130.88s/it] 63%|██████▎ | 1150/1817 [41:50:57<24:17:05, 131.07s/it] {'loss': 1.8628, 'learning_rate': 2e-05, 'epoch': 0.63} 63%|██████▎ | 1150/1817 [41:50:57<24:17:05, 131.07s/it] 63%|██████▎ | 1151/1817 [41:53:02<23:54:21, 129.22s/it] {'loss': 1.8981, 'learning_rate': 2e-05, 'epoch': 0.63} 63%|██████▎ | 1151/1817 [41:53:02<23:54:21, 129.22s/it] 63%|██████▎ | 1152/1817 [41:55:17<24:10:09, 130.84s/it] {'loss': 1.8749, 'learning_rate': 2e-05, 'epoch': 0.63} 63%|██████▎ | 1152/1817 [41:55:17<24:10:09, 130.84s/it] 63%|██████▎ | 1153/1817 [41:57:28<24:09:01, 130.94s/it] {'loss': 1.8755, 'learning_rate': 2e-05, 'epoch': 0.63} 63%|██████▎ | 1153/1817 [41:57:28<24:09:01, 130.94s/it] 64%|██████▎ | 1154/1817 [41:59:37<24:01:58, 130.50s/it] {'loss': 1.8683, 'learning_rate': 2e-05, 'epoch': 0.63} 64%|██████▎ | 1154/1817 [41:59:37<24:01:58, 130.50s/it] 64%|██████▎ | 1155/1817 [42:01:48<24:00:48, 130.59s/it] {'loss': 1.8803, 'learning_rate': 2e-05, 'epoch': 0.64} 64%|██████▎ | 1155/1817 [42:01:48<24:00:48, 130.59s/it] 64%|██████▎ | 1156/1817 [42:03:59<23:56:45, 130.42s/it] {'loss': 1.8516, 'learning_rate': 2e-05, 'epoch': 0.64} 64%|██████▎ | 1156/1817 [42:03:58<23:56:45, 130.42s/it] 64%|██████▎ | 1157/1817 [42:06:09<23:54:47, 130.44s/it] {'loss': 1.8788, 'learning_rate': 2e-05, 'epoch': 0.64} 64%|██████▎ | 1157/1817 [42:06:09<23:54:48, 130.44s/it] 64%|██████▎ | 1158/1817 [42:08:20<23:54:52, 130.64s/it] {'loss': 1.9136, 'learning_rate': 2e-05, 'epoch': 0.64} 64%|██████▎ | 1158/1817 [42:08:20<23:54:52, 130.64s/it] 64%|██████▍ | 1159/1817 [42:10:32<23:56:30, 130.99s/it] {'loss': 1.8637, 'learning_rate': 2e-05, 'epoch': 0.64} 64%|██████▍ | 1159/1817 [42:10:32<23:56:28, 130.98s/it] 64%|██████▍ | 1160/1817 [42:12:45<24:00:32, 131.56s/it] {'loss': 1.8859, 'learning_rate': 2e-05, 'epoch': 0.64} 64%|██████▍ | 1160/1817 [42:12:45<24:00:29, 131.55s/it] 64%|██████▍ | 1161/1817 [42:14:59<24:06:32, 132.31s/it] {'loss': 1.8597, 'learning_rate': 2e-05, 'epoch': 0.64} 64%|██████▍ | 1161/1817 [42:14:59<24:06:30, 132.30s/it] 64%|██████▍ | 1162/1817 [42:17:07<23:50:16, 131.02s/it] {'loss': 1.8645, 'learning_rate': 2e-05, 'epoch': 0.64} 64%|██████▍ | 1162/1817 [42:17:07<23:50:16, 131.02s/it] 64%|██████▍ | 1163/1817 [42:19:16<23:42:16, 130.48s/it] {'loss': 1.8789, 'learning_rate': 2e-05, 'epoch': 0.64} 64%|██████▍ | 1163/1817 [42:19:16<23:42:16, 130.48s/it] 64%|██████▍ | 1164/1817 [42:21:29<23:47:29, 131.16s/it] {'loss': 1.8784, 'learning_rate': 2e-05, 'epoch': 0.64} 64%|██████▍ | 1164/1817 [42:21:29<23:47:29, 131.16s/it] 64%|██████▍ | 1165/1817 [42:23:36<23:34:31, 130.17s/it] {'loss': 1.8852, 'learning_rate': 2e-05, 'epoch': 0.64} 64%|██████▍ | 1165/1817 [42:23:37<23:34:31, 130.17s/it] 64%|██████▍ | 1166/1817 [42:25:48<23:36:11, 130.52s/it] {'loss': 1.8868, 'learning_rate': 2e-05, 'epoch': 0.64} 64%|██████▍ | 1166/1817 [42:25:48<23:36:11, 130.52s/it] 64%|██████▍ | 1167/1817 [42:27:56<23:27:39, 129.94s/it] {'loss': 1.8976, 'learning_rate': 2e-05, 'epoch': 0.64} 64%|██████▍ | 1167/1817 [42:27:56<23:27:39, 129.94s/it] 64%|██████▍ | 1168/1817 [42:30:06<23:25:48, 129.97s/it] {'loss': 1.9341, 'learning_rate': 2e-05, 'epoch': 0.64} 64%|██████▍ | 1168/1817 [42:30:06<23:25:48, 129.97s/it] 64%|██████▍ | 1169/1817 [42:32:20<23:36:44, 131.18s/it] {'loss': 1.8666, 'learning_rate': 2e-05, 'epoch': 0.64} 64%|██████▍ | 1169/1817 [42:32:20<23:36:44, 131.18s/it] 64%|██████▍ | 1170/1817 [42:34:29<23:24:52, 130.28s/it] {'loss': 1.8925, 'learning_rate': 2e-05, 'epoch': 0.64} 64%|██████▍ | 1170/1817 [42:34:29<23:24:52, 130.28s/it] 64%|██████▍ | 1171/1817 [42:36:37<23:16:36, 129.72s/it] {'loss': 1.8777, 'learning_rate': 2e-05, 'epoch': 0.64} 64%|██████▍ | 1171/1817 [42:36:37<23:16:36, 129.72s/it] 65%|██████▍ | 1172/1817 [42:38:43<23:03:16, 128.68s/it] {'loss': 1.9076, 'learning_rate': 2e-05, 'epoch': 0.64} 65%|██████▍ | 1172/1817 [42:38:43<23:03:16, 128.68s/it] 65%|██████▍ | 1173/1817 [42:40:55<23:10:46, 129.58s/it] {'loss': 1.872, 'learning_rate': 2e-05, 'epoch': 0.65} 65%|██████▍ | 1173/1817 [42:40:55<23:10:46, 129.58s/it] 65%|██████▍ | 1174/1817 [42:43:09<23:22:45, 130.90s/it] {'loss': 1.8977, 'learning_rate': 2e-05, 'epoch': 0.65} 65%|██████▍ | 1174/1817 [42:43:09<23:22:46, 130.90s/it] 65%|██████▍ | 1175/1817 [42:45:20<23:19:49, 130.82s/it] {'loss': 1.9084, 'learning_rate': 2e-05, 'epoch': 0.65} 65%|██████▍ | 1175/1817 [42:45:20<23:19:49, 130.82s/it] 65%|██████▍ | 1176/1817 [42:47:30<23:17:07, 130.78s/it] {'loss': 1.8769, 'learning_rate': 2e-05, 'epoch': 0.65} 65%|██████▍ | 1176/1817 [42:47:30<23:17:07, 130.78s/it] 65%|██████▍ | 1177/1817 [42:49:41<23:13:56, 130.68s/it] {'loss': 1.902, 'learning_rate': 2e-05, 'epoch': 0.65} 65%|██████▍ | 1177/1817 [42:49:41<23:13:56, 130.68s/it] 65%|██████▍ | 1178/1817 [42:51:51<23:10:50, 130.60s/it] {'loss': 1.8502, 'learning_rate': 2e-05, 'epoch': 0.65} 65%|██████▍ | 1178/1817 [42:51:51<23:10:51, 130.60s/it] 65%|██████▍ | 1179/1817 [42:53:59<23:00:25, 129.82s/it] {'loss': 1.893, 'learning_rate': 2e-05, 'epoch': 0.65} 65%|██████▍ | 1179/1817 [42:53:59<23:00:25, 129.82s/it] 65%|██████▍ | 1180/1817 [42:56:11<23:05:04, 130.46s/it] {'loss': 1.8775, 'learning_rate': 2e-05, 'epoch': 0.65} 65%|██████▍ | 1180/1817 [42:56:11<23:05:04, 130.46s/it] 65%|██████▍ | 1181/1817 [42:58:24<23:09:33, 131.09s/it] {'loss': 1.8738, 'learning_rate': 2e-05, 'epoch': 0.65} 65%|██████▍ | 1181/1817 [42:58:24<23:09:33, 131.09s/it] 65%|██████▌ | 1182/1817 [43:00:33<23:00:14, 130.42s/it] {'loss': 1.8535, 'learning_rate': 2e-05, 'epoch': 0.65} 65%|██████▌ | 1182/1817 [43:00:32<23:00:14, 130.42s/it] 65%|██████▌ | 1183/1817 [43:02:44<23:01:55, 130.78s/it] {'loss': 1.8783, 'learning_rate': 2e-05, 'epoch': 0.65} 65%|██████▌ | 1183/1817 [43:02:44<23:01:55, 130.78s/it] 65%|██████▌ | 1184/1817 [43:04:56<23:02:17, 131.02s/it] {'loss': 1.8712, 'learning_rate': 2e-05, 'epoch': 0.65} 65%|██████▌ | 1184/1817 [43:04:56<23:02:17, 131.02s/it] 65%|██████▌ | 1185/1817 [43:07:07<23:00:31, 131.06s/it] {'loss': 1.9222, 'learning_rate': 2e-05, 'epoch': 0.65} 65%|██████▌ | 1185/1817 [43:07:07<23:00:31, 131.06s/it] 65%|██████▌ | 1186/1817 [43:09:19<23:00:37, 131.28s/it] {'loss': 1.8569, 'learning_rate': 2e-05, 'epoch': 0.65} 65%|██████▌ | 1186/1817 [43:09:19<23:00:37, 131.28s/it] 65%|██████▌ | 1187/1817 [43:11:30<22:58:00, 131.24s/it] {'loss': 1.9174, 'learning_rate': 2e-05, 'epoch': 0.65} 65%|██████▌ | 1187/1817 [43:11:30<22:57:57, 131.23s/it] 65%|██████▌ | 1188/1817 [43:13:40<22:51:15, 130.80s/it] {'loss': 1.8663, 'learning_rate': 2e-05, 'epoch': 0.65} 65%|██████▌ | 1188/1817 [43:13:40<22:51:12, 130.80s/it] 65%|██████▌ | 1189/1817 [43:15:54<22:58:40, 131.72s/it] {'loss': 1.9532, 'learning_rate': 2e-05, 'epoch': 0.65} 65%|██████▌ | 1189/1817 [43:15:53<22:58:38, 131.72s/it] 65%|██████▌ | 1190/1817 [43:18:03<22:49:42, 131.07s/it] {'loss': 1.893, 'learning_rate': 2e-05, 'epoch': 0.65} 65%|██████▌ | 1190/1817 [43:18:03<22:49:43, 131.07s/it] 66%|██████▌ | 1191/1817 [43:20:16<22:54:20, 131.73s/it] {'loss': 1.892, 'learning_rate': 2e-05, 'epoch': 0.66} 66%|██████▌ | 1191/1817 [43:20:16<22:54:20, 131.73s/it] 66%|██████▌ | 1192/1817 [43:22:25<22:43:40, 130.91s/it] {'loss': 1.8703, 'learning_rate': 2e-05, 'epoch': 0.66} 66%|██████▌ | 1192/1817 [43:22:25<22:43:41, 130.91s/it] 66%|██████▌ | 1193/1817 [43:24:35<22:36:54, 130.47s/it] {'loss': 1.8892, 'learning_rate': 2e-05, 'epoch': 0.66} 66%|██████▌ | 1193/1817 [43:24:35<22:36:54, 130.47s/it] 66%|██████▌ | 1194/1817 [43:26:44<22:29:05, 129.93s/it] {'loss': 1.8654, 'learning_rate': 2e-05, 'epoch': 0.66} 66%|██████▌ | 1194/1817 [43:26:43<22:29:05, 129.93s/it] 66%|██████▌ | 1195/1817 [43:28:57<22:39:06, 131.10s/it] {'loss': 1.8621, 'learning_rate': 2e-05, 'epoch': 0.66} 66%|██████▌ | 1195/1817 [43:28:57<22:39:06, 131.10s/it] 66%|██████▌ | 1196/1817 [43:31:11<22:45:14, 131.91s/it] {'loss': 1.8876, 'learning_rate': 2e-05, 'epoch': 0.66} 66%|██████▌ | 1196/1817 [43:31:11<22:45:14, 131.91s/it] 66%|██████▌ | 1197/1817 [43:33:21<22:35:49, 131.21s/it] {'loss': 1.8773, 'learning_rate': 2e-05, 'epoch': 0.66} 66%|██████▌ | 1197/1817 [43:33:21<22:35:49, 131.21s/it] 66%|██████▌ | 1198/1817 [43:35:33<22:35:48, 131.42s/it] {'loss': 1.8513, 'learning_rate': 2e-05, 'epoch': 0.66} 66%|██████▌ | 1198/1817 [43:35:32<22:35:48, 131.42s/it] 66%|██████▌ | 1199/1817 [43:37:45<22:36:16, 131.68s/it] {'loss': 1.9009, 'learning_rate': 2e-05, 'epoch': 0.66} 66%|██████▌ | 1199/1817 [43:37:45<22:36:16, 131.68s/it] 66%|██████▌ | 1200/1817 [43:39:55<22:27:55, 131.08s/it] {'loss': 1.8695, 'learning_rate': 2e-05, 'epoch': 0.66} 66%|██████▌ | 1200/1817 [43:39:54<22:27:56, 131.08s/it] 66%|██████▌ | 1201/1817 [43:42:01<22:13:01, 129.84s/it] {'loss': 1.9021, 'learning_rate': 2e-05, 'epoch': 0.66} 66%|██████▌ | 1201/1817 [43:42:01<22:13:01, 129.84s/it] 66%|██████▌ | 1202/1817 [43:44:14<22:17:18, 130.47s/it] {'loss': 1.9249, 'learning_rate': 2e-05, 'epoch': 0.66} 66%|██████▌ | 1202/1817 [43:44:13<22:17:18, 130.47s/it] 66%|██████▌ | 1203/1817 [43:46:21<22:06:32, 129.63s/it] {'loss': 1.9047, 'learning_rate': 2e-05, 'epoch': 0.66} 66%|██████▌ | 1203/1817 [43:46:21<22:06:32, 129.63s/it] 66%|██████▋ | 1204/1817 [43:48:33<22:09:51, 130.17s/it] {'loss': 1.8443, 'learning_rate': 2e-05, 'epoch': 0.66} 66%|██████▋ | 1204/1817 [43:48:32<22:09:51, 130.17s/it] 66%|██████▋ | 1205/1817 [43:50:42<22:05:07, 129.91s/it] {'loss': 1.8377, 'learning_rate': 2e-05, 'epoch': 0.66} 66%|██████▋ | 1205/1817 [43:50:42<22:05:07, 129.91s/it] 66%|██████▋ | 1206/1817 [43:52:51<22:01:40, 129.79s/it] {'loss': 1.866, 'learning_rate': 2e-05, 'epoch': 0.66} 66%|██████▋ | 1206/1817 [43:52:51<22:01:41, 129.79s/it] 66%|██████▋ | 1207/1817 [43:55:02<22:01:38, 130.00s/it] {'loss': 1.8705, 'learning_rate': 2e-05, 'epoch': 0.66} 66%|██████▋ | 1207/1817 [43:55:02<22:01:38, 130.00s/it] 66%|██████▋ | 1208/1817 [43:57:15<22:07:27, 130.78s/it] {'loss': 1.8827, 'learning_rate': 2e-05, 'epoch': 0.66} 66%|██████▋ | 1208/1817 [43:57:14<22:07:28, 130.79s/it] 67%|██████▋ | 1209/1817 [43:59:24<22:00:35, 130.32s/it] {'loss': 1.8548, 'learning_rate': 2e-05, 'epoch': 0.67} 67%|██████▋ | 1209/1817 [43:59:24<22:00:35, 130.32s/it] 67%|██████▋ | 1210/1817 [44:01:36<22:03:55, 130.87s/it] {'loss': 1.8358, 'learning_rate': 2e-05, 'epoch': 0.67} 67%|██████▋ | 1210/1817 [44:01:36<22:03:56, 130.87s/it] 67%|██████▋ | 1211/1817 [44:03:48<22:05:08, 131.20s/it] {'loss': 1.8877, 'learning_rate': 2e-05, 'epoch': 0.67} 67%|██████▋ | 1211/1817 [44:03:48<22:05:08, 131.20s/it] 67%|██████▋ | 1212/1817 [44:05:59<22:03:16, 131.23s/it] {'loss': 1.9016, 'learning_rate': 2e-05, 'epoch': 0.67} 67%|██████▋ | 1212/1817 [44:05:59<22:03:17, 131.23s/it] 67%|██████▋ | 1213/1817 [44:08:11<22:03:14, 131.45s/it] {'loss': 1.8679, 'learning_rate': 2e-05, 'epoch': 0.67} 67%|██████▋ | 1213/1817 [44:08:11<22:03:14, 131.45s/it] 67%|██████▋ | 1214/1817 [44:10:25<22:09:18, 132.27s/it] {'loss': 1.8732, 'learning_rate': 2e-05, 'epoch': 0.67} 67%|██████▋ | 1214/1817 [44:10:25<22:09:16, 132.27s/it] 67%|██████▋ | 1215/1817 [44:12:35<22:00:14, 131.59s/it] {'loss': 1.8472, 'learning_rate': 2e-05, 'epoch': 0.67} 67%|██████▋ | 1215/1817 [44:12:35<22:00:11, 131.58s/it] 67%|██████▋ | 1216/1817 [44:14:49<22:04:13, 132.20s/it] {'loss': 1.8641, 'learning_rate': 2e-05, 'epoch': 0.67} 67%|██████▋ | 1216/1817 [44:14:49<22:04:12, 132.20s/it] 67%|██████▋ | 1217/1817 [44:16:58<21:51:40, 131.17s/it] {'loss': 1.873, 'learning_rate': 2e-05, 'epoch': 0.67} 67%|██████▋ | 1217/1817 [44:16:58<21:51:41, 131.17s/it] 67%|██████▋ | 1218/1817 [44:19:09<21:49:26, 131.16s/it] {'loss': 1.9007, 'learning_rate': 2e-05, 'epoch': 0.67} 67%|██████▋ | 1218/1817 [44:19:09<21:49:26, 131.16s/it] 67%|██████▋ | 1219/1817 [44:21:20<21:47:56, 131.23s/it] {'loss': 1.8814, 'learning_rate': 2e-05, 'epoch': 0.67} 67%|██████▋ | 1219/1817 [44:21:20<21:47:56, 131.23s/it] 67%|██████▋ | 1220/1817 [44:23:31<21:42:41, 130.92s/it] {'loss': 1.8837, 'learning_rate': 2e-05, 'epoch': 0.67} 67%|██████▋ | 1220/1817 [44:23:30<21:42:41, 130.92s/it] 67%|██████▋ | 1221/1817 [44:25:38<21:31:33, 130.02s/it] {'loss': 1.9575, 'learning_rate': 2e-05, 'epoch': 0.67} 67%|██████▋ | 1221/1817 [44:25:38<21:31:33, 130.02s/it] 67%|██████▋ | 1222/1817 [44:27:51<21:35:52, 130.68s/it] {'loss': 1.892, 'learning_rate': 2e-05, 'epoch': 0.67} 67%|██████▋ | 1222/1817 [44:27:50<21:35:52, 130.68s/it] 67%|██████▋ | 1223/1817 [44:30:02<21:36:47, 130.99s/it] {'loss': 1.8732, 'learning_rate': 2e-05, 'epoch': 0.67} 67%|██████▋ | 1223/1817 [44:30:02<21:36:49, 130.99s/it] 67%|██████▋ | 1224/1817 [44:32:10<21:24:49, 130.00s/it] {'loss': 1.8502, 'learning_rate': 2e-05, 'epoch': 0.67} 67%|██████▋ | 1224/1817 [44:32:10<21:24:49, 130.00s/it] 67%|██████▋ | 1225/1817 [44:34:20<21:23:08, 130.05s/it] {'loss': 1.9164, 'learning_rate': 2e-05, 'epoch': 0.67} 67%|██████▋ | 1225/1817 [44:34:20<21:23:08, 130.05s/it] 67%|██████▋ | 1226/1817 [44:36:33<21:27:44, 130.74s/it] {'loss': 1.8431, 'learning_rate': 2e-05, 'epoch': 0.67} 67%|██████▋ | 1226/1817 [44:36:32<21:27:45, 130.74s/it] 68%|██████▊ | 1227/1817 [44:38:41<21:18:01, 129.97s/it] {'loss': 1.9021, 'learning_rate': 2e-05, 'epoch': 0.68} 68%|██████▊ | 1227/1817 [44:38:40<21:18:01, 129.97s/it] 68%|██████▊ | 1228/1817 [44:40:51<21:17:07, 130.10s/it] {'loss': 1.8617, 'learning_rate': 2e-05, 'epoch': 0.68} 68%|██████▊ | 1228/1817 [44:40:51<21:17:07, 130.10s/it] 68%|██████▊ | 1229/1817 [44:43:02<21:16:31, 130.26s/it] {'loss': 1.8558, 'learning_rate': 2e-05, 'epoch': 0.68} 68%|██████▊ | 1229/1817 [44:43:02<21:16:31, 130.26s/it] 68%|██████▊ | 1230/1817 [44:45:09<21:06:13, 129.43s/it] {'loss': 1.8585, 'learning_rate': 2e-05, 'epoch': 0.68} 68%|██████▊ | 1230/1817 [44:45:09<21:06:14, 129.43s/it] 68%|██████▊ | 1231/1817 [44:47:19<21:03:35, 129.38s/it] {'loss': 1.9039, 'learning_rate': 2e-05, 'epoch': 0.68} 68%|██████▊ | 1231/1817 [44:47:18<21:03:36, 129.38s/it] 68%|██████▊ | 1232/1817 [44:49:30<21:06:13, 129.87s/it] {'loss': 1.88, 'learning_rate': 2e-05, 'epoch': 0.68} 68%|██████▊ | 1232/1817 [44:49:29<21:06:13, 129.87s/it] 68%|██████▊ | 1233/1817 [44:51:44<21:17:01, 131.20s/it] {'loss': 1.9104, 'learning_rate': 2e-05, 'epoch': 0.68} 68%|██████▊ | 1233/1817 [44:51:44<21:17:01, 131.20s/it] 68%|██████▊ | 1234/1817 [44:53:57<21:21:04, 131.84s/it] {'loss': 1.8651, 'learning_rate': 2e-05, 'epoch': 0.68} 68%|██████▊ | 1234/1817 [44:53:57<21:21:05, 131.84s/it] 68%|██████▊ | 1235/1817 [44:56:07<21:11:42, 131.10s/it] {'loss': 1.8573, 'learning_rate': 2e-05, 'epoch': 0.68} 68%|██████▊ | 1235/1817 [44:56:06<21:11:42, 131.10s/it] 68%|██████▊ | 1236/1817 [44:58:16<21:05:21, 130.67s/it] {'loss': 1.8721, 'learning_rate': 2e-05, 'epoch': 0.68} 68%|██████▊ | 1236/1817 [44:58:16<21:05:21, 130.67s/it] 68%|██████▊ | 1237/1817 [45:00:28<21:07:04, 131.08s/it] {'loss': 1.9248, 'learning_rate': 2e-05, 'epoch': 0.68} 68%|██████▊ | 1237/1817 [45:00:28<21:07:05, 131.08s/it] 68%|██████▊ | 1238/1817 [45:02:38<21:00:22, 130.61s/it] {'loss': 1.8989, 'learning_rate': 2e-05, 'epoch': 0.68} 68%|██████▊ | 1238/1817 [45:02:38<21:00:23, 130.61s/it] 68%|██████▊ | 1239/1817 [45:04:50<21:03:07, 131.12s/it] {'loss': 1.876, 'learning_rate': 2e-05, 'epoch': 0.68} 68%|██████▊ | 1239/1817 [45:04:50<21:03:07, 131.12s/it] 68%|██████▊ | 1240/1817 [45:06:59<20:55:15, 130.53s/it] {'loss': 1.9199, 'learning_rate': 2e-05, 'epoch': 0.68} 68%|██████▊ | 1240/1817 [45:06:59<20:55:16, 130.53s/it] 68%|██████▊ | 1241/1817 [45:09:12<20:59:31, 131.20s/it] {'loss': 1.8958, 'learning_rate': 2e-05, 'epoch': 0.68} 68%|██████▊ | 1241/1817 [45:09:12<20:59:31, 131.20s/it] 68%|██████▊ | 1242/1817 [45:11:22<20:55:38, 131.02s/it] {'loss': 1.8644, 'learning_rate': 2e-05, 'epoch': 0.68} 68%|██████▊ | 1242/1817 [45:11:22<20:55:38, 131.02s/it] 68%|██████▊ | 1243/1817 [45:13:36<21:00:54, 131.80s/it] {'loss': 1.8436, 'learning_rate': 2e-05, 'epoch': 0.68} 68%|██████▊ | 1243/1817 [45:13:36<21:00:52, 131.80s/it] 68%|██████▊ | 1244/1817 [45:15:45<20:48:38, 130.75s/it] {'loss': 1.8671, 'learning_rate': 2e-05, 'epoch': 0.68} 68%|██████▊ | 1244/1817 [45:15:44<20:48:37, 130.75s/it] 69%|██████▊ | 1245/1817 [45:17:58<20:55:11, 131.66s/it] {'loss': 1.8905, 'learning_rate': 2e-05, 'epoch': 0.69} 69%|██████▊ | 1245/1817 [45:17:58<20:55:12, 131.67s/it] 69%|██████▊ | 1246/1817 [45:20:08<20:47:19, 131.07s/it] {'loss': 1.9101, 'learning_rate': 2e-05, 'epoch': 0.69} 69%|██████▊ | 1246/1817 [45:20:08<20:47:19, 131.07s/it] 69%|██████▊ | 1247/1817 [45:22:18<20:44:06, 130.96s/it] {'loss': 1.8963, 'learning_rate': 2e-05, 'epoch': 0.69} 69%|██████▊ | 1247/1817 [45:22:18<20:44:06, 130.96s/it] 69%|██████▊ | 1248/1817 [45:24:29<20:40:40, 130.83s/it] {'loss': 1.8828, 'learning_rate': 2e-05, 'epoch': 0.69} 69%|██████▊ | 1248/1817 [45:24:29<20:40:40, 130.83s/it] 69%|██████▊ | 1249/1817 [45:26:38<20:33:59, 130.35s/it] {'loss': 1.8661, 'learning_rate': 2e-05, 'epoch': 0.69} 69%|██████▊ | 1249/1817 [45:26:38<20:33:59, 130.35s/it] 69%|██████▉ | 1250/1817 [45:28:46<20:24:00, 129.52s/it] {'loss': 1.884, 'learning_rate': 2e-05, 'epoch': 0.69} 69%|██████▉ | 1250/1817 [45:28:46<20:24:00, 129.52s/it] 69%|██████▉ | 1251/1817 [45:30:56<20:22:20, 129.58s/it] {'loss': 1.8528, 'learning_rate': 2e-05, 'epoch': 0.69} 69%|██████▉ | 1251/1817 [45:30:56<20:22:20, 129.58s/it] 69%|██████▉ | 1252/1817 [45:33:09<20:32:09, 130.85s/it] {'loss': 1.9145, 'learning_rate': 2e-05, 'epoch': 0.69} 69%|██████▉ | 1252/1817 [45:33:09<20:32:09, 130.85s/it] 69%|██████▉ | 1253/1817 [45:35:17<20:19:35, 129.74s/it] {'loss': 1.8783, 'learning_rate': 2e-05, 'epoch': 0.69} 69%|██████▉ | 1253/1817 [45:35:16<20:19:35, 129.74s/it] 69%|██████▉ | 1254/1817 [45:37:24<20:09:56, 128.95s/it] {'loss': 1.894, 'learning_rate': 2e-05, 'epoch': 0.69} 69%|██████▉ | 1254/1817 [45:37:24<20:09:56, 128.95s/it] 69%|██████▉ | 1255/1817 [45:39:37<20:21:05, 130.37s/it] {'loss': 1.8741, 'learning_rate': 2e-05, 'epoch': 0.69} 69%|██████▉ | 1255/1817 [45:39:37<20:21:05, 130.37s/it] 69%|██████▉ | 1256/1817 [45:41:47<20:18:06, 130.28s/it] {'loss': 1.8616, 'learning_rate': 2e-05, 'epoch': 0.69} 69%|██████▉ | 1256/1817 [45:41:47<20:18:06, 130.28s/it] 69%|██████▉ | 1257/1817 [45:43:59<20:18:35, 130.56s/it] {'loss': 1.8569, 'learning_rate': 2e-05, 'epoch': 0.69} 69%|██████▉ | 1257/1817 [45:43:59<20:18:35, 130.56s/it] 69%|██████▉ | 1258/1817 [45:46:10<20:17:58, 130.73s/it] {'loss': 1.9009, 'learning_rate': 2e-05, 'epoch': 0.69} 69%|██████▉ | 1258/1817 [45:46:10<20:17:58, 130.73s/it] 69%|██████▉ | 1259/1817 [45:48:20<20:15:11, 130.67s/it] {'loss': 1.8648, 'learning_rate': 2e-05, 'epoch': 0.69} 69%|██████▉ | 1259/1817 [45:48:20<20:15:11, 130.67s/it] 69%|██████▉ | 1260/1817 [45:50:27<20:02:53, 129.58s/it] {'loss': 1.8977, 'learning_rate': 2e-05, 'epoch': 0.69} 69%|██████▉ | 1260/1817 [45:50:27<20:02:53, 129.58s/it] 69%|██████▉ | 1261/1817 [45:52:40<20:08:05, 130.37s/it] {'loss': 1.8856, 'learning_rate': 2e-05, 'epoch': 0.69} 69%|██████▉ | 1261/1817 [45:52:39<20:08:05, 130.37s/it] 69%|██████▉ | 1262/1817 [45:54:53<20:15:04, 131.36s/it] {'loss': 1.8776, 'learning_rate': 2e-05, 'epoch': 0.69} 69%|██████▉ | 1262/1817 [45:54:53<20:15:04, 131.36s/it] 70%|██████▉ | 1263/1817 [45:57:06<20:15:44, 131.67s/it] {'loss': 1.8862, 'learning_rate': 2e-05, 'epoch': 0.69} 70%|██████▉ | 1263/1817 [45:57:06<20:15:44, 131.67s/it] 70%|██████▉ | 1264/1817 [45:59:18<20:15:30, 131.88s/it] {'loss': 1.8781, 'learning_rate': 2e-05, 'epoch': 0.7} 70%|██████▉ | 1264/1817 [45:59:18<20:15:30, 131.88s/it] 70%|██████▉ | 1265/1817 [46:01:30<20:13:17, 131.88s/it] {'loss': 1.8614, 'learning_rate': 2e-05, 'epoch': 0.7} 70%|██████▉ | 1265/1817 [46:01:30<20:13:17, 131.88s/it] 70%|██████▉ | 1266/1817 [46:03:41<20:09:02, 131.66s/it] {'loss': 1.8786, 'learning_rate': 2e-05, 'epoch': 0.7} 70%|██████▉ | 1266/1817 [46:03:41<20:09:02, 131.66s/it] 70%|██████▉ | 1267/1817 [46:05:58<20:20:19, 133.13s/it] {'loss': 1.8319, 'learning_rate': 2e-05, 'epoch': 0.7} 70%|██████▉ | 1267/1817 [46:05:57<20:20:20, 133.13s/it] 70%|██████▉ | 1268/1817 [46:08:09<20:14:09, 132.69s/it] {'loss': 1.8941, 'learning_rate': 2e-05, 'epoch': 0.7} 70%|██████▉ | 1268/1817 [46:08:09<20:14:09, 132.70s/it] 70%|██████▉ | 1269/1817 [46:10:20<20:07:33, 132.21s/it] {'loss': 1.8386, 'learning_rate': 2e-05, 'epoch': 0.7} 70%|██████▉ | 1269/1817 [46:10:20<20:07:32, 132.21s/it] 70%|██████▉ | 1270/1817 [46:12:30<19:59:10, 131.54s/it] {'loss': 1.8694, 'learning_rate': 2e-05, 'epoch': 0.7} 70%|██████▉ | 1270/1817 [46:12:30<19:59:07, 131.53s/it] 70%|██████▉ | 1271/1817 [46:14:42<19:57:43, 131.62s/it] {'loss': 1.8487, 'learning_rate': 2e-05, 'epoch': 0.7} 70%|██████▉ | 1271/1817 [46:14:42<19:57:43, 131.62s/it] 70%|███████ | 1272/1817 [46:16:50<19:44:48, 130.44s/it] {'loss': 1.8761, 'learning_rate': 2e-05, 'epoch': 0.7} 70%|███████ | 1272/1817 [46:16:50<19:44:48, 130.44s/it] 70%|███████ | 1273/1817 [46:19:00<19:41:39, 130.33s/it] {'loss': 1.8745, 'learning_rate': 2e-05, 'epoch': 0.7} 70%|███████ | 1273/1817 [46:19:00<19:41:39, 130.33s/it] 70%|███████ | 1274/1817 [46:21:11<19:42:12, 130.63s/it] {'loss': 1.8708, 'learning_rate': 2e-05, 'epoch': 0.7} 70%|███████ | 1274/1817 [46:21:11<19:42:12, 130.63s/it] 70%|███████ | 1275/1817 [46:23:24<19:46:31, 131.35s/it] {'loss': 1.864, 'learning_rate': 2e-05, 'epoch': 0.7} 70%|███████ | 1275/1817 [46:23:24<19:46:31, 131.35s/it] 70%|███████ | 1276/1817 [46:25:35<19:42:30, 131.15s/it] {'loss': 1.8706, 'learning_rate': 2e-05, 'epoch': 0.7} 70%|███████ | 1276/1817 [46:25:35<19:42:30, 131.15s/it] 70%|███████ | 1277/1817 [46:27:49<19:47:32, 131.95s/it] {'loss': 1.8689, 'learning_rate': 2e-05, 'epoch': 0.7} 70%|███████ | 1277/1817 [46:27:49<19:47:32, 131.95s/it] 70%|███████ | 1278/1817 [46:29:57<19:35:40, 130.87s/it] {'loss': 1.8621, 'learning_rate': 2e-05, 'epoch': 0.7} 70%|███████ | 1278/1817 [46:29:57<19:35:40, 130.87s/it] 70%|███████ | 1279/1817 [46:32:08<19:34:20, 130.97s/it] {'loss': 1.9056, 'learning_rate': 2e-05, 'epoch': 0.7} 70%|███████ | 1279/1817 [46:32:08<19:34:20, 130.97s/it] 70%|███████ | 1280/1817 [46:34:24<19:44:53, 132.39s/it] {'loss': 1.8715, 'learning_rate': 2e-05, 'epoch': 0.7} 70%|███████ | 1280/1817 [46:34:24<19:44:53, 132.39s/it] 71%|███████ | 1281/1817 [46:36:37<19:43:04, 132.43s/it] {'loss': 1.8655, 'learning_rate': 2e-05, 'epoch': 0.7} 71%|███████ | 1281/1817 [46:36:36<19:43:04, 132.43s/it] 71%|███████ | 1282/1817 [46:38:49<19:39:54, 132.33s/it] {'loss': 1.8723, 'learning_rate': 2e-05, 'epoch': 0.71} 71%|███████ | 1282/1817 [46:38:48<19:39:54, 132.33s/it] 71%|███████ | 1283/1817 [46:40:59<19:32:39, 131.76s/it] {'loss': 1.944, 'learning_rate': 2e-05, 'epoch': 0.71} 71%|███████ | 1283/1817 [46:40:59<19:32:40, 131.76s/it] 71%|███████ | 1284/1817 [46:43:09<19:25:12, 131.17s/it] {'loss': 1.8464, 'learning_rate': 2e-05, 'epoch': 0.71} 71%|███████ | 1284/1817 [46:43:09<19:25:12, 131.17s/it] 71%|███████ | 1285/1817 [46:45:19<19:19:49, 130.81s/it] {'loss': 1.895, 'learning_rate': 2e-05, 'epoch': 0.71} 71%|███████ | 1285/1817 [46:45:19<19:19:49, 130.81s/it] 71%|███████ | 1286/1817 [46:47:30<19:18:51, 130.95s/it] {'loss': 1.8915, 'learning_rate': 2e-05, 'epoch': 0.71} 71%|███████ | 1286/1817 [46:47:30<19:18:51, 130.95s/it] 71%|███████ | 1287/1817 [46:49:42<19:18:23, 131.14s/it] {'loss': 1.918, 'learning_rate': 2e-05, 'epoch': 0.71} 71%|███████ | 1287/1817 [46:49:42<19:18:23, 131.14s/it] 71%|███████ | 1288/1817 [46:51:56<19:24:29, 132.08s/it] {'loss': 1.8647, 'learning_rate': 2e-05, 'epoch': 0.71} 71%|███████ | 1288/1817 [46:51:56<19:24:29, 132.08s/it] 71%|███████ | 1289/1817 [46:54:07<19:18:23, 131.64s/it] {'loss': 1.8687, 'learning_rate': 2e-05, 'epoch': 0.71} 71%|███████ | 1289/1817 [46:54:06<19:18:23, 131.64s/it] 71%|███████ | 1290/1817 [46:56:19<19:18:11, 131.86s/it] {'loss': 1.8776, 'learning_rate': 2e-05, 'epoch': 0.71} 71%|███████ | 1290/1817 [46:56:19<19:18:11, 131.86s/it] 71%|███████ | 1291/1817 [46:58:29<19:11:30, 131.35s/it] {'loss': 1.889, 'learning_rate': 2e-05, 'epoch': 0.71} 71%|███████ | 1291/1817 [46:58:29<19:11:30, 131.35s/it] 71%|███████ | 1292/1817 [47:00:39<19:05:46, 130.95s/it] {'loss': 1.8872, 'learning_rate': 2e-05, 'epoch': 0.71} 71%|███████ | 1292/1817 [47:00:39<19:05:46, 130.95s/it] 71%|███████ | 1293/1817 [47:02:53<19:10:02, 131.68s/it] {'loss': 1.8969, 'learning_rate': 2e-05, 'epoch': 0.71} 71%|███████ | 1293/1817 [47:02:52<19:10:02, 131.68s/it] 71%|███████ | 1294/1817 [47:05:04<19:06:32, 131.53s/it] {'loss': 1.8585, 'learning_rate': 2e-05, 'epoch': 0.71} 71%|███████ | 1294/1817 [47:05:04<19:06:33, 131.54s/it] 71%|███████▏ | 1295/1817 [47:07:13<18:57:39, 130.77s/it] {'loss': 1.9201, 'learning_rate': 2e-05, 'epoch': 0.71} 71%|███████▏ | 1295/1817 [47:07:13<18:57:39, 130.77s/it] 71%|███████▏ | 1296/1817 [47:09:25<19:00:15, 131.32s/it] {'loss': 1.8733, 'learning_rate': 2e-05, 'epoch': 0.71} 71%|███████▏ | 1296/1817 [47:09:25<19:00:15, 131.32s/it] 71%|███████▏ | 1297/1817 [47:11:33<18:48:37, 130.23s/it] {'loss': 1.8838, 'learning_rate': 2e-05, 'epoch': 0.71} 71%|███████▏ | 1297/1817 [47:11:33<18:48:35, 130.22s/it] 71%|███████▏ | 1298/1817 [47:13:44<18:48:39, 130.48s/it] {'loss': 1.8795, 'learning_rate': 2e-05, 'epoch': 0.71} 71%|███████▏ | 1298/1817 [47:13:44<18:48:37, 130.48s/it] 71%|███████▏ | 1299/1817 [47:15:53<18:43:05, 130.09s/it] {'loss': 1.8481, 'learning_rate': 2e-05, 'epoch': 0.71} 71%|███████▏ | 1299/1817 [47:15:53<18:43:04, 130.09s/it] 72%|███████▏ | 1300/1817 [47:18:01<18:34:48, 129.38s/it] {'loss': 1.9141, 'learning_rate': 2e-05, 'epoch': 0.72} 72%|███████▏ | 1300/1817 [47:18:01<18:34:48, 129.38s/it] 72%|███████▏ | 1301/1817 [47:20:11<18:35:01, 129.65s/it] {'loss': 1.9117, 'learning_rate': 2e-05, 'epoch': 0.72} 72%|███████▏ | 1301/1817 [47:20:11<18:35:01, 129.65s/it] 72%|███████▏ | 1302/1817 [47:22:22<18:35:11, 129.92s/it] {'loss': 1.8819, 'learning_rate': 2e-05, 'epoch': 0.72} 72%|███████▏ | 1302/1817 [47:22:22<18:35:11, 129.93s/it] 72%|███████▏ | 1303/1817 [47:24:40<18:53:10, 132.28s/it] {'loss': 1.8715, 'learning_rate': 2e-05, 'epoch': 0.72} 72%|███████▏ | 1303/1817 [47:24:39<18:53:10, 132.28s/it] 72%|███████▏ | 1304/1817 [47:26:47<18:39:36, 130.95s/it] {'loss': 1.8756, 'learning_rate': 2e-05, 'epoch': 0.72} 72%|███████▏ | 1304/1817 [47:26:47<18:39:37, 130.95s/it] 72%|███████▏ | 1305/1817 [47:28:59<18:38:03, 131.02s/it] {'loss': 1.872, 'learning_rate': 2e-05, 'epoch': 0.72} 72%|███████▏ | 1305/1817 [47:28:58<18:38:03, 131.02s/it] 72%|███████▏ | 1306/1817 [47:31:10<18:37:43, 131.24s/it] {'loss': 1.8981, 'learning_rate': 2e-05, 'epoch': 0.72} 72%|███████▏ | 1306/1817 [47:31:10<18:37:44, 131.24s/it] 72%|███████▏ | 1307/1817 [47:33:24<18:40:40, 131.84s/it] {'loss': 1.9005, 'learning_rate': 2e-05, 'epoch': 0.72} 72%|███████▏ | 1307/1817 [47:33:23<18:40:40, 131.84s/it] 72%|███████▏ | 1308/1817 [47:35:33<18:31:47, 131.06s/it] {'loss': 1.8732, 'learning_rate': 2e-05, 'epoch': 0.72} 72%|███████▏ | 1308/1817 [47:35:33<18:31:48, 131.06s/it] 72%|███████▏ | 1309/1817 [47:37:40<18:20:34, 129.99s/it] {'loss': 1.8604, 'learning_rate': 2e-05, 'epoch': 0.72} 72%|███████▏ | 1309/1817 [47:37:40<18:20:34, 129.99s/it] 72%|███████▏ | 1310/1817 [47:39:55<18:30:25, 131.41s/it] {'loss': 1.8578, 'learning_rate': 2e-05, 'epoch': 0.72} 72%|███████▏ | 1310/1817 [47:39:55<18:30:25, 131.41s/it] 72%|███████▏ | 1311/1817 [47:42:06<18:28:08, 131.40s/it] {'loss': 1.8708, 'learning_rate': 2e-05, 'epoch': 0.72} 72%|███████▏ | 1311/1817 [47:42:06<18:28:08, 131.40s/it] 72%|███████▏ | 1312/1817 [47:44:16<18:21:22, 130.86s/it] {'loss': 1.8824, 'learning_rate': 2e-05, 'epoch': 0.72} 72%|███████▏ | 1312/1817 [47:44:16<18:21:22, 130.86s/it] 72%|███████▏ | 1313/1817 [47:46:26<18:17:49, 130.69s/it] {'loss': 1.8953, 'learning_rate': 2e-05, 'epoch': 0.72} 72%|███████▏ | 1313/1817 [47:46:26<18:17:49, 130.69s/it] 72%|███████▏ | 1314/1817 [47:48:39<18:21:41, 131.41s/it] {'loss': 1.8471, 'learning_rate': 2e-05, 'epoch': 0.72} 72%|███████▏ | 1314/1817 [47:48:39<18:21:41, 131.41s/it] 72%|███████▏ | 1315/1817 [47:50:52<18:23:19, 131.87s/it] {'loss': 1.8964, 'learning_rate': 2e-05, 'epoch': 0.72} 72%|███████▏ | 1315/1817 [47:50:52<18:23:19, 131.87s/it] 72%|███████▏ | 1316/1817 [47:53:04<18:19:35, 131.69s/it] {'loss': 1.8698, 'learning_rate': 2e-05, 'epoch': 0.72} 72%|███████▏ | 1316/1817 [47:53:03<18:19:35, 131.69s/it] 72%|███████▏ | 1317/1817 [47:55:17<18:20:31, 132.06s/it] {'loss': 1.8725, 'learning_rate': 2e-05, 'epoch': 0.72} 72%|███████▏ | 1317/1817 [47:55:16<18:20:31, 132.06s/it] 73%|███████▎ | 1318/1817 [47:57:31<18:25:03, 132.87s/it] {'loss': 1.8585, 'learning_rate': 2e-05, 'epoch': 0.73} 73%|███████▎ | 1318/1817 [47:57:31<18:25:03, 132.87s/it] 73%|███████▎ | 1319/1817 [47:59:44<18:22:19, 132.81s/it] {'loss': 1.9112, 'learning_rate': 2e-05, 'epoch': 0.73} 73%|███████▎ | 1319/1817 [47:59:44<18:22:20, 132.81s/it] 73%|███████▎ | 1320/1817 [48:01:55<18:16:34, 132.38s/it] {'loss': 1.8853, 'learning_rate': 2e-05, 'epoch': 0.73} 73%|███████▎ | 1320/1817 [48:01:55<18:16:34, 132.38s/it] 73%|███████▎ | 1321/1817 [48:04:06<18:09:27, 131.79s/it] {'loss': 1.8723, 'learning_rate': 2e-05, 'epoch': 0.73} 73%|███████▎ | 1321/1817 [48:04:06<18:09:27, 131.79s/it] 73%|███████▎ | 1322/1817 [48:06:16<18:02:10, 131.17s/it] {'loss': 1.8844, 'learning_rate': 2e-05, 'epoch': 0.73} 73%|███████▎ | 1322/1817 [48:06:15<18:02:10, 131.17s/it] 73%|███████▎ | 1323/1817 [48:08:25<17:54:36, 130.52s/it] {'loss': 1.8858, 'learning_rate': 2e-05, 'epoch': 0.73} 73%|███████▎ | 1323/1817 [48:08:24<17:54:36, 130.52s/it] 73%|███████▎ | 1324/1817 [48:10:32<17:44:34, 129.56s/it] {'loss': 1.8681, 'learning_rate': 2e-05, 'epoch': 0.73} 73%|███████▎ | 1324/1817 [48:10:32<17:44:34, 129.56s/it] 73%|███████▎ | 1325/1817 [48:12:40<17:37:39, 128.98s/it] {'loss': 1.8607, 'learning_rate': 2e-05, 'epoch': 0.73} 73%|███████▎ | 1325/1817 [48:12:39<17:37:37, 128.98s/it] 73%|███████▎ | 1326/1817 [48:14:52<17:44:24, 130.07s/it] {'loss': 1.8898, 'learning_rate': 2e-05, 'epoch': 0.73} 73%|███████▎ | 1326/1817 [48:14:52<17:44:24, 130.07s/it] 73%|███████▎ | 1327/1817 [48:17:06<17:51:39, 131.22s/it] {'loss': 1.8886, 'learning_rate': 2e-05, 'epoch': 0.73} 73%|███████▎ | 1327/1817 [48:17:06<17:51:39, 131.22s/it] 73%|███████▎ | 1328/1817 [48:19:17<17:48:56, 131.16s/it] {'loss': 1.8718, 'learning_rate': 2e-05, 'epoch': 0.73} 73%|███████▎ | 1328/1817 [48:19:17<17:48:56, 131.16s/it] 73%|███████▎ | 1329/1817 [48:21:25<17:39:25, 130.26s/it] {'loss': 1.8453, 'learning_rate': 2e-05, 'epoch': 0.73} 73%|███████▎ | 1329/1817 [48:21:25<17:39:25, 130.26s/it] 73%|███████▎ | 1330/1817 [48:23:35<17:36:55, 130.22s/it] {'loss': 1.8659, 'learning_rate': 2e-05, 'epoch': 0.73} 73%|███████▎ | 1330/1817 [48:23:35<17:36:56, 130.22s/it] 73%|███████▎ | 1331/1817 [48:25:45<17:34:58, 130.24s/it] {'loss': 1.8908, 'learning_rate': 2e-05, 'epoch': 0.73} 73%|███████▎ | 1331/1817 [48:25:45<17:34:58, 130.24s/it] 73%|███████▎ | 1332/1817 [48:27:57<17:35:56, 130.63s/it] {'loss': 1.8787, 'learning_rate': 2e-05, 'epoch': 0.73} 73%|███████▎ | 1332/1817 [48:27:57<17:35:57, 130.63s/it] 73%|███████▎ | 1333/1817 [48:30:06<17:30:14, 130.19s/it] {'loss': 1.8953, 'learning_rate': 2e-05, 'epoch': 0.73} 73%|███████▎ | 1333/1817 [48:30:06<17:30:15, 130.20s/it] 73%|███████▎ | 1334/1817 [48:32:17<17:29:44, 130.40s/it] {'loss': 1.8546, 'learning_rate': 2e-05, 'epoch': 0.73} 73%|███████▎ | 1334/1817 [48:32:17<17:29:46, 130.41s/it] 73%|███████▎ | 1335/1817 [48:34:30<17:32:04, 130.96s/it] {'loss': 1.8683, 'learning_rate': 2e-05, 'epoch': 0.73} 73%|███████▎ | 1335/1817 [48:34:29<17:32:04, 130.96s/it] 74%|███████▎ | 1336/1817 [48:36:41<17:31:56, 131.22s/it] {'loss': 1.8966, 'learning_rate': 2e-05, 'epoch': 0.74} 74%|███████▎ | 1336/1817 [48:36:41<17:31:56, 131.22s/it] 74%|███████▎ | 1337/1817 [48:38:52<17:29:28, 131.18s/it] {'loss': 1.8854, 'learning_rate': 2e-05, 'epoch': 0.74} 74%|███████▎ | 1337/1817 [48:38:52<17:29:28, 131.18s/it] 74%|███████▎ | 1338/1817 [48:41:00<17:19:39, 130.23s/it] {'loss': 1.849, 'learning_rate': 2e-05, 'epoch': 0.74} 74%|███████▎ | 1338/1817 [48:41:00<17:19:40, 130.23s/it] 74%|███████▎ | 1339/1817 [48:43:12<17:21:42, 130.76s/it] {'loss': 1.8665, 'learning_rate': 2e-05, 'epoch': 0.74} 74%|███████▎ | 1339/1817 [48:43:12<17:21:43, 130.76s/it] 74%|███████▎ | 1340/1817 [48:45:21<17:13:48, 130.04s/it] {'loss': 1.8751, 'learning_rate': 2e-05, 'epoch': 0.74} 74%|███████▎ | 1340/1817 [48:45:21<17:13:48, 130.04s/it] 74%|███████▍ | 1341/1817 [48:47:29<17:07:13, 129.48s/it] {'loss': 1.8637, 'learning_rate': 2e-05, 'epoch': 0.74} 74%|███████▍ | 1341/1817 [48:47:29<17:07:14, 129.48s/it] 74%|███████▍ | 1342/1817 [48:49:39<17:05:37, 129.55s/it] {'loss': 1.9077, 'learning_rate': 2e-05, 'epoch': 0.74} 74%|███████▍ | 1342/1817 [48:49:38<17:05:37, 129.55s/it] 74%|███████▍ | 1343/1817 [48:51:49<17:06:25, 129.93s/it] {'loss': 1.8622, 'learning_rate': 2e-05, 'epoch': 0.74} 74%|███████▍ | 1343/1817 [48:51:49<17:06:25, 129.93s/it] 74%|███████▍ | 1344/1817 [48:54:02<17:11:05, 130.79s/it] {'loss': 1.9119, 'learning_rate': 2e-05, 'epoch': 0.74} 74%|███████▍ | 1344/1817 [48:54:02<17:11:05, 130.79s/it] 74%|███████▍ | 1345/1817 [48:56:16<17:16:54, 131.81s/it] {'loss': 1.847, 'learning_rate': 2e-05, 'epoch': 0.74} 74%|███████▍ | 1345/1817 [48:56:16<17:16:54, 131.81s/it] 74%|███████▍ | 1346/1817 [48:58:27<17:11:20, 131.38s/it] {'loss': 1.886, 'learning_rate': 2e-05, 'epoch': 0.74} 74%|███████▍ | 1346/1817 [48:58:27<17:11:20, 131.38s/it] 74%|███████▍ | 1347/1817 [49:00:37<17:05:38, 130.93s/it] {'loss': 1.9127, 'learning_rate': 2e-05, 'epoch': 0.74} 74%|███████▍ | 1347/1817 [49:00:37<17:05:38, 130.93s/it] 74%|███████▍ | 1348/1817 [49:02:47<17:01:00, 130.62s/it] {'loss': 1.9164, 'learning_rate': 2e-05, 'epoch': 0.74} 74%|███████▍ | 1348/1817 [49:02:46<17:01:00, 130.62s/it] 74%|███████▍ | 1349/1817 [49:05:02<17:11:05, 132.19s/it] {'loss': 1.8546, 'learning_rate': 2e-05, 'epoch': 0.74} 74%|███████▍ | 1349/1817 [49:05:02<17:11:05, 132.19s/it] 74%|███████▍ | 1350/1817 [49:07:14<17:06:33, 131.89s/it] {'loss': 1.841, 'learning_rate': 2e-05, 'epoch': 0.74} 74%|███████▍ | 1350/1817 [49:07:13<17:06:33, 131.89s/it] 74%|███████▍ | 1351/1817 [49:09:23<16:58:53, 131.19s/it] {'loss': 1.8822, 'learning_rate': 2e-05, 'epoch': 0.74} 74%|███████▍ | 1351/1817 [49:09:23<16:58:47, 131.17s/it] 74%|███████▍ | 1352/1817 [49:11:35<16:57:27, 131.28s/it] {'loss': 1.9006, 'learning_rate': 2e-05, 'epoch': 0.74} 74%|███████▍ | 1352/1817 [49:11:35<16:57:27, 131.29s/it] 74%|███████▍ | 1353/1817 [49:13:42<16:45:32, 130.03s/it] {'loss': 1.8814, 'learning_rate': 2e-05, 'epoch': 0.74} 74%|███████▍ | 1353/1817 [49:13:42<16:45:31, 130.02s/it] 75%|███████▍ | 1354/1817 [49:15:53<16:46:27, 130.43s/it] {'loss': 1.8943, 'learning_rate': 2e-05, 'epoch': 0.75} 75%|███████▍ | 1354/1817 [49:15:53<16:46:26, 130.42s/it] 75%|███████▍ | 1355/1817 [49:18:00<16:36:40, 129.44s/it] {'loss': 1.8307, 'learning_rate': 2e-05, 'epoch': 0.75} 75%|███████▍ | 1355/1817 [49:18:00<16:36:40, 129.44s/it] 75%|███████▍ | 1356/1817 [49:20:11<16:38:31, 129.96s/it] {'loss': 1.8514, 'learning_rate': 2e-05, 'epoch': 0.75} 75%|███████▍ | 1356/1817 [49:20:11<16:38:31, 129.96s/it] 75%|███████▍ | 1357/1817 [49:22:22<16:37:36, 130.12s/it] {'loss': 1.8833, 'learning_rate': 2e-05, 'epoch': 0.75} 75%|███████▍ | 1357/1817 [49:22:22<16:37:36, 130.12s/it] 75%|███████▍ | 1358/1817 [49:24:35<16:41:07, 130.87s/it] {'loss': 1.8592, 'learning_rate': 2e-05, 'epoch': 0.75} 75%|███████▍ | 1358/1817 [49:24:34<16:41:07, 130.87s/it] 75%|███████▍ | 1359/1817 [49:26:47<16:43:20, 131.44s/it] {'loss': 1.876, 'learning_rate': 2e-05, 'epoch': 0.75} 75%|███████▍ | 1359/1817 [49:26:47<16:43:20, 131.44s/it] 75%|███████▍ | 1360/1817 [49:28:56<16:34:45, 130.60s/it] {'loss': 1.826, 'learning_rate': 2e-05, 'epoch': 0.75} 75%|███████▍ | 1360/1817 [49:28:56<16:34:45, 130.60s/it] 75%|███████▍ | 1361/1817 [49:31:09<16:37:16, 131.22s/it] {'loss': 1.8787, 'learning_rate': 2e-05, 'epoch': 0.75} 75%|███████▍ | 1361/1817 [49:31:08<16:37:17, 131.22s/it] 75%|███████▍ | 1362/1817 [49:33:21<16:37:17, 131.51s/it] {'loss': 1.8438, 'learning_rate': 2e-05, 'epoch': 0.75} 75%|███████▍ | 1362/1817 [49:33:21<16:37:18, 131.51s/it] 75%|███████▌ | 1363/1817 [49:35:31<16:32:12, 131.13s/it] {'loss': 1.9007, 'learning_rate': 2e-05, 'epoch': 0.75} 75%|███████▌ | 1363/1817 [49:35:31<16:32:13, 131.13s/it] 75%|███████▌ | 1364/1817 [49:37:44<16:35:17, 131.83s/it] {'loss': 1.8553, 'learning_rate': 2e-05, 'epoch': 0.75} 75%|███████▌ | 1364/1817 [49:37:44<16:35:17, 131.83s/it] 75%|███████▌ | 1365/1817 [49:39:56<16:32:41, 131.77s/it] {'loss': 1.873, 'learning_rate': 2e-05, 'epoch': 0.75} 75%|███████▌ | 1365/1817 [49:39:56<16:32:41, 131.77s/it] 75%|███████▌ | 1366/1817 [49:42:06<16:26:24, 131.23s/it] {'loss': 1.8543, 'learning_rate': 2e-05, 'epoch': 0.75} 75%|███████▌ | 1366/1817 [49:42:06<16:26:24, 131.23s/it] 75%|███████▌ | 1367/1817 [49:44:13<16:15:39, 130.09s/it] {'loss': 1.8754, 'learning_rate': 2e-05, 'epoch': 0.75} 75%|███████▌ | 1367/1817 [49:44:13<16:15:39, 130.09s/it] 75%|███████▌ | 1368/1817 [49:46:24<16:14:45, 130.26s/it] {'loss': 1.8999, 'learning_rate': 2e-05, 'epoch': 0.75} 75%|███████▌ | 1368/1817 [49:46:24<16:14:45, 130.26s/it] 75%|███████▌ | 1369/1817 [49:48:35<16:13:12, 130.34s/it] {'loss': 1.8657, 'learning_rate': 2e-05, 'epoch': 0.75} 75%|███████▌ | 1369/1817 [49:48:35<16:13:12, 130.34s/it] 75%|███████▌ | 1370/1817 [49:50:48<16:17:50, 131.25s/it] {'loss': 1.872, 'learning_rate': 2e-05, 'epoch': 0.75} 75%|███████▌ | 1370/1817 [49:50:48<16:17:50, 131.25s/it] 75%|███████▌ | 1371/1817 [49:52:57<16:11:11, 130.65s/it] {'loss': 1.8836, 'learning_rate': 2e-05, 'epoch': 0.75} 75%|███████▌ | 1371/1817 [49:52:57<16:11:11, 130.65s/it] 76%|███████▌ | 1372/1817 [49:55:08<16:09:41, 130.74s/it] {'loss': 1.8686, 'learning_rate': 2e-05, 'epoch': 0.75} 76%|███████▌ | 1372/1817 [49:55:08<16:09:41, 130.74s/it] 76%|███████▌ | 1373/1817 [49:57:21<16:10:47, 131.19s/it] {'loss': 1.9047, 'learning_rate': 2e-05, 'epoch': 0.76} 76%|███████▌ | 1373/1817 [49:57:20<16:10:47, 131.19s/it] 76%|███████▌ | 1374/1817 [49:59:36<16:17:52, 132.44s/it] {'loss': 1.8771, 'learning_rate': 2e-05, 'epoch': 0.76} 76%|███████▌ | 1374/1817 [49:59:36<16:17:52, 132.44s/it] 76%|███████▌ | 1375/1817 [50:01:46<16:10:16, 131.71s/it] {'loss': 1.8535, 'learning_rate': 2e-05, 'epoch': 0.76} 76%|███████▌ | 1375/1817 [50:01:46<16:10:16, 131.71s/it] 76%|███████▌ | 1376/1817 [50:04:02<16:17:36, 133.01s/it] {'loss': 1.8832, 'learning_rate': 2e-05, 'epoch': 0.76} 76%|███████▌ | 1376/1817 [50:04:02<16:17:36, 133.01s/it] 76%|███████▌ | 1377/1817 [50:06:13<16:10:57, 132.40s/it] {'loss': 1.9163, 'learning_rate': 2e-05, 'epoch': 0.76} 76%|███████▌ | 1377/1817 [50:06:13<16:10:58, 132.40s/it] 76%|███████▌ | 1378/1817 [50:08:25<16:07:57, 132.30s/it] {'loss': 1.8608, 'learning_rate': 2e-05, 'epoch': 0.76} 76%|███████▌ | 1378/1817 [50:08:25<16:07:57, 132.30s/it] 76%|███████▌ | 1379/1817 [50:10:36<16:02:29, 131.85s/it] {'loss': 1.8221, 'learning_rate': 2e-05, 'epoch': 0.76} 76%|███████▌ | 1379/1817 [50:10:36<16:02:27, 131.84s/it] 76%|███████▌ | 1380/1817 [50:12:47<15:58:27, 131.60s/it] {'loss': 1.8456, 'learning_rate': 2e-05, 'epoch': 0.76} 76%|███████▌ | 1380/1817 [50:12:47<15:58:25, 131.59s/it] 76%|███████▌ | 1381/1817 [50:14:58<15:54:50, 131.40s/it] {'loss': 1.8719, 'learning_rate': 2e-05, 'epoch': 0.76} 76%|███████▌ | 1381/1817 [50:14:58<15:54:49, 131.40s/it] 76%|███████▌ | 1382/1817 [50:17:10<15:53:56, 131.58s/it] {'loss': 1.8242, 'learning_rate': 2e-05, 'epoch': 0.76} 76%|███████▌ | 1382/1817 [50:17:10<15:53:57, 131.58s/it] 76%|███████▌ | 1383/1817 [50:19:20<15:48:21, 131.11s/it] {'loss': 1.8773, 'learning_rate': 2e-05, 'epoch': 0.76} 76%|███████▌ | 1383/1817 [50:19:20<15:48:21, 131.11s/it] 76%|███████▌ | 1384/1817 [50:21:31<15:47:28, 131.29s/it] {'loss': 1.9126, 'learning_rate': 2e-05, 'epoch': 0.76} 76%|███████▌ | 1384/1817 [50:21:31<15:47:28, 131.29s/it] 76%|███████▌ | 1385/1817 [50:23:42<15:44:18, 131.15s/it] {'loss': 1.8744, 'learning_rate': 2e-05, 'epoch': 0.76} 76%|███████▌ | 1385/1817 [50:23:42<15:44:18, 131.15s/it] 76%|███████▋ | 1386/1817 [50:25:54<15:43:43, 131.38s/it] {'loss': 1.8727, 'learning_rate': 2e-05, 'epoch': 0.76} 76%|███████▋ | 1386/1817 [50:25:54<15:43:43, 131.38s/it] 76%|███████▋ | 1387/1817 [50:28:05<15:39:27, 131.09s/it] {'loss': 1.8672, 'learning_rate': 2e-05, 'epoch': 0.76} 76%|███████▋ | 1387/1817 [50:28:04<15:39:28, 131.09s/it] 76%|███████▋ | 1388/1817 [50:30:15<15:35:23, 130.82s/it] {'loss': 1.8343, 'learning_rate': 2e-05, 'epoch': 0.76} 76%|███████▋ | 1388/1817 [50:30:15<15:35:23, 130.82s/it] 76%|███████▋ | 1389/1817 [50:32:26<15:34:13, 130.97s/it] {'loss': 1.9104, 'learning_rate': 2e-05, 'epoch': 0.76} 76%|███████▋ | 1389/1817 [50:32:26<15:34:14, 130.97s/it] 76%|███████▋ | 1390/1817 [50:34:36<15:29:42, 130.64s/it] {'loss': 1.8916, 'learning_rate': 2e-05, 'epoch': 0.76} 76%|███████▋ | 1390/1817 [50:34:36<15:29:42, 130.64s/it] 77%|███████▋ | 1391/1817 [50:36:43<15:20:00, 129.58s/it] {'loss': 1.8685, 'learning_rate': 2e-05, 'epoch': 0.77} 77%|███████▋ | 1391/1817 [50:36:43<15:20:00, 129.58s/it] 77%|███████▋ | 1392/1817 [50:38:51<15:13:39, 128.99s/it] {'loss': 1.88, 'learning_rate': 2e-05, 'epoch': 0.77} 77%|███████▋ | 1392/1817 [50:38:51<15:13:39, 128.99s/it] 77%|███████▋ | 1393/1817 [50:41:01<15:13:59, 129.34s/it] {'loss': 1.8535, 'learning_rate': 2e-05, 'epoch': 0.77} 77%|███████▋ | 1393/1817 [50:41:01<15:13:59, 129.34s/it] 77%|███████▋ | 1394/1817 [50:43:09<15:09:46, 129.05s/it] {'loss': 1.8606, 'learning_rate': 2e-05, 'epoch': 0.77} 77%|███████▋ | 1394/1817 [50:43:09<15:09:46, 129.05s/it] 77%|███████▋ | 1395/1817 [50:45:27<15:24:54, 131.50s/it] {'loss': 1.868, 'learning_rate': 2e-05, 'epoch': 0.77} 77%|███████▋ | 1395/1817 [50:45:26<15:24:55, 131.50s/it] 77%|███████▋ | 1396/1817 [50:47:34<15:14:31, 130.34s/it] {'loss': 1.8892, 'learning_rate': 2e-05, 'epoch': 0.77} 77%|███████▋ | 1396/1817 [50:47:34<15:14:31, 130.34s/it] 77%|███████▋ | 1397/1817 [50:49:43<15:09:39, 129.95s/it] {'loss': 1.9273, 'learning_rate': 2e-05, 'epoch': 0.77} 77%|███████▋ | 1397/1817 [50:49:43<15:09:40, 129.95s/it] 77%|███████▋ | 1398/1817 [50:51:55<15:10:53, 130.44s/it] {'loss': 1.8649, 'learning_rate': 2e-05, 'epoch': 0.77} 77%|███████▋ | 1398/1817 [50:51:55<15:10:53, 130.44s/it] 77%|███████▋ | 1399/1817 [50:54:04<15:06:04, 130.06s/it] {'loss': 1.8475, 'learning_rate': 2e-05, 'epoch': 0.77} 77%|███████▋ | 1399/1817 [50:54:04<15:06:04, 130.06s/it] 77%|███████▋ | 1400/1817 [50:56:12<14:59:31, 129.43s/it] {'loss': 1.8477, 'learning_rate': 2e-05, 'epoch': 0.77} 77%|███████▋ | 1400/1817 [50:56:12<14:59:31, 129.43s/it] 77%|███████▋ | 1401/1817 [50:58:19<14:53:23, 128.85s/it] {'loss': 1.8906, 'learning_rate': 2e-05, 'epoch': 0.77} 77%|███████▋ | 1401/1817 [50:58:19<14:53:24, 128.86s/it] 77%|███████▋ | 1402/1817 [51:00:30<14:54:35, 129.34s/it] {'loss': 1.8739, 'learning_rate': 2e-05, 'epoch': 0.77} 77%|███████▋ | 1402/1817 [51:00:30<14:54:36, 129.34s/it] 77%|███████▋ | 1403/1817 [51:02:39<14:51:58, 129.27s/it] {'loss': 1.9052, 'learning_rate': 2e-05, 'epoch': 0.77} 77%|███████▋ | 1403/1817 [51:02:39<14:51:59, 129.27s/it] 77%|███████▋ | 1404/1817 [51:04:49<14:52:00, 129.59s/it] {'loss': 1.8702, 'learning_rate': 2e-05, 'epoch': 0.77} 77%|███████▋ | 1404/1817 [51:04:49<14:52:00, 129.59s/it] 77%|███████▋ | 1405/1817 [51:06:58<14:47:53, 129.30s/it] {'loss': 1.9367, 'learning_rate': 2e-05, 'epoch': 0.77} 77%|███████▋ | 1405/1817 [51:06:58<14:47:53, 129.30s/it] 77%|███████▋ | 1406/1817 [51:09:08<14:46:46, 129.46s/it] {'loss': 1.8584, 'learning_rate': 2e-05, 'epoch': 0.77} 77%|███████▋ | 1406/1817 [51:09:08<14:46:41, 129.44s/it] 77%|███████▋ | 1407/1817 [51:11:18<14:47:07, 129.82s/it] {'loss': 1.855, 'learning_rate': 2e-05, 'epoch': 0.77} 77%|███████▋ | 1407/1817 [51:11:18<14:47:07, 129.82s/it] 77%|███████▋ | 1408/1817 [51:13:29<14:46:07, 129.99s/it] {'loss': 1.8467, 'learning_rate': 2e-05, 'epoch': 0.77} 77%|███████▋ | 1408/1817 [51:13:29<14:46:07, 129.99s/it] 78%|███████▊ | 1409/1817 [51:15:38<14:42:13, 129.74s/it] {'loss': 1.852, 'learning_rate': 2e-05, 'epoch': 0.78} 78%|███████▊ | 1409/1817 [51:15:38<14:42:13, 129.74s/it] 78%|███████▊ | 1410/1817 [51:17:53<14:50:26, 131.27s/it] {'loss': 1.8757, 'learning_rate': 2e-05, 'epoch': 0.78} 78%|███████▊ | 1410/1817 [51:17:53<14:50:26, 131.27s/it] 78%|███████▊ | 1411/1817 [51:20:02<14:44:09, 130.66s/it] {'loss': 1.8712, 'learning_rate': 2e-05, 'epoch': 0.78} 78%|███████▊ | 1411/1817 [51:20:02<14:44:09, 130.66s/it] 78%|███████▊ | 1412/1817 [51:22:13<14:42:21, 130.72s/it] {'loss': 1.8631, 'learning_rate': 2e-05, 'epoch': 0.78} 78%|███████▊ | 1412/1817 [51:22:13<14:42:21, 130.72s/it] 78%|███████▊ | 1413/1817 [51:24:20<14:33:03, 129.66s/it] {'loss': 1.8885, 'learning_rate': 2e-05, 'epoch': 0.78} 78%|███████▊ | 1413/1817 [51:24:20<14:33:03, 129.66s/it] 78%|███████▊ | 1414/1817 [51:26:29<14:29:15, 129.42s/it] {'loss': 1.9031, 'learning_rate': 2e-05, 'epoch': 0.78} 78%|███████▊ | 1414/1817 [51:26:29<14:29:15, 129.42s/it] 78%|███████▊ | 1415/1817 [51:28:42<14:34:24, 130.51s/it] {'loss': 1.8281, 'learning_rate': 2e-05, 'epoch': 0.78} 78%|███████▊ | 1415/1817 [51:28:42<14:34:24, 130.51s/it] 78%|███████▊ | 1416/1817 [51:30:53<14:34:06, 130.79s/it] {'loss': 1.8569, 'learning_rate': 2e-05, 'epoch': 0.78} 78%|███████▊ | 1416/1817 [51:30:53<14:34:06, 130.79s/it] 78%|███████▊ | 1417/1817 [51:33:03<14:30:22, 130.56s/it] {'loss': 1.8456, 'learning_rate': 2e-05, 'epoch': 0.78} 78%|███████▊ | 1417/1817 [51:33:03<14:30:22, 130.56s/it] 78%|███████▊ | 1418/1817 [51:35:15<14:30:38, 130.92s/it] {'loss': 1.8642, 'learning_rate': 2e-05, 'epoch': 0.78} 78%|███████▊ | 1418/1817 [51:35:15<14:30:38, 130.92s/it] 78%|███████▊ | 1419/1817 [51:37:23<14:22:30, 130.03s/it] {'loss': 1.8733, 'learning_rate': 2e-05, 'epoch': 0.78} 78%|███████▊ | 1419/1817 [51:37:23<14:22:30, 130.03s/it] 78%|███████▊ | 1420/1817 [51:39:36<14:25:21, 130.78s/it] {'loss': 1.8524, 'learning_rate': 2e-05, 'epoch': 0.78} 78%|███████▊ | 1420/1817 [51:39:36<14:25:20, 130.78s/it] 78%|███████▊ | 1421/1817 [51:41:49<14:27:21, 131.42s/it] {'loss': 1.8815, 'learning_rate': 2e-05, 'epoch': 0.78} 78%|███████▊ | 1421/1817 [51:41:48<14:27:21, 131.42s/it] 78%|███████▊ | 1422/1817 [51:44:00<14:24:27, 131.31s/it] {'loss': 1.885, 'learning_rate': 2e-05, 'epoch': 0.78} 78%|███████▊ | 1422/1817 [51:43:59<14:24:27, 131.31s/it] 78%|███████▊ | 1423/1817 [51:46:10<14:19:52, 130.94s/it] {'loss': 1.851, 'learning_rate': 2e-05, 'epoch': 0.78} 78%|███████▊ | 1423/1817 [51:46:10<14:19:52, 130.94s/it] 78%|███████▊ | 1424/1817 [51:48:21<14:18:07, 131.01s/it] {'loss': 1.888, 'learning_rate': 2e-05, 'epoch': 0.78} 78%|███████▊ | 1424/1817 [51:48:21<14:18:07, 131.01s/it] 78%|███████▊ | 1425/1817 [51:50:30<14:12:38, 130.51s/it] {'loss': 1.9037, 'learning_rate': 2e-05, 'epoch': 0.78} 78%|███████▊ | 1425/1817 [51:50:30<14:12:38, 130.51s/it] 78%|███████▊ | 1426/1817 [51:52:42<14:13:37, 130.99s/it] {'loss': 1.848, 'learning_rate': 2e-05, 'epoch': 0.78} 78%|███████▊ | 1426/1817 [51:52:42<14:13:37, 130.99s/it] 79%|███████▊ | 1427/1817 [51:54:54<14:12:06, 131.09s/it] {'loss': 1.8778, 'learning_rate': 2e-05, 'epoch': 0.79} 79%|███████▊ | 1427/1817 [51:54:54<14:12:06, 131.09s/it] 79%|███████▊ | 1428/1817 [51:57:02<14:03:56, 130.17s/it] {'loss': 1.8874, 'learning_rate': 2e-05, 'epoch': 0.79} 79%|███████▊ | 1428/1817 [51:57:02<14:03:57, 130.17s/it] 79%|███████▊ | 1429/1817 [51:59:12<14:01:04, 130.06s/it] {'loss': 1.9082, 'learning_rate': 2e-05, 'epoch': 0.79} 79%|███████▊ | 1429/1817 [51:59:11<14:01:04, 130.06s/it] 79%|███████▊ | 1430/1817 [52:01:23<14:02:11, 130.57s/it] {'loss': 1.887, 'learning_rate': 2e-05, 'epoch': 0.79} 79%|███████▊ | 1430/1817 [52:01:23<14:02:11, 130.57s/it] 79%|███████▉ | 1431/1817 [52:03:34<14:01:29, 130.80s/it] {'loss': 1.8443, 'learning_rate': 2e-05, 'epoch': 0.79} 79%|███████▉ | 1431/1817 [52:03:34<14:01:29, 130.80s/it] 79%|███████▉ | 1432/1817 [52:05:47<14:02:27, 131.29s/it] {'loss': 1.8664, 'learning_rate': 2e-05, 'epoch': 0.79} 79%|███████▉ | 1432/1817 [52:05:47<14:02:27, 131.29s/it] 79%|███████▉ | 1433/1817 [52:07:54<13:52:27, 130.07s/it] {'loss': 1.8806, 'learning_rate': 2e-05, 'epoch': 0.79} 79%|███████▉ | 1433/1817 [52:07:54<13:52:27, 130.07s/it] 79%|███████▉ | 1434/1817 [52:10:04<13:50:00, 130.03s/it] {'loss': 1.8636, 'learning_rate': 2e-05, 'epoch': 0.79} 79%|███████▉ | 1434/1817 [52:10:04<13:50:00, 130.03s/it] 79%|███████▉ | 1435/1817 [52:12:18<13:54:18, 131.04s/it] {'loss': 1.8992, 'learning_rate': 2e-05, 'epoch': 0.79} 79%|███████▉ | 1435/1817 [52:12:17<13:54:18, 131.04s/it] 79%|███████▉ | 1436/1817 [52:14:27<13:49:07, 130.57s/it] {'loss': 1.884, 'learning_rate': 2e-05, 'epoch': 0.79} 79%|███████▉ | 1436/1817 [52:14:27<13:49:07, 130.57s/it] 79%|███████▉ | 1437/1817 [52:16:38<13:47:23, 130.64s/it] {'loss': 1.8766, 'learning_rate': 2e-05, 'epoch': 0.79} 79%|███████▉ | 1437/1817 [52:16:38<13:47:23, 130.64s/it] 79%|███████▉ | 1438/1817 [52:18:48<13:44:27, 130.52s/it] {'loss': 1.8679, 'learning_rate': 2e-05, 'epoch': 0.79} 79%|███████▉ | 1438/1817 [52:18:48<13:44:27, 130.52s/it] 79%|███████▉ | 1439/1817 [52:20:56<13:37:42, 129.80s/it] {'loss': 1.8536, 'learning_rate': 2e-05, 'epoch': 0.79} 79%|███████▉ | 1439/1817 [52:20:56<13:37:43, 129.80s/it] 79%|███████▉ | 1440/1817 [52:23:09<13:41:48, 130.79s/it] {'loss': 1.8634, 'learning_rate': 2e-05, 'epoch': 0.79} 79%|███████▉ | 1440/1817 [52:23:09<13:41:48, 130.79s/it] 79%|███████▉ | 1441/1817 [52:25:22<13:43:50, 131.46s/it] {'loss': 1.87, 'learning_rate': 2e-05, 'epoch': 0.79} 79%|███████▉ | 1441/1817 [52:25:22<13:43:50, 131.46s/it] 79%|███████▉ | 1442/1817 [52:27:31<13:36:08, 130.58s/it] {'loss': 1.8926, 'learning_rate': 2e-05, 'epoch': 0.79} 79%|███████▉ | 1442/1817 [52:27:31<13:36:08, 130.58s/it] 79%|███████▉ | 1443/1817 [52:29:41<13:32:55, 130.42s/it] {'loss': 1.8831, 'learning_rate': 2e-05, 'epoch': 0.79} 79%|███████▉ | 1443/1817 [52:29:41<13:32:55, 130.42s/it] 79%|███████▉ | 1444/1817 [52:31:53<13:33:21, 130.84s/it] {'loss': 1.8954, 'learning_rate': 2e-05, 'epoch': 0.79} 79%|███████▉ | 1444/1817 [52:31:53<13:33:21, 130.84s/it] 80%|███████▉ | 1445/1817 [52:34:06<13:35:03, 131.46s/it] {'loss': 1.8975, 'learning_rate': 2e-05, 'epoch': 0.8} 80%|███████▉ | 1445/1817 [52:34:06<13:35:03, 131.46s/it] 80%|███████▉ | 1446/1817 [52:36:16<13:31:25, 131.23s/it] {'loss': 1.8715, 'learning_rate': 2e-05, 'epoch': 0.8} 80%|███████▉ | 1446/1817 [52:36:16<13:31:25, 131.23s/it] 80%|███████▉ | 1447/1817 [52:38:30<13:32:49, 131.81s/it] {'loss': 1.8265, 'learning_rate': 2e-05, 'epoch': 0.8} 80%|███████▉ | 1447/1817 [52:38:29<13:32:49, 131.81s/it] 80%|███████▉ | 1448/1817 [52:40:38<13:23:57, 130.73s/it] {'loss': 1.8603, 'learning_rate': 2e-05, 'epoch': 0.8} 80%|███████▉ | 1448/1817 [52:40:38<13:23:57, 130.73s/it] 80%|███████▉ | 1449/1817 [52:42:49<13:22:31, 130.85s/it] {'loss': 1.8608, 'learning_rate': 2e-05, 'epoch': 0.8} 80%|███████▉ | 1449/1817 [52:42:49<13:22:31, 130.85s/it] 80%|███████▉ | 1450/1817 [52:44:59<13:19:50, 130.76s/it] {'loss': 1.9213, 'learning_rate': 2e-05, 'epoch': 0.8} 80%|███████▉ | 1450/1817 [52:44:59<13:19:50, 130.76s/it] 80%|███████▉ | 1451/1817 [52:47:09<13:14:38, 130.27s/it] {'loss': 1.8926, 'learning_rate': 2e-05, 'epoch': 0.8} 80%|███████▉ | 1451/1817 [52:47:08<13:14:38, 130.27s/it] 80%|███████▉ | 1452/1817 [52:49:20<13:13:53, 130.50s/it] {'loss': 1.9171, 'learning_rate': 2e-05, 'epoch': 0.8} 80%|███████▉ | 1452/1817 [52:49:19<13:13:53, 130.50s/it] 80%|███████▉ | 1453/1817 [52:51:31<13:13:08, 130.74s/it] {'loss': 1.8958, 'learning_rate': 2e-05, 'epoch': 0.8} 80%|███████▉ | 1453/1817 [52:51:31<13:13:08, 130.74s/it] 80%|████████ | 1454/1817 [52:53:46<13:19:43, 132.19s/it] {'loss': 1.8461, 'learning_rate': 2e-05, 'epoch': 0.8} 80%|████████ | 1454/1817 [52:53:46<13:19:43, 132.19s/it] 80%|████████ | 1455/1817 [52:55:57<13:15:05, 131.78s/it] {'loss': 1.9011, 'learning_rate': 2e-05, 'epoch': 0.8} 80%|████████ | 1455/1817 [52:55:57<13:15:06, 131.79s/it] 80%|████████ | 1456/1817 [52:58:08<13:10:33, 131.39s/it] {'loss': 1.8344, 'learning_rate': 2e-05, 'epoch': 0.8} 80%|████████ | 1456/1817 [52:58:08<13:10:34, 131.40s/it] 80%|████████ | 1457/1817 [53:00:22<13:14:11, 132.36s/it] {'loss': 1.845, 'learning_rate': 2e-05, 'epoch': 0.8} 80%|████████ | 1457/1817 [53:00:22<13:14:12, 132.37s/it] 80%|████████ | 1458/1817 [53:02:32<13:07:22, 131.60s/it] {'loss': 1.8783, 'learning_rate': 2e-05, 'epoch': 0.8} 80%|████████ | 1458/1817 [53:02:32<13:07:22, 131.60s/it] 80%|████████ | 1459/1817 [53:04:43<13:03:18, 131.28s/it] {'loss': 1.8769, 'learning_rate': 2e-05, 'epoch': 0.8} 80%|████████ | 1459/1817 [53:04:43<13:03:18, 131.28s/it] 80%|████████ | 1460/1817 [53:06:55<13:01:58, 131.43s/it] {'loss': 1.85, 'learning_rate': 2e-05, 'epoch': 0.8} 80%|████████ | 1460/1817 [53:06:54<13:01:58, 131.43s/it] 80%|████████ | 1461/1817 [53:09:04<12:57:25, 131.03s/it] {'loss': 1.8799, 'learning_rate': 2e-05, 'epoch': 0.8} 80%|████████ | 1461/1817 [53:09:04<12:57:25, 131.03s/it] 80%|████████ | 1462/1817 [53:11:15<12:54:13, 130.86s/it] {'loss': 1.8903, 'learning_rate': 2e-05, 'epoch': 0.8} 80%|████████ | 1462/1817 [53:11:15<12:54:13, 130.86s/it] 81%|████████ | 1463/1817 [53:13:23<12:46:23, 129.90s/it] {'loss': 1.8638, 'learning_rate': 2e-05, 'epoch': 0.8} 81%|████████ | 1463/1817 [53:13:23<12:46:23, 129.90s/it] 81%|████████ | 1464/1817 [53:15:34<12:47:16, 130.42s/it] {'loss': 1.92, 'learning_rate': 2e-05, 'epoch': 0.81} 81%|████████ | 1464/1817 [53:15:34<12:47:16, 130.41s/it] 81%|████████ | 1465/1817 [53:17:46<12:47:28, 130.82s/it] {'loss': 1.8596, 'learning_rate': 2e-05, 'epoch': 0.81} 81%|████████ | 1465/1817 [53:17:46<12:47:28, 130.82s/it] 81%|████████ | 1466/1817 [53:19:56<12:43:48, 130.57s/it] {'loss': 1.8986, 'learning_rate': 2e-05, 'epoch': 0.81} 81%|████████ | 1466/1817 [53:19:56<12:43:48, 130.57s/it] 81%|████████ | 1467/1817 [53:22:08<12:44:11, 131.00s/it] {'loss': 1.8703, 'learning_rate': 2e-05, 'epoch': 0.81} 81%|████████ | 1467/1817 [53:22:08<12:44:11, 131.00s/it] 81%|████████ | 1468/1817 [53:24:20<12:42:56, 131.16s/it] {'loss': 1.9154, 'learning_rate': 2e-05, 'epoch': 0.81} 81%|████████ | 1468/1817 [53:24:19<12:42:56, 131.16s/it] 81%|████████ | 1469/1817 [53:26:30<12:38:38, 130.80s/it] {'loss': 1.9298, 'learning_rate': 2e-05, 'epoch': 0.81} 81%|████████ | 1469/1817 [53:26:29<12:38:38, 130.80s/it] 81%|████████ | 1470/1817 [53:28:42<12:39:39, 131.35s/it] {'loss': 1.8913, 'learning_rate': 2e-05, 'epoch': 0.81} 81%|████████ | 1470/1817 [53:28:42<12:39:39, 131.35s/it] 81%|████████ | 1471/1817 [53:30:51<12:33:29, 130.66s/it] {'loss': 1.8949, 'learning_rate': 2e-05, 'epoch': 0.81} 81%|████████ | 1471/1817 [53:30:51<12:33:29, 130.66s/it] 81%|████████ | 1472/1817 [53:33:02<12:30:52, 130.59s/it] {'loss': 1.888, 'learning_rate': 2e-05, 'epoch': 0.81} 81%|████████ | 1472/1817 [53:33:02<12:30:52, 130.59s/it] 81%|████████ | 1473/1817 [53:35:11<12:25:39, 130.06s/it] {'loss': 1.8862, 'learning_rate': 2e-05, 'epoch': 0.81} 81%|████████ | 1473/1817 [53:35:10<12:25:39, 130.06s/it] 81%|████████ | 1474/1817 [53:37:20<12:22:04, 129.81s/it] {'loss': 1.8609, 'learning_rate': 2e-05, 'epoch': 0.81} 81%|████████ | 1474/1817 [53:37:20<12:22:04, 129.81s/it] 81%|████████ | 1475/1817 [53:39:30<12:20:53, 129.98s/it] {'loss': 1.8804, 'learning_rate': 2e-05, 'epoch': 0.81} 81%|████████ | 1475/1817 [53:39:30<12:20:53, 129.98s/it] 81%|████████ | 1476/1817 [53:41:42<12:22:29, 130.64s/it] {'loss': 1.8548, 'learning_rate': 2e-05, 'epoch': 0.81} 81%|████████ | 1476/1817 [53:41:42<12:22:29, 130.64s/it] 81%|████████▏ | 1477/1817 [53:43:54<12:21:48, 130.91s/it] {'loss': 1.8737, 'learning_rate': 2e-05, 'epoch': 0.81} 81%|████████▏ | 1477/1817 [53:43:54<12:21:48, 130.91s/it] 81%|████████▏ | 1478/1817 [53:46:02<12:16:01, 130.27s/it] {'loss': 1.8435, 'learning_rate': 2e-05, 'epoch': 0.81} 81%|████████▏ | 1478/1817 [53:46:02<12:16:01, 130.27s/it] 81%|████████▏ | 1479/1817 [53:48:15<12:17:37, 130.94s/it] {'loss': 1.9236, 'learning_rate': 2e-05, 'epoch': 0.81} 81%|████████▏ | 1479/1817 [53:48:15<12:17:37, 130.94s/it] 81%|████████▏ | 1480/1817 [53:50:26<12:14:32, 130.78s/it] {'loss': 1.8821, 'learning_rate': 2e-05, 'epoch': 0.81} 81%|████████▏ | 1480/1817 [53:50:25<12:14:33, 130.78s/it] 82%|████████▏ | 1481/1817 [53:52:35<12:10:12, 130.40s/it] {'loss': 1.871, 'learning_rate': 2e-05, 'epoch': 0.81} 82%|████████▏ | 1481/1817 [53:52:35<12:10:12, 130.40s/it] 82%|████████▏ | 1482/1817 [53:54:46<12:09:08, 130.59s/it] {'loss': 1.8917, 'learning_rate': 2e-05, 'epoch': 0.82} 82%|████████▏ | 1482/1817 [53:54:46<12:09:08, 130.59s/it] 82%|████████▏ | 1483/1817 [53:56:55<12:03:57, 130.05s/it] {'loss': 1.8655, 'learning_rate': 2e-05, 'epoch': 0.82} 82%|████████▏ | 1483/1817 [53:56:55<12:03:57, 130.05s/it] 82%|████████▏ | 1484/1817 [53:59:07<12:05:52, 130.79s/it] {'loss': 1.8887, 'learning_rate': 2e-05, 'epoch': 0.82} 82%|████████▏ | 1484/1817 [53:59:07<12:05:52, 130.79s/it] 82%|████████▏ | 1485/1817 [54:01:15<11:58:02, 129.77s/it] {'loss': 1.903, 'learning_rate': 2e-05, 'epoch': 0.82} 82%|████████▏ | 1485/1817 [54:01:15<11:58:03, 129.77s/it] 82%|████████▏ | 1486/1817 [54:03:25<11:57:21, 130.03s/it] {'loss': 1.8654, 'learning_rate': 2e-05, 'epoch': 0.82} 82%|████████▏ | 1486/1817 [54:03:25<11:57:21, 130.03s/it] 82%|████████▏ | 1487/1817 [54:05:36<11:56:14, 130.23s/it] {'loss': 1.8732, 'learning_rate': 2e-05, 'epoch': 0.82} 82%|████████▏ | 1487/1817 [54:05:36<11:56:14, 130.23s/it] 82%|████████▏ | 1488/1817 [54:07:47<11:54:47, 130.36s/it] {'loss': 1.8589, 'learning_rate': 2e-05, 'epoch': 0.82} 82%|████████▏ | 1488/1817 [54:07:47<11:54:47, 130.36s/it] 82%|████████▏ | 1489/1817 [54:09:59<11:56:24, 131.05s/it] {'loss': 1.8873, 'learning_rate': 2e-05, 'epoch': 0.82} 82%|████████▏ | 1489/1817 [54:09:59<11:56:24, 131.05s/it] 82%|████████▏ | 1490/1817 [54:12:09<11:51:55, 130.63s/it] {'loss': 1.8711, 'learning_rate': 2e-05, 'epoch': 0.82} 82%|████████▏ | 1490/1817 [54:12:09<11:51:55, 130.63s/it] 82%|████████▏ | 1491/1817 [54:14:19<11:47:52, 130.28s/it] {'loss': 1.8395, 'learning_rate': 2e-05, 'epoch': 0.82} 82%|████████▏ | 1491/1817 [54:14:18<11:47:47, 130.27s/it] 82%|████████▏ | 1492/1817 [54:16:28<11:45:01, 130.16s/it] {'loss': 1.8723, 'learning_rate': 2e-05, 'epoch': 0.82} 82%|████████▏ | 1492/1817 [54:16:28<11:45:01, 130.16s/it] 82%|████████▏ | 1493/1817 [54:18:37<11:40:13, 129.67s/it] {'loss': 1.8528, 'learning_rate': 2e-05, 'epoch': 0.82} 82%|████████▏ | 1493/1817 [54:18:37<11:40:13, 129.67s/it] 82%|████████▏ | 1494/1817 [54:20:48<11:40:08, 130.06s/it] {'loss': 1.865, 'learning_rate': 2e-05, 'epoch': 0.82} 82%|████████▏ | 1494/1817 [54:20:48<11:40:08, 130.06s/it] 82%|████████▏ | 1495/1817 [54:23:04<11:47:14, 131.78s/it] {'loss': 1.8554, 'learning_rate': 2e-05, 'epoch': 0.82} 82%|████████▏ | 1495/1817 [54:23:04<11:47:14, 131.78s/it] 82%|████████▏ | 1496/1817 [54:25:15<11:43:48, 131.55s/it] {'loss': 1.8831, 'learning_rate': 2e-05, 'epoch': 0.82} 82%|████████▏ | 1496/1817 [54:25:15<11:43:49, 131.56s/it] 82%|████████▏ | 1497/1817 [54:27:26<11:41:25, 131.52s/it] {'loss': 1.8769, 'learning_rate': 2e-05, 'epoch': 0.82} 82%|████████▏ | 1497/1817 [54:27:26<11:41:25, 131.52s/it] 82%|████████▏ | 1498/1817 [54:29:38<11:38:54, 131.46s/it] {'loss': 1.8453, 'learning_rate': 2e-05, 'epoch': 0.82} 82%|████████▏ | 1498/1817 [54:29:37<11:38:54, 131.46s/it] 82%|████████▏ | 1499/1817 [54:31:54<11:45:00, 133.02s/it] {'loss': 1.8827, 'learning_rate': 2e-05, 'epoch': 0.82} 82%|████████▏ | 1499/1817 [54:31:54<11:45:00, 133.02s/it] 83%|████████▎ | 1500/1817 [54:34:05<11:39:31, 132.40s/it] {'loss': 1.8742, 'learning_rate': 2e-05, 'epoch': 0.83} 83%|████████▎ | 1500/1817 [54:34:05<11:39:32, 132.40s/it][INFO|trainer.py:2830] 2024-01-31 18:19:42,247 >> Saving model checkpoint to ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1500 [INFO|configuration_utils.py:457] 2024-01-31 18:19:42,253 >> Configuration saved in ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1500/config.json [INFO|configuration_utils.py:362] 2024-01-31 18:19:42,257 >> Configuration saved in ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1500/generation_config.json [INFO|modeling_utils.py:1759] 2024-01-31 18:19:42,293 >> Model weights saved in ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1500/pytorch_model.bin [INFO|tokenization_utils_base.py:2164] 2024-01-31 18:19:42,295 >> tokenizer config file saved in ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1500/tokenizer_config.json [INFO|tokenization_utils_base.py:2171] 2024-01-31 18:19:42,295 >> Special tokens file saved in ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1500/special_tokens_map.json [INFO|tokenization_utils_base.py:2221] 2024-01-31 18:19:42,296 >> added tokens file saved in ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1500/added_tokens.json [2024-01-31 18:19:56,975] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step1500 is about to be saved! [2024-01-31 18:19:56,987] [INFO] [engine.py:3492:save_16bit_model] Saving model weights to ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1500/pytorch_model.bin, tag: global_step1500 [2024-01-31 18:19:56,987] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1500/pytorch_model.bin... [2024-01-31 18:20:28,996] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1500/global_step1500/zero_pp_rank_24_mp_rank_00_model_states.pt...[[2024-01-31 18:20:29,053] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1500/global_step1500/zero_pp_rank_24_mp_rank_00_model_states.pt[2[2024-01-31 18:20:29,939] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1500/global_step1500/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt... tes.pt [2024-01-31 18:20:29,028] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1500/global_step1500/zero_pp_rank_0_mp_rank_00_model_states.pt... [2024-01-31 18:20:29,912] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1500/global_step1500/bf16_zero_pp_rank_24_mp_rank_00_optim_stat[2024-01-31 18:20:29,945] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1500/global_step1500/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt... [2024-01-31 18:20:44,340] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1500/global_step1500/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt. [2024-01-31 18:20:44,377] [INFO] [engine.py:3381:_save_zero_checkpoint] zero checkpoint saved ./checkpoints_ct/ac/allm-ac-13b/checkpoint-1500/global_step1500/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt [2[2024-01-31 18:20:44,461] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step1500 is ready now[INFO|trainer.py:2908] 2024-01-31 18:20:44,527 >> Deleting older checkpoint [checkpoints_ct/ac/allm-ac-13b/checkpoint-1000] due to args.save_total_limit 83%|████████▎ | 1501/1817 [54:37:17<13:10:42, 150.13s/it] {'loss': 1.8876, 'learning_rate': 2e-05, 'epoch': 0.83} 83%|████████▎ | 1501/1817 [54:37:16<13:10:42, 150.13s/it] 83%|████████▎ | 1502/1817 [54:39:29<12:40:25, 144.84s/it] {'loss': 1.8828, 'learning_rate': 2e-05, 'epoch': 0.83} 83%|████████▎ | 1502/1817 [54:39:29<12:40:26, 144.84s/it] 83%|████████▎ | 1503/1817 [54:41:42<12:18:48, 141.17s/it] {'loss': 1.8875, 'learning_rate': 2e-05, 'epoch': 0.83} 83%|████████▎ | 1503/1817 [54:41:42<12:18:48, 141.17s/it] 83%|████████▎ | 1504/1817 [54:43:57<12:06:23, 139.25s/it] {'loss': 1.865, 'learning_rate': 2e-05, 'epoch': 0.83} 83%|████████▎ | 1504/1817 [54:43:56<12:06:24, 139.25s/it] 83%|████████▎ | 1505/1817 [54:46:10<11:55:26, 137.59s/it] {'loss': 1.8396, 'learning_rate': 2e-05, 'epoch': 0.83} 83%|████████▎ | 1505/1817 [54:46:10<11:55:26, 137.59s/it] 83%|████████▎ | 1506/1817 [54:48:21<11:42:24, 135.51s/it] {'loss': 1.8587, 'learning_rate': 2e-05, 'epoch': 0.83} 83%|████████▎ | 1506/1817 [54:48:21<11:42:24, 135.51s/it] 83%|████████▎ | 1507/1817 [54:50:32<11:33:08, 134.16s/it] {'loss': 1.8716, 'learning_rate': 2e-05, 'epoch': 0.83} 83%|████████▎ | 1507/1817 [54:50:32<11:33:08, 134.16s/it] 83%|████████▎ | 1508/1817 [54:52:43<11:26:25, 133.29s/it] {'loss': 1.9094, 'learning_rate': 2e-05, 'epoch': 0.83} 83%|████████▎ | 1508/1817 [54:52:43<11:26:25, 133.29s/it] 83%|████████▎ | 1509/1817 [54:54:56<11:24:06, 133.27s/it] {'loss': 1.8805, 'learning_rate': 2e-05, 'epoch': 0.83} 83%|████████▎ | 1509/1817 [54:54:56<11:24:07, 133.27s/it] 83%|████████▎ | 1510/1817 [54:57:04<11:13:55, 131.71s/it] {'loss': 1.8155, 'learning_rate': 2e-05, 'epoch': 0.83} 83%|████████▎ | 1510/1817 [54:57:04<11:13:55, 131.71s/it] 83%|████████▎ | 1511/1817 [54:59:16<11:10:55, 131.55s/it] {'loss': 1.862, 'learning_rate': 2e-05, 'epoch': 0.83} 83%|████████▎ | 1511/1817 [54:59:15<11:10:56, 131.56s/it] 83%|████████▎ | 1512/1817 [55:01:29<11:11:47, 132.16s/it] {'loss': 1.8737, 'learning_rate': 2e-05, 'epoch': 0.83} 83%|████████▎ | 1512/1817 [55:01:29<11:11:47, 132.16s/it] 83%|████████▎ | 1513/1817 [55:03:40<11:08:01, 131.85s/it] {'loss': 1.9139, 'learning_rate': 2e-05, 'epoch': 0.83} 83%|████████▎ | 1513/1817 [55:03:40<11:08:01, 131.85s/it] 83%|████████▎ | 1514/1817 [55:05:51<11:04:13, 131.53s/it] {'loss': 1.9087, 'learning_rate': 2e-05, 'epoch': 0.83} 83%|████████▎ | 1514/1817 [55:05:51<11:04:13, 131.53s/it] 83%|████████▎ | 1515/1817 [55:08:05<11:05:14, 132.17s/it] {'loss': 1.8477, 'learning_rate': 2e-05, 'epoch': 0.83} 83%|████████▎ | 1515/1817 [55:08:05<11:05:14, 132.17s/it] 83%|████████▎ | 1516/1817 [55:10:17<11:02:30, 132.06s/it] {'loss': 1.8587, 'learning_rate': 2e-05, 'epoch': 0.83} 83%|████████▎ | 1516/1817 [55:10:16<11:02:29, 132.06s/it] 83%|████████▎ | 1517/1817 [55:12:28<10:59:40, 131.94s/it] {'loss': 1.8647, 'learning_rate': 2e-05, 'epoch': 0.83} 83%|████████▎ | 1517/1817 [55:12:28<10:59:39, 131.93s/it] 84%|████████▎ | 1518/1817 [55:14:37<10:53:15, 131.09s/it] {'loss': 1.8471, 'learning_rate': 2e-05, 'epoch': 0.84} 84%|████████▎ | 1518/1817 [55:14:37<10:53:14, 131.08s/it] 84%|████████▎ | 1519/1817 [55:16:46<10:48:03, 130.48s/it] {'loss': 1.8925, 'learning_rate': 2e-05, 'epoch': 0.84} 84%|████████▎ | 1519/1817 [55:16:46<10:48:03, 130.48s/it] 84%|████████▎ | 1520/1817 [55:18:54<10:41:57, 129.69s/it] {'loss': 1.8907, 'learning_rate': 2e-05, 'epoch': 0.84} 84%|████████▎ | 1520/1817 [55:18:54<10:41:58, 129.69s/it] 84%|████████▎ | 1521/1817 [55:21:03<10:37:42, 129.26s/it] {'loss': 1.8933, 'learning_rate': 2e-05, 'epoch': 0.84} 84%|████████▎ | 1521/1817 [55:21:02<10:37:42, 129.26s/it] 84%|████████▍ | 1522/1817 [55:23:10<10:33:32, 128.86s/it] {'loss': 1.8827, 'learning_rate': 2e-05, 'epoch': 0.84} 84%|████████▍ | 1522/1817 [55:23:10<10:33:32, 128.86s/it] 84%|████████▍ | 1523/1817 [55:25:21<10:34:08, 129.42s/it] {'loss': 1.8804, 'learning_rate': 2e-05, 'epoch': 0.84} 84%|████████▍ | 1523/1817 [55:25:21<10:34:08, 129.42s/it] 84%|████████▍ | 1524/1817 [55:27:35<10:37:59, 130.65s/it] {'loss': 1.8636, 'learning_rate': 2e-05, 'epoch': 0.84} 84%|████████▍ | 1524/1817 [55:27:34<10:38:00, 130.65s/it] 84%|████████▍ | 1525/1817 [55:29:41<10:30:12, 129.49s/it] {'loss': 1.9036, 'learning_rate': 2e-05, 'epoch': 0.84} 84%|████████▍ | 1525/1817 [55:29:41<10:30:12, 129.49s/it] 84%|████████▍ | 1526/1817 [55:31:51<10:28:02, 129.49s/it] {'loss': 1.8571, 'learning_rate': 2e-05, 'epoch': 0.84} 84%|████████▍ | 1526/1817 [55:31:51<10:28:03, 129.50s/it] 84%|████████▍ | 1527/1817 [55:33:59<10:23:15, 128.95s/it] {'loss': 1.8457, 'learning_rate': 2e-05, 'epoch': 0.84} 84%|████████▍ | 1527/1817 [55:33:58<10:23:15, 128.95s/it] 84%|████████▍ | 1528/1817 [55:36:08<10:21:13, 128.98s/it] {'loss': 1.8557, 'learning_rate': 2e-05, 'epoch': 0.84} 84%|████████▍ | 1528/1817 [55:36:07<10:21:14, 128.98s/it] 84%|████████▍ | 1529/1817 [55:38:23<10:27:56, 130.82s/it] {'loss': 1.8877, 'learning_rate': 2e-05, 'epoch': 0.84} 84%|████████▍ | 1529/1817 [55:38:23<10:27:57, 130.82s/it] 84%|████████▍ | 1530/1817 [55:40:29<10:19:23, 129.49s/it] {'loss': 1.8704, 'learning_rate': 2e-05, 'epoch': 0.84} 84%|████████▍ | 1530/1817 [55:40:29<10:19:23, 129.49s/it] 84%|████████▍ | 1531/1817 [55:42:40<10:18:33, 129.77s/it] {'loss': 1.8547, 'learning_rate': 2e-05, 'epoch': 0.84} 84%|████████▍ | 1531/1817 [55:42:39<10:18:33, 129.77s/it] 84%|████████▍ | 1532/1817 [55:44:51<10:18:24, 130.19s/it] {'loss': 1.8567, 'learning_rate': 2e-05, 'epoch': 0.84} 84%|████████▍ | 1532/1817 [55:44:51<10:18:24, 130.19s/it] 84%|████████▍ | 1533/1817 [55:46:59<10:13:32, 129.62s/it] {'loss': 1.8774, 'learning_rate': 2e-05, 'epoch': 0.84} 84%|████████▍ | 1533/1817 [55:46:59<10:13:32, 129.62s/it] 84%|████████▍ | 1534/1817 [55:49:12<10:16:30, 130.71s/it] {'loss': 1.8966, 'learning_rate': 2e-05, 'epoch': 0.84} 84%|████████▍ | 1534/1817 [55:49:12<10:16:30, 130.71s/it] 84%|████████▍ | 1535/1817 [55:51:23<10:14:43, 130.79s/it] {'loss': 1.8533, 'learning_rate': 2e-05, 'epoch': 0.84} 84%|████████▍ | 1535/1817 [55:51:23<10:14:43, 130.79s/it] 85%|████████▍ | 1536/1817 [55:53:33<10:10:58, 130.46s/it] {'loss': 1.8697, 'learning_rate': 2e-05, 'epoch': 0.85} 85%|████████▍ | 1536/1817 [55:53:33<10:10:58, 130.46s/it] 85%|████████▍ | 1537/1817 [55:55:46<10:12:58, 131.35s/it] {'loss': 1.8756, 'learning_rate': 2e-05, 'epoch': 0.85} 85%|████████▍ | 1537/1817 [55:55:46<10:12:58, 131.35s/it] 85%|████████▍ | 1538/1817 [55:57:56<10:08:39, 130.89s/it] {'loss': 1.848, 'learning_rate': 2e-05, 'epoch': 0.85} 85%|████████▍ | 1538/1817 [55:57:56<10:08:39, 130.89s/it] 85%|████████▍ | 1539/1817 [56:00:08<10:07:54, 131.20s/it] {'loss': 1.8651, 'learning_rate': 2e-05, 'epoch': 0.85} 85%|████████▍ | 1539/1817 [56:00:08<10:07:55, 131.21s/it] 85%|████████▍ | 1540/1817 [56:02:21<10:08:13, 131.74s/it] {'loss': 1.85, 'learning_rate': 2e-05, 'epoch': 0.85} 85%|████████▍ | 1540/1817 [56:02:21<10:08:13, 131.75s/it] 85%|████████▍ | 1541/1817 [56:04:32<10:04:31, 131.42s/it] {'loss': 1.8694, 'learning_rate': 2e-05, 'epoch': 0.85} 85%|████████▍ | 1541/1817 [56:04:32<10:04:31, 131.42s/it] 85%|████████▍ | 1542/1817 [56:06:41<9:59:15, 130.75s/it] {'loss': 1.8519, 'learning_rate': 2e-05, 'epoch': 0.85} 85%|████████▍ | 1542/1817 [56:06:41<9:59:15, 130.75s/it] 85%|████████▍ | 1543/1817 [56:08:51<9:56:33, 130.63s/it] {'loss': 1.8332, 'learning_rate': 2e-05, 'epoch': 0.85} 85%|████████▍ | 1543/1817 [56:08:51<9:56:33, 130.63s/it] 85%|████████▍ | 1544/1817 [56:11:01<9:53:22, 130.41s/it] {'loss': 1.8425, 'learning_rate': 2e-05, 'epoch': 0.85} 85%|████████▍ | 1544/1817 [56:11:01<9:53:20, 130.41s/it] 85%|████████▌ | 1545/1817 [56:13:12<9:51:53, 130.56s/it] {'loss': 1.8681, 'learning_rate': 2e-05, 'epoch': 0.85} 85%|████████▌ | 1545/1817 [56:13:12<9:51:49, 130.55s/it] 85%|████████▌ | 1546/1817 [56:15:25<9:52:39, 131.22s/it] {'loss': 1.8785, 'learning_rate': 2e-05, 'epoch': 0.85} 85%|████████▌ | 1546/1817 [56:15:25<9:52:39, 131.21s/it] 85%|████████▌ | 1547/1817 [56:17:36<9:50:09, 131.15s/it] {'loss': 1.8633, 'learning_rate': 2e-05, 'epoch': 0.85} 85%|████████▌ | 1547/1817 [56:17:36<9:50:09, 131.15s/it] 85%|████████▌ | 1548/1817 [56:19:46<9:46:21, 130.79s/it] {'loss': 1.8451, 'learning_rate': 2e-05, 'epoch': 0.85} 85%|████████▌ | 1548/1817 [56:19:46<9:46:21, 130.79s/it] 85%|████████▌ | 1549/1817 [56:21:57<9:44:42, 130.91s/it] {'loss': 1.8167, 'learning_rate': 2e-05, 'epoch': 0.85} 85%|████████▌ | 1549/1817 [56:21:57<9:44:43, 130.91s/it] 85%|████████▌ | 1550/1817 [56:24:08<9:42:57, 131.00s/it] {'loss': 1.8816, 'learning_rate': 2e-05, 'epoch': 0.85} 85%|████████▌ | 1550/1817 [56:24:08<9:42:57, 131.00s/it] 85%|████████▌ | 1551/1817 [56:26:21<9:43:38, 131.65s/it] {'loss': 1.8685, 'learning_rate': 2e-05, 'epoch': 0.85} 85%|████████▌ | 1551/1817 [56:26:21<9:43:38, 131.65s/it] 85%|████████▌ | 1552/1817 [56:28:33<9:40:41, 131.48s/it] {'loss': 1.8638, 'learning_rate': 2e-05, 'epoch': 0.85} 85%|████████▌ | 1552/1817 [56:28:32<9:40:42, 131.48s/it] 85%|████████▌ | 1553/1817 [56:30:42<9:35:14, 130.74s/it] {'loss': 1.809, 'learning_rate': 2e-05, 'epoch': 0.85} 85%|████████▌ | 1553/1817 [56:30:41<9:35:15, 130.74s/it] 86%|████████▌ | 1554/1817 [56:32:57<9:39:16, 132.15s/it] {'loss': 1.8622, 'learning_rate': 2e-05, 'epoch': 0.86} 86%|████████▌ | 1554/1817 [56:32:57<9:39:16, 132.15s/it] 86%|████████▌ | 1555/1817 [56:35:07<9:33:49, 131.41s/it] {'loss': 1.8921, 'learning_rate': 2e-05, 'epoch': 0.86} 86%|████████▌ | 1555/1817 [56:35:06<9:33:49, 131.41s/it] 86%|████████▌ | 1556/1817 [56:37:21<9:34:49, 132.14s/it] {'loss': 1.8507, 'learning_rate': 2e-05, 'epoch': 0.86} 86%|████████▌ | 1556/1817 [56:37:20<9:34:49, 132.14s/it] 86%|████████▌ | 1557/1817 [56:39:30<9:28:36, 131.22s/it] {'loss': 1.8893, 'learning_rate': 2e-05, 'epoch': 0.86} 86%|████████▌ | 1557/1817 [56:39:29<9:28:36, 131.22s/it] 86%|████████▌ | 1558/1817 [56:41:42<9:28:27, 131.69s/it] {'loss': 1.8445, 'learning_rate': 2e-05, 'epoch': 0.86} 86%|████████▌ | 1558/1817 [56:41:42<9:28:27, 131.69s/it] 86%|████████▌ | 1559/1817 [56:43:51<9:22:47, 130.88s/it] {'loss': 1.8833, 'learning_rate': 2e-05, 'epoch': 0.86} 86%|████████▌ | 1559/1817 [56:43:51<9:22:48, 130.88s/it] 86%|████████▌ | 1560/1817 [56:46:08<9:27:42, 132.54s/it] {'loss': 1.8659, 'learning_rate': 2e-05, 'epoch': 0.86} 86%|████████▌ | 1560/1817 [56:46:08<9:27:42, 132.54s/it] 86%|████████▌ | 1561/1817 [56:48:20<9:24:33, 132.32s/it] {'loss': 1.8436, 'learning_rate': 2e-05, 'epoch': 0.86} 86%|████████▌ | 1561/1817 [56:48:19<9:24:33, 132.32s/it] 86%|████████▌ | 1562/1817 [56:50:31<9:20:42, 131.93s/it] {'loss': 1.8476, 'learning_rate': 2e-05, 'epoch': 0.86} 86%|████████▌ | 1562/1817 [56:50:30<9:20:42, 131.93s/it] 86%|████████▌ | 1563/1817 [56:52:40<9:15:48, 131.29s/it] {'loss': 1.8938, 'learning_rate': 2e-05, 'epoch': 0.86} 86%|████████▌ | 1563/1817 [56:52:40<9:15:48, 131.29s/it] 86%|████████▌ | 1564/1817 [56:54:51<9:13:03, 131.16s/it] {'loss': 1.8922, 'learning_rate': 2e-05, 'epoch': 0.86} 86%|████████▌ | 1564/1817 [56:54:51<9:13:03, 131.16s/it] 86%|████████▌ | 1565/1817 [56:57:00<9:07:10, 130.28s/it] {'loss': 1.8548, 'learning_rate': 2e-05, 'epoch': 0.86} 86%|████████▌ | 1565/1817 [56:56:59<9:07:10, 130.28s/it] 86%|████████▌ | 1566/1817 [56:59:11<9:06:28, 130.63s/it] {'loss': 1.8763, 'learning_rate': 2e-05, 'epoch': 0.86} 86%|████████▌ | 1566/1817 [56:59:11<9:06:28, 130.63s/it] 86%|████████▌ | 1567/1817 [57:01:21<9:03:24, 130.42s/it] {'loss': 1.8875, 'learning_rate': 2e-05, 'epoch': 0.86} 86%|████████▌ | 1567/1817 [57:01:21<9:03:24, 130.42s/it] 86%|████████▋ | 1568/1817 [57:03:34<9:04:54, 131.30s/it] {'loss': 1.8145, 'learning_rate': 2e-05, 'epoch': 0.86} 86%|████████▋ | 1568/1817 [57:03:34<9:04:54, 131.30s/it] 86%|████████▋ | 1569/1817 [57:05:43<8:59:46, 130.59s/it] {'loss': 1.8663, 'learning_rate': 2e-05, 'epoch': 0.86} 86%|████████▋ | 1569/1817 [57:05:43<8:59:46, 130.59s/it] 86%|████████▋ | 1570/1817 [57:07:54<8:57:46, 130.64s/it] {'loss': 1.8326, 'learning_rate': 2e-05, 'epoch': 0.86} 86%|████████▋ | 1570/1817 [57:07:54<8:57:47, 130.64s/it] 86%|████████▋ | 1571/1817 [57:10:05<8:56:16, 130.80s/it] {'loss': 1.859, 'learning_rate': 2e-05, 'epoch': 0.86} 86%|████████▋ | 1571/1817 [57:10:05<8:56:16, 130.80s/it] 87%|████████▋ | 1572/1817 [57:12:13<8:50:28, 129.91s/it] {'loss': 1.8685, 'learning_rate': 2e-05, 'epoch': 0.86} 87%|████████▋ | 1572/1817 [57:12:13<8:50:26, 129.90s/it] 87%|████████▋ | 1573/1817 [57:14:22<8:47:40, 129.75s/it] {'loss': 1.8913, 'learning_rate': 2e-05, 'epoch': 0.87} 87%|████████▋ | 1573/1817 [57:14:22<8:47:39, 129.75s/it] 87%|████████▋ | 1574/1817 [57:16:32<8:45:10, 129.67s/it] {'loss': 1.9075, 'learning_rate': 2e-05, 'epoch': 0.87} 87%|████████▋ | 1574/1817 [57:16:32<8:45:11, 129.68s/it] 87%|████████▋ | 1575/1817 [57:18:44<8:45:28, 130.28s/it] {'loss': 1.8691, 'learning_rate': 2e-05, 'epoch': 0.87} 87%|████████▋ | 1575/1817 [57:18:43<8:45:28, 130.28s/it] 87%|████████▋ | 1576/1817 [57:20:54<8:43:16, 130.27s/it] {'loss': 1.8347, 'learning_rate': 2e-05, 'epoch': 0.87} 87%|████████▋ | 1576/1817 [57:20:54<8:43:16, 130.27s/it] 87%|████████▋ | 1577/1817 [57:23:00<8:36:08, 129.04s/it] {'loss': 1.8934, 'learning_rate': 2e-05, 'epoch': 0.87} 87%|████████▋ | 1577/1817 [57:23:00<8:36:08, 129.04s/it] 87%|████████▋ | 1578/1817 [57:25:10<8:35:38, 129.45s/it] {'loss': 1.867, 'learning_rate': 2e-05, 'epoch': 0.87} 87%|████████▋ | 1578/1817 [57:25:10<8:35:38, 129.45s/it] 87%|████████▋ | 1579/1817 [57:27:24<8:38:41, 130.76s/it] {'loss': 1.8744, 'learning_rate': 2e-05, 'epoch': 0.87} 87%|████████▋ | 1579/1817 [57:27:24<8:38:41, 130.76s/it] 87%|████████▋ | 1580/1817 [57:29:35<8:36:41, 130.81s/it] {'loss': 1.9009, 'learning_rate': 2e-05, 'epoch': 0.87} 87%|████████▋ | 1580/1817 [57:29:35<8:36:41, 130.81s/it] 87%|████████▋ | 1581/1817 [57:31:46<8:34:31, 130.81s/it] {'loss': 1.9074, 'learning_rate': 2e-05, 'epoch': 0.87} 87%|████████▋ | 1581/1817 [57:31:46<8:34:31, 130.81s/it] 87%|████████▋ | 1582/1817 [57:33:51<8:26:05, 129.21s/it] {'loss': 1.852, 'learning_rate': 2e-05, 'epoch': 0.87} 87%|████████▋ | 1582/1817 [57:33:51<8:26:05, 129.21s/it] 87%|████████▋ | 1583/1817 [57:36:06<8:30:27, 130.89s/it] {'loss': 1.8801, 'learning_rate': 2e-05, 'epoch': 0.87} 87%|████████▋ | 1583/1817 [57:36:06<8:30:27, 130.89s/it] 87%|████████▋ | 1584/1817 [57:38:17<8:27:46, 130.76s/it] {'loss': 1.9031, 'learning_rate': 2e-05, 'epoch': 0.87} 87%|████████▋ | 1584/1817 [57:38:16<8:27:46, 130.76s/it] 87%|████████▋ | 1585/1817 [57:40:34<8:32:44, 132.61s/it] {'loss': 1.8945, 'learning_rate': 2e-05, 'epoch': 0.87} 87%|████████▋ | 1585/1817 [57:40:33<8:32:44, 132.61s/it] 87%|████████▋ | 1586/1817 [57:42:50<8:34:34, 133.66s/it] {'loss': 1.8655, 'learning_rate': 2e-05, 'epoch': 0.87} 87%|████████▋ | 1586/1817 [57:42:49<8:34:34, 133.66s/it] 87%|████████▋ | 1587/1817 [57:45:02<8:30:56, 133.29s/it] {'loss': 1.8964, 'learning_rate': 2e-05, 'epoch': 0.87} 87%|████████▋ | 1587/1817 [57:45:02<8:30:56, 133.29s/it] 87%|████████▋ | 1588/1817 [57:47:18<8:31:48, 134.10s/it] {'loss': 1.8408, 'learning_rate': 2e-05, 'epoch': 0.87} 87%|████████▋ | 1588/1817 [57:47:18<8:31:49, 134.10s/it] 87%|████████▋ | 1589/1817 [57:49:31<8:27:50, 133.64s/it] {'loss': 1.8964, 'learning_rate': 2e-05, 'epoch': 0.87} 87%|████████▋ | 1589/1817 [57:49:30<8:27:50, 133.64s/it] 88%|████████▊ | 1590/1817 [57:51:42<8:23:13, 133.01s/it] {'loss': 1.8489, 'learning_rate': 2e-05, 'epoch': 0.87} 88%|████████▊ | 1590/1817 [57:51:42<8:23:13, 133.01s/it] 88%|████████▊ | 1591/1817 [57:53:58<8:24:09, 133.85s/it] {'loss': 1.8262, 'learning_rate': 2e-05, 'epoch': 0.88} 88%|████████▊ | 1591/1817 [57:53:58<8:24:09, 133.85s/it] 88%|████████▊ | 1592/1817 [57:56:17<8:27:19, 135.29s/it] {'loss': 1.8767, 'learning_rate': 2e-05, 'epoch': 0.88} 88%|████████▊ | 1592/1817 [57:56:16<8:27:19, 135.29s/it] 88%|████████▊ | 1593/1817 [57:58:30<8:23:25, 134.85s/it] {'loss': 1.9, 'learning_rate': 2e-05, 'epoch': 0.88} 88%|████████▊ | 1593/1817 [57:58:30<8:23:26, 134.85s/it] 88%|████████▊ | 1594/1817 [58:00:43<8:18:32, 134.14s/it] {'loss': 1.8541, 'learning_rate': 2e-05, 'epoch': 0.88} 88%|████████▊ | 1594/1817 [58:00:43<8:18:32, 134.14s/it] 88%|████████▊ | 1595/1817 [58:02:56<8:15:25, 133.90s/it] {'loss': 1.8435, 'learning_rate': 2e-05, 'epoch': 0.88} 88%|████████▊ | 1595/1817 [58:02:56<8:15:25, 133.90s/it] 88%|████████▊ | 1596/1817 [58:05:15<8:18:12, 135.26s/it] {'loss': 1.8552, 'learning_rate': 2e-05, 'epoch': 0.88} 88%|████████▊ | 1596/1817 [58:05:14<8:18:12, 135.26s/it] 88%|████████▊ | 1597/1817 [58:07:29<8:14:46, 134.94s/it] {'loss': 1.8935, 'learning_rate': 2e-05, 'epoch': 0.88} 88%|████████▊ | 1597/1817 [58:07:29<8:14:46, 134.94s/it] 88%|████████▊ | 1598/1817 [58:09:41<8:09:05, 134.00s/it] {'loss': 1.8801, 'learning_rate': 2e-05, 'epoch': 0.88} 88%|████████▊ | 1598/1817 [58:09:40<8:09:05, 134.00s/it] 88%|████████▊ | 1599/1817 [58:11:55<8:07:37, 134.21s/it] {'loss': 1.8799, 'learning_rate': 2e-05, 'epoch': 0.88} 88%|████████▊ | 1599/1817 [58:11:55<8:07:36, 134.20s/it] 88%|████████▊ | 1600/1817 [58:14:07<8:02:38, 133.45s/it] {'loss': 1.8747, 'learning_rate': 2e-05, 'epoch': 0.88} 88%|████████▊ | 1600/1817 [58:14:07<8:02:37, 133.44s/it] 88%|████████▊ | 1601/1817 [58:16:19<7:58:42, 132.98s/it] {'loss': 1.8347, 'learning_rate': 2e-05, 'epoch': 0.88} 88%|████████▊ | 1601/1817 [58:16:19<7:58:43, 132.98s/it] 88%|████████▊ | 1602/1817 [58:18:36<8:00:26, 134.08s/it] {'loss': 1.8626, 'learning_rate': 2e-05, 'epoch': 0.88} 88%|████████▊ | 1602/1817 [58:18:35<8:00:26, 134.08s/it] 88%|████████▊ | 1603/1817 [58:20:48<7:55:52, 133.42s/it] {'loss': 1.8357, 'learning_rate': 2e-05, 'epoch': 0.88} 88%|████████▊ | 1603/1817 [58:20:47<7:55:52, 133.42s/it] 88%|████████▊ | 1604/1817 [58:23:06<7:58:44, 134.86s/it] {'loss': 1.857, 'learning_rate': 2e-05, 'epoch': 0.88} 88%|████████▊ | 1604/1817 [58:23:05<7:58:44, 134.86s/it] 88%|████████▊ | 1605/1817 [58:25:26<8:01:58, 136.41s/it] {'loss': 1.9032, 'learning_rate': 2e-05, 'epoch': 0.88} 88%|████████▊ | 1605/1817 [58:25:25<8:01:58, 136.41s/it] 88%|████████▊ | 1606/1817 [58:27:48<8:06:22, 138.31s/it] {'loss': 1.8792, 'learning_rate': 2e-05, 'epoch': 0.88} 88%|████████▊ | 1606/1817 [58:27:48<8:06:22, 138.31s/it] 88%|████████▊ | 1607/1817 [58:30:06<8:02:59, 138.00s/it] {'loss': 1.8645, 'learning_rate': 2e-05, 'epoch': 0.88} 88%|████████▊ | 1607/1817 [58:30:06<8:02:59, 138.00s/it] 88%|████████▊ | 1608/1817 [58:32:16<7:52:24, 135.62s/it] {'loss': 1.8823, 'learning_rate': 2e-05, 'epoch': 0.88} 88%|████████▊ | 1608/1817 [58:32:16<7:52:24, 135.62s/it] 89%|████████▊ | 1609/1817 [58:34:25<7:42:59, 133.56s/it] {'loss': 1.8317, 'learning_rate': 2e-05, 'epoch': 0.89} 89%|████████▊ | 1609/1817 [58:34:24<7:43:00, 133.56s/it] 89%|████████▊ | 1610/1817 [58:36:37<7:39:47, 133.27s/it] {'loss': 1.8453, 'learning_rate': 2e-05, 'epoch': 0.89} 89%|████████▊ | 1610/1817 [58:36:37<7:39:47, 133.27s/it] 89%|████████▊ | 1611/1817 [58:38:52<7:38:47, 133.63s/it] {'loss': 1.852, 'learning_rate': 2e-05, 'epoch': 0.89} 89%|████████▊ | 1611/1817 [58:38:51<7:38:47, 133.63s/it] 89%|████████▊ | 1612/1817 [58:41:02<7:32:47, 132.53s/it] {'loss': 1.8709, 'learning_rate': 2e-05, 'epoch': 0.89} 89%|████████▊ | 1612/1817 [58:41:01<7:32:48, 132.53s/it] 89%|████████▉ | 1613/1817 [58:43:17<7:33:36, 133.41s/it] {'loss': 1.8459, 'learning_rate': 2e-05, 'epoch': 0.89} 89%|████████▉ | 1613/1817 [58:43:17<7:33:36, 133.41s/it] 89%|████████▉ | 1614/1817 [58:45:35<7:35:50, 134.73s/it] {'loss': 1.8539, 'learning_rate': 2e-05, 'epoch': 0.89} 89%|████████▉ | 1614/1817 [58:45:35<7:35:50, 134.73s/it] 89%|████████▉ | 1615/1817 [58:47:53<7:37:29, 135.89s/it] {'loss': 1.8793, 'learning_rate': 2e-05, 'epoch': 0.89} 89%|████████▉ | 1615/1817 [58:47:53<7:37:29, 135.89s/it] 89%|████████▉ | 1616/1817 [58:50:12<7:38:30, 136.87s/it] {'loss': 1.8687, 'learning_rate': 2e-05, 'epoch': 0.89} 89%|████████▉ | 1616/1817 [58:50:12<7:38:30, 136.87s/it] 89%|████████▉ | 1617/1817 [58:52:27<7:33:49, 136.15s/it] {'loss': 1.867, 'learning_rate': 2e-05, 'epoch': 0.89} 89%|████████▉ | 1617/1817 [58:52:27<7:33:50, 136.15s/it] 89%|████████▉ | 1618/1817 [58:54:41<7:29:31, 135.54s/it] {'loss': 1.8669, 'learning_rate': 2e-05, 'epoch': 0.89} 89%|████████▉ | 1618/1817 [58:54:41<7:29:31, 135.54s/it] 89%|████████▉ | 1619/1817 [58:56:50<7:20:50, 133.59s/it] {'loss': 1.8717, 'learning_rate': 2e-05, 'epoch': 0.89} 89%|████████▉ | 1619/1817 [58:56:50<7:20:50, 133.59s/it] 89%|████████▉ | 1620/1817 [58:59:04<7:19:07, 133.74s/it] {'loss': 1.897, 'learning_rate': 2e-05, 'epoch': 0.89} 89%|████████▉ | 1620/1817 [58:59:04<7:19:07, 133.74s/it] 89%|████████▉ | 1621/1817 [59:01:24<7:22:15, 135.39s/it] {'loss': 1.8852, 'learning_rate': 2e-05, 'epoch': 0.89} 89%|████████▉ | 1621/1817 [59:01:23<7:22:15, 135.39s/it] 89%|████████▉ | 1622/1817 [59:03:34<7:15:03, 133.86s/it] {'loss': 1.904, 'learning_rate': 2e-05, 'epoch': 0.89} 89%|████████▉ | 1622/1817 [59:03:34<7:15:03, 133.86s/it] 89%|████████▉ | 1623/1817 [59:05:47<7:12:14, 133.68s/it] {'loss': 1.8781, 'learning_rate': 2e-05, 'epoch': 0.89} 89%|████████▉ | 1623/1817 [59:05:47<7:12:14, 133.68s/it] 89%|████████▉ | 1624/1817 [59:08:01<7:10:27, 133.82s/it] {'loss': 1.8769, 'learning_rate': 2e-05, 'epoch': 0.89} 89%|████████▉ | 1624/1817 [59:08:01<7:10:27, 133.82s/it] 89%|████████▉ | 1625/1817 [59:10:25<7:17:45, 136.80s/it] {'loss': 1.8595, 'learning_rate': 2e-05, 'epoch': 0.89} 89%|████████▉ | 1625/1817 [59:10:25<7:17:45, 136.80s/it] 89%|████████▉ | 1626/1817 [59:12:41<7:14:25, 136.47s/it] {'loss': 1.8533, 'learning_rate': 2e-05, 'epoch': 0.89} 89%|████████▉ | 1626/1817 [59:12:40<7:14:24, 136.46s/it] 90%|████████▉ | 1627/1817 [59:14:50<7:05:45, 134.45s/it] {'loss': 1.89, 'learning_rate': 2e-05, 'epoch': 0.9} 90%|████████▉ | 1627/1817 [59:14:50<7:05:45, 134.45s/it] 90%|████████▉ | 1628/1817 [59:17:04<7:02:47, 134.22s/it] {'loss': 1.8205, 'learning_rate': 2e-05, 'epoch': 0.9} 90%|████████▉ | 1628/1817 [59:17:04<7:02:47, 134.22s/it] 90%|████████▉ | 1629/1817 [59:19:12<6:54:19, 132.23s/it] {'loss': 1.9202, 'learning_rate': 2e-05, 'epoch': 0.9} 90%|████████▉ | 1629/1817 [59:19:12<6:54:19, 132.23s/it] 90%|████████▉ | 1630/1817 [59:21:20<6:48:28, 131.06s/it] {'loss': 1.8871, 'learning_rate': 2e-05, 'epoch': 0.9} 90%|████████▉ | 1630/1817 [59:21:20<6:48:28, 131.06s/it] 90%|████████▉ | 1631/1817 [59:23:38<6:52:57, 133.21s/it] {'loss': 1.9076, 'learning_rate': 2e-05, 'epoch': 0.9} 90%|████████▉ | 1631/1817 [59:23:38<6:52:57, 133.21s/it] 90%|████████▉ | 1632/1817 [59:25:57<6:56:13, 134.99s/it] {'loss': 1.8333, 'learning_rate': 2e-05, 'epoch': 0.9} 90%|████████▉ | 1632/1817 [59:25:57<6:56:13, 134.99s/it] 90%|████████▉ | 1633/1817 [59:28:10<6:52:05, 134.38s/it] {'loss': 1.8212, 'learning_rate': 2e-05, 'epoch': 0.9} 90%|████████▉ | 1633/1817 [59:28:10<6:52:05, 134.38s/it] 90%|████████▉ | 1634/1817 [59:30:18<6:43:55, 132.44s/it] {'loss': 1.8993, 'learning_rate': 2e-05, 'epoch': 0.9} 90%|████████▉ | 1634/1817 [59:30:18<6:43:55, 132.44s/it] 90%|████████▉ | 1635/1817 [59:32:35<6:45:43, 133.75s/it] {'loss': 1.8713, 'learning_rate': 2e-05, 'epoch': 0.9} 90%|████████▉ | 1635/1817 [59:32:35<6:45:43, 133.75s/it] 90%|█████████ | 1636/1817 [59:34:48<6:43:15, 133.68s/it] {'loss': 1.8474, 'learning_rate': 2e-05, 'epoch': 0.9} 90%|█████████ | 1636/1817 [59:34:48<6:43:15, 133.68s/it] 90%|█████████ | 1637/1817 [59:37:03<6:41:50, 133.95s/it] {'loss': 1.8749, 'learning_rate': 2e-05, 'epoch': 0.9} 90%|█████████ | 1637/1817 [59:37:03<6:41:50, 133.95s/it] 90%|█████████ | 1638/1817 [59:39:13<6:35:54, 132.71s/it] {'loss': 1.8737, 'learning_rate': 2e-05, 'epoch': 0.9} 90%|█████████ | 1638/1817 [59:39:13<6:35:54, 132.71s/it] 90%|█████████ | 1639/1817 [59:41:24<6:32:12, 132.21s/it] {'loss': 1.872, 'learning_rate': 2e-05, 'epoch': 0.9} 90%|█████████ | 1639/1817 [59:41:24<6:32:12, 132.21s/it] 90%|█████████ | 1640/1817 [59:43:39<6:33:02, 133.24s/it] {'loss': 1.8854, 'learning_rate': 2e-05, 'epoch': 0.9} 90%|█████████ | 1640/1817 [59:43:39<6:33:02, 133.24s/it] 90%|█████████ | 1641/1817 [59:45:54<6:32:10, 133.70s/it] {'loss': 1.8928, 'learning_rate': 2e-05, 'epoch': 0.9} 90%|█████████ | 1641/1817 [59:45:54<6:32:10, 133.70s/it] 90%|█████████ | 1642/1817 [59:48:09<6:30:48, 133.99s/it] {'loss': 1.9048, 'learning_rate': 2e-05, 'epoch': 0.9} 90%|█████████ | 1642/1817 [59:48:09<6:30:48, 133.99s/it] 90%|█████████ | 1643/1817 [59:50:24<6:29:16, 134.23s/it] {'loss': 1.8525, 'learning_rate': 2e-05, 'epoch': 0.9} 90%|█████████ | 1643/1817 [59:50:24<6:29:16, 134.23s/it] 90%|█████████ | 1644/1817 [59:52:37<6:26:27, 134.03s/it] {'loss': 1.8911, 'learning_rate': 2e-05, 'epoch': 0.9} 90%|█████████ | 1644/1817 [59:52:37<6:26:27, 134.03s/it] 91%|█████████ | 1645/1817 [59:54:48<6:21:20, 133.03s/it] {'loss': 1.838, 'learning_rate': 2e-05, 'epoch': 0.91} 91%|█████████ | 1645/1817 [59:54:48<6:21:20, 133.03s/it] 91%|█████████ | 1646/1817 [59:57:04<6:21:53, 134.00s/it] {'loss': 1.8125, 'learning_rate': 2e-05, 'epoch': 0.91} 91%|█████████ | 1646/1817 [59:57:04<6:21:53, 134.00s/it] 91%|█████████ | 1647/1817 [59:59:12<6:14:26, 132.15s/it] {'loss': 1.9012, 'learning_rate': 2e-05, 'epoch': 0.91} 91%|█████████ | 1647/1817 [59:59:12<6:14:26, 132.16s/it] 91%|█████████ | 1648/1817 [60:01:23<6:11:18, 131.83s/it] {'loss': 1.8352, 'learning_rate': 2e-05, 'epoch': 0.91} 91%|█████████ | 1648/1817 [60:01:23<6:11:18, 131.83s/it] 91%|█████████ | 1649/1817 [60:03:43<6:15:30, 134.11s/it] {'loss': 1.8793, 'learning_rate': 2e-05, 'epoch': 0.91} 91%|█████████ | 1649/1817 [60:03:43<6:15:30, 134.11s/it] 91%|█████████ | 1650/1817 [60:05:53<6:09:55, 132.90s/it] {'loss': 1.8904, 'learning_rate': 2e-05, 'epoch': 0.91} 91%|█████████ | 1650/1817 [60:05:53<6:09:55, 132.91s/it] 91%|█████████ | 1651/1817 [60:08:01<6:03:38, 131.43s/it] {'loss': 1.8419, 'learning_rate': 2e-05, 'epoch': 0.91} 91%|█████████ | 1651/1817 [60:08:01<6:03:37, 131.43s/it] 91%|█████████ | 1652/1817 [60:10:15<6:03:31, 132.19s/it] {'loss': 1.8828, 'learning_rate': 2e-05, 'epoch': 0.91} 91%|█████████ | 1652/1817 [60:10:15<6:03:28, 132.17s/it] 91%|█████████ | 1653/1817 [60:12:30<6:03:45, 133.08s/it] {'loss': 1.8462, 'learning_rate': 2e-05, 'epoch': 0.91} 91%|█████████ | 1653/1817 [60:12:30<6:03:42, 133.06s/it] 91%|█████████ | 1654/1817 [60:14:43<6:01:04, 132.91s/it] {'loss': 1.9209, 'learning_rate': 2e-05, 'epoch': 0.91} 91%|█████████ | 1654/1817 [60:14:42<6:01:02, 132.90s/it] 91%|█████████ | 1655/1817 [60:16:59<6:01:59, 134.07s/it] {'loss': 1.8781, 'learning_rate': 2e-05, 'epoch': 0.91} 91%|█████████ | 1655/1817 [60:16:59<6:01:59, 134.07s/it] 91%|█████████ | 1656/1817 [60:19:09<5:56:18, 132.79s/it] {'loss': 1.8657, 'learning_rate': 2e-05, 'epoch': 0.91} 91%|█████████ | 1656/1817 [60:19:09<5:56:19, 132.79s/it] 91%|█████████ | 1657/1817 [60:21:22<5:54:06, 132.79s/it] {'loss': 1.9012, 'learning_rate': 2e-05, 'epoch': 0.91} 91%|█████████ | 1657/1817 [60:21:22<5:54:06, 132.79s/it] 91%|█████████ | 1658/1817 [60:23:41<5:56:42, 134.61s/it] {'loss': 1.8799, 'learning_rate': 2e-05, 'epoch': 0.91} 91%|█████████ | 1658/1817 [60:23:40<5:56:42, 134.61s/it] 91%|█████████▏| 1659/1817 [60:25:57<5:55:38, 135.05s/it] {'loss': 1.8579, 'learning_rate': 2e-05, 'epoch': 0.91} 91%|█████████▏| 1659/1817 [60:25:57<5:55:38, 135.05s/it] 91%|█████████▏| 1660/1817 [60:28:08<5:50:47, 134.06s/it] {'loss': 1.8629, 'learning_rate': 2e-05, 'epoch': 0.91} 91%|█████████▏| 1660/1817 [60:28:09<5:50:46, 134.06s/it] 91%|█████████▏| 1661/1817 [60:30:20<5:46:48, 133.39s/it] {'loss': 1.8847, 'learning_rate': 2e-05, 'epoch': 0.91} 91%|█████████▏| 1661/1817 [60:30:20<5:46:48, 133.39s/it] 91%|█████████▏| 1662/1817 [60:32:39<5:48:45, 135.00s/it] {'loss': 1.8533, 'learning_rate': 2e-05, 'epoch': 0.91} 91%|█████████▏| 1662/1817 [60:32:39<5:48:45, 135.00s/it] 92%|█████████▏| 1663/1817 [60:34:49<5:42:51, 133.58s/it] {'loss': 1.8537, 'learning_rate': 2e-05, 'epoch': 0.92} 92%|█████████▏| 1663/1817 [60:34:49<5:42:51, 133.58s/it] 92%|█████████▏| 1664/1817 [60:37:05<5:42:13, 134.21s/it] {'loss': 1.874, 'learning_rate': 2e-05, 'epoch': 0.92} 92%|█████████▏| 1664/1817 [60:37:05<5:42:13, 134.21s/it] 92%|█████████▏| 1665/1817 [60:39:24<5:43:34, 135.62s/it] {'loss': 1.8667, 'learning_rate': 2e-05, 'epoch': 0.92} 92%|█████████▏| 1665/1817 [60:39:24<5:43:34, 135.62s/it] 92%|█████████▏| 1666/1817 [60:41:41<5:42:18, 136.02s/it] {'loss': 1.8552, 'learning_rate': 2e-05, 'epoch': 0.92} 92%|█████████▏| 1666/1817 [60:41:41<5:42:18, 136.02s/it] 92%|█████████▏| 1667/1817 [60:43:54<5:38:14, 135.30s/it] {'loss': 1.8666, 'learning_rate': 2e-05, 'epoch': 0.92} 92%|█████████▏| 1667/1817 [60:43:54<5:38:14, 135.30s/it] 92%|█████████▏| 1668/1817 [60:46:07<5:34:21, 134.64s/it] {'loss': 1.897, 'learning_rate': 2e-05, 'epoch': 0.92} 92%|█████████▏| 1668/1817 [60:46:07<5:34:21, 134.64s/it] 92%|█████████▏| 1669/1817 [60:48:21<5:31:09, 134.26s/it] {'loss': 1.8951, 'learning_rate': 2e-05, 'epoch': 0.92} 92%|█████████▏| 1669/1817 [60:48:21<5:31:10, 134.26s/it] 92%|█████████▏| 1670/1817 [60:50:35<5:28:59, 134.28s/it] {'loss': 1.8607, 'learning_rate': 2e-05, 'epoch': 0.92} 92%|█████████▏| 1670/1817 [60:50:35<5:28:59, 134.28s/it] 92%|█████████▏| 1671/1817 [60:52:57<5:31:50, 136.37s/it] {'loss': 1.8659, 'learning_rate': 2e-05, 'epoch': 0.92} 92%|█████████▏| 1671/1817 [60:52:56<5:31:50, 136.37s/it] 92%|█████████▏| 1672/1817 [60:55:10<5:27:45, 135.62s/it] {'loss': 1.8903, 'learning_rate': 2e-05, 'epoch': 0.92} 92%|█████████▏| 1672/1817 [60:55:10<5:27:45, 135.62s/it] 92%|█████████▏| 1673/1817 [60:57:22<5:22:12, 134.26s/it] {'loss': 1.8719, 'learning_rate': 2e-05, 'epoch': 0.92} 92%|█████████▏| 1673/1817 [60:57:21<5:22:13, 134.26s/it] 92%|█████████▏| 1674/1817 [60:59:34<5:18:27, 133.62s/it] {'loss': 1.9339, 'learning_rate': 2e-05, 'epoch': 0.92} 92%|█████████▏| 1674/1817 [60:59:33<5:18:27, 133.62s/it] 92%|█████████▏| 1675/1817 [61:01:51<5:18:55, 134.76s/it] {'loss': 1.8857, 'learning_rate': 2e-05, 'epoch': 0.92} 92%|█████████▏| 1675/1817 [61:01:51<5:18:55, 134.76s/it] 92%|█████████▏| 1676/1817 [61:04:04<5:15:54, 134.43s/it] {'loss': 1.9035, 'learning_rate': 2e-05, 'epoch': 0.92} 92%|█████████▏| 1676/1817 [61:04:04<5:15:54, 134.43s/it] 92%|█████████▏| 1677/1817 [61:06:19<5:13:57, 134.56s/it] {'loss': 1.8748, 'learning_rate': 2e-05, 'epoch': 0.92} 92%|█████████▏| 1677/1817 [61:06:19<5:13:57, 134.56s/it] 92%|█████████▏| 1678/1817 [61:08:31<5:09:42, 133.69s/it] {'loss': 1.8685, 'learning_rate': 2e-05, 'epoch': 0.92} 92%|█████████▏| 1678/1817 [61:08:31<5:09:42, 133.68s/it] 92%|█████████▏| 1679/1817 [61:10:49<5:10:30, 135.00s/it] {'loss': 1.9209, 'learning_rate': 2e-05, 'epoch': 0.92} 92%|█████████▏| 1679/1817 [61:10:49<5:10:30, 135.01s/it] 92%|█████████▏| 1680/1817 [61:13:05<5:08:37, 135.16s/it] {'loss': 1.9362, 'learning_rate': 2e-05, 'epoch': 0.92} 92%|█████████▏| 1680/1817 [61:13:05<5:08:37, 135.16s/it] 93%|█████████▎| 1681/1817 [61:15:19<5:05:39, 134.85s/it] {'loss': 1.8942, 'learning_rate': 2e-05, 'epoch': 0.92} 93%|█████████▎| 1681/1817 [61:15:19<5:05:39, 134.85s/it] 93%|█████████▎| 1682/1817 [61:17:32<5:02:05, 134.26s/it] {'loss': 1.8841, 'learning_rate': 2e-05, 'epoch': 0.93} 93%|█████████▎| 1682/1817 [61:17:32<5:02:06, 134.27s/it] 93%|█████████▎| 1683/1817 [61:19:46<4:59:57, 134.31s/it] {'loss': 1.8662, 'learning_rate': 2e-05, 'epoch': 0.93} 93%|█████████▎| 1683/1817 [61:19:46<4:59:57, 134.31s/it] 93%|█████████▎| 1684/1817 [61:21:59<4:56:43, 133.86s/it] {'loss': 1.8387, 'learning_rate': 2e-05, 'epoch': 0.93} 93%|█████████▎| 1684/1817 [61:21:59<4:56:43, 133.86s/it] 93%|█████████▎| 1685/1817 [61:24:13<4:54:36, 133.91s/it] {'loss': 1.897, 'learning_rate': 2e-05, 'epoch': 0.93} 93%|█████████▎| 1685/1817 [61:24:13<4:54:37, 133.92s/it] 93%|█████████▎| 1686/1817 [61:26:28<4:52:55, 134.16s/it] {'loss': 1.8769, 'learning_rate': 2e-05, 'epoch': 0.93} 93%|█████████▎| 1686/1817 [61:26:28<4:52:55, 134.16s/it] 93%|█████████▎| 1687/1817 [61:28:37<4:47:51, 132.86s/it] {'loss': 1.8549, 'learning_rate': 2e-05, 'epoch': 0.93} 93%|█████████▎| 1687/1817 [61:28:37<4:47:51, 132.86s/it] 93%|█████████▎| 1688/1817 [61:30:47<4:43:35, 131.90s/it] {'loss': 1.8705, 'learning_rate': 2e-05, 'epoch': 0.93} 93%|█████████▎| 1688/1817 [61:30:47<4:43:35, 131.90s/it] 93%|█████████▎| 1689/1817 [61:33:00<4:42:07, 132.24s/it] {'loss': 1.9146, 'learning_rate': 2e-05, 'epoch': 0.93} 93%|█████████▎| 1689/1817 [61:33:00<4:42:07, 132.24s/it] 93%|█████████▎| 1690/1817 [61:35:10<4:38:05, 131.38s/it] {'loss': 1.908, 'learning_rate': 2e-05, 'epoch': 0.93} 93%|█████████▎| 1690/1817 [61:35:10<4:38:05, 131.38s/it] 93%|█████████▎| 1691/1817 [61:37:19<4:34:33, 130.74s/it] {'loss': 1.8196, 'learning_rate': 2e-05, 'epoch': 0.93} 93%|█████████▎| 1691/1817 [61:37:19<4:34:33, 130.74s/it] 93%|█████████▎| 1692/1817 [61:39:28<4:31:29, 130.31s/it] {'loss': 1.8368, 'learning_rate': 2e-05, 'epoch': 0.93} 93%|█████████▎| 1692/1817 [61:39:28<4:31:29, 130.32s/it] 93%|█████████▎| 1693/1817 [61:41:38<4:29:03, 130.19s/it] {'loss': 1.8883, 'learning_rate': 2e-05, 'epoch': 0.93} 93%|█████████▎| 1693/1817 [61:41:38<4:29:03, 130.19s/it] 93%|█████████▎| 1694/1817 [61:43:52<4:29:07, 131.28s/it] {'loss': 1.8544, 'learning_rate': 2e-05, 'epoch': 0.93} 93%|█████████▎| 1694/1817 [61:43:52<4:29:07, 131.28s/it] 93%|█████████▎| 1695/1817 [61:46:06<4:28:48, 132.20s/it] {'loss': 1.879, 'learning_rate': 2e-05, 'epoch': 0.93} 93%|█████████▎| 1695/1817 [61:46:06<4:28:48, 132.20s/it] 93%|█████████▎| 1696/1817 [61:48:15<4:24:27, 131.14s/it] {'loss': 1.8944, 'learning_rate': 2e-05, 'epoch': 0.93} 93%|█████████▎| 1696/1817 [61:48:15<4:24:28, 131.14s/it] 93%|█████████▎| 1697/1817 [61:50:27<4:23:02, 131.52s/it] {'loss': 1.8654, 'learning_rate': 2e-05, 'epoch': 0.93} 93%|█████████▎| 1697/1817 [61:50:27<4:23:02, 131.52s/it] 93%|█████████▎| 1698/1817 [61:52:40<4:21:17, 131.74s/it] {'loss': 1.9071, 'learning_rate': 2e-05, 'epoch': 0.93} 93%|█████████▎| 1698/1817 [61:52:40<4:21:17, 131.74s/it] 94%|█████████▎| 1699/1817 [61:54:49<4:17:30, 130.94s/it] {'loss': 1.8566, 'learning_rate': 2e-05, 'epoch': 0.93} 94%|█████████▎| 1699/1817 [61:54:49<4:17:30, 130.94s/it] 94%|█████████▎| 1700/1817 [61:57:02<4:16:32, 131.56s/it] {'loss': 1.9074, 'learning_rate': 2e-05, 'epoch': 0.94} 94%|█████████▎| 1700/1817 [61:57:02<4:16:32, 131.56s/it] 94%|█████████▎| 1701/1817 [61:59:14<4:14:36, 131.69s/it] {'loss': 1.8927, 'learning_rate': 2e-05, 'epoch': 0.94} 94%|█████████▎| 1701/1817 [61:59:14<4:14:36, 131.69s/it] 94%|█████████▎| 1702/1817 [62:01:28<4:14:15, 132.65s/it] {'loss': 1.8955, 'learning_rate': 2e-05, 'epoch': 0.94} 94%|█████████▎| 1702/1817 [62:01:28<4:14:15, 132.65s/it] 94%|█████████▎| 1703/1817 [62:03:41<4:11:47, 132.52s/it] {'loss': 1.9155, 'learning_rate': 2e-05, 'epoch': 0.94} 94%|█████████▎| 1703/1817 [62:03:41<4:11:47, 132.52s/it] 94%|█████████▍| 1704/1817 [62:05:48<4:06:52, 131.09s/it] {'loss': 1.8717, 'learning_rate': 2e-05, 'epoch': 0.94} 94%|█████████▍| 1704/1817 [62:05:48<4:06:52, 131.09s/it] 94%|█████████▍| 1705/1817 [62:08:01<4:05:28, 131.50s/it] {'loss': 1.8442, 'learning_rate': 2e-05, 'epoch': 0.94} 94%|█████████▍| 1705/1817 [62:08:01<4:05:28, 131.50s/it] 94%|█████████▍| 1706/1817 [62:10:10<4:01:55, 130.77s/it] {'loss': 1.8978, 'learning_rate': 2e-05, 'epoch': 0.94} 94%|█████████▍| 1706/1817 [62:10:10<4:01:55, 130.77s/it] 94%|█████████▍| 1707/1817 [62:12:22<4:00:13, 131.04s/it] {'loss': 1.9095, 'learning_rate': 2e-05, 'epoch': 0.94} 94%|█████████▍| 1707/1817 [62:12:22<4:00:13, 131.04s/it] 94%|█████████▍| 1708/1817 [62:14:32<3:57:30, 130.74s/it] {'loss': 1.8556, 'learning_rate': 2e-05, 'epoch': 0.94} 94%|█████████▍| 1708/1817 [62:14:32<3:57:30, 130.74s/it] 94%|█████████▍| 1709/1817 [62:16:38<3:53:12, 129.56s/it] {'loss': 1.925, 'learning_rate': 2e-05, 'epoch': 0.94} 94%|█████████▍| 1709/1817 [62:16:39<3:53:12, 129.56s/it] 94%|█████████▍| 1710/1817 [62:18:46<3:50:06, 129.03s/it] {'loss': 1.8797, 'learning_rate': 2e-05, 'epoch': 0.94} 94%|█████████▍| 1710/1817 [62:18:46<3:50:06, 129.03s/it] 94%|█████████▍| 1711/1817 [62:20:58<3:49:36, 129.96s/it] {'loss': 1.8596, 'learning_rate': 2e-05, 'epoch': 0.94} 94%|█████████▍| 1711/1817 [62:20:58<3:49:36, 129.96s/it] 94%|█████████▍| 1712/1817 [62:23:11<3:48:58, 130.84s/it] {'loss': 1.8717, 'learning_rate': 2e-05, 'epoch': 0.94} 94%|█████████▍| 1712/1817 [62:23:11<3:48:58, 130.84s/it] 94%|█████████▍| 1713/1817 [62:25:21<3:46:04, 130.43s/it] {'loss': 1.8875, 'learning_rate': 2e-05, 'epoch': 0.94} 94%|█████████▍| 1713/1817 [62:25:21<3:46:04, 130.43s/it] 94%|█████████▍| 1714/1817 [62:27:29<3:42:28, 129.60s/it] {'loss': 1.84, 'learning_rate': 2e-05, 'epoch': 0.94} 94%|█████████▍| 1714/1817 [62:27:28<3:42:28, 129.60s/it] 94%|█████████▍| 1715/1817 [62:29:44<3:43:14, 131.31s/it] {'loss': 1.8697, 'learning_rate': 2e-05, 'epoch': 0.94} 94%|█████████▍| 1715/1817 [62:29:44<3:43:14, 131.31s/it] 94%|█████████▍| 1716/1817 [62:31:52<3:39:22, 130.33s/it] {'loss': 1.859, 'learning_rate': 2e-05, 'epoch': 0.94} 94%|█████████▍| 1716/1817 [62:31:52<3:39:22, 130.33s/it] 94%|█████████▍| 1717/1817 [62:34:02<3:36:59, 130.19s/it] {'loss': 1.9052, 'learning_rate': 2e-05, 'epoch': 0.94} 94%|█████████▍| 1717/1817 [62:34:02<3:36:59, 130.19s/it] 95%|█████████▍| 1718/1817 [62:36:13<3:35:26, 130.57s/it] {'loss': 1.8825, 'learning_rate': 2e-05, 'epoch': 0.95} 95%|█████████▍| 1718/1817 [62:36:13<3:35:26, 130.57s/it] 95%|█████████▍| 1719/1817 [62:38:25<3:34:04, 131.07s/it] {'loss': 1.8936, 'learning_rate': 2e-05, 'epoch': 0.95} 95%|█████████▍| 1719/1817 [62:38:25<3:34:04, 131.07s/it] 95%|█████████▍| 1720/1817 [62:40:37<3:32:20, 131.34s/it] {'loss': 1.8468, 'learning_rate': 2e-05, 'epoch': 0.95} 95%|█████████▍| 1720/1817 [62:40:37<3:32:20, 131.34s/it] 95%|█████████▍| 1721/1817 [62:42:50<3:30:50, 131.78s/it] {'loss': 1.8993, 'learning_rate': 2e-05, 'epoch': 0.95} 95%|█████████▍| 1721/1817 [62:42:50<3:30:50, 131.78s/it] 95%|█████████▍| 1722/1817 [62:45:05<3:30:13, 132.77s/it] {'loss': 1.8449, 'learning_rate': 2e-05, 'epoch': 0.95} 95%|█████████▍| 1722/1817 [62:45:05<3:30:13, 132.77s/it] 95%|█████████▍| 1723/1817 [62:47:17<3:27:27, 132.42s/it] {'loss': 1.8636, 'learning_rate': 2e-05, 'epoch': 0.95} 95%|█████████▍| 1723/1817 [62:47:17<3:27:27, 132.42s/it] 95%|█████████▍| 1724/1817 [62:49:27<3:24:01, 131.63s/it] {'loss': 1.8804, 'learning_rate': 2e-05, 'epoch': 0.95} 95%|█████████▍| 1724/1817 [62:49:27<3:24:01, 131.63s/it] 95%|█████████▍| 1725/1817 [62:51:41<3:23:03, 132.43s/it] {'loss': 1.8738, 'learning_rate': 2e-05, 'epoch': 0.95} 95%|█████████▍| 1725/1817 [62:51:41<3:23:03, 132.43s/it] 95%|█████████▍| 1726/1817 [62:53:51<3:19:33, 131.58s/it] {'loss': 1.8988, 'learning_rate': 2e-05, 'epoch': 0.95} 95%|█████████▍| 1726/1817 [62:53:50<3:19:33, 131.58s/it] 95%|█████████▌| 1727/1817 [62:56:00<3:16:29, 130.99s/it] {'loss': 1.8514, 'learning_rate': 2e-05, 'epoch': 0.95} 95%|█████████▌| 1727/1817 [62:56:00<3:16:29, 130.99s/it] 95%|█████████▌| 1728/1817 [62:58:10<3:13:50, 130.68s/it] {'loss': 1.8495, 'learning_rate': 2e-05, 'epoch': 0.95} 95%|█████████▌| 1728/1817 [62:58:10<3:13:50, 130.68s/it] 95%|█████████▌| 1729/1817 [63:00:18<3:10:23, 129.82s/it] {'loss': 1.8892, 'learning_rate': 2e-05, 'epoch': 0.95} 95%|█████████▌| 1729/1817 [63:00:18<3:10:23, 129.82s/it] 95%|█████████▌| 1730/1817 [63:02:28<3:08:14, 129.82s/it] {'loss': 1.8706, 'learning_rate': 2e-05, 'epoch': 0.95} 95%|█████████▌| 1730/1817 [63:02:28<3:08:14, 129.82s/it] 95%|█████████▌| 1731/1817 [63:04:35<3:04:53, 129.00s/it] {'loss': 1.8389, 'learning_rate': 2e-05, 'epoch': 0.95} 95%|█████████▌| 1731/1817 [63:04:35<3:04:53, 129.00s/it] 95%|█████████▌| 1732/1817 [63:06:46<3:03:33, 129.57s/it] {'loss': 1.8946, 'learning_rate': 2e-05, 'epoch': 0.95} 95%|█████████▌| 1732/1817 [63:06:46<3:03:33, 129.57s/it] 95%|█████████▌| 1733/1817 [63:08:56<3:01:49, 129.87s/it] {'loss': 1.8783, 'learning_rate': 2e-05, 'epoch': 0.95} 95%|█████████▌| 1733/1817 [63:08:56<3:01:49, 129.87s/it] 95%|█████████▌| 1734/1817 [63:11:07<3:00:09, 130.24s/it] {'loss': 1.8355, 'learning_rate': 2e-05, 'epoch': 0.95} 95%|█████████▌| 1734/1817 [63:11:07<3:00:09, 130.24s/it] 95%|█████████▌| 1735/1817 [63:13:20<2:58:43, 130.77s/it] {'loss': 1.8433, 'learning_rate': 2e-05, 'epoch': 0.95} 95%|█████████▌| 1735/1817 [63:13:19<2:58:43, 130.77s/it] 96%|█████████▌| 1736/1817 [63:15:31<2:56:47, 130.95s/it] {'loss': 1.8575, 'learning_rate': 2e-05, 'epoch': 0.96} 96%|█████████▌| 1736/1817 [63:15:31<2:56:46, 130.95s/it] 96%|█████████▌| 1737/1817 [63:17:42<2:54:32, 130.91s/it] {'loss': 1.8635, 'learning_rate': 2e-05, 'epoch': 0.96} 96%|█████████▌| 1737/1817 [63:17:42<2:54:32, 130.91s/it] 96%|█████████▌| 1738/1817 [63:19:51<2:51:52, 130.53s/it] {'loss': 1.8848, 'learning_rate': 2e-05, 'epoch': 0.96} 96%|█████████▌| 1738/1817 [63:19:51<2:51:52, 130.53s/it] 96%|█████████▌| 1739/1817 [63:22:04<2:50:31, 131.17s/it] {'loss': 1.8945, 'learning_rate': 2e-05, 'epoch': 0.96} 96%|█████████▌| 1739/1817 [63:22:04<2:50:31, 131.18s/it] 96%|█████████▌| 1740/1817 [63:24:18<2:49:36, 132.16s/it] {'loss': 1.8499, 'learning_rate': 2e-05, 'epoch': 0.96} 96%|█████████▌| 1740/1817 [63:24:18<2:49:36, 132.16s/it] 96%|█████████▌| 1741/1817 [63:26:25<2:45:16, 130.48s/it] {'loss': 1.9152, 'learning_rate': 2e-05, 'epoch': 0.96} 96%|█████████▌| 1741/1817 [63:26:25<2:45:16, 130.48s/it] 96%|█████████▌| 1742/1817 [63:28:35<2:42:49, 130.26s/it] {'loss': 1.8679, 'learning_rate': 2e-05, 'epoch': 0.96} 96%|█████████▌| 1742/1817 [63:28:35<2:42:49, 130.26s/it] 96%|█████████▌| 1743/1817 [63:30:41<2:39:18, 129.17s/it] {'loss': 1.834, 'learning_rate': 2e-05, 'epoch': 0.96} 96%|█████████▌| 1743/1817 [63:30:41<2:39:18, 129.17s/it] 96%|█████████▌| 1744/1817 [63:32:53<2:38:13, 130.04s/it] {'loss': 1.9023, 'learning_rate': 2e-05, 'epoch': 0.96} 96%|█████████▌| 1744/1817 [63:32:53<2:38:13, 130.04s/it] 96%|█████████▌| 1745/1817 [63:35:02<2:35:29, 129.57s/it] {'loss': 1.8412, 'learning_rate': 2e-05, 'epoch': 0.96} 96%|█████████▌| 1745/1817 [63:35:02<2:35:29, 129.57s/it] 96%|█████████▌| 1746/1817 [63:37:17<2:35:13, 131.18s/it] {'loss': 1.8567, 'learning_rate': 2e-05, 'epoch': 0.96} 96%|█████████▌| 1746/1817 [63:37:17<2:35:13, 131.18s/it] 96%|█████████▌| 1747/1817 [63:39:26<2:32:11, 130.46s/it] {'loss': 1.9078, 'learning_rate': 2e-05, 'epoch': 0.96} 96%|█████████▌| 1747/1817 [63:39:26<2:32:11, 130.46s/it] 96%|█████████▌| 1748/1817 [63:41:36<2:29:54, 130.36s/it] {'loss': 1.8781, 'learning_rate': 2e-05, 'epoch': 0.96} 96%|█████████▌| 1748/1817 [63:41:36<2:29:54, 130.36s/it] 96%|█████████▋| 1749/1817 [63:43:47<2:28:00, 130.60s/it] {'loss': 1.8181, 'learning_rate': 2e-05, 'epoch': 0.96} 96%|█████████▋| 1749/1817 [63:43:47<2:28:00, 130.60s/it] 96%|█████████▋| 1750/1817 [63:46:00<2:26:41, 131.37s/it] {'loss': 1.8478, 'learning_rate': 2e-05, 'epoch': 0.96} 96%|█████████▋| 1750/1817 [63:46:00<2:26:41, 131.37s/it] 96%|█████████▋| 1751/1817 [63:48:11<2:24:31, 131.39s/it] {'loss': 1.8724, 'learning_rate': 2e-05, 'epoch': 0.96} 96%|█████████▋| 1751/1817 [63:48:11<2:24:31, 131.39s/it] 96%|█████████▋| 1752/1817 [63:50:25<2:22:53, 131.90s/it] {'loss': 1.8737, 'learning_rate': 2e-05, 'epoch': 0.96} 96%|█████████▋| 1752/1817 [63:50:25<2:22:53, 131.90s/it] 96%|█████████▋| 1753/1817 [63:52:34<2:19:51, 131.12s/it] {'loss': 1.8798, 'learning_rate': 2e-05, 'epoch': 0.96} 96%|█████████▋| 1753/1817 [63:52:34<2:19:51, 131.12s/it] 97%|█████████▋| 1754/1817 [63:54:42<2:16:38, 130.13s/it] {'loss': 1.8806, 'learning_rate': 2e-05, 'epoch': 0.97} 97%|█████████▋| 1754/1817 [63:54:42<2:16:38, 130.13s/it] 97%|█████████▋| 1755/1817 [63:56:49<2:13:28, 129.17s/it] {'loss': 1.8774, 'learning_rate': 2e-05, 'epoch': 0.97} 97%|█████████▋| 1755/1817 [63:56:49<2:13:28, 129.17s/it] 97%|█████████▋| 1756/1817 [63:59:00<2:11:58, 129.81s/it] {'loss': 1.8783, 'learning_rate': 2e-05, 'epoch': 0.97} 97%|█████████▋| 1756/1817 [63:59:00<2:11:58, 129.81s/it] 97%|█████████▋| 1757/1817 [64:01:11<2:10:00, 130.01s/it] {'loss': 1.8749, 'learning_rate': 2e-05, 'epoch': 0.97} 97%|█████████▋| 1757/1817 [64:01:10<2:10:00, 130.01s/it] 97%|█████████▋| 1758/1817 [64:03:19<2:07:28, 129.63s/it] {'loss': 1.862, 'learning_rate': 2e-05, 'epoch': 0.97} 97%|█████████▋| 1758/1817 [64:03:19<2:07:28, 129.63s/it] 97%|█████████▋| 1759/1817 [64:05:35<2:07:06, 131.49s/it] {'loss': 1.8775, 'learning_rate': 2e-05, 'epoch': 0.97} 97%|█████████▋| 1759/1817 [64:05:35<2:07:06, 131.49s/it] 97%|█████████▋| 1760/1817 [64:07:45<2:04:23, 130.94s/it] {'loss': 1.8552, 'learning_rate': 2e-05, 'epoch': 0.97} 97%|█████████▋| 1760/1817 [64:07:45<2:04:23, 130.94s/it] 97%|█████████▋| 1761/1817 [64:09:57<2:02:25, 131.17s/it] {'loss': 1.8731, 'learning_rate': 2e-05, 'epoch': 0.97} 97%|█████████▋| 1761/1817 [64:09:56<2:02:25, 131.17s/it] 97%|█████████▋| 1762/1817 [64:12:11<2:01:13, 132.25s/it] {'loss': 1.8578, 'learning_rate': 2e-05, 'epoch': 0.97} 97%|█████████▋| 1762/1817 [64:12:11<2:01:12, 132.23s/it] 97%|█████████▋| 1763/1817 [64:14:19<1:57:42, 130.78s/it] {'loss': 1.89, 'learning_rate': 2e-05, 'epoch': 0.97} 97%|█████████▋| 1763/1817 [64:14:18<1:57:41, 130.77s/it] 97%|█████████▋| 1764/1817 [64:16:27<1:54:58, 130.17s/it] {'loss': 1.8835, 'learning_rate': 2e-05, 'epoch': 0.97} 97%|█████████▋| 1764/1817 [64:16:27<1:54:58, 130.17s/it] 97%|█████████▋| 1765/1817 [64:18:41<1:53:41, 131.18s/it] {'loss': 1.872, 'learning_rate': 2e-05, 'epoch': 0.97} 97%|█████████▋| 1765/1817 [64:18:41<1:53:41, 131.18s/it] 97%|█████████▋| 1766/1817 [64:20:54<1:51:51, 131.61s/it] {'loss': 1.9179, 'learning_rate': 2e-05, 'epoch': 0.97} 97%|█████████▋| 1766/1817 [64:20:53<1:51:51, 131.61s/it] 97%|█████████▋| 1767/1817 [64:23:04<1:49:16, 131.14s/it] {'loss': 1.8593, 'learning_rate': 2e-05, 'epoch': 0.97} 97%|█████████▋| 1767/1817 [64:23:03<1:49:16, 131.14s/it] 97%|█████████▋| 1768/1817 [64:25:13<1:46:36, 130.54s/it] {'loss': 1.8647, 'learning_rate': 2e-05, 'epoch': 0.97} 97%|█████████▋| 1768/1817 [64:25:12<1:46:36, 130.54s/it] 97%|█████████▋| 1769/1817 [64:27:25<1:44:45, 130.96s/it] {'loss': 1.8653, 'learning_rate': 2e-05, 'epoch': 0.97} 97%|█████████▋| 1769/1817 [64:27:24<1:44:45, 130.96s/it] 97%|█████████▋| 1770/1817 [64:29:34<1:42:19, 130.62s/it] {'loss': 1.8078, 'learning_rate': 2e-05, 'epoch': 0.97} 97%|█████████▋| 1770/1817 [64:29:34<1:42:19, 130.62s/it] 97%|█████████▋| 1771/1817 [64:31:43<1:39:43, 130.08s/it] {'loss': 1.829, 'learning_rate': 2e-05, 'epoch': 0.97} 97%|█████████▋| 1771/1817 [64:31:43<1:39:43, 130.08s/it] 98%|█████████▊| 1772/1817 [64:33:56<1:38:07, 130.83s/it] {'loss': 1.8342, 'learning_rate': 2e-05, 'epoch': 0.97} 98%|█████████▊| 1772/1817 [64:33:56<1:38:07, 130.83s/it] 98%|█████████▊| 1773/1817 [64:36:02<1:34:59, 129.54s/it] {'loss': 1.8369, 'learning_rate': 2e-05, 'epoch': 0.98} 98%|█████████▊| 1773/1817 [64:36:02<1:35:00, 129.55s/it] 98%|█████████▊| 1774/1817 [64:38:15<1:33:33, 130.54s/it] {'loss': 1.8374, 'learning_rate': 2e-05, 'epoch': 0.98} 98%|█████████▊| 1774/1817 [64:38:15<1:33:33, 130.54s/it] 98%|█████████▊| 1775/1817 [64:40:27<1:31:33, 130.81s/it] {'loss': 1.9067, 'learning_rate': 2e-05, 'epoch': 0.98} 98%|█████████▊| 1775/1817 [64:40:26<1:31:33, 130.81s/it] 98%|█████████▊| 1776/1817 [64:42:35<1:28:50, 130.02s/it] {'loss': 1.8967, 'learning_rate': 2e-05, 'epoch': 0.98} 98%|█████████▊| 1776/1817 [64:42:35<1:28:50, 130.02s/it] 98%|█████████▊| 1777/1817 [64:44:42<1:26:11, 129.28s/it] {'loss': 1.8906, 'learning_rate': 2e-05, 'epoch': 0.98} 98%|█████████▊| 1777/1817 [64:44:42<1:26:11, 129.28s/it] 98%|█████████▊| 1778/1817 [64:46:53<1:24:21, 129.79s/it] {'loss': 1.8684, 'learning_rate': 2e-05, 'epoch': 0.98} 98%|█████████▊| 1778/1817 [64:46:53<1:24:21, 129.79s/it] 98%|█████████▊| 1779/1817 [64:49:07<1:22:52, 130.86s/it] {'loss': 1.8642, 'learning_rate': 2e-05, 'epoch': 0.98} 98%|█████████▊| 1779/1817 [64:49:07<1:22:52, 130.86s/it] 98%|█████████▊| 1780/1817 [64:51:18<1:20:49, 131.08s/it] {'loss': 1.8712, 'learning_rate': 2e-05, 'epoch': 0.98} 98%|█████████▊| 1780/1817 [64:51:18<1:20:49, 131.08s/it] 98%|█████████▊| 1781/1817 [64:53:30<1:18:41, 131.17s/it] {'loss': 1.8705, 'learning_rate': 2e-05, 'epoch': 0.98} 98%|█████████▊| 1781/1817 [64:53:29<1:18:41, 131.17s/it] 98%|█████████▊| 1782/1817 [64:55:39<1:16:07, 130.50s/it] {'loss': 1.8623, 'learning_rate': 2e-05, 'epoch': 0.98} 98%|█████████▊| 1782/1817 [64:55:38<1:16:07, 130.50s/it] 98%|█████████▊| 1783/1817 [64:57:51<1:14:13, 130.98s/it] {'loss': 1.839, 'learning_rate': 2e-05, 'epoch': 0.98} 98%|█████████▊| 1783/1817 [64:57:51<1:14:13, 130.98s/it] 98%|█████████▊| 1784/1817 [65:00:00<1:11:49, 130.59s/it] {'loss': 1.8821, 'learning_rate': 2e-05, 'epoch': 0.98} 98%|█████████▊| 1784/1817 [65:00:00<1:11:49, 130.59s/it] 98%|█████████▊| 1785/1817 [65:02:16<1:10:27, 132.10s/it] {'loss': 1.8676, 'learning_rate': 2e-05, 'epoch': 0.98} 98%|█████████▊| 1785/1817 [65:02:16<1:10:27, 132.10s/it] 98%|█████████▊| 1786/1817 [65:04:26<1:07:58, 131.56s/it] {'loss': 1.8502, 'learning_rate': 2e-05, 'epoch': 0.98} 98%|█████████▊| 1786/1817 [65:04:26<1:07:58, 131.56s/it] 98%|█████████▊| 1787/1817 [65:06:40<1:06:02, 132.08s/it] {'loss': 1.8297, 'learning_rate': 2e-05, 'epoch': 0.98} 98%|█████████▊| 1787/1817 [65:06:39<1:06:02, 132.08s/it] 98%|█████████▊| 1788/1817 [65:08:53<1:03:58, 132.36s/it] {'loss': 1.8825, 'learning_rate': 2e-05, 'epoch': 0.98} 98%|█████████▊| 1788/1817 [65:08:52<1:03:58, 132.36s/it] 98%|█████████▊| 1789/1817 [65:11:03<1:01:31, 131.83s/it] {'loss': 1.8766, 'learning_rate': 2e-05, 'epoch': 0.98} 98%|█████████▊| 1789/1817 [65:11:03<1:01:30, 131.81s/it] 99%|█████████▊| 1790/1817 [65:13:14<59:10, 131.51s/it] {'loss': 1.859, 'learning_rate': 2e-05, 'epoch': 0.98} 99%|█████████▊| 1790/1817 [65:13:14<59:10, 131.51s/it] 99%|█████████▊| 1791/1817 [65:15:28<57:20, 132.31s/it] {'loss': 1.8339, 'learning_rate': 2e-05, 'epoch': 0.99} 99%|█████████▊| 1791/1817 [65:15:28<57:20, 132.31s/it] 99%|█████████▊| 1792/1817 [65:17:37<54:46, 131.45s/it] {'loss': 1.8778, 'learning_rate': 2e-05, 'epoch': 0.99} 99%|█████████▊| 1792/1817 [65:17:37<54:46, 131.45s/it] 99%|█████████▊| 1793/1817 [65:19:49<52:31, 131.33s/it] {'loss': 1.8697, 'learning_rate': 2e-05, 'epoch': 0.99} 99%|█████████▊| 1793/1817 [65:19:48<52:31, 131.33s/it] 99%|█████████▊| 1794/1817 [65:21:56<49:53, 130.15s/it] {'loss': 1.8438, 'learning_rate': 2e-05, 'epoch': 0.99} 99%|█████████▊| 1794/1817 [65:21:56<49:53, 130.15s/it] 99%|█████████▉| 1795/1817 [65:24:09<48:01, 130.96s/it] {'loss': 1.8587, 'learning_rate': 2e-05, 'epoch': 0.99} 99%|█████████▉| 1795/1817 [65:24:09<48:01, 130.96s/it] 99%|█████████▉| 1796/1817 [65:26:22<46:03, 131.58s/it] {'loss': 1.8422, 'learning_rate': 2e-05, 'epoch': 0.99} 99%|█████████▉| 1796/1817 [65:26:22<46:03, 131.58s/it] 99%|█████████▉| 1797/1817 [65:28:32<43:42, 131.15s/it] {'loss': 1.8591, 'learning_rate': 2e-05, 'epoch': 0.99} 99%|█████████▉| 1797/1817 [65:28:32<43:42, 131.15s/it] 99%|█████████▉| 1798/1817 [65:30:39<41:05, 129.78s/it] {'loss': 1.8402, 'learning_rate': 2e-05, 'epoch': 0.99} 99%|█████████▉| 1798/1817 [65:30:38<41:05, 129.78s/it] 99%|█████████▉| 1799/1817 [65:32:49<39:00, 130.02s/it] {'loss': 1.8282, 'learning_rate': 2e-05, 'epoch': 0.99} 99%|█████████▉| 1799/1817 [65:32:49<39:00, 130.02s/it] 99%|█████████▉| 1800/1817 [65:34:58<36:42, 129.54s/it] {'loss': 1.8519, 'learning_rate': 2e-05, 'epoch': 0.99} 99%|█████████▉| 1800/1817 [65:34:57<36:42, 129.54s/it] 99%|█████████▉| 1801/1817 [65:37:06<34:29, 129.33s/it] {'loss': 1.859, 'learning_rate': 2e-05, 'epoch': 0.99} 99%|█████████▉| 1801/1817 [65:37:06<34:29, 129.33s/it] 99%|█████████▉| 1802/1817 [65:39:16<32:21, 129.40s/it] {'loss': 1.8665, 'learning_rate': 2e-05, 'epoch': 0.99} 99%|█████████▉| 1802/1817 [65:39:16<32:21, 129.40s/it] 99%|█████████▉| 1803/1817 [65:41:27<30:17, 129.80s/it] {'loss': 1.9243, 'learning_rate': 2e-05, 'epoch': 0.99} 99%|█████████▉| 1803/1817 [65:41:27<30:17, 129.80s/it] 99%|█████████▉| 1804/1817 [65:43:34<27:55, 128.87s/it] {'loss': 1.8668, 'learning_rate': 2e-05, 'epoch': 0.99} 99%|█████████▉| 1804/1817 [65:43:33<27:55, 128.87s/it] 99%|█████████▉| 1805/1817 [65:45:44<25:51, 129.30s/it] {'loss': 1.8742, 'learning_rate': 2e-05, 'epoch': 0.99} 99%|█████████▉| 1805/1817 [65:45:44<25:51, 129.30s/it] 99%|█████████▉| 1806/1817 [65:47:55<23:47, 129.79s/it] {'loss': 1.8725, 'learning_rate': 2e-05, 'epoch': 0.99} 99%|█████████▉| 1806/1817 [65:47:54<23:47, 129.79s/it] 99%|█████████▉| 1807/1817 [65:50:06<21:41, 130.18s/it] {'loss': 1.8672, 'learning_rate': 2e-05, 'epoch': 0.99} 99%|█████████▉| 1807/1817 [65:50:06<21:41, 130.18s/it] 100%|█████████▉| 1808/1817 [65:52:15<19:29, 129.94s/it] {'loss': 1.8796, 'learning_rate': 2e-05, 'epoch': 0.99} 100%|█████████▉| 1808/1817 [65:52:15<19:29, 129.94s/it] 100%|█████████▉| 1809/1817 [65:54:24<17:16, 129.62s/it] {'loss': 1.849, 'learning_rate': 2e-05, 'epoch': 1.0} 100%|█████████▉| 1809/1817 [65:54:24<17:16, 129.62s/it] 100%|█████████▉| 1810/1817 [65:56:37<15:13, 130.46s/it] {'loss': 1.8265, 'learning_rate': 2e-05, 'epoch': 1.0} 100%|█████████▉| 1810/1817 [65:56:36<15:13, 130.46s/it] 100%|█████████▉| 1811/1817 [65:58:49<13:05, 130.95s/it] {'loss': 1.8675, 'learning_rate': 2e-05, 'epoch': 1.0} 100%|█████████▉| 1811/1817 [65:58:48<13:05, 130.95s/it] 100%|█████████▉| 1812/1817 [66:00:59<10:53, 130.79s/it] {'loss': 1.8935, 'learning_rate': 2e-05, 'epoch': 1.0} 100%|█████████▉| 1812/1817 [66:00:59<10:53, 130.79s/it] 100%|█████████▉| 1813/1817 [66:03:09<08:42, 130.66s/it] {'loss': 1.8882, 'learning_rate': 2e-05, 'epoch': 1.0} 100%|█████████▉| 1813/1817 [66:03:09<08:42, 130.66s/it] 100%|█████████▉| 1814/1817 [66:05:21<06:32, 130.86s/it] {'loss': 1.9208, 'learning_rate': 2e-05, 'epoch': 1.0} 100%|█████████▉| 1814/1817 [66:05:20<06:32, 130.86s/it] 100%|█████████▉| 1815/1817 [66:07:34<04:22, 131.49s/it] {'loss': 1.8801, 'learning_rate': 2e-05, 'epoch': 1.0} 100%|█████████▉| 1815/1817 [66:07:33<04:22, 131.49s/it] 100%|█████████▉| 1816/1817 [66:09:43<02:10, 130.93s/it] {'loss': 1.8958, 'learning_rate': 2e-05, 'epoch': 1.0} 100%|█████████▉| 1816/1817 [66:09:43<02:10, 130.93s/it] 100%|██████████| 1817/1817 [66:11:52<00:00, 130.28s/it] {'loss': 1.8254, 'learning_rate': 2e-05, 'epoch': 1.0} 100%|██████████| 1817/1817 [66:11:52<00:00, 130.28s/it][INFO|trainer.py:2025] 2024-02-01 05:57:30,082 >> Training completed. Do not forget to share your model on huggingface.co/models =) {'train_runtime': 238313.358, 'train_samples_per_second': 3.905, 'train_steps_per_second': 0.008, 'train_loss': 1.9326286626242646, 'epoch': 1.0} 100%|██████████| 1817/1817 [66:11:53<00:00, 130.28s/it] 100%|██████████| 1817/1817 [66:11:53<00:00, 131.16s/it] [INFO|trainer.py:2830] 2024-02-01 05:57:30,173 >> Saving model checkpoint to ./checkpoints_ct/ac/allm-ac-13b [INFO|configuration_utils.py:457] 2024-02-01 05:57:30,213 >> Configuration saved in ./checkpoints_ct/ac/allm-ac-13b/config.json [INFO|configuration_utils.py:362] 2024-02-01 05:57:30,221 >> Configuration saved in ./checkpoints_ct/ac/allm-ac-13b/generation_config.json [INFO|modeling_utils.py:1759] 2024-02-01 05:57:30,261 >> Model weights saved in ./checkpoints_ct/ac/allm-ac-13b/pytorch_model.bin [INFO|tokenization_utils_base.py:2164] 2024-02-01 05:57:30,263 >> tokenizer config file saved in ./checkpoints_ct/ac/allm-ac-13b/tokenizer_config.json [INFO|tokenization_utils_base.py:2171] 2024-02-01 05:57:30,267 >> Special tokens file saved in ./checkpoints_ct/ac/allm-ac-13b/special_tokens_map.json [INFO|tokenization_utils_base.py:2221] 2024-02-01 05:57:30,270 >> added tokens file saved in ./checkpoints_ct/ac/allm-ac-13b/added_tokens.json [2024-02-01 05:57:45,256] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step1817 is about to be saved! [2024-02-01 05:57:45,259] [INFO] [engine.py:3492:save_16bit_model] Saving model weights to ./checkpoints_ct/ac/allm-ac-13b/pytorch_model.bin, tag: global_step1817 [2024-02-01 05:57:45,259] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./checkpoints_ct/ac/allm-ac-13b/pytorch_model.bin... [2024-02-01 05:58:18,895] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./checkpoints_ct/ac/allm-ac-13b/pytorch_model.bin. [2024-02-01 05:58:18,896] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step1817 is ready now! ***** train metrics ***** epoch = 1.0 train_loss = 1.9326 train_runtime = 2 days, 18:11:53.35 train_samples = 930514 train_samples_per_second = 3.905 train_steps_per_second = 0.008 [INFO|modelcard.py:451] 2024-02-01 05:58:18,961 >> Dropping the following result as it does not have all the necessary fields: {'task': {'name': 'Causal Language Modeling', 'type': 'text-generation'}, 'dataset': {'name': '/apdcephfs/share_733425/vinnylywang/jianhuipang/datasets/RedPajama-Data-1T-Sample', 'type': '/apdcephfs/share_733425/vinnylywang/jianhuipang/datasets/RedPajama-Data-1T-Sample', 'config': None, 'split': 'None'}}