| 2024-05-25 20:26:13,701 INFO MainThread:33493 [wandb_setup.py:_flush():76] Current SDK version is 0.17.0 | |
| 2024-05-25 20:26:13,701 INFO MainThread:33493 [wandb_setup.py:_flush():76] Configure stats pid to 33493 | |
| 2024-05-25 20:26:13,701 INFO MainThread:33493 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings | |
| 2024-05-25 20:26:13,701 INFO MainThread:33493 [wandb_setup.py:_flush():76] Loading settings from /root/mistral-finetune/wandb/settings | |
| 2024-05-25 20:26:13,701 INFO MainThread:33493 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} | |
| 2024-05-25 20:26:13,701 WARNING MainThread:33493 [wandb_setup.py:_flush():76] Could not find program at -m train | |
| 2024-05-25 20:26:13,701 INFO MainThread:33493 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': None, 'program': '-m train'} | |
| 2024-05-25 20:26:13,701 INFO MainThread:33493 [wandb_setup.py:_flush():76] Applying login settings: {'api_key': '***REDACTED***'} | |
| 2024-05-25 20:26:13,701 INFO MainThread:33493 [wandb_setup.py:_flush():76] Applying login settings: {'api_key': '***REDACTED***'} | |
| 2024-05-25 20:26:13,701 INFO MainThread:33493 [wandb_setup.py:_flush():76] Applying login settings: {} | |
| 2024-05-25 20:26:13,701 INFO MainThread:33493 [wandb_init.py:_log_setup():520] Logging user logs to /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/logs/debug.log | |
| 2024-05-25 20:26:13,701 INFO MainThread:33493 [wandb_init.py:_log_setup():521] Logging internal logs to /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/logs/debug-internal.log | |
| 2024-05-25 20:26:13,701 INFO MainThread:33493 [wandb_init.py:init():560] calling init triggers | |
| 2024-05-25 20:26:13,701 INFO MainThread:33493 [wandb_init.py:init():567] wandb.init called with sweep_config: {} | |
| config: {'data': {'data': '', 'shuffle': False, 'instruct_data': '/root/data/mol_instructions_train.jsonl', 'eval_instruct_data': '', 'instruct': {'shuffle': True, 'dynamic_chunk_fn_call': True}}, 'model_id_or_path': '/root/mistral_models/7B-v0.3', 'run_dir': '/root/mistral-finetune/runs2', 'optim': {'lr': 6e-05, 'weight_decay': 0.1, 'pct_start': 0.05}, 'seed': 0, 'num_microbatches': 1, 'seq_len': 32768, 'batch_size': 1, 'max_norm': 1.0, 'max_steps': 500, 'log_freq': 1, 'ckpt_freq': 100, 'ckpt_only_lora': False, 'no_ckpt': False, 'num_ckpt_keep': 3, 'eval_freq': 100, 'no_eval': True, 'checkpoint': True, 'world_size': 1, 'wandb': {'project': 'CHEMISTral7b-ft', 'offline': False, 'key': 'aaf77f83a4e316f6a8b47fa975ab6b5e73c7c8df', 'run_name': 'run1'}, 'mlflow': {'tracking_uri': None, 'experiment_name': None}, 'lora': {'enable': True, 'rank': 64, 'dropout': 0.0, 'scaling': 2.0}} | |
| 2024-05-25 20:26:13,701 INFO MainThread:33493 [wandb_init.py:init():610] starting backend | |
| 2024-05-25 20:26:13,701 INFO MainThread:33493 [wandb_init.py:init():614] setting up manager | |
| 2024-05-25 20:26:13,702 INFO MainThread:33493 [backend.py:_multiprocessing_setup():105] multiprocessing start_methods=fork,spawn,forkserver, using: spawn | |
| 2024-05-25 20:26:13,702 INFO MainThread:33493 [wandb_init.py:init():622] backend started and connected | |
| 2024-05-25 20:26:13,704 INFO MainThread:33493 [wandb_init.py:init():711] updated telemetry | |
| 2024-05-25 20:26:13,706 INFO MainThread:33493 [wandb_init.py:init():744] communicating run to backend with 90.0 second timeout | |
| 2024-05-25 20:26:14,054 INFO MainThread:33493 [wandb_run.py:_on_init():2396] communicating current version | |
| 2024-05-25 20:26:14,086 INFO MainThread:33493 [wandb_run.py:_on_init():2405] got version response | |
| 2024-05-25 20:26:14,087 INFO MainThread:33493 [wandb_init.py:init():795] starting run threads in backend | |
| 2024-05-25 20:26:15,147 INFO MainThread:33493 [wandb_run.py:_console_start():2374] atexit reg | |
| 2024-05-25 20:26:15,147 INFO MainThread:33493 [wandb_run.py:_redirect():2229] redirect: wrap_raw | |
| 2024-05-25 20:26:15,148 INFO MainThread:33493 [wandb_run.py:_redirect():2294] Wrapping output streams. | |
| 2024-05-25 20:26:15,148 INFO MainThread:33493 [wandb_run.py:_redirect():2319] Redirects installed. | |
| 2024-05-25 20:26:15,148 INFO MainThread:33493 [wandb_init.py:init():838] run started, returning control to user process | |
| 2024-05-25 21:29:34,196 INFO MainThread:33493 [wandb_run.py:_finish():2103] finishing run artsy/CHEMISTral7b-ft/sapihyp4 | |
| 2024-05-25 21:29:34,196 INFO MainThread:33493 [wandb_run.py:_atexit_cleanup():2343] got exitcode: 0 | |
| 2024-05-25 21:29:34,196 INFO MainThread:33493 [wandb_run.py:_restore():2326] restore | |
| 2024-05-25 21:29:34,196 INFO MainThread:33493 [wandb_run.py:_restore():2332] restore done | |
| 2024-05-25 21:29:38,445 INFO MainThread:33493 [wandb_run.py:_footer_history_summary_info():3994] rendering history | |
| 2024-05-25 21:29:38,445 INFO MainThread:33493 [wandb_run.py:_footer_history_summary_info():4026] rendering summary | |
| 2024-05-25 21:29:38,448 INFO MainThread:33493 [wandb_run.py:_footer_sync_info():3953] logging synced files | |