| # wizardLM-LlaMA-LoRA-7B | |
| A LoRA trained on the WizardLM dataset, with a LlaMA 7B as the basemodel. | |
| ## Instruction example | |
| It was trained with the alpaca-short template, without any inputs, so prompt as follows: | |
| ``` | |
| ### Instruction: | |
| Write a poem about the transformers Python library. | |
| Mention the word "large language models" in that poem. | |
| ### Response: | |
| I'm not sure if this is what you meant, but here goes! | |
| The Transformers are large language models | |
| that help us make sense of text. | |
| They take our sentences and turn them into vectors, | |
| which can be used to find similarities between texts. | |
| We use these for things like search engines or spam filters; | |
| they also have uses in machine learning too. | |
| ``` | |
| ## Trained with the following params | |
| ``` | |
| base_model: /root/alpaca-lora/llama-7b-hf | |
| data_path: victor123/evol_instruct_70k | |
| output_dir: /loras/wizardLM-lama-lora | |
| batch_size: 64 | |
| micro_batch_size: 8 | |
| num_epochs: 3 | |
| learning_rate: 2e-05 | |
| cutoff_len: 2048 | |
| val_set_size: 2000 | |
| lora_r: 16 | |
| lora_alpha: 16 | |
| lora_dropout: 0.05 | |
| lora_target_modules: ['q_proj', 'k_proj', 'v_proj', 'o_proj'] | |
| train_on_inputs: True | |
| add_eos_token: False | |
| group_by_length: True | |
| wandb_project: | |
| wandb_run_name: | |
| wandb_watch: | |
| wandb_log_model: | |
| resume_from_checkpoint: False | |
| prompt template: alpaca_short | |
| ``` | |
| ## Training Details | |
| - Trained with https://github.com/tloen/alpaca-lora. Note: ince the `victor123/evol_instruct_70k` dataset only contains instruction and output, comment out the line `data_point["input"],` around line 151 in `alpaca-lora\finetune.py` | |
| - Trained on [RunPod](https://runpod.io?ref=qgrfwczf | |
| ) community cloud with 1x A100 80GB vram (Note: less GPU was needed) | |
| - Took 14:47:39 to train 3 epochs | |
| - Cost around $37 to train | |
| ## Evaluation | |
| - No evaluation has been done on this model. If someone wants to share I would happily pull. | |
| - Empirically it looks promising for complex instruction following. |