Actually masked loss

README.md

@@ -5,6 +5,27 @@ license: mit
 This repo contains a low-rank adapter for LLaMA-7b
 fit on the [Stanford Alpaca](https://github.com/tatsu-lab/stanford_alpaca) dataset.
 
-It doesn't contain the foundation model itself, so it's MIT licensed!
+This version of the weights was trained with the following hyperparameters:
 
-Instructions for running it can be found at https://github.com/tloen/alpaca-lora.
+- Epochs: 10 (load from best epoch)
+- Batch size: 128
+- Cutoff length: 512
+- Learning rate: 3e-4
+- Lora _r_: 16
+- Lora target modules: q_proj, k_proj, v_proj, o_proj
+
+That is:
+
+```
+python finetune.py \
+    --base_model='decapoda-research/llama-7b-hf' \
+    --num_epochs=10 \
+    --cutoff_len=512 \
+    --group_by_length \
+    --output_dir='./lora-alpaca-512-qkvo' \
+    --lora_target_modules='[q_proj,k_proj,v_proj,o_proj]' \
+    --lora_r=16 \
+    --micro_batch_size=8
+```
+
+Instructions for running it can be found at https://github.com/tloen/alpaca-lora.

adapter_config.json

@@ -9,10 +9,12 @@
   "merge_weights": false,
   "modules_to_save": null,
   "peft_type": "LORA",
-  "r": 8,
+  "r": 16,
   "target_modules": [
     "q_proj",
-    "v_proj"
+    "k_proj",
+    "v_proj",
+    "o_proj"
   ],
   "task_type": "CAUSAL_LM"
 }

adapter_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:321e826099a0eacb1cf39916923eb6feb4327e8e5e09fe9f09a6d6d2a8595448
-size 16822989
+oid sha256:2e7187f51fbdeff8815046d30f0a325e43491040e6eac8cec5e2ba64f1e87807
+size 67201357