Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,17 @@
|
|
| 1 |
---
|
| 2 |
license: mit
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: mit
|
| 3 |
---
|
| 4 |
+
|
| 5 |
+
This repo contains a low-rank adapter for LLaMA-13b finetuned on Ntropy proprietary dataset (consumer financial transactions).
|
| 6 |
+
|
| 7 |
+
This version of the weights was trained with the following hyperparameters:
|
| 8 |
+
|
| 9 |
+
- Base Model: decapoda-research/llama-13b-hf
|
| 10 |
+
- Epochs: 10 (load from best epoch)
|
| 11 |
+
- Batch size: 16
|
| 12 |
+
- Cutoff length: 1024
|
| 13 |
+
- Learning rate: 3e-4
|
| 14 |
+
- Lora r: 16
|
| 15 |
+
- Lora target modules: q_proj, k_proj, v_proj, o_proj
|
| 16 |
+
|
| 17 |
+
Instructions for running the adapter can be found at https://github.com/ntropy-network/enrichment_models/blob/main/notebooks/llama.ipynb
|