Update README.md
Browse files
README.md
CHANGED
@@ -1,3 +1,29 @@
|
|
1 |
---
|
2 |
license: gpl-3.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
---
|
2 |
license: gpl-3.0
|
3 |
+
datasets:
|
4 |
+
- yahma/alpaca-cleaned
|
5 |
+
language:
|
6 |
+
- en
|
7 |
+
pipeline_tag: text2text-generation
|
8 |
+
tags:
|
9 |
+
- alpaca
|
10 |
+
- llama
|
11 |
+
- chat
|
12 |
---
|
13 |
+
|
14 |
+
This repository comes with LoRA checkpoint to make LLaMA into a chatbot like language model. The checkpoint is the output of instruction following fine-tuning process with the following settings on 8xA100(40G) DGX system.
|
15 |
+
- Dataset: [cleaned-up Alpaca dataset](https://github.com/gururise/AlpacaDataCleaned) up to 04/06/23
|
16 |
+
- Training script: borrowed from the official [Alpaca-LoRA](https://github.com/tloen/alpaca-lora) implementation
|
17 |
+
- Training script:
|
18 |
+
```shell
|
19 |
+
python finetune.py \
|
20 |
+
--base_model='decapoda-research/llama-13b-hf' \
|
21 |
+
--num_epochs=10 \
|
22 |
+
--cutoff_len=512 \
|
23 |
+
--group_by_length \
|
24 |
+
--output_dir='./lora-alpaca' \
|
25 |
+
--lora_target_modules='[q_proj,k_proj,v_proj,o_proj]' \
|
26 |
+
--lora_r=16 \
|
27 |
+
--batch_size=... \
|
28 |
+
--micro_batch_size=...
|
29 |
+
```
|