chansung commited on
Commit
1c43b74
1 Parent(s): 5c49a53

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +26 -0
README.md CHANGED
@@ -1,3 +1,29 @@
1
  ---
2
  license: gpl-3.0
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: gpl-3.0
3
+ datasets:
4
+ - yahma/alpaca-cleaned
5
+ language:
6
+ - en
7
+ pipeline_tag: text2text-generation
8
+ tags:
9
+ - alpaca
10
+ - llama
11
+ - chat
12
  ---
13
+
14
+ This repository comes with LoRA checkpoint to make LLaMA into a chatbot like language model. The checkpoint is the output of instruction following fine-tuning process with the following settings on 8xA100(40G) DGX system.
15
+ - Dataset: [cleaned-up Alpaca dataset](https://github.com/gururise/AlpacaDataCleaned) up to 04/06/23
16
+ - Training script: borrowed from the official [Alpaca-LoRA](https://github.com/tloen/alpaca-lora) implementation
17
+ - Training script:
18
+ ```shell
19
+ python finetune.py \
20
+ --base_model='decapoda-research/llama-13b-hf' \
21
+ --num_epochs=10 \
22
+ --cutoff_len=512 \
23
+ --group_by_length \
24
+ --output_dir='./lora-alpaca' \
25
+ --lora_target_modules='[q_proj,k_proj,v_proj,o_proj]' \
26
+ --lora_r=16 \
27
+ --batch_size=... \
28
+ --micro_batch_size=...
29
+ ```