---
license: apache-2.0
datasets:
- c-s-ale/alpaca-gpt4-data
pipeline_tag: text2text-generation
---

This repo provides the training checkpoint of LLaMA on the alpaca_data_gpt4 dataset via LoRA [MLP] on 8xA100(80G). 

He et al. 2022 gave an insight that FFN can better utilize modification at larger capacities.

The codes is provided by [tloen/alpaca-lora: Instruct-tune LLaMA on consumer hardware (github.com)](https://github.com/tloen/alpaca-lora).

We modify the running scripts to 
```bash
torchrun --nproc_per_node=8 finetune.py \
    --base_model '/cache1/chtan/large_models/llama-hf/llama-65b' \
    --data_path './alpaca_data_gpt4.json' \
    --output_dir './gpt4-alpaca-lora_mlp-65b' \
    --batch_size 128 \
    --micro_batch_size 2 \
    --num_epochs 10 \
    --learning_rate 1e-4 \
    --cutoff_len 512 \
    --val_set_size 2000 \
    --lora_r 16 \
    --lora_alpha 16 \
    --lora_dropout 0.05 \
    --lora_target_modules '[gate_proj,down_proj,up_proj]' \
    --train_on_inputs \
    --group_by_length
```

> [1] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, Graham Neubig: Towards a Unified View of Parameter-Efficient Transfer Learning. ICLR 2022