---
license: llama2
---

## Lazy LoRA

### Benefits

0. using the updated [Meta's LLaMA-2 models](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf).
1. support [4-bit qlora](https://arxiv.org/abs/2305.14314), extreme GPU memory and inference time saving;
2. comparable MMLU evaluation dataset results:

 |               | eval   | test   | comp-eval | comp-test |
 |---------------|--------|--------|-----------|-----------|
 |llama2-7b-chat | 49.38% | 48.22% |           |           |
 |ckpt-1600      | 46.51% | 47.44% | -2.87%    | -0.78%    |
 |ckpt-1500      | 47.66% | 46.88% | -1.72%    | -1.33%    |

 llama2-7b-chat: '0ede8dd71e923db6258295621d817ca8714516d4'

### Introduction
Determine the rank of LoRA layers by the singular values of pretrained weight matrices.
Also, combines:
1. LoRA: [LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS](https://arxiv.org/abs/2106.09685)
2. Prefix Tuning: [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://aclanthology.org/2021.acl-long.3
53/), [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://arxiv.or
g/pdf/2110.07602.pdf)
3. Prompt Tuning: [The Power of Scale for Parameter-Efficient Prompt Tuning](https://arxiv.org/abs/2104.08691)
4. LLaMA adapter: [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention] (https://arxiv.org/abs/2303.16199)
in one model.

This allows you to perform LoRA (additional low rank adapters inserted to each linear layer), and prompt learning (additional virtual tokens attached to the input and to the attention layers acting as `past_key_values`)

## Usage:
```python
import sys
sys.path.insert(1, '/workspace/asr/peft/src')
# TODO set this path to the lazy-lora source code path, 
# or you can install it from source code:
# TODO, please install lazylora for usage:
# git clone git@github.com:Xianchao-Wu/peft.git
# cd peft
# python setup.py install

from transformers import (AutoTokenizer, 
	AutoModelForCausalLM, BitsAndBytesConfig)
from peft import PeftModel, PeftConfig
import os
import torch

#import ipdb; ipdb.set_trace()
cache_dir="/workspace/asr/peft/qlora"
# TODO set this cache_dir to the path where you 
# stored (or, want to store) llama2-7b-chat-hf model

lazylora_dir=os.getcwd() 
# the path that contains 'adapter_config.json' 
#     and 'adapter_model.bin'

config = PeftConfig.from_pretrained(lazylora_dir)

tokenizer = AutoTokenizer.from_pretrained(
    config.base_model_name_or_path,
    cache_dir=cache_dir,
    use_auth_token=True
)

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type='nf4',
    bnb_4bit_compute_dtype=torch.bfloat16
)

model = AutoModelForCausalLM.from_pretrained(
    config.base_model_name_or_path,
    quantization_config=bnb_config,
    device_map="auto",
    cache_dir=cache_dir,
    use_auth_token=True
)
#model.print_trainable_parameters()
print(sum(p.numel() for p in model.parameters()))
# 3,500,412,928 -> half-size of 7B due to 4-bit loading

model = PeftModel.from_pretrained(model, lazylora_dir)
print('after adding lazy lora parameters:')
model.print_trainable_parameters()
# trainable params: 0 || all params: 3,660,359,168 || trainable%: 0.0


```

## MMLU result:

### MMLU eval result:

```json
{"mmlu_loss": 1.44828412985973,
 "mmlu_eval_accuracy_professional_accounting": 0.3548387096774194,
 "mmlu_eval_accuracy_high_school_physics": 0.35294117647058826,
 "mmlu_eval_accuracy_elementary_mathematics": 0.24390243902439024,
 "mmlu_eval_accuracy_high_school_statistics": 0.391304347826087,
 "mmlu_eval_accuracy_machine_learning": 0.36363636363636365,
 "mmlu_eval_accuracy_sociology": 0.6363636363636364,
 "mmlu_eval_accuracy_human_sexuality": 0.4166666666666667,
 "mmlu_eval_accuracy_professional_medicine": 0.3870967741935484,
 "mmlu_eval_accuracy_computer_security": 0.2727272727272727,
 "mmlu_eval_accuracy_astronomy": 0.375,
 "mmlu_eval_accuracy_high_school_computer_science": 0.5555555555555556,
 "mmlu_eval_accuracy_high_school_world_history": 0.5384615384615384,
 "mmlu_eval_accuracy_high_school_psychology": 0.7166666666666667,
 "mmlu_eval_accuracy_professional_law": 0.3176470588235294,
 "mmlu_eval_accuracy_econometrics": 0.16666666666666666,
 "mmlu_eval_accuracy_high_school_chemistry": 0.2727272727272727,
 "mmlu_eval_accuracy_global_facts": 0.5,
 "mmlu_eval_accuracy_high_school_government_and_politics": 0.5238095238095238,
 "mmlu_eval_accuracy_electrical_engineering": 0.25,
 "mmlu_eval_accuracy_college_chemistry": 0.375,
 "mmlu_eval_accuracy_high_school_biology": 0.4375,
 "mmlu_eval_accuracy_high_school_microeconomics": 0.4230769230769231,
 "mmlu_eval_accuracy_public_relations": 0.5833333333333334,
 "mmlu_eval_accuracy_high_school_macroeconomics": 0.4883720930232558,
 "mmlu_eval_accuracy_world_religions": 0.7368421052631579,
 "mmlu_eval_accuracy_medical_genetics": 0.8181818181818182,
 "mmlu_eval_accuracy_college_biology": 0.375,
 "mmlu_eval_accuracy_jurisprudence": 0.36363636363636365,
 "mmlu_eval_accuracy_college_mathematics": 0.36363636363636365,
 "mmlu_eval_accuracy_miscellaneous": 0.627906976744186,
 "mmlu_eval_accuracy_high_school_us_history": 0.8181818181818182,
 "mmlu_eval_accuracy_virology": 0.4444444444444444,
 "mmlu_eval_accuracy_anatomy": 0.5,
 "mmlu_eval_accuracy_college_medicine": 0.3181818181818182,
 "mmlu_eval_accuracy_human_aging": 0.7391304347826086,
 "mmlu_eval_accuracy_college_physics": 0.45454545454545453,
 "mmlu_eval_accuracy_philosophy": 0.38235294117647056,
 "mmlu_eval_accuracy_clinical_knowledge": 0.4482758620689655,
 "mmlu_eval_accuracy_formal_logic": 0.21428571428571427,
 "mmlu_eval_accuracy_nutrition": 0.6363636363636364,
 "mmlu_eval_accuracy_high_school_mathematics": 0.20689655172413793,
 "mmlu_eval_accuracy_logical_fallacies": 0.6666666666666666,
 "mmlu_eval_accuracy_professional_psychology": 0.42028985507246375,
 "mmlu_eval_accuracy_prehistory": 0.5428571428571428,
 "mmlu_eval_accuracy_high_school_geography": 0.7272727272727273,
 "mmlu_eval_accuracy_management": 0.6363636363636364,
 "mmlu_eval_accuracy_marketing": 0.76,
 "mmlu_eval_accuracy_international_law": 0.9230769230769231,
 "mmlu_eval_accuracy_us_foreign_policy": 0.7272727272727273,
 "mmlu_eval_accuracy_moral_scenarios": 0.32,
 "mmlu_eval_accuracy_high_school_european_history": 0.5,
 "mmlu_eval_accuracy_business_ethics": 0.45454545454545453,
 "mmlu_eval_accuracy_moral_disputes": 0.39473684210526316,
 "mmlu_eval_accuracy_conceptual_physics": 0.38461538461538464,
 "mmlu_eval_accuracy_security_studies": 0.5925925925925926,
 "mmlu_eval_accuracy_abstract_algebra": 0.36363636363636365,
 "mmlu_eval_accuracy_college_computer_science": 0.36363636363636365,
 "mmlu_eval_accuracy": 0.4766441930115949}
```

### MMLU test result:

```json
{"mmlu_loss": 1.4452685356679218,
 "mmlu_test_accuracy_moral_scenarios": 0.23575418994413408,
 "mmlu_test_accuracy_security_studies": 0.5020408163265306,
 "mmlu_test_accuracy_astronomy": 0.4934210526315789,
 "mmlu_test_accuracy_medical_genetics": 0.52,
 "mmlu_test_accuracy_logical_fallacies": 0.5521472392638037,
 "mmlu_test_accuracy_professional_psychology": 0.4444444444444444,
 "mmlu_test_accuracy_high_school_psychology": 0.6110091743119266,
 "mmlu_test_accuracy_high_school_us_history": 0.6372549019607843,
 "mmlu_test_accuracy_high_school_physics": 0.33112582781456956,
 "mmlu_test_accuracy_prehistory": 0.5308641975308642,
 "mmlu_test_accuracy_human_sexuality": 0.549618320610687,
 "mmlu_test_accuracy_management": 0.6213592233009708,
 "mmlu_test_accuracy_international_law": 0.6363636363636364,
 "mmlu_test_accuracy_moral_disputes": 0.49421965317919075,
 "mmlu_test_accuracy_conceptual_physics": 0.4127659574468085,
 "mmlu_test_accuracy_econometrics": 0.3508771929824561,
 "mmlu_test_accuracy_college_medicine": 0.3815028901734104,
 "mmlu_test_accuracy_high_school_biology": 0.5064516129032258,
 "mmlu_test_accuracy_high_school_statistics": 0.27314814814814814,
 "mmlu_test_accuracy_high_school_macroeconomics": 0.43333333333333335,
 "mmlu_test_accuracy_college_mathematics": 0.28,
 "mmlu_test_accuracy_elementary_mathematics": 0.30687830687830686,
 "mmlu_test_accuracy_public_relations": 0.509090909090909,
 "mmlu_test_accuracy_high_school_european_history": 0.5515151515151515,
 "mmlu_test_accuracy_human_aging": 0.5381165919282511,
 "mmlu_test_accuracy_high_school_geography": 0.5555555555555556,
 "mmlu_test_accuracy_formal_logic": 0.25396825396825395,
 "mmlu_test_accuracy_miscellaneous": 0.665389527458493,
 "mmlu_test_accuracy_high_school_computer_science": 0.4,
 "mmlu_test_accuracy_global_facts": 0.33,
 "mmlu_test_accuracy_world_religions": 0.6666666666666666,
 "mmlu_test_accuracy_machine_learning": 0.33035714285714285,
 "mmlu_test_accuracy_sociology": 0.6169154228855721,
 "mmlu_test_accuracy_clinical_knowledge": 0.49433962264150944,
 "mmlu_test_accuracy_virology": 0.4397590361445783,
 "mmlu_test_accuracy_high_school_government_and_politics": 0.6839378238341969,
 "mmlu_test_accuracy_high_school_world_history": 0.6329113924050633,
 "mmlu_test_accuracy_college_biology": 0.5138888888888888,
 "mmlu_test_accuracy_philosophy": 0.5627009646302251,
 "mmlu_test_accuracy_college_physics": 0.2549019607843137,
 "mmlu_test_accuracy_college_computer_science": 0.34,
 "mmlu_test_accuracy_high_school_chemistry": 0.3793103448275862,
 "mmlu_test_accuracy_nutrition": 0.5163398692810458,
 "mmlu_test_accuracy_professional_accounting": 0.35106382978723405,
 "mmlu_test_accuracy_jurisprudence": 0.5925925925925926,
 "mmlu_test_accuracy_high_school_mathematics": 0.25925925925925924,
 "mmlu_test_accuracy_marketing": 0.6923076923076923,
 "mmlu_test_accuracy_business_ethics": 0.48,
 "mmlu_test_accuracy_high_school_microeconomics": 0.4495798319327731,
 "mmlu_test_accuracy_college_chemistry": 0.3,
 "mmlu_test_accuracy_us_foreign_policy": 0.72,
 "mmlu_test_accuracy_computer_security": 0.6,
 "mmlu_test_accuracy_anatomy": 0.4740740740740741,
 "mmlu_test_accuracy_professional_law": 0.3220338983050847,
 "mmlu_test_accuracy_abstract_algebra": 0.27,
 "mmlu_test_accuracy_electrical_engineering": 0.4827586206896552,
 "mmlu_test_accuracy_professional_medicine": 0.3897058823529412,
 "mmlu_test_accuracy": 0.46883545484585126}
```

## License and intended use 

This lazy-lora adapter is based on [Meta's LLaMA-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf), and using the [oasst1 dataset](https://huggingface.co/datasets/OpenAssistant/oasst1), following [Guanaco](https://huggingface.co/timdettmers/guanaco-65b).

lazy lora adapter weights are available under LLAMA-2 license. Note the use of the lazy lora adapter weights, requires access to the LLaMA model weighs. Lazy lora is based on LLaMA and therefore should be used according to the LLaMA license.


## Risks and Biases

The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. The model was trained on various public datasets; it is possible that this model could generate lewd, biased, or otherwise offensive outputs.