license: apache-2.0
OpenAlpaca: A Fully Open-Source Instruction-Following Model Based On OpenLLaMA
In this repo, we release a permissively licensed open-source instruction-following model based on OpenLLaMA. In this release, we release a public preview of the 7B OpenAlpaca model based on the previewed version of OpenLLaMA that is 7B model trained with 700 billion tokens. We provide PyTorch weights of OpenAlpaca. Stay tuned for our forthcoming updates!
[Project Page] (https://github.com/yxuansu/OpenAlpaca)
Dataset and Training
We train our model on the dolly 15k dataset released by Databricks. The training configurations are provided in the table below. The training takes on 8 x A100(40G) GPUs and lasts for around 30 minutes.
| Batch Size | 64 |
| Learning rate | 2e-5 |
| Epochs | 3 |
| Max length | 1024 |
Example Usage
Below shows an example on how to use OpenAlpaca
import torch
from transformers import LlamaForCausalLM, LlamaTokenizer
# the previewed version of OpenAlpaca
model_path = r'openllmplayground/openalpaca_7b_700bt_preview'
tokenizer = LlamaTokenizer.from_pretrained(model_path)
model = LlamaForCausalLM.from_pretrained(model_path).cuda()
# same prompt as provided in https://crfm.stanford.edu/2023/03/13/alpaca.html
instruction = r'What is an alpaca? How is it different from a llama?'
'''
instruction = r'Write an e-mail to congratulate new Standford admits and mention that you are excited about meeting all of them in person.'
instruction = r'What is the capital of Tanzania?'
instruction = r'Write a well-thought out abstract for a machine learning paper that proves that 42 is the optimal seed for training neural networks.'
'''
prompt_no_input = r'### Instruction:\n{instruction}\n\n### Response:'
tokens = tokenizer.encode(prompt_no_input)
bos_token_id, eos_token_id = 1, 2 # see https://github.com/openlm-research/open_llama#preview-weights-release-and-usage
tokens = [bos_token_id] + tokens + [eos_token_id] + [bos_token_id]
tokens = torch.LongTensor(tokens[-1024:]).unsqueeze(0).cuda()
instance = {'input_ids': tokens,
'top_k': 50,
'top_p': 0.9,
'generate_len': 128}
length = len(tokens[0])
with torch.no_grad():
rest = model.generate(
input_ids=tokens,
max_length=length+instance['generate_len'],
use_cache=True,
do_sample=True,
top_p=instance['top_p'],
top_k=instance['top_k']
)
output = rest[0][length:]
string = tokenizer.decode(output, skip_special_tokens=False)
string = string.replace('<s>', '').replace('</s>', '').strip()
print(f'[!] Generation results: {string}')
License and Usage
OpenAlpaca is permissively licensed under the Apache 2.0 license and can be used freely for academic/commercial purposes.
Contact
We would love to get feedback from the community. If you have any questions, please open an issue or contact us.
OpenAlpaca is developed by: Yixuan Su*, Tian Lan*, and Deng Cai (The first two members* contributed equally.)
Reference:
If you found OpenAlpaca useful in your research or applications, please kindly cite using the following BibTeX:
@misc{openalpaca,
author = {Yixuan Su and Tian Lan and Deng Cai},
title = {OpenAlpaca: A Fully Open-Source Instruction-Following Model Based On OpenLLaMA},
year = {2023},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/yxuansu/OpenAlpaca}},
}
@software{openlm2023openllama,
author = {Xinyang Geng and Hao Liu},
title = {OpenLLaMA: An Open Reproduction of LLaMA},
month = May,
year = 2023,
url = {https://github.com/openlm-research/open_llama}
}
@misc{alpaca,
author = {Rohan Taori and Ishaan Gulrajani and Tianyi Zhang and Yann Dubois and Xuechen Li and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto },
title = {Stanford Alpaca: An Instruction-following LLaMA model},
year = {2023},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/tatsu-lab/stanford_alpaca}},
}
@article{touvron2023llama,
title={Llama: Open and efficient foundation language models},
author={Hugo Touvron and Thibaut Lavril and Gautier Izacard and Xavier Martinet and Marie{-}Anne Lachaux and Timoth{\'{e}}e Lacroix and Baptiste Rozi{\`{e}}re and Naman Goyal and Eric Hambro and Faisal Azhar and Aur{\'{e}}lien Rodriguez and Armand Joulin and Edouard Grave and Guillaume Lample},
journal={arXiv preprint arXiv:2302.13971},
year={2023}
}