|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- shareAI/ShareGPT-Chinese-English-90k |
|
language: |
|
- zh |
|
- en |
|
pipeline_tag: text-generation |
|
--- |
|
 |
|
|
|
<div align="center"> |
|
<h2> |
|
Aurora: Activating chinese chat capability for Mistral-8x7B sparse Mixture-of-Experts through Instruction-Tuning |
|
</h2> |
|
</div> |
|
|
|
1. <h1>Please follow our Github: <a href="https://github.com/WangRongsheng/Aurora">https://github.com/WangRongsheng/Aurora</a></h1> |
|
2. <h1>Please follow our Paper: <a href="https://arxiv.org/abs/2312.14557">https://arxiv.org/abs/2312.14557</a></h1> |
|
|
|
## Overview |
|
|
|
Existing research has demonstrated that refining large language models (LLMs) through the utilization of machine-generated instruction-following data empowers these models to exhibit impressive zero-shot capabilities for novel tasks, without requiring human-authored instructions. In this paper, we systematically investigate, preprocess, and integrate three Chinese instruction-following datasets with the aim of enhancing the Chinese conversational capabilities of Mixtral-8x7B sparse Mixture-of-Experts model. Through instruction fine-tuning on this carefully processed dataset, we successfully construct the Mixtral-8x7B sparse Mixture-of-Experts model named "Aurora." To assess the performance of Aurora, we utilize three widely recognized benchmark tests: C-Eval, MMLU, and CMMLU. Empirical studies validate the effectiveness of instruction fine-tuning applied to Mixtral-8x7B sparse Mixture-of-Experts model. This work is pioneering in the execution of instruction fine-tuning on a sparse expert-mixed model, marking a significant breakthrough in enhancing the capabilities of this model architecture. |
|
|
|
 |
|
|
|
## Usage |
|
|
|
```python |
|
import gradio as gr |
|
import torch |
|
from transformers import AutoModelForCausalLM, AutoTokenizer, StoppingCriteria, StoppingCriteriaList, TextIteratorStreamer |
|
from threading import Thread |
|
from peft import PeftModel |
|
import time |
|
|
|
model_name_or_path = "mistralai/Mixtral-8x7B-Instruct-v0.1" # download weights from https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1 |
|
lora_weights = "wangrongsheng/Aurora" # download weights from https://huggingface.co/wangrongsheng/Aurora |
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path) |
|
model0 = AutoModelForCausalLM.from_pretrained(model_name_or_path, load_in_4bit=True, device_map="auto", torch_dtype=torch.bfloat16) |
|
model = PeftModel.from_pretrained( |
|
model0, |
|
lora_weights, |
|
) |
|
|
|
class StopOnTokens(StoppingCriteria): |
|
def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool: |
|
stop_ids = [0,] |
|
for stop_id in stop_ids: |
|
if input_ids[0][-1] == stop_id: |
|
return True |
|
return False |
|
|
|
def convert_history_to_text(history): |
|
text = "" |
|
if len(history) > 1: |
|
text = "<s> " + "".join( |
|
[ |
|
"".join( |
|
[ |
|
f"[INST]{item[0]}[/INST] {item[1]} ", |
|
] |
|
) |
|
for item in history[:-1] |
|
] |
|
) + "</s> " |
|
text += "".join( |
|
[ |
|
"".join( |
|
[ |
|
f"[INST]{history[-1][0]}[/INST]", |
|
] |
|
) |
|
] |
|
) |
|
return text |
|
|
|
def predict(message, history): |
|
|
|
history_transformer_format = history + [[message, ""]] |
|
stop = StopOnTokens() |
|
|
|
messages = convert_history_to_text(history_transformer_format) |
|
|
|
model_inputs = tokenizer([messages], return_tensors="pt").to("cuda") |
|
streamer = TextIteratorStreamer(tokenizer, timeout=10., skip_prompt=True, skip_special_tokens=True) |
|
generate_kwargs = dict( |
|
model_inputs, |
|
streamer=streamer, |
|
max_new_tokens=4096, |
|
do_sample=True, |
|
top_p=0.95, |
|
top_k=1000, |
|
temperature=1.0, |
|
num_beams=1, |
|
pad_token_id=tokenizer.eos_token_id, |
|
stopping_criteria=StoppingCriteriaList([stop]) |
|
) |
|
t = Thread(target=model.generate, kwargs=generate_kwargs) |
|
t.start() |
|
|
|
partial_message = "" |
|
t1 = time.time() |
|
count = 0 |
|
for new_token in streamer: |
|
if new_token != '<': |
|
partial_message += new_token |
|
count += 1 |
|
yield partial_message |
|
t2 = time.time() |
|
speed = count/(t2-t1) |
|
print("inference speed: %f tok/s" % speed) |
|
|
|
|
|
gr.ChatInterface(predict,chatbot=gr.Chatbot(height=600,),title="MoE").queue().launch() |
|
``` |
|
|
|
## Citation |
|
If you find our work helpful, feel free to give us a cite. |
|
```latex |
|
@misc{wang2023auroraactivating, |
|
title={Aurora:Activating Chinese chat capability for Mixtral-8x7B sparse Mixture-of-Experts through Instruction-Tuning}, |
|
author={Rongsheng Wang and Haoming Chen and Ruizhe Zhou and Yaofei Duan and Kunyan Cai and Han Ma and Jiaxi Cui and Jian Li and Patrick Cheong-Iao Pang and Yapeng Wang and Tao Tan}, |
|
year={2023}, |
|
eprint={2312.14557}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL} |
|
} |
|
``` |
