metadata

language:
  - en
tags:
  - ctranslate2
  - int8
  - float16
  - causal-lm
license: cc-by-nc-sa-4.0
datasets:
  - dmayhem93/ChatCombined
  - tatsu-lab/alpaca
  - nomic-ai/gpt4all_prompt_generations
  - Dahoas/full-hh-rlhf
  - jeffwan/sharegpt_vicuna
  - HuggingFaceH4/databricks_dolly_15k

# Fast-Inference with Ctranslate2

Speedup inference while reducing memory by 2x-4x using int8 inference in C++ on CPU or GPU.

quantized version of stabilityai/stablelm-tuned-alpha-7b

pip install hf-hub-ctranslate2>=2.0.8

Converted on 2023-05-22 using

ct2-transformers-converter --model stabilityai/stablelm-tuned-alpha-7b --output_dir /home/michael/tmp-ct2fast-stablelm-tuned-alpha-7b --force --copy_files tokenizer.json README.md tokenizer_config.json generation_config.json special_tokens_map.json .gitattributes --quantization float16

Checkpoint compatible to ctranslate2>=3.13.0 and hf-hub-ctranslate2>=2.0.6

compute_type=int8_float16 for device="cuda"
compute_type=int8 for device="cpu"

from hf_hub_ctranslate2 import TranslatorCT2fromHfHub, GeneratorCT2fromHfHub
from transformers import AutoTokenizer

model_name = "michaelfeil/ct2fast-stablelm-tuned-alpha-7b"
# use either TranslatorCT2fromHfHub or GeneratorCT2fromHfHub here, depending on model.
model = GeneratorCT2fromHfHub(
        # load in int8 on CUDA
        model_name_or_path=model_name, 
        device="cuda",
        compute_type="int8_float16",
        # tokenizer=AutoTokenizer.from_pretrained("stabilityai/stablelm-tuned-alpha-7b")
)
outputs = model.generate(
    text=["def print_hello_world():", "def hello_name(name:"],
    max_length=64
)
print(outputs)

Licence and other remarks:

This is just a quantized version. Licence conditions are intended to be idential to original huggingface repo.

Original description

StableLM-Tuned-Alpha

Model Description

StableLM-Tuned-Alpha is a suite of 3B and 7B parameter decoder-only language models built on top of the StableLM-Base-Alpha models and further fine-tuned on various chat and instruction-following datasets.

Usage

Get started chatting with StableLM-Tuned-Alpha by using the following code snippet:

from transformers import AutoModelForCausalLM, AutoTokenizer, StoppingCriteria, StoppingCriteriaList

tokenizer = AutoTokenizer.from_pretrained("StabilityAI/stablelm-tuned-alpha-7b")
model = AutoModelForCausalLM.from_pretrained("StabilityAI/stablelm-tuned-alpha-7b")
model.half().cuda()

class StopOnTokens(StoppingCriteria):
    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:
        stop_ids = [50278, 50279, 50277, 1, 0]
        for stop_id in stop_ids:
            if input_ids[0][-1] == stop_id:
                return True
        return False

system_prompt = """<|SYSTEM|># StableLM Tuned (Alpha version)
- StableLM is a helpful and harmless open-source AI language model developed by StabilityAI.
- StableLM is excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.
- StableLM is more than just an information source, StableLM is also able to write poetry, short stories, and make jokes.
- StableLM will refuse to participate in anything that could harm a human.
"""

prompt = f"{system_prompt}<|USER|>What's your mood today?<|ASSISTANT|>"

inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
tokens = model.generate(
  **inputs,
  max_new_tokens=64,
  temperature=0.7,
  do_sample=True,
  stopping_criteria=StoppingCriteriaList([StopOnTokens()])
)
print(tokenizer.decode(tokens[0], skip_special_tokens=True))

<|SYSTEM|># StableLM Tuned (Alpha version)
- StableLM is a helpful and harmless open-source AI language model developed by StabilityAI.
- StableLM is excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.
- StableLM is more than just an information source, StableLM is also able to write poetry, short stories, and make jokes.
- StableLM will refuse to participate in anything that could harm a human.

Model Details

Developed by: Stability AI
Model type: StableLM-Tuned-Alpha models are auto-regressive language models based on the NeoX transformer architecture.
Language(s): English
Library: HuggingFace Transformers
License: Fine-tuned checkpoints (StableLM-Tuned-Alpha) are licensed under the Non-Commercial Creative Commons license (CC BY-NC-SA-4.0), in-line with the original non-commercial license specified by Stanford Alpaca.
Contact: For questions and comments about the model, please email lm@stability.ai

Training

Parameters	Hidden Size	Layers	Heads	Sequence Length
3B	4096	16	32	4096
7B	6144	16	48	4096

Training Dataset

StableLM-Tuned-Alpha models are fine-tuned on a combination of five datasets: Alpaca, a dataset of 52,000 instructions and demonstrations generated by OpenAI's text-davinci-003 engine. GPT4All Prompt Generations, which consists of 400k prompts and responses generated by GPT-4; Anthropic HH, made up of preferences about AI assistant helpfulness and harmlessness; DataBricks Dolly, comprising 15k instruction/responses generated by Databricks employees in capability domains from the InstructGPT paper, including brainstorming, classification, closed QA, generation, information extraction, open QA and summarization; and ShareGPT Vicuna (English subset), a dataset of conversations retrieved from ShareGPT.

Training Procedure

Models are learned via supervised fine-tuning on the aforementioned datasets, trained in mixed-precision (FP16), and optimized with AdamW. We outline the following hyperparameters:

Parameters	Batch Size	Learning Rate	Warm-up	Weight Decay	Betas
3B	256	2e-5	50	0.01	(0.9, 0.99)
7B	128	2e-5	100	0.01	(0.9, 0.99)

Use and Limitations

Intended Use

These models are intended to be used by the open-source community chat-like applications in adherence with the CC BY-NC-SA-4.0 license.

Limitations and bias

Although the aforementioned datasets help to steer the base language models into "safer" distributions of text, not all biases and toxicity can be mitigated through fine-tuning. We ask that users be mindful of such potential issues that can arise in generated responses. Do not treat model outputs as substitutes for human judgment or as sources of truth. Please use responsibly.

Acknowledgements

This work would not have been possible without the helpful hand of Dakota Mahan (@dmayhem93).

Citations

@misc{alpaca,
  author = {Rohan Taori and Ishaan Gulrajani and Tianyi Zhang and Yann Dubois and Xuechen Li and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto },
  title = {Stanford Alpaca: An Instruction-following LLaMA model},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/tatsu-lab/stanford_alpaca}},
}

@misc{vicuna2023,
    title = {Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality},
    url = {https://vicuna.lmsys.org},
    author = {Chiang, Wei-Lin and Li, Zhuohan and Lin, Zi and Sheng, Ying and Wu, Zhanghao and Zhang, Hao and Zheng, Lianmin and Zhuang, Siyuan and Zhuang, Yonghao and Gonzalez, Joseph E. and Stoica, Ion and Xing, Eric P.},
    month = {March},
    year = {2023}
}

@misc{gpt4all,
  author = {Yuvanesh Anand and Zach Nussbaum and Brandon Duderstadt and Benjamin Schmidt and Andriy Mulyar},
  title = {GPT4All: Training an Assistant-style Chatbot with Large Scale Data Distillation from GPT-3.5-Turbo},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/nomic-ai/gpt4all}},
}