metadata

license: other
inference: false

OpenAssistant LLaMA 30B SFT 7 GGML

This is a repo of GGML format models for OpenAssistant's LLaMA 30B SFT 7.

It is the result of merging the XORs from the above repo with the original Llama 30B weights, and then quantising to 4bit and 5bit GGML for CPU inference using llama.cpp.

This is epoch 7 of OpenAssistant's training of their Llama 30B model.

Repositories available

PROMPT TEMPLATE

This model requires the following prompt template:

<|prompter|> prompt goes here
<|assistant|>:

THE FILES IN MAIN BRANCH REQUIRES LATEST LLAMA.CPP (May 19th 2023 - commit 2d5db48)!

llama.cpp recently made another breaking change to its quantisation methods - https://github.com/ggerganov/llama.cpp/pull/1508

I have quantised the GGML files in this repo with the latest version. Therefore you will require llama.cpp compiled on May 19th or later (commit 2d5db48 or later) to use them.

For files compatible with the previous version of llama.cpp, please see branch previous_llama_ggmlv2.

Provided files

Name	Quant method	Bits	Size	RAM required	Use case
`OpenAssistant-30B-epoch7.ggmlv3.q4_0.bin`	q4_0	4bit	20.3GB	23GB	4-bit.
`OpenAssistant-30B-epoch7.ggmlv3.q4_1.bin`	q4_1	4bit	22.4GB	25GB	4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models.
`OpenAssistant-30B-epoch7.ggmlv3.q5_0.bin`	q5_0	5bit	22.4GB	25GB	5-bit. Higher accuracy, higher resource usage and slower inference.
`OpenAssistant-30B-epoch7.ggmlv3.q5_1.bin`	q5_1	5bit	24.4GB	27GB	5-bit. Even higher accuracy, resource usage and slower inference.
`OpenAssistant-30B-epoch7.ggmlv3.q8_9.bin`	q8_0	8bit	24.4GB	27GB	8-bit. Almost indistinguishable from float16. Huge resource use and slow. Not recommended for normal use.

How to run in `llama.cpp`

I use the following command line; adjust for your tastes and needs:

./main -t 18 -m OpenAssistant-30B-epoch7.ggmlv3.q4_0.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "<|prompter|>Write a very story about llamas <|assistant|>:"

Change -t 18 to the number of physical CPU cores you have. For example if your system has 8 cores/16 threads, use -t 8.

How to run in `text-generation-webui`

GGML models can be loaded into text-generation-webui by installing the llama.cpp module, then placing the ggml model file in a model folder as usual.

Further instructions here: text-generation-webui/docs/llama.cpp-models.md.

Note: at this time text-generation-webui will likely not support the newly updated llama.cpp quantisation methods.

Thireus has written a great guide on how to update it to the latest llama.cpp code so that you can likely get support for the new quantisation methods sooner.

Original model card

llama-30b-sft-7:
  dtype: fp16
  log_dir: "llama_log_30b"
  learning_rate: 1e-5
  model_name: /home/ubuntu/Open-Assistant/model/model_training/.saved/llama-30b-super-pretrain/checkpoint-3500
  #model_name: OpenAssistant/llama-30b-super-pretrain
  output_dir: llama_model_30b
  deepspeed_config: configs/zero3_config_sft.json
  weight_decay: 0.0
  residual_dropout: 0.0
  max_length: 2048
  use_flash_attention: true
  warmup_steps: 20
  gradient_checkpointing: true
  gradient_accumulation_steps: 12
  per_device_train_batch_size: 2
  per_device_eval_batch_size: 3
  eval_steps: 101
  save_steps: 485
  num_train_epochs: 4
  save_total_limit: 3
  use_custom_sampler: true
  sort_by_length: false
  #save_strategy: steps
  save_strategy: epoch
  datasets:
    - oasst_export:
        lang: "bg,ca,cs,da,de,en,es,fr,hr,hu,it,nl,pl,pt,ro,ru,sl,sr,sv,uk"
        input_file_path: 2023-04-12_oasst_release_ready_synth.jsonl.gz
        val_split: 0.05
    - vicuna:
        val_split: 0.05
        max_val_set: 800
        fraction: 1.0
    - dolly15k:
        val_split: 0.05
        max_val_set: 300
    - grade_school_math_instructions:
        val_split: 0.05
    - code_alpaca:
        val_split: 0.05
        max_val_set: 250

OASST dataset paper: https://arxiv.org/abs/2304.07327