Pippa-13b-qlora

This is a repository of my Llama-2-13b Qlora checkpoints of the PIPPA-13b-ShareGPT dataset.

You can read more about the dataset on its relevant page. It's a ShareGPT reformat of the PIPPA dataset by PygmalionAI. The reformat was done to allow for axolotl compatability.

Architecture

Model Architecture: Llama-2-13b
Training Algorithm: QLora
Dataset Used: PIPPA-ShareGPT (pippa_sharegpt_trimmed.jsonl)

Training Details

Dataset: PIPPA-ShareGPT
Datset type: ShareGPT
Training Parameters: See Here
Training Environment: Axolotl
sequence_len: 4096

Instruct Format

ShareGPT gets converted to vicuna format. The dataset uses modified roles of USER and CHARACTER instead of USER and ASSISTANT.

SYSTEM: Enter roleplay mode...
USER: {prompt}
CHARACTER:

Notes

This Qlora was produced as an experiment to see how the public version of PIPPA can affect a model. As a result, I have no idea if this lora is of great quality or absolute garbage.

Acknowledgments

Thanks to:

PygmalionAI: The creators of the PIPPA dataset
Axolotl: Finetuning suite

Donate?

All my infrastructure and cloud expenses are paid out of pocket. If you'd like to donate, you can do so here: https://ko-fi.com/kingbri

You should not feel obligated to donate, but if you do, I'd appreciate it.

Axolotl stuff

Training procedure

The following bitsandbytes quantization config was used during training:

quant_method: bitsandbytes
load_in_8bit: False
load_in_4bit: True
llm_int8_threshold: 6.0
llm_int8_skip_modules: None
llm_int8_enable_fp32_cpu_offload: False
llm_int8_has_fp16_weight: False
bnb_4bit_quant_type: nf4
bnb_4bit_use_double_quant: True
bnb_4bit_compute_dtype: bfloat16

Framework versions

PEFT 0.6.0.dev0