---
license: other
library_name: peft
tags:
- llama2
base_model: NousResearch/Llama-2-7b-hf
---


# Model Card for PIPPA ShareGPT Subset Lora 7b

It is an experimental Lora focused on Roleplay that uses a subset of [PIPPA ShareGPT](https://huggingface.co/datasets/kingbri/PIPPA-shareGPT)

## Usage

Custom
```
SYSTEM: Do thing
USER: {prompt}
CHARACTER:
```

## Bias, Risks, and Limitations

This Lora is not intended for supplying factual information or advice in any form

## Training Details

### Training Data

1k of conversation from [PIPPA ShareGPT](https://huggingface.co/datasets/kingbri/PIPPA-shareGPT)

### Training Procedure 

The version of this Lora uploaded on this repository was trained using a 8x RTX A6000 cluster in 8-bit with regular LoRA adapters and 32-bit AdamW optimizer.

#### Training Hyperparameters

Training using a fork of [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) with two paths [Patch 1](https://github.com/bdashore3/axolotl/commit/995557bdf3c6c8b3e839b224ef9513fc2b097f30) [Patch 2](https://github.com/bdashore3/axolotl/commit/8970280de2ea01e41c044406051922715f4086cb)

- load_in_8bit: true
- lora_r: 16
- lora_alpha: 16
- lora_dropout: 0.01
- gradient_accumulation_steps: 6
- micro_batch_size: 4
- num_epochs: 3
- learning_rate: 0.000065

## Environmental Impact

Finetuning this model on 8xNVIDIA A6000 48GB in parallel takes about 30 minutes (7B)