README.md

---
license: cc-by-nc-4.0
---

## This repo is old, and uses an old prompt format, you may want to look at [airoboros-13b](https://huggingface.com/jondurbin/airoboros-13b)

## airoboros-gpt-3.5-turbo-100k-7b

This is a 7b parameter, fine-tuned on 100k synthetic instruction/response pairs generated by gpt-3.5-turbo using my version of self-instruct [airoboros](https://github.com/jondurbin/airoboros)

Context length for this model is 2048.

Links:

* [airoboros](https://github.com/jondurbin/airoboros)
* [instructions.jsonl](https://storage.googleapis.com/airoboros-dump/gpt-3.5-turbo-100k/instructions.jsonl)
* [topics.txt](https://storage.googleapis.com/airoboros-dump/gpt-3.5-turbo-100k/topics-d732f92dd90a1a5337a4a02ddeaec72b.txt)

### Prompt generation

```
airoboros generate-instructions --instruction-count 100000 --concurrency 100 --temperature 1.0
```

### Fine-tuning

The instructions.jsonl file was converted to conversation style expected by the FastChat training scripts, and then trained with:
```
torchrun --nproc_per_node=8 --master_port=20001 train_mem.py \
        --model_name_or_path /workspace/llama-7b-hf \
        --data_path ./as_conversations.json \
        --bf16 True \
        --output_dir /workspace/airoboros-gpt-3.5-100k-7b \
        --num_train_epochs 3 \
        --per_device_train_batch_size 4 \
        --per_device_eval_batch_size 32 \
        --gradient_accumulation_steps 4 \
        --evaluation_strategy "steps" \
        --eval_steps 1500 \
        --save_strategy "steps" \
        --save_steps 1500 \
        --save_total_limit 8 \
        --learning_rate 2e-5 \
        --weight_decay 0. \
        --warmup_ratio 0.04 \
        --lr_scheduler_type "cosine" \
        --logging_steps 1 \
        --fsdp "full_shard auto_wrap offload" \
        --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
        --tf32 True \
        --model_max_length 2048 \
        --gradient_checkpointing True \
        --lazy_preprocess True
```

Training took roughly 22 hours on 8x nvidia A100 80GB.

Conversion to conversation style:
```
import json
import uuid
inputs = [json.loads(line) for line in open("instructions.jsonl").readlines()]
conversations = []
for row in inputs:
    inputs = row['instruction']
    conversations.append({
        "id": str(uuid.uuid4()),
        "conversations": [
            {
                "from": "human",
                "value": inputs,
            },
            {
                "from": "gpt",
                "value": row['response']
            },
        ],
    })
with open("as_conversations.json", "w") as outfile:
    outfile.write(json.dumps(conversations, indent=2))
```

## Evaluation

I used the same questions from [WizardVicunaLM](https://github.com/melodysdreamj/WizardVicunaLM):

| instruction | gpt3.5 | wizard-vicuna-13b | vicuna-13b | wizard-7b | airoboros-gpt-3.5-turbo-100k-7b |
| --- | --- | --- | --- | --- | --- |
| "Write a compelling product launch announcement email to inform our customers of our new software solution." | 95 | 92 | 89 | 90 | 91 |
| "Draft an apology email to a customer who experienced a delay in their order, and provide reassurance that the issue has been resolved." | 94 | 96 | 90 | 89 | 91 |
| "As a pirate captain, what would you say to your crew to motivate them to search for hidden treasure?" | 95 | 90 | 80 | 70 | 85 |
| "Imagine you are a time traveler from the year 3000. What technological advancements would you tell people about?" | 95 | 92 | 90 | 88 | 85 |
| "As a space colonist on Mars, describe your daily life and the challenges you face living on another planet." | 95 | 90 | 87 | 85 | 88 |
| "How can you assess the credibility of a source of information, such as a news article or blog post, without relying solely on the reputation of the author or publisher?" | 93 | 85 | 89 | 87 | 90 |
| "How can observing the behavior of other people in a social situation provide clues about cultural norms and expectations?" | 95 | 90 | 85 | 92 | 80 |
| "How many text messages are sent globally in a minute? Try to explain your answer. Your explanation should take the reader through your reasoning step-by-step." | 90 | 70 | 65 | 80 | 85 |
| "What are the main differences between Python and JavaScript programming languages?"| 90 | 85 | 80 | 88 | 82 |
| "What are the differences between plant-based and animal-based protein sources?"| 85 | 92 | 90 | 80 | 94 |
| "Describe a scenario where artificial intelligence could be used to improve the quality and efficiency of healthcare delivery." | 95 | 90 | 92 | 89 | 91 |
| "How do cultural, social, and economic factors influence people's food choices, and how can this knowledge be used to promote healthier diets?" | 90 | 85 | 87 | 83 | 84 |
| "How many words are spoken daily on Earth? Try to explain your answer. Your explanation should take the reader through your reasoning step-by-step." | 90 | 70 | 80 | 75 | 65 |
| "How many lightning strikes occur on Earth each day? Try to explain your answer. Your explanation should take the reader through your reasoning step-by-step." | 90 | 80 | 60 | 70 | 85 |

If we use gpt-3.5 as the baseline (as wizardvicuna/vicuna did), we get the following scores:

| gpt3.5 | wizard-vicuna-13b | vicuna-13b | wizard-7b | airoboros-gpt-3.5-turbo-100k-7b |
| --- | --- | --- | --- | --- |
| 1.0 | __0.968421052631579__ | 0.9368421052631579 | 0.9473684210526315 | 0.9578947368421052 |
| 1.0 | __1.0212765957446808__ | 0.9574468085106383 | 0.9468085106382979 | 0.9680851063829787 |
| 1.0 | __0.9473684210526315__ | 0.8421052631578947 | 0.7368421052631579 | 0.8947368421052632 |
| 1.0 | __0.968421052631579__ | 0.9473684210526315 | 0.9263157894736842 | 0.8947368421052632 |
| 1.0 | __0.9473684210526315__ | 0.9157894736842105 | 0.8947368421052632 | 0.9263157894736842 |
| 1.0 | 0.9139784946236559 | 0.956989247311828 | 0.9354838709677419 | __0.967741935483871__ |
| 1.0 | 0.9473684210526315 | 0.8947368421052632 | __0.968421052631579__ | 0.8421052631578947 |
| 1.0 | 0.7777777777777778 | 0.7222222222222222 | 0.8888888888888888 | __0.9444444444444444__ |
| 1.0 | 0.9444444444444444 | 0.8888888888888888 | __0.9777777777777777__ | 0.9111111111111111 |
| 1.0 | 1.0823529411764705 | 1.0588235294117647 | 0.9411764705882353 | __1.1058823529411765__ |
| 1.0 | 0.9473684210526315 | __0.968421052631579__ | 0.9368421052631579 | 0.9578947368421052 |
| 1.0 | 0.9444444444444444 | __0.9666666666666667__ | 0.9222222222222223 | 0.9333333333333333 |
| 1.0 | 0.7777777777777778 | __0.8888888888888888__ | 0.8333333333333334 | 0.7222222222222222 |
| 1.0 | 0.8888888888888888 | 0.6666666666666666 | 0.7777777777777778 | __0.9444444444444444__ |

Average scores:

```
gpt3.5                             1.000000
wizard-vicuna-13b                  0.934090
vicuna-13b                         0.900847
wizard-7b                          0.902428
airoboros-gpt-3.5-turbo-100k-7b    0.926496
```
As you can see, the __7b__ airoboros model performs well, even compared to 13b models.


## Usage

The easiest way to use this model is via fastchat:

```
python -m fastchat.serve.cli --model-path ./airoboros-gpt-3.5-turbo-100k-7b --temperature 1.0
```

For example:
```
Human: Write an email introducing a new instruction-tuned AI model named airoboros.  The key features about airoboros are: 1. it was fine-tuned with entirely machine-generated synthetic instruction/response pairs  2. the 7 billion parameter model performs very well in comparison to 13b models  3. the synthetic prompt generation tool is free, open-source, and highly customizable.
Assistant: Subject: Introducing airoboros - a new instruction-tuned AI model

Dear [Recipient],

I am excited to introduce a new instruction-tuned AI model named airoboros. This model was developed by our team with the aim of providing high-quality natural language processing capabilities at scale.

airoboros is a 7 billion parameter model that was fine-tuned with entirely machine-generated synthetic instruction/response pairs. This approach allowed us to achieve state-of-the-art performance in various NLP tasks such as question answering, sentiment analysis, and machine translation. In fact, when compared to other popular 13b models, airoboros performed better on several benchmarks.

Moreover, airoboros's synthetic prompt generation tool is free, open-source, and highly customizable. This means that developers can easily integrate our model into their applications without having to worry about licensing fees or restrictions.

We believe that airoboros will be an excellent addition to your workflow and hope you find it useful. If you have any questions or concerns, please do not hesitate to contact us.

Best regards,

[Your Name]

[Your Company Name]
```

Keep in mind, this is an instruction-tuned model and not well-suited to chat.

### Usage and License Notices

All airoboros models and datasets are intended and licensed for research use only.  I've used the 'cc-nc-4.0' license, but really it is subject to a custom/special license because:

- the base model is LLaMa, which has it's own special research license
- the dataset(s) were generated with OpenAI (gpt-4 and/or gpt-3.5-turbo), which has a clausing saying the data can't be used to create models to compete with openai

So, to reiterate: this model (and datasets) cannot be used commercially.