---
license: other
---

## airoboros-gpt-3.5-turbo-100k-7b

This is a 7b parameter, fine-tuned on 100k synthetic instruction/response pairs generated by gpt-3.5-turbo using (airoboros)[https://github.com/jondurbin/airoboros]

Links:

* (airoboros)[https://github.com/jondurbin/airoboros]
* (instructions.jsonl)[https://storage.googleapis.com/airoboros-dump/gpt-3.5-turbo-100k/instructions.jsonl]
* (topics.txt)[https://storage.googleapis.com/airoboros-dump/gpt-3.5-turbo-100k/topics-d732f92dd90a1a5337a4a02ddeaec72b.txt]

### Prompt generation

```
airoboros generate-instructions --instruction-count 100000 --concurrency 100 --temperature 1.0
```

### Fine-tuning

The instructions.jsonl file was converted to conversation style expected by the FastChat training scripts, and then trained with:
```
torchrun --nproc_per_node=8 --master_port=20001 train_mem.py \
        --model_name_or_path /workspace/llama-7b-hf \
        --data_path ./as_conversations.json \
        --bf16 True \
        --output_dir /workspace/airoboros-gpt-3.5-100k-7b \
        --num_train_epochs 3 \
        --per_device_train_batch_size 4 \
        --per_device_eval_batch_size 32 \
        --gradient_accumulation_steps 4 \
        --evaluation_strategy "steps" \
        --eval_steps 1500 \
        --save_strategy "steps" \
        --save_steps 1500 \
        --save_total_limit 8 \
        --learning_rate 2e-5 \
        --weight_decay 0. \
        --warmup_ratio 0.04 \
        --lr_scheduler_type "cosine" \
        --logging_steps 1 \
        --fsdp "full_shard auto_wrap offload" \
        --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
        --tf32 True \
        --model_max_length 2048 \
        --gradient_checkpointing True \
        --lazy_preprocess True
```

Training took roughly 22 hours on 8x nvidia A100 80GB.

Conversion to conversation style:
```
import json
import uuid
inputs = [json.loads(line) for line in open("instructions.jsonl").readlines()]
conversations = []
for row in inputs:
    inputs = row['instruction']
    conversations.append({
        "id": str(uuid.uuid4()),
        "conversations": [
            {
                "from": "human",
                "value": inputs,
            },
            {
                "from": "gpt",
                "value": row['response']
            },
        ],
    })
with open("as_conversations.json", "w") as outfile:
    outfile.write(json.dumps(conversations, indent=2)
```

## Evaluation

I used the same questions from (WizardVicunaLM)[]:

| instruction | gpt3.5 | wizard-vicuna-13b | vicuna-13b | wizard-7b | airoboros-gpt-3.5-turbo-100k-7b |
| --- | --- | --- | --- | --- | --- |
| "Write a compelling product launch announcement email to inform our customers of our new software solution." | 95 | 92 | 89 | 90 | 91 |
| "Draft an apology email to a customer who experienced a delay in their order, and provide reassurance that the issue has been resolved." | 94 | 96 | 90 | 89 | 91 |
| "As a pirate captain, what would you say to your crew to motivate them to search for hidden treasure?" | 95 | 90 | 80 | 70 | 85 |
| "Imagine you are a time traveler from the year 3000. What technological advancements would you tell people about?" | 95 | 92 | 90 | 88 | 85 |
| "As a space colonist on Mars, describe your daily life and the challenges you face living on another planet." | 95 | 90 | 87 | 85 | 88 |
| "How can you assess the credibility of a source of information, such as a news article or blog post, without relying solely on the reputation of the author or publisher?" | 93 | 85 | 89 | 87 | 90 |
| "How can observing the behavior of other people in a social situation provide clues about cultural norms and expectations?" | 95 | 90 | 85 | 92 | 80 |
| "How many text messages are sent globally in a minute? Try to explain your answer. Your explanation should take the reader through your reasoning step-by-step." | 90 | 70 | 65 | 80 | 85 |
| "What are the main differences between Python and JavaScript programming languages?"| 90 | 85 | 80 | 88 | 82 |
| "What are the differences between plant-based and animal-based protein sources?"| 85 | 92 | 90 | 80 | 94 |
| "Describe a scenario where artificial intelligence could be used to improve the quality and efficiency of healthcare delivery." | 95 | 90 | 92 | 89 | 91 |
| "How do cultural, social, and economic factors influence people's food choices, and how can this knowledge be used to promote healthier diets?" | 90 | 85 | 87 | 83 | 84 |
| "How many words are spoken daily on Earth? Try to explain your answer. Your explanation should take the reader through your reasoning step-by-step." | 90 | 70 | 80 | 75 | 65 |
| "How many lightning strikes occur on Earth each day? Try to explain your answer. Your explanation should take the reader through your reasoning step-by-step." | 90 | 80 | 60 | 70 | 85 |

If we use gpt-3.5 as the baseline (as wizardvicuna/vicuna did), we get the following scores:

| gpt3.5 | wizard-vicuna-13b | vicuna-13b | wizard-7b | airoboros-gpt-3.5-turbo-100k-7b |
| --- | --- | --- | --- | --- |
| 1.0 | __0.968421052631579__ | 0.9368421052631579 | 0.9473684210526315 | 0.9578947368421052 |
| 1.0 | __1.0212765957446808__ | 0.9574468085106383 | 0.9468085106382979 | 0.9680851063829787 |
| 1.0 | __0.9473684210526315__ | 0.8421052631578947 | 0.7368421052631579 | 0.8947368421052632 |
| 1.0 | __0.968421052631579__ | 0.9473684210526315 | 0.9263157894736842 | 0.8947368421052632 |
| 1.0 | __0.9473684210526315__ | 0.9157894736842105 | 0.8947368421052632 | 0.9263157894736842 |
| 1.0 | 0.9139784946236559 | 0.956989247311828 | 0.9354838709677419 | __0.967741935483871__ |
| 1.0 | 0.9473684210526315 | 0.8947368421052632 | __0.968421052631579__ | 0.8421052631578947 |
| 1.0 | 0.7777777777777778 | 0.7222222222222222 | 0.8888888888888888 | __0.9444444444444444__ |
| 1.0 | 0.9444444444444444 | 0.8888888888888888 | __0.9777777777777777__ | 0.9111111111111111 |
| 1.0 | 1.0823529411764705 | 1.0588235294117647 | 0.9411764705882353 | __1.1058823529411765__ |
| 1.0 | 0.9473684210526315 | __0.968421052631579__ | 0.9368421052631579 | 0.9578947368421052 |
| 1.0 | 0.9444444444444444 | __0.9666666666666667__ | 0.9222222222222223 | 0.9333333333333333 |
| 1.0 | 0.7777777777777778 | __0.8888888888888888__ | 0.8333333333333334 | 0.7222222222222222 |
| 1.0 | 0.8888888888888888 | 0.6666666666666666 | 0.7777777777777778 | __0.9444444444444444__ |

Average scores:

```
gpt3.5                             1.000000
wizard-vicuna-13b                  0.934090
vicuna-13b                         0.900847
wizard-7b                          0.902428
airoboros-gpt-3.5-turbo-100k-7b    0.926496
```
As you can see, the __7b__ airoboros model performs well, even compared to 13b models.

## License
The model is licensed under the LLaMA model, and the dataset is licensed under the terms of OpenAI because it uses ChatGPT. Everything else is free.