File size: 3,625 Bytes
3cb457a 0784209 3cb457a 015e969 3cb457a 015e969 3cb457a 0784209 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 |
---
license: wtfpl
base_model: CausalLM/7B
tags:
- generated_from_trainer
model-index:
- name: workspace/causal-dolphin-v0.1
results: []
datasets:
- ehartford/dolphin
- THUDM/AgentInstruct
---
[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
# Causal-Dolphin-Agent-v0.1
This model is a LoRA fine-tune of [CausalLM/7B](https://huggingface.co/CausalLM/7B) on Eric's wonderful [Dolphin](https://huggingface.co/datasets/ehartford/dolphin) dataset, with [THUDM/AgentInstruct](https://huggingface.co/datasets/THUDM/AgentInstruct) mixed in both training runs.
Causal-Dolphin-Agent was trained for 3 epochs on 5 million GPT3.5 augmented FLAN instructions & AgentInstruct dataset in ChatML format. It was then trained a further 3 epochs on 1 million GPT4 augmented FLAN instructions with AgentInstruct shuffled in as well.
It achieves the following results on the evaluation set:
- Loss: 2.8435
## Prompt Format
Causal-Dolphin-Agent uses ChatML as the prompt format:
```
<|im_start|>system
You are Dolphin, a helpful AI assistant.<|im_end|>
<|im_start|>user
If Danny owns a bike, then Edward owns a bike. If Edward owns a bike, then Freddy owns a bike. If Danny owns a bike, which of the following statements must be true? Let's think step by step.
I. Edward owns a bike.
II. Freddy owns a bike.
III. Freddy does not own a bike.
Choose one answer:
I only
II only
III only
I and II only
I and III only
<|im_end|>
<|im_start|>assistant
```
## Training and evaluation data
[ehartford/dolphin](https://huggingface.co/datasets/ehartford/dolphin)
[THUDM/AgentInstruct](https://huggingface.co/datasets/THUDM/AgentInstruct)
## Training procedure
Causal-Dolphin-Agent was trained for 3 epochs on 5 million GPT3.5 augmented FLAN instructions & AgentInstruct dataset in ChatML format. It was then trained a further 3 epochs on 1 million GPT4 augmented FLAN instructions with AgentInstruct shuffled in as well.
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 6e-06
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-05
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 100
- num_epochs: 3
### Training results
| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 2.7774 | 0.0 | 1 | 5.1009 |
| 3.2798 | 0.15 | 46 | 5.1010 |
| 2.0722 | 0.3 | 92 | 5.0489 |
| 2.5919 | 0.45 | 138 | 4.8834 |
| 2.0011 | 0.6 | 184 | 4.6678 |
| 1.3733 | 0.75 | 230 | 4.4628 |
| 1.7321 | 0.9 | 276 | 4.2757 |
| 1.3994 | 1.05 | 322 | 4.1029 |
| 1.2308 | 1.2 | 368 | 3.8916 |
| 0.8229 | 1.35 | 414 | 3.6451 |
| 0.9592 | 1.5 | 460 | 3.4106 |
| 0.8528 | 1.65 | 506 | 3.2250 |
| 0.7362 | 1.8 | 552 | 3.0852 |
| 0.8077 | 1.95 | 598 | 2.9881 |
| 0.6912 | 2.1 | 644 | 2.9315 |
| 0.7776 | 2.25 | 690 | 2.8911 |
| 0.6916 | 2.41 | 736 | 2.8678 |
| 0.8674 | 2.56 | 782 | 2.8534 |
| 0.7797 | 2.71 | 828 | 2.8545 |
| 0.6838 | 2.86 | 874 | 2.8435 |
### Framework versions
- Transformers 4.34.1
- Pytorch 2.1.0+cu121
- Datasets 2.14.6
- Tokenizers 0.14.1 |