---
base_model: NousResearch/Llama-2-7b-chat-hf
tags:
- generated_from_trainer
model-index:
- name: aiql01
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
# aiql01

This model is a fine-tuned version of [NousResearch/Llama-2-7b-chat-hf](https://huggingface.co/NousResearch/Llama-2-7b-chat-hf) on the Agent Instruct dataset.
It achieves the following results on the evaluation set:
- Loss: 0.3156

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 0.5579        | 0.09  | 20   | 0.6611          |
| 0.3651        | 0.18  | 40   | 0.4642          |
| 0.3704        | 0.26  | 60   | 0.3942          |
| 0.3868        | 0.35  | 80   | 0.3672          |
| 0.2843        | 0.44  | 100  | 0.3513          |
| 0.2535        | 0.53  | 120  | 0.3479          |
| 0.3797        | 0.61  | 140  | 0.3406          |
| 0.3702        | 0.7   | 160  | 0.3347          |
| 0.2986        | 0.79  | 180  | 0.3345          |
| 0.2952        | 0.88  | 200  | 0.3249          |
| 0.2109        | 0.96  | 220  | 0.3272          |
| 0.2744        | 1.05  | 240  | 0.3275          |
| 0.1762        | 1.14  | 260  | 0.3249          |
| 0.28          | 1.23  | 280  | 0.3247          |
| 0.2997        | 1.31  | 300  | 0.3263          |
| 0.2115        | 1.4   | 320  | 0.3212          |
| 0.252         | 1.49  | 340  | 0.3229          |
| 0.2687        | 1.58  | 360  | 0.3246          |
| 0.1996        | 1.66  | 380  | 0.3205          |
| 0.2227        | 1.75  | 400  | 0.3164          |
| 0.5474        | 1.84  | 420  | 0.3145          |
| 0.2224        | 1.93  | 440  | 0.3121          |
| 0.3695        | 2.02  | 460  | 0.3129          |
| 0.1738        | 2.1   | 480  | 0.3182          |
| 0.1299        | 2.19  | 500  | 0.3178          |
| 0.2314        | 2.28  | 520  | 0.3159          |
| 0.2391        | 2.37  | 540  | 0.3179          |
| 0.2131        | 2.45  | 560  | 0.3178          |
| 0.1541        | 2.54  | 580  | 0.3154          |
| 0.2768        | 2.63  | 600  | 0.3153          |
| 0.1362        | 2.72  | 620  | 0.3155          |
| 0.2494        | 2.8   | 640  | 0.3157          |
| 0.1641        | 2.89  | 660  | 0.3154          |
| 0.1397        | 2.98  | 680  | 0.3156          |


### Framework versions

- Transformers 4.34.1
- Pytorch 2.0.1+cu118
- Datasets 2.14.6
- Tokenizers 0.14.1