---
license: apache-2.0
base_model: keeeeenw/MicroLlama
tags:
- generated_from_trainer
model-index:
- name: medusa-microllama_305M_stage2
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/momorami-kaist/medusa_test/runs/eg00n44l)
# medusa-microllama_305M_stage2

This model is a fine-tuned version of [keeeeenw/MicroLlama](https://huggingface.co/keeeeenw/MicroLlama) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 3.5262

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0005
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 40
- num_epochs: 2

### Training results

| Training Loss | Epoch  | Step | Validation Loss |
|:-------------:|:------:|:----:|:---------------:|
| 4.6913        | 0.0244 | 40   | 4.7578          |
| 4.8782        | 0.0489 | 80   | 4.8017          |
| 4.642         | 0.0733 | 120  | 4.7973          |
| 4.4601        | 0.0978 | 160  | 4.7589          |
| 4.4806        | 0.1222 | 200  | 4.6955          |
| 4.4856        | 0.1467 | 240  | 4.6196          |
| 4.4671        | 0.1711 | 280  | 4.5750          |
| 4.3228        | 0.1955 | 320  | 4.5563          |
| 4.1184        | 0.2200 | 360  | 4.5274          |
| 3.9986        | 0.2444 | 400  | 4.5031          |
| 4.2603        | 0.2689 | 440  | 4.4637          |
| 4.1344        | 0.2933 | 480  | 4.4349          |
| 4.1973        | 0.3178 | 520  | 4.4106          |
| 4.3961        | 0.3422 | 560  | 4.4202          |
| 4.1814        | 0.3666 | 600  | 4.3732          |
| 4.1685        | 0.3911 | 640  | 4.3877          |
| 4.3108        | 0.4155 | 680  | 4.3262          |
| 4.6294        | 0.4400 | 720  | 4.3108          |
| 4.3653        | 0.4644 | 760  | 4.2880          |
| 4.1505        | 0.4888 | 800  | 4.2835          |
| 3.8278        | 0.5133 | 840  | 4.2623          |
| 4.3567        | 0.5377 | 880  | 4.2253          |
| 4.2782        | 0.5622 | 920  | 4.1919          |
| 4.1025        | 0.5866 | 960  | 4.1846          |
| 4.2819        | 0.6111 | 1000 | 4.1637          |
| 3.9919        | 0.6355 | 1040 | 4.1323          |
| 4.1932        | 0.6599 | 1080 | 4.1017          |
| 4.0949        | 0.6844 | 1120 | 4.1085          |
| 3.7266        | 0.7088 | 1160 | 4.0668          |
| 4.1255        | 0.7333 | 1200 | 4.0500          |
| 4.3707        | 0.7577 | 1240 | 4.0207          |
| 4.1965        | 0.7822 | 1280 | 4.0065          |
| 3.4585        | 0.8066 | 1320 | 3.9363          |
| 3.7242        | 0.8310 | 1360 | 3.8893          |
| 3.9228        | 0.8555 | 1400 | 3.8569          |
| 4.2051        | 0.8799 | 1440 | 3.8412          |
| 3.6795        | 0.9044 | 1480 | 3.8245          |
| 3.2453        | 0.9288 | 1520 | 3.8132          |
| 3.5941        | 0.9533 | 1560 | 3.7907          |
| 3.6246        | 0.9777 | 1600 | 3.7573          |
| 2.8637        | 1.0021 | 1640 | 3.7530          |
| 2.8495        | 1.0266 | 1680 | 3.7741          |
| 3.0246        | 1.0510 | 1720 | 3.7690          |
| 2.99          | 1.0755 | 1760 | 3.7464          |
| 3.1902        | 1.0999 | 1800 | 3.7347          |
| 2.8099        | 1.1244 | 1840 | 3.7278          |
| 2.7652        | 1.1488 | 1880 | 3.7245          |
| 2.6362        | 1.1732 | 1920 | 3.7034          |
| 2.8562        | 1.1977 | 1960 | 3.6871          |
| 3.1712        | 1.2221 | 2000 | 3.6786          |
| 2.7405        | 1.2466 | 2040 | 3.6709          |
| 2.734         | 1.2710 | 2080 | 3.6404          |
| 3.1788        | 1.2954 | 2120 | 3.6310          |
| 2.9609        | 1.3199 | 2160 | 3.6176          |
| 3.0737        | 1.3443 | 2200 | 3.6136          |
| 2.751         | 1.3688 | 2240 | 3.5960          |
| 2.7105        | 1.3932 | 2280 | 3.5872          |
| 2.8158        | 1.4177 | 2320 | 3.5848          |
| 3.03          | 1.4421 | 2360 | 3.5679          |
| 2.8122        | 1.4665 | 2400 | 3.5718          |
| 2.5581        | 1.4910 | 2440 | 3.5568          |
| 2.9845        | 1.5154 | 2480 | 3.5496          |
| 2.83          | 1.5399 | 2520 | 3.5440          |
| 2.7004        | 1.5643 | 2560 | 3.5402          |
| 2.8271        | 1.5888 | 2600 | 3.5406          |
| 2.5315        | 1.6132 | 2640 | 3.5316          |
| 2.6001        | 1.6376 | 2680 | 3.5346          |
| 2.4959        | 1.6621 | 2720 | 3.5298          |
| 2.9174        | 1.6865 | 2760 | 3.5304          |
| 2.7219        | 1.7110 | 2800 | 3.5286          |
| 2.5395        | 1.7354 | 2840 | 3.5279          |
| 2.7464        | 1.7599 | 2880 | 3.5284          |
| 2.7532        | 1.7843 | 2920 | 3.5274          |
| 2.6472        | 1.8087 | 2960 | 3.5270          |
| 2.8263        | 1.8332 | 3000 | 3.5268          |
| 2.916         | 1.8576 | 3040 | 3.5263          |
| 3.0202        | 1.8821 | 3080 | 3.5262          |
| 2.7152        | 1.9065 | 3120 | 3.5261          |
| 2.7628        | 1.9310 | 3160 | 3.5261          |
| 2.783         | 1.9554 | 3200 | 3.5263          |
| 3.2587        | 1.9798 | 3240 | 3.5262          |


### Framework versions

- Transformers 4.43.0
- Pytorch 2.3.1
- Datasets 2.15.0
- Tokenizers 0.19.1