metadata
base_model: unsloth/llama-3-8b
library_name: peft
license: llama3
tags:
- unsloth
- generated_from_trainer
model-index:
- name: Meta-Llama-3-8B_metamath_ortho
results: []
Meta-Llama-3-8B_metamath_ortho
This model is a fine-tuned version of unsloth/llama-3-8b on an unknown dataset. It achieves the following results on the evaluation set:
- Loss: 0.4760
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0003
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 8
- total_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.02
- num_epochs: 1
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 0.9348 | 0.0211 | 13 | 0.7026 |
| 0.6609 | 0.0421 | 26 | 0.6960 |
| 0.6695 | 0.0632 | 39 | 0.6785 |
| 0.6578 | 0.0842 | 52 | 0.6770 |
| 0.6222 | 0.1053 | 65 | 0.6748 |
| 0.6331 | 0.1264 | 78 | 0.6662 |
| 0.6413 | 0.1474 | 91 | 0.6609 |
| 0.631 | 0.1685 | 104 | 0.6538 |
| 0.6115 | 0.1896 | 117 | 0.6490 |
| 0.6097 | 0.2106 | 130 | 0.6458 |
| 0.6052 | 0.2317 | 143 | 0.6533 |
| 0.6131 | 0.2527 | 156 | 0.6335 |
| 0.6012 | 0.2738 | 169 | 0.6165 |
| 0.6002 | 0.2949 | 182 | 0.6204 |
| 0.5849 | 0.3159 | 195 | 0.6220 |
| 0.5689 | 0.3370 | 208 | 0.6159 |
| 0.5781 | 0.3580 | 221 | 0.6110 |
| 0.5765 | 0.3791 | 234 | 0.6027 |
| 0.5899 | 0.4002 | 247 | 0.5983 |
| 0.5638 | 0.4212 | 260 | 0.5905 |
| 0.5716 | 0.4423 | 273 | 0.5874 |
| 0.5729 | 0.4633 | 286 | 0.5809 |
| 0.5691 | 0.4844 | 299 | 0.5729 |
| 0.5441 | 0.5055 | 312 | 0.5659 |
| 0.5468 | 0.5265 | 325 | 0.5584 |
| 0.536 | 0.5476 | 338 | 0.5544 |
| 0.5277 | 0.5687 | 351 | 0.5474 |
| 0.5052 | 0.5897 | 364 | 0.5397 |
| 0.5185 | 0.6108 | 377 | 0.5309 |
| 0.5161 | 0.6318 | 390 | 0.5262 |
| 0.5056 | 0.6529 | 403 | 0.5227 |
| 0.5091 | 0.6740 | 416 | 0.5164 |
| 0.492 | 0.6950 | 429 | 0.5115 |
| 0.4936 | 0.7161 | 442 | 0.5070 |
| 0.4818 | 0.7371 | 455 | 0.5005 |
| 0.4762 | 0.7582 | 468 | 0.4986 |
| 0.4685 | 0.7793 | 481 | 0.4938 |
| 0.4614 | 0.8003 | 494 | 0.4904 |
| 0.4942 | 0.8214 | 507 | 0.4870 |
| 0.4767 | 0.8424 | 520 | 0.4837 |
| 0.4589 | 0.8635 | 533 | 0.4819 |
| 0.4806 | 0.8846 | 546 | 0.4796 |
| 0.4647 | 0.9056 | 559 | 0.4782 |
| 0.461 | 0.9267 | 572 | 0.4773 |
| 0.4718 | 0.9478 | 585 | 0.4766 |
| 0.4684 | 0.9688 | 598 | 0.4761 |
| 0.4716 | 0.9899 | 611 | 0.4760 |
Framework versions
- PEFT 0.12.0
- Transformers 4.44.0
- Pytorch 2.4.0+cu121
- Datasets 2.20.0
- Tokenizers 0.19.1