---
base_model: microsoft/Phi-3.5-mini-instruct
library_name: peft
license: mit
tags:
- trl
- sft
- generated_from_trainer
model-index:
- name: phi-3-mini-LoRA
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# phi-3-mini-LoRA

This model is a fine-tuned version of [microsoft/Phi-3.5-mini-instruct](https://huggingface.co/microsoft/Phi-3.5-mini-instruct) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.4630

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.2
- training_steps: 250

### Training results

| Training Loss | Epoch   | Step | Validation Loss |
|:-------------:|:-------:|:----:|:---------------:|
| 1.6578        | 1.3333  | 5    | 1.6863          |
| 1.6277        | 2.6667  | 10   | 1.6771          |
| 1.615         | 4.0     | 15   | 1.6615          |
| 1.5879        | 5.3333  | 20   | 1.6372          |
| 1.5835        | 6.6667  | 25   | 1.6028          |
| 1.5908        | 8.0     | 30   | 1.5586          |
| 1.5143        | 9.3333  | 35   | 1.5012          |
| 1.4633        | 10.6667 | 40   | 1.4352          |
| 1.3414        | 12.0    | 45   | 1.3606          |
| 1.3229        | 13.3333 | 50   | 1.2811          |
| 1.2218        | 14.6667 | 55   | 1.2119          |
| 1.1352        | 16.0    | 60   | 1.1488          |
| 1.0852        | 17.3333 | 65   | 1.0885          |
| 0.9989        | 18.6667 | 70   | 1.0299          |
| 0.9959        | 20.0    | 75   | 0.9757          |
| 0.921         | 21.3333 | 80   | 0.9205          |
| 0.8727        | 22.6667 | 85   | 0.8683          |
| 0.8067        | 24.0    | 90   | 0.8200          |
| 0.7785        | 25.3333 | 95   | 0.7783          |
| 0.7139        | 26.6667 | 100  | 0.7396          |
| 0.7081        | 28.0    | 105  | 0.7095          |
| 0.6705        | 29.3333 | 110  | 0.6824          |
| 0.6177        | 30.6667 | 115  | 0.6613          |
| 0.6106        | 32.0    | 120  | 0.6418          |
| 0.575         | 33.3333 | 125  | 0.6239          |
| 0.5904        | 34.6667 | 130  | 0.6083          |
| 0.5917        | 36.0    | 135  | 0.5927          |
| 0.5051        | 37.3333 | 140  | 0.5801          |
| 0.5169        | 38.6667 | 145  | 0.5656          |
| 0.5442        | 40.0    | 150  | 0.5542          |
| 0.5112        | 41.3333 | 155  | 0.5432          |
| 0.5061        | 42.6667 | 160  | 0.5321          |
| 0.5071        | 44.0    | 165  | 0.5234          |
| 0.4373        | 45.3333 | 170  | 0.5119          |
| 0.4476        | 46.6667 | 175  | 0.5049          |
| 0.3914        | 48.0    | 180  | 0.4972          |
| 0.465         | 49.3333 | 185  | 0.4914          |
| 0.4122        | 50.6667 | 190  | 0.4890          |
| 0.4209        | 52.0    | 195  | 0.4837          |
| 0.3933        | 53.3333 | 200  | 0.4784          |
| 0.3583        | 54.6667 | 205  | 0.4760          |
| 0.3952        | 56.0    | 210  | 0.4727          |
| 0.3858        | 57.3333 | 215  | 0.4708          |
| 0.3433        | 58.6667 | 220  | 0.4707          |
| 0.4041        | 60.0    | 225  | 0.4680          |
| 0.3558        | 61.3333 | 230  | 0.4665          |
| 0.382         | 62.6667 | 235  | 0.4650          |
| 0.3625        | 64.0    | 240  | 0.4638          |
| 0.3513        | 65.3333 | 245  | 0.4644          |
| 0.3541        | 66.6667 | 250  | 0.4630          |


### Framework versions

- PEFT 0.13.2
- Transformers 4.45.2
- Pytorch 2.4.0
- Datasets 3.0.1
- Tokenizers 0.20.0