Hugging Face's logo

imdatta0
/
lora_final

PEFT
Safetensors
unsloth
Generated from Trainer
Model card Files Community
lora_final / Mistral-7B-v0.3_pct_default_r32 /README.md
imdatta0's picture
imdatta0
End of training
8b1b653 verified
preview code
|
raw
history blame
3.92 kB
metadata
base_model: mistralai/Mistral-7B-v0.3
library_name: peft
license: apache-2.0
tags:
  - unsloth
  - generated_from_trainer
model-index:
  - name: Mistral-7B-v0.3_pct_default_r32
    results: []

Mistral-7B-v0.3_pct_default_r32

This model is a fine-tuned version of mistralai/Mistral-7B-v0.3 on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 2.0448

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0001
  • train_batch_size: 1
  • eval_batch_size: 1
  • seed: 42
  • gradient_accumulation_steps: 64
  • total_train_batch_size: 64
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.02
  • num_epochs: 1

Training results

Training Loss Epoch Step Validation Loss
1.9915 0.0206 8 2.0385
2.054 0.0413 16 2.0376
2.0356 0.0619 24 2.0604
2.0385 0.0825 32 2.0639
2.1223 0.1032 40 2.0833
2.0677 0.1238 48 2.0910
2.0729 0.1444 56 2.0872
2.1197 0.1651 64 2.0973
2.1053 0.1857 72 2.0919
2.0848 0.2063 80 2.1035
2.1015 0.2270 88 2.1114
2.0872 0.2476 96 2.1133
2.0948 0.2682 104 2.1221
2.097 0.2889 112 2.1219
2.147 0.3095 120 2.1240
2.1315 0.3301 128 2.1189
2.1563 0.3508 136 2.1368
2.1836 0.3714 144 2.1271
2.1245 0.3920 152 2.1198
2.0947 0.4127 160 2.1240
2.1472 0.4333 168 2.1354
2.1348 0.4539 176 2.1261
2.1099 0.4746 184 2.1275
2.1006 0.4952 192 2.1196
2.1339 0.5158 200 2.1170
2.0841 0.5364 208 2.1105
2.1344 0.5571 216 2.1079
2.0732 0.5777 224 2.1043
2.0417 0.5983 232 2.1035
2.1003 0.6190 240 2.0967
2.0501 0.6396 248 2.1007
2.078 0.6602 256 2.0862
2.0507 0.6809 264 2.0840
2.0235 0.7015 272 2.0762
2.0743 0.7221 280 2.0723
2.1028 0.7428 288 2.0721
2.0987 0.7634 296 2.0662
2.0985 0.7840 304 2.0663
2.0548 0.8047 312 2.0602
2.0365 0.8253 320 2.0563
2.0102 0.8459 328 2.0564
2.0497 0.8666 336 2.0522
2.0721 0.8872 344 2.0471
2.0812 0.9078 352 2.0468
2.0475 0.9285 360 2.0462
2.0687 0.9491 368 2.0452
2.065 0.9697 376 2.0450
1.991 0.9904 384 2.0448

Framework versions

  • PEFT 0.12.0
  • Transformers 4.44.2
  • Pytorch 2.3.0+cu121
  • Datasets 2.21.0
  • Tokenizers 0.19.1