Edit model card

phi-1_5-finetuned-qlora-cluster-gsm8k-v3-smallsubset

This model is a fine-tuned version of microsoft/phi-1_5 on the None dataset. It achieves the following results on the evaluation set:

  • Loss: 1.4985

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0002
  • train_batch_size: 8
  • eval_batch_size: 8
  • seed: 42
  • gradient_accumulation_steps: 4
  • total_train_batch_size: 32
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • num_epochs: 25
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss
No log 0.99 31 1.2173
No log 1.98 62 1.1653
No log 2.98 93 1.1535
No log 4.0 125 1.1498
No log 4.99 156 1.1562
No log 5.98 187 1.1682
1.0347 6.98 218 1.1832
1.0347 8.0 250 1.1934
1.0347 8.99 281 1.2183
1.0347 9.98 312 1.2468
1.0347 10.98 343 1.2760
1.0347 12.0 375 1.3096
0.7791 12.99 406 1.3348
0.7791 13.98 437 1.3695
0.7791 14.98 468 1.3935
0.7791 16.0 500 1.4104
0.7791 16.99 531 1.4235
0.7791 17.98 562 1.4546
0.7791 18.98 593 1.4709
0.5995 20.0 625 1.4790
0.5995 20.99 656 1.4889
0.5995 21.98 687 1.4942
0.5995 22.98 718 1.4954
0.5995 24.0 750 1.4982
0.5995 24.8 775 1.4985

Framework versions

  • PEFT 0.11.1
  • Transformers 4.37.2
  • Pytorch 2.3.0
  • Datasets 2.19.1
  • Tokenizers 0.15.1
Downloads last month
4
Safetensors
Model size
1.42B params
Tensor type
FP16
·
Inference API
Unable to determine this model’s pipeline type. Check the docs .

Model tree for nikhilajjarapu/phi-1_5-finetuned-qlora-cluster-gsm8k-v3-smallsubset

Base model

microsoft/phi-1_5
Adapter
(202)
this model