sade-adrien's picture
Update README.md
b40233b
metadata
license: apache-2.0
base_model: mistralai/Mistral-7B-Instruct-v0.1
tags:
  - generated_from_trainer
model-index:
  - name: Mistral-7B-Instruct-v0.1-LC-PI-.5
    results: []

Mistral-7B-Instruct-v0.1-LC-PI-.5

This model is a fine-tuned version of mistralai/Mistral-7B-Instruct-v0.1 on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 1.9295

Model description

This model is a fine-tuning of Mistral-7B-Instruct-v0.1. This FT was using a Position Interpolation factor of 0.5 (Linear RoPE scaling). Please note that the RoPE scaling factor should be determined by L/L' where L is the pre-training max context length and L' is the new max context length. In our case, we are just making experiments (and for us we would have had L/L' = 8096/7200 > 1 which did not require any PI scaling).

Intended uses & limitations

More information needed

Training and evaluation data

Data is a 9k sample from the RedPajama datset. The context is <=7200 with a decreasing exponential distribution of scale 1500.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 1e-05
  • train_batch_size: 1
  • eval_batch_size: 1
  • seed: 42
  • gradient_accumulation_steps: 32
  • total_train_batch_size: 32
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_steps: 20
  • training_steps: 300

Training results

Training Loss Epoch Step Validation Loss
2.163 0.18 50 2.0175
2.1576 0.36 100 1.9574
2.0073 0.55 150 1.9391
1.8824 0.73 200 1.9320
2.0718 0.91 250 1.9298
1.9498 1.09 300 1.9295

Framework versions

  • Transformers 4.34.1
  • Pytorch 2.0.0+cu117
  • Datasets 2.14.6
  • Tokenizers 0.14.1