base_model: openaccess-ai-collective/tiny-mistral
library_name: peft
tags:
- generated_from_trainer
- fine-tuning
- text-generation
model-index:
- name: tiny-mistral-alpaca-finance
results: []
datasets:
- gbharti/finance-alpaca
Tiny Mistral fine-tuned on finance dataset
This model is a fine-tuned version of the openaccess-ai-collective/tiny-mistral language model.
It has been fine-tuned on a specialized finance dataset using Parameter-Efficient Fine-Tuning (PEFT) with Low-Rank Adaptation (LoRA).
The model is designed to generate responses based on financial instructions and contexts.
Intended uses & limitations
This model is intended for text generation tasks specifically related to financial instructions and contexts. It can be used for generating responses based on given financial prompts.
Limitations:
- The model may not perform well on financial topics not covered in the training data.
- The quality of responses may vary depending on the specificity and complexity of the financial queries.
- The model may generate responses that are not factually accurate or may include biases present in the training data.
Training and evaluation data
The model was fine-tuned on the gbharti/finance-alpaca dataset, which includes financial instructions and outputs.
The dataset was processed to format instructions with or without additional context.
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 4
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 8
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 100
- num_epochs: 2
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.3155 | 0.2580 | 500 | 1.3207 |
| 1.1306 | 0.5160 | 1000 | 1.1318 |
| 0.9935 | 0.7739 | 1500 | 0.9970 |
| 0.7188 | 1.0319 | 2000 | 0.8934 |
| 0.6962 | 1.2899 | 2500 | 0.8238 |
| 0.6427 | 1.5479 | 3000 | 0.7610 |
| 0.6014 | 1.8059 | 3500 | 0.7193 |
Framework versions
- PEFT 0.12.0
- Transformers 4.42.4
- Pytorch 2.3.1+cu121
- Datasets 2.20.0
- Tokenizers 0.19.1