GaetanMichelet's picture
End of training
23639ab verified
metadata
base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
datasets:
  - GaetanMichelet/chat-60_ft_task-2
library_name: peft
license: llama3.1
tags:
  - alignment-handbook
  - trl
  - sft
  - generated_from_trainer
model-index:
  - name: Llama-31-8B_task-2_60-samples_config-2
    results: []

Llama-31-8B_task-2_60-samples_config-2

This model is a fine-tuned version of meta-llama/Meta-Llama-3.1-8B-Instruct on the GaetanMichelet/chat-60_ft_task-2 dataset. It achieves the following results on the evaluation set:

  • Loss: 0.7086

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0001
  • train_batch_size: 1
  • eval_batch_size: 1
  • seed: 42
  • distributed_type: multi-GPU
  • gradient_accumulation_steps: 16
  • total_train_batch_size: 16
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 50

Training results

Training Loss Epoch Step Validation Loss
1.0749 0.6957 2 1.0893
1.0585 1.7391 5 1.0147
0.935 2.7826 8 0.8862
0.8267 3.8261 11 0.8299
0.7982 4.8696 14 0.7848
0.7295 5.9130 17 0.7512
0.7036 6.9565 20 0.7342
0.6598 8.0 23 0.7226
0.6493 8.6957 25 0.7157
0.5819 9.7391 28 0.7086
0.5695 10.7826 31 0.7087
0.5157 11.8261 34 0.7257
0.4755 12.8696 37 0.7442
0.4184 13.9130 40 0.7554
0.3399 14.9565 43 0.8076
0.2958 16.0 46 0.8420
0.2339 16.6957 48 0.9179

Framework versions

  • PEFT 0.12.0
  • Transformers 4.44.0
  • Pytorch 2.1.2+cu121
  • Datasets 2.20.0
  • Tokenizers 0.19.1