mistral_mindmate

This model is a fine-tuned version of mistralai/Mistral-7B-Instruct-v0.2 on the generator dataset. It achieves the following results on the evaluation set:

• Loss: 1.7242

Model description

Fine tuned version of Mistral-7B

Intended uses & limitations

To create a mental health chatbot. Not intended to replace therapists. Just a small prototype

Training and evaluation data

Amod/mental_health_counseling_conversations

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• training_steps: 100

Training results

Training Loss	Epoch	Step	Validation Loss
1.934	0.1754	20	1.8779
1.8349	0.3509	40	1.7850
1.8512	0.5263	60	1.7561
1.7467	0.7018	80	1.7396
1.7246	0.8772	100	1.7242

Framework versions

• PEFT 0.12.0
• Transformers 4.44.0.dev0
• Pytorch 2.2.1+cu121
• Datasets 2.18.0
• Tokenizers 0.19.1