mistral7b_instruct_generation

This model is a fine-tuned version of mistralai/Mistral-7B-v0.1 on the generator dataset. It achieves the following results on the evaluation set:

• Loss: 1.8140

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• lr_scheduler_warmup_steps: 0.03
• training_steps: 500

Training results

Training Loss	Epoch	Step	Validation Loss
1.8937	0.0028	20	1.8344
1.8442	0.0056	40	1.8027
1.887	0.0084	60	1.7948
1.7711	0.0112	80	1.7884
1.7809	0.0140	100	1.7881
1.8632	0.0168	120	1.7957
1.8436	0.0196	140	1.8012
1.8959	0.0224	160	1.7860
1.8644	0.0252	180	1.7912
1.8285	0.0280	200	1.7884
1.8769	0.0307	220	1.7982
1.8726	0.0335	240	1.8046
1.8227	0.0363	260	1.7954
1.8554	0.0391	280	1.7927
1.9006	0.0419	300	1.7952
1.769	0.0447	320	1.8006
1.8014	0.0475	340	1.7895
1.8772	0.0503	360	1.7966
1.7403	0.0531	380	1.7988
1.8945	0.0559	400	1.7979
1.8937	0.0587	420	1.7987
1.9235	0.0615	440	1.8044
1.8496	0.0643	460	1.7951
1.8778	0.0671	480	1.7974
1.8681	0.0699	500	1.8140

Framework versions

• PEFT 0.11.1
• Transformers 4.41.1
• Pytorch 2.3.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1