llama2_instruct_generation

This model is a fine-tuned version of NousResearch/Llama-2-7b-hf on the generator dataset. It achieves the following results on the evaluation set:

• Loss: 1.6728

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• lr_scheduler_warmup_steps: 0.03
• training_steps: 500

Training results

Training Loss	Epoch	Step	Validation Loss
1.8965	0.0027	20	1.8107
1.8264	0.0054	40	1.7808
1.9263	0.0081	60	1.7669
1.8825	0.0108	80	1.7538
1.8391	0.0135	100	1.7324
1.8253	0.0163	120	1.7068
1.9297	0.0190	140	1.7029
1.8338	0.0217	160	1.6985
1.7641	0.0244	180	1.6939
1.7714	0.0271	200	1.6916
1.715	0.0298	220	1.6886
1.7786	0.0325	240	1.6866
1.716	0.0352	260	1.6860
1.899	0.0379	280	1.6857
1.8455	0.0406	300	1.6851
1.6802	0.0433	320	1.6836
1.7304	0.0461	340	1.6818
1.8154	0.0488	360	1.6793
1.8813	0.0515	380	1.6793
1.8423	0.0542	400	1.6789
1.7785	0.0569	420	1.6767
1.8999	0.0596	440	1.6763
1.7594	0.0623	460	1.6752
1.7182	0.0650	480	1.6745
1.8259	0.0677	500	1.6728

Framework versions

• PEFT 0.11.1
• Transformers 4.41.1
• Pytorch 2.3.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1