longt5_xl_sfd_memsum_30

This model is a fine-tuned version of google/long-t5-tglobal-xl on the learn3r/summ_screen_memsum_oracle dataset. It achieves the following results on the evaluation set:

• Loss: 5.1322

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 32
• total_train_batch_size: 256
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• num_epochs: 30.0

Training results

Training Loss	Epoch	Step	Validation Loss
2.6697	0.97	14	2.4168
2.2272	1.95	28	2.2644
1.9024	2.99	43	2.2556
1.6554	3.97	57	2.4007
1.3619	4.94	71	2.4233
1.1577	5.98	86	2.6797
0.9584	6.96	100	2.8449
0.7197	8.0	115	3.0255
0.5756	8.97	129	3.1467
0.485	9.95	143	3.2976
0.4027	10.99	158	3.8111
0.2938	11.97	172	3.7330
0.2665	12.94	186	4.1417
0.2019	13.98	201	4.0316
0.1706	14.96	215	4.1357
0.1418	16.0	230	4.1022
0.1286	16.97	244	4.1198
0.1022	17.95	258	4.1862
0.1122	18.99	273	4.6386
0.093	19.97	287	4.6829
0.0783	20.94	301	4.6637
0.0698	21.98	316	4.7190
0.0688	22.96	330	5.0200
0.0633	24.0	345	4.7576
0.0609	24.97	359	4.7805
0.0553	25.95	373	4.7338
0.0503	26.99	388	5.1409
0.0471	27.97	402	5.1463
0.0472	28.94	416	5.1636
0.0376	29.22	420	5.1322

Framework versions

• Transformers 4.36.2
• Pytorch 2.1.2+cu121
• Datasets 2.16.1
• Tokenizers 0.15.0