longt5-arvix-finetuned

This model is a fine-tuned version of google/long-t5-tglobal-base on the arxiv-summarization dataset. It achieves the following results on the evaluation set:

• Loss: 30.0035
• Rouge1: 0.0875
• Rouge2: 0.0242
• Rougel: 0.0708
• Rougelsum: 0.0709
• Gen Len: 19.0

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 4
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 10

Training results

Training Loss	Epoch	Step	Validation Loss	Rouge1	Rouge2	Rougel	Rougelsum	Gen Len
29.9697	1.0	1015	31.5396	0.089	0.0243	0.072	0.072	19.0
26.9874	2.0	2031	30.1941	0.0879	0.0242	0.0711	0.0712	19.0
26.6468	3.0	3046	30.0299	0.0877	0.0243	0.071	0.0712	19.0
26.4548	4.0	4062	30.0120	0.087	0.0239	0.0704	0.0705	19.0
26.5997	5.0	5077	30.0093	0.0875	0.0241	0.0708	0.0709	19.0
26.411	6.0	6093	30.0050	0.0875	0.024	0.0709	0.0709	19.0
26.5478	7.0	7108	30.0062	0.0876	0.0242	0.071	0.0711	19.0
26.4063	8.0	8124	30.0035	0.0876	0.0242	0.071	0.071	19.0
26.4737	9.0	9139	30.0070	0.0874	0.0241	0.0708	0.0709	19.0
26.5836	10.0	10150	30.0035	0.0875	0.0242	0.0708	0.0709	19.0

Framework versions

• PEFT 0.9.0
• Transformers 4.38.2
• Pytorch 2.2.1+cu121
• Datasets 2.18.0
• Tokenizers 0.15.2