t5-small-mlm-pubmed-15

This model is a fine-tuned version of t5-small on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.5389
• Rouge2 Precision: 0.7165
• Rouge2 Recall: 0.5375
• Rouge2 Fmeasure: 0.5981

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 40
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Rouge2 Precision	Rouge2 Recall	Rouge2 Fmeasure
1.1024	0.75	500	0.7890	0.6854	0.4813	0.5502
0.8788	1.51	1000	0.7176	0.6906	0.4989	0.5638
0.8086	2.26	1500	0.6830	0.6872	0.5052	0.5663
0.7818	3.02	2000	0.6650	0.6912	0.5104	0.5711
0.7466	3.77	2500	0.6458	0.6965	0.5167	0.5774
0.731	4.52	3000	0.6355	0.6955	0.5161	0.5763
0.7126	5.28	3500	0.6249	0.6924	0.517	0.576
0.6998	6.03	4000	0.6166	0.6995	0.5207	0.5809
0.6855	6.79	4500	0.6076	0.6981	0.5215	0.5813
0.676	7.54	5000	0.6015	0.7003	0.5242	0.5836
0.6688	8.3	5500	0.5962	0.7004	0.5235	0.583
0.6569	9.05	6000	0.5900	0.6997	0.5234	0.5827
0.6503	9.8	6500	0.5880	0.703	0.5257	0.5856
0.6455	10.56	7000	0.5818	0.7008	0.5259	0.5849
0.635	11.31	7500	0.5796	0.7017	0.5271	0.5861
0.6323	12.07	8000	0.5769	0.7053	0.5276	0.5877
0.6241	12.82	8500	0.5730	0.7011	0.5243	0.5838
0.6224	13.57	9000	0.5696	0.7046	0.5286	0.5879
0.6139	14.33	9500	0.5685	0.7047	0.5295	0.5886
0.6118	15.08	10000	0.5653	0.704	0.5297	0.5886
0.6089	15.84	10500	0.5633	0.703	0.5272	0.5865
0.598	16.59	11000	0.5613	0.7059	0.5293	0.5889
0.6003	17.35	11500	0.5602	0.7085	0.532	0.5918
0.5981	18.1	12000	0.5587	0.7106	0.5339	0.5938
0.5919	18.85	12500	0.5556	0.708	0.5319	0.5914
0.5897	19.61	13000	0.5556	0.7106	0.5327	0.5931
0.5899	20.36	13500	0.5526	0.7114	0.534	0.5939
0.5804	21.12	14000	0.5521	0.7105	0.5328	0.5928
0.5764	21.87	14500	0.5520	0.715	0.537	0.5976
0.5793	22.62	15000	0.5506	0.713	0.5346	0.5951
0.5796	23.38	15500	0.5492	0.7124	0.5352	0.5952
0.5672	24.13	16000	0.5482	0.7124	0.5346	0.5948
0.5737	24.89	16500	0.5470	0.7134	0.5352	0.5956
0.5685	25.64	17000	0.5463	0.7117	0.5346	0.5946
0.5658	26.4	17500	0.5457	0.7145	0.5359	0.5965
0.5657	27.15	18000	0.5447	0.7145	0.5367	0.597
0.5645	27.9	18500	0.5441	0.7141	0.5362	0.5964
0.565	28.66	19000	0.5436	0.7151	0.5367	0.5972
0.5579	29.41	19500	0.5426	0.7162	0.5378	0.5982
0.563	30.17	20000	0.5424	0.7155	0.5373	0.5977
0.556	30.92	20500	0.5418	0.7148	0.536	0.5966
0.5576	31.67	21000	0.5411	0.7141	0.5356	0.5961
0.5546	32.43	21500	0.5409	0.7149	0.5364	0.5967
0.556	33.18	22000	0.5405	0.7143	0.5356	0.596
0.5536	33.94	22500	0.5401	0.7165	0.5377	0.5982
0.5527	34.69	23000	0.5397	0.7188	0.5389	0.5999
0.5531	35.44	23500	0.5395	0.7172	0.538	0.5989
0.5508	36.2	24000	0.5392	0.7166	0.538	0.5985
0.5495	36.95	24500	0.5391	0.7176	0.5387	0.5993
0.5539	37.71	25000	0.5391	0.7169	0.5372	0.598
0.5452	38.46	25500	0.5390	0.7179	0.5384	0.5991
0.5513	39.22	26000	0.5390	0.717	0.5377	0.5984
0.5506	39.97	26500	0.5389	0.7165	0.5375	0.5981

Framework versions

• Transformers 4.12.5
• Pytorch 1.10.0+cu111
• Datasets 1.15.1
• Tokenizers 0.10.3