vihealthbert-w_mlm-ViMedNLI

This model is a fine-tuned version of demdecuong/vihealthbert-base-word on the None dataset. It achieves the following results on the evaluation set:

• Loss: 1.1156
• Accuracy: 0.8341

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 19161
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• training_steps: 30000

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
5.5327	10.5263	1000	2.7528	0.5890
1.9051	21.0526	2000	1.4678	0.7783
1.1194	31.5789	3000	1.1543	0.8020
0.831	42.1053	4000	1.0972	0.8147
0.6805	52.6316	5000	0.9968	0.8256
0.5937	63.1579	6000	1.0310	0.8243
0.5258	73.6842	7000	1.1045	0.8151
0.4569	84.2105	8000	1.0393	0.8254
0.4007	94.7368	9000	1.0684	0.8217
0.3632	105.2632	10000	1.1223	0.8182
0.3343	115.7895	11000	1.1048	0.8230
0.2998	126.3158	12000	1.0996	0.8218
0.2817	136.8421	13000	1.0880	0.8320
0.2568	147.3684	14000	1.1189	0.8216
0.2396	157.8947	15000	1.1026	0.8267
0.219	168.4211	16000	1.1284	0.8241
0.2028	178.9474	17000	1.1205	0.8243
0.1927	189.4737	18000	1.1104	0.8313
0.1841	200.0	19000	1.0284	0.8348
0.1687	210.5263	20000	1.1662	0.8266
0.1627	221.0526	21000	1.1330	0.8278
0.1564	231.5789	22000	1.1413	0.8265
0.1483	242.1053	23000	1.1836	0.8246
0.1439	252.6316	24000	1.2169	0.8179
0.1396	263.1579	25000	1.1871	0.8266
0.1364	273.6842	26000	1.1696	0.8301
0.1314	284.2105	27000	1.1557	0.8324
0.1295	294.7368	28000	1.1712	0.8298
0.1296	305.2632	29000	1.1821	0.8273
0.1251	315.7895	30000	1.1567	0.8262

Framework versions

• Transformers 4.40.2
• Pytorch 2.2.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1