DGurgurov/maltese-cn-lang-adapter

mt

This model is an adapter fine-tuned on top of of bert-base-multilingual-cased on the Maltese ConceptNet dataset. It achieves the following results on the evaluation set:

• Loss: 0.8117
• Accuracy: 0.8590

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• distributed_type: multi-GPU
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• training_steps: 50000

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
2.6689	1.04	500	2.1345	0.6677
2.1415	2.07	1000	1.8436	0.6926
1.9421	3.11	1500	1.7874	0.6907
1.7588	4.14	2000	1.7605	0.7013
1.6729	5.18	2500	1.7568	0.6957
1.596	6.21	3000	1.5006	0.7273
1.5778	7.25	3500	1.3924	0.7451
1.4821	8.28	4000	1.6097	0.7099
1.4183	9.32	4500	1.3552	0.7491
1.4197	10.35	5000	1.2847	0.7513
1.3156	11.39	5500	1.3173	0.7496
1.2882	12.42	6000	1.2817	0.7738
1.2692	13.46	6500	1.1892	0.7751
1.2368	14.49	7000	1.2363	0.7816
1.1975	15.53	7500	1.2442	0.7700
1.1907	16.56	8000	1.2569	0.7720
1.1231	17.6	8500	1.1386	0.7761
1.0873	18.63	9000	1.2105	0.7856
1.1242	19.67	9500	1.2142	0.7738
1.0367	20.7	10000	1.2121	0.7712
1.0869	21.74	10500	1.0782	0.7955
1.0353	22.77	11000	0.9918	0.8069
1.0324	23.81	11500	1.0908	0.7971
1.0145	24.84	12000	1.0945	0.7975
0.9951	25.88	12500	1.0005	0.8028
0.9483	26.92	13000	0.9638	0.8187
0.9304	27.95	13500	0.9761	0.8205
0.8835	28.99	14000	1.0620	0.8046
0.9097	30.02	14500	0.9138	0.8060
0.9293	31.06	15000	0.9180	0.8176
0.9043	32.09	15500	0.9215	0.8208
0.8581	33.13	16000	0.9625	0.8225
0.8638	34.16	16500	0.8586	0.8368
0.874	35.2	17000	1.0044	0.8135
0.8235	36.23	17500	0.9755	0.8184
0.8589	37.27	18000	0.9042	0.8292
0.8107	38.3	18500	0.8821	0.8272
0.8346	39.34	19000	0.9061	0.8248
0.8393	40.37	19500	0.9796	0.8235
0.789	41.41	20000	0.9015	0.8331
0.8121	42.44	20500	0.8589	0.8386
0.7709	43.48	21000	0.8836	0.8351
0.7922	44.51	21500	0.9524	0.8180
0.7457	45.55	22000	0.8350	0.8364
0.7386	46.58	22500	0.9025	0.8341
0.7515	47.62	23000	0.9092	0.8390
0.7324	48.65	23500	0.8322	0.8421
0.7314	49.69	24000	0.7968	0.8477
0.7442	50.72	24500	0.9305	0.8324
0.7074	51.76	25000	1.0011	0.8208
0.739	52.8	25500	0.8732	0.8331
0.7243	53.83	26000	0.7857	0.8480
0.6842	54.87	26500	0.7945	0.8377
0.6991	55.9	27000	0.9628	0.8275
0.6896	56.94	27500	0.8363	0.8410
0.6925	57.97	28000	0.8433	0.8392
0.7081	59.01	28500	1.0086	0.8223
0.6598	60.04	29000	0.9251	0.8333
0.6677	61.08	29500	0.8823	0.8437
0.695	62.11	30000	0.7751	0.8560
0.7108	63.15	30500	0.8452	0.8481
0.6721	64.18	31000	0.8560	0.8413
0.6571	65.22	31500	0.9800	0.8163
0.6891	66.25	32000	0.8106	0.8457
0.6541	67.29	32500	0.8197	0.8430
0.6559	68.32	33000	0.8678	0.8388
0.6554	69.36	33500	0.7396	0.8662
0.618	70.39	34000	0.8518	0.8376
0.6558	71.43	34500	0.7706	0.8409
0.6034	72.46	35000	0.7829	0.8518
0.6336	73.5	35500	0.7835	0.8591
0.6287	74.53	36000	0.7548	0.8575
0.6065	75.57	36500	0.8542	0.8508
0.6029	76.6	37000	0.8203	0.8405
0.6208	77.64	37500	0.7082	0.8661
0.64	78.67	38000	0.8505	0.8410
0.6144	79.71	38500	0.7246	0.8604
0.6507	80.75	39000	0.7150	0.8611
0.6177	81.78	39500	0.9332	0.84
0.6159	82.82	40000	0.6427	0.8733
0.5944	83.85	40500	0.7721	0.8411
0.6044	84.89	41000	0.8968	0.8449
0.6	85.92	41500	0.7673	0.8538
0.5899	86.96	42000	0.8039	0.8505
0.5812	87.99	42500	0.7467	0.8567
0.5977	89.03	43000	0.9534	0.8316
0.6019	90.06	43500	0.9170	0.8316
0.563	91.1	44000	0.7761	0.8569
0.6347	92.13	44500	0.7811	0.8577
0.5855	93.17	45000	0.7562	0.8606
0.6026	94.2	45500	0.7490	0.8636
0.5846	95.24	46000	0.7456	0.8487
0.5635	96.27	46500	0.8115	0.8495
0.5903	97.31	47000	0.8137	0.8448
0.576	98.34	47500	0.8441	0.8424
0.5745	99.38	48000	0.7266	0.8609
0.5915	100.41	48500	0.9169	0.8446
0.601	101.45	49000	0.7671	0.8576
0.5713	102.48	49500	0.7868	0.8487
0.5541	103.52	50000	0.7907	0.8569

Framework versions

• Transformers 4.35.2
• Pytorch 2.0.0
• Datasets 2.15.0
• Tokenizers 0.15.0