metadata

tags:
  - generated_from_trainer
datasets:
  - sentiment_reduced
metrics:
  - accuracy
model-index:
  - name: EstBERT128_Sentiment
    results:
      - task:
          name: Text Classification
          type: text-classification
        dataset:
          name: Estonian Sentiment Corpus
          type: sentiment
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.747863233089447
language: et
license: cc-by-4.0
widget:
  - text: >-
      Enam kui kümme aastat tagasi tegutses huumorisaates «Wremja» inspektor
      Kukeke, kes kogu aeg vingus väikese palga pärast ja vaatas, mida saaks töö
      juurest koju tassida. Stsenaristid Andrus Kivirähk ja Mart Juur olid
      Kukekese isikusse kokku valanud kõik, mis 1990. aastate Eesti
      politseinikke halvast küljest iseloomustas.
    example_title: negative
  - text: >-
      Isiklikult kohtasin natukegi Kukekese moodi politseinikku viimati kaheksa
      aasta eest Lätis. Eranditult kõik viimase kümnendi kokkupuuted
      politseiametnikega on kinnitanud: vaatamata raskustele on Eesti riik
      suutnud korrakaitsjateks värvata inimesi, kes on arukad, kohusetundlikud,
      lugupidamist sisendavas füüsilises vormis ja hea väljendusoskusega.
    example_title: positive
  - text: >-
      Pisut retooriline küsimus, kelle või mille jaoks on Estonian Ai, nõuab
      taas vastust. Oleme jõudnud olukorda, kus vastus peaks olema juba
      konkreetne. Siinkohal tuleks hoiduda rahvusliku lennukompanii mõistest,
      mis pärineb ajast, kui lennundusäri oli peaaegu sajaprotsendiliselt riigi
      kontrolli all ning riigid ja nende grupeeringud reguleerisid äärmise
      põhjalikkusega lennundusturgu.
    example_title: neutral

EstBERT128_sentiment

This model is a fine-tuned version of tartuNLP/EstBERT on the reduced version of the Estonian Valence corpus, where the items with Mixed labels were removed. The data (containing Positive, Negative and Neutral labels) was split into 70/10/20 train/dev/test splits.

It achieves the following results on the developments split:

Loss: 2.2440
Accuracy: 0.7926

It achieves the following results on the test split:

Loss: 2.7633
Accuracy: 0.7479

How to use?

You can use this model with the Transformers pipeline for text classification.

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from transformers import pipeline

tokenizer = AutoTokenizer.from_pretrained("tartuNLP/EstBERT128_sentiment")
model = AutoModelForSequenceClassification.from_pretrained("tartuNLP/EstBERT128_sentiment")

nlp = pipeline("text-classification", model=model, tokenizer=tokenizer)
text = "Viimastel nädalatel on üha valjemaks muutunud hääled, mis läbisegi süüdistavad regionaalminister Madis Kallast röövretke korraldamises rikastesse valdadesse ja teisalt tegevusetuses."
result = nlp(text)

print(result)

[{'label': 'negatiivne', 'score': 0.9999992847442627}]

Model description

A single linear layer classifier is fit on top of the last layer [CLS] token representation of the EstBERT model. The model is fully fine-tuned during training.

Intended uses & limitations

This model is intended to be used as it is. We hope that it can prove to be useful to somebody but we do not guarantee that the model is useful for anything or that the predictions are accurate on new data.

Citation information

If you use this model, please cite:

@inproceedings{tanvir2021estbert,
  title={EstBERT: A Pretrained Language-Specific BERT for Estonian},
  author={Tanvir, Hasan and Kittask, Claudia and Eiche, Sandra and Sirts, Kairit},
  booktitle={Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)},
  pages={11--19},
  year={2021}
}

Training and evaluation data

The model was trained and evaluated on the sentiment categories of the Estonian Valence corpus. The data was split into train/dev/test parts with 70/10/20 proportions.

The Estonian Valence corpus has four sentiment labels:

positive
negative
neutral
mixed

Following Pajupuu et al., 2016, the items with mixed labels were removed. Thus, the model was trained and evaluated on the reduced version of the dataset containing only three labels (positive, negative and neutral).

Training procedure

The model was trained for maximu 100 epochs using early stopping procedure. After every epoch, the accuracy was calculated on the development set. If the development set accuracy did not improve for 20 epochs, the training was stopped.

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-05
train_batch_size: 16
eval_batch_size: 16
seed: 2
gradient_accumulation_steps: 4
total_train_batch_size: 64
optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-06
lr_scheduler_type: polynomial
num_epochs: 100
mixed_precision_training: Native AMP

Training results

The final model was taken after 44th epoch.

Training Loss	Epoch	Step	Validation Loss	Accuracy
0.836	1	38	0.6966	0.7216
0.5336	2	76	0.5948	0.7699
0.2913	3	114	0.7197	0.7358
0.1048	4	152	0.9570	0.7557
0.0424	5	190	1.2144	0.7528
0.0262	6	228	1.2675	0.7727
0.0169	7	266	1.4788	0.75
0.0048	8	304	1.5053	0.7699
0.0084	9	342	1.5368	0.7614
0.0087	10	380	1.6678	0.7699
0.0082	11	418	1.7598	0.7642
0.0104	12	456	1.6951	0.7528
0.0115	13	494	1.7123	0.7727
0.0111	14	532	1.7577	0.7528
0.0028	15	570	1.7383	0.7727
0.0032	16	608	2.0254	0.7727
0.0107	17	646	2.2123	0.7415
0.0056	18	684	1.9406	0.7614
0.0078	19	722	2.2002	0.7642
0.0041	20	760	2.0157	0.7670
0.0087	21	798	2.1228	0.7642
0.0113	22	836	2.3692	0.7727
0.0025	23	874	2.2211	0.75
0.0083	24	912	2.2120	0.7841
0.0104	25	950	2.1478	0.7614
0.0041	26	988	2.1118	0.7756
0.002	27	1026	1.9929	0.7699
0.001	28	1064	2.0295	0.7841
0.003	29	1102	2.3142	0.7699
0.006	30	1140	2.2957	0.7642
0.0005	31	1178	2.0661	0.7642
0.0007	32	1216	2.4220	0.7614
0.0088	33	1254	2.2842	0.7614
0.0	34	1292	2.4060	0.7585
0.0	35	1330	2.2088	0.7585
0.0	36	1368	2.2181	0.7614
0.0	37	1406	2.2560	0.7784
0.0	38	1444	2.4803	0.7585
0.0	39	1482	2.1163	0.7812
0.0087	40	1520	2.3410	0.75
0.0021	41	1558	2.3583	0.75
0.0054	42	1596	2.3546	0.7642
0.0051	43	1634	2.2295	0.7812
0.0	44	1672	2.2440	0.7926
0.0019	45	1710	2.3248	0.7784
0.0044	46	1748	2.3058	0.7841
0.0006	47	1786	2.3588	0.7784
0.0007	48	1824	2.6541	0.7670
0.0001	49	1862	2.4621	0.7614
0.0	50	1900	2.4696	0.7727
0.0	51	1938	2.4981	0.7670
0.0031	52	1976	2.6702	0.7670
0.0	53	2014	2.4448	0.7756
0.0	54	2052	2.4214	0.7756
0.0	55	2090	2.4308	0.7841
0.0001	56	2128	2.5869	0.7642
0.0007	57	2166	2.4803	0.7727
0.0	58	2204	2.4557	0.7784
0.0	59	2242	2.4702	0.7784
0.0	60	2280	2.5165	0.7784
0.0013	61	2318	2.6322	0.7727
0.0001	62	2356	2.6253	0.7756
0.0011	63	2394	2.6303	0.7841
0.0002	64	2432	2.5646	0.7614

Framework versions

Transformers 4.14.1
Pytorch 1.10.1+cu113
Datasets 1.16.1
Tokenizers 0.10.3

Contact

Kairit Sirts: kairit.sirts@ut.ee