Funnyworld1412's picture
Add SetFit ABSA model
9bf4223 verified
metadata
library_name: setfit
tags:
  - setfit
  - absa
  - sentence-transformers
  - text-classification
  - generated_from_setfit_trainer
base_model: sentence-transformers/bert-base-nli-mean-tokens
metrics:
  - accuracy
widget:
  - text: >-
      gamenya seru bagus paket:gamenya seru bagus paket worth it gak lag mudah
      mainnya tugas hadiah bagus modenya sayangnya game kadang ngebug gapapa
      kasih
  - text: >-
      tolong perbaiki analog nya pengaturan posisi:tolong perbaiki analog nya
      pengaturan posisi berpindah pindah
  - text: >-
      visualisasi bagus segi graphic:visualisasi bagus segi graphic bagus ya
      game cocok sih mantra nya banyakin contoh mantra penghilang
  - text: >-
      jaringan udah bagus game jaringan nya bagus:game nya udah bagus jaringan
      game nya bermasalah jaringan udah bagus game jaringan nya bagus mohon
      nambahin karakter
  - text: >-
      kali game stuk loading server pakai jaringan:game bagus cma kendala kali
      game stuk loading server pakai jaringan wifi masuk jaringan jaringan
      bermasalah main game online lancar game susah akses tolong diperbaiki
      supercell detik bermain coc lancar masuk kendala
pipeline_tag: text-classification
inference: false
model-index:
  - name: SetFit Polarity Model with sentence-transformers/bert-base-nli-mean-tokens
    results:
      - task:
          type: text-classification
          name: Text Classification
        dataset:
          name: Unknown
          type: unknown
          split: test
        metrics:
          - type: accuracy
            value: 0.8478260869565217
            name: Accuracy

SetFit Polarity Model with sentence-transformers/bert-base-nli-mean-tokens

This is a SetFit model that can be used for Aspect Based Sentiment Analysis (ABSA). This SetFit model uses sentence-transformers/bert-base-nli-mean-tokens as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification. In particular, this model is in charge of classifying aspect polarities.

The model has been trained using an efficient few-shot learning technique that involves:

  1. Fine-tuning a Sentence Transformer with contrastive learning.
  2. Training a classification head with features from the fine-tuned Sentence Transformer.

This model was trained within the context of a larger system for ABSA, which looks like so:

  1. Use a spaCy model to select possible aspect span candidates.
  2. Use a SetFit model to filter these possible aspect span candidates.
  3. Use this SetFit model to classify the filtered aspect span candidates.

Model Details

Model Description

Model Sources

Model Labels

Label Examples
negatif
  • 'seru tolong diperbaiki pencarian lawan bermain ketemu player:kapada supercell game nya bagus seru tolong diperbaiki pencarian lawan bermain ketemu player trophy mahkotanya jaraknya dapet berpengaruh peleton akun perbedaan level'
  • 'bugnya nakal banget y:bugnya nakal banget y coc cr aja sukanya ngebug pas match suka hitam match relog kalo udah relog lawan udah 1 2 mahkota kecewa sih bintang nya 1 aja bug nya diurus bintang lawannya kadang g setara levelnya dahlah gk suka banget kalo main 2 vs 2 temen suka banget afk coba fitur report'
  • 'kadang g setara levelnya dahlah gk suka:bugnya nakal banget y coc cr aja sukanya ngebug pas match suka hitam match relog kalo udah relog lawan udah 1 2 mahkota kecewa sih bintang nya 1 aja bug nya diurus bintang lawannya kadang g setara levelnya dahlah gk suka banget kalo main 2 vs 2 temen suka banget afk coba fitur report'
positif
  • 'kapada supercell game nya bagus seru:kapada supercell game nya bagus seru tolong diperbaiki pencarian lawan bermain ketemu player trophy mahkotanya jaraknya dapet berpengaruh peleton akun perbedaan level'
  • 'fairrrr mending uninstall gamenya maen game yg:overall gamenya bagus pencarian match dikasih musuh yg levelnya levelku yg pertandingan fair menganggu kenyamanan pemainnya kalo nyariin musuh gapapa nyarinya kasih yg fair levelnya gaush buru buru ngasih yg gak fairrrr pas arena 4 udh dikasih musuh yg pletonnya 2 yg level 11 gak fairrrr mending uninstall gamenya maen game yg yg org gak fairr'
  • 'gameplay menyenangkan pemain afk:gameplay menyenangkan pemain afk pertengahan menyerah 2vs2 mode mengganggu tolong tambahkan fitur lapor pemain'

Evaluation

Metrics

Label Accuracy
all 0.8478

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import AbsaModel

# Download from the 🤗 Hub
model = AbsaModel.from_pretrained(
    "Funnyworld1412/ABSA_bert-base_MiniLM-L6-aspect",
    "Funnyworld1412/ABSA_bert-base_MiniLM-L6-polarity",
)
# Run inference
preds = model("The food was great, but the venue is just way too busy.")

Training Details

Training Set Metrics

Training set Min Median Max
Word count 3 28.3626 83
Label Training Sample Count
negatif 738
positif 528

Training Hyperparameters

  • batch_size: (4, 4)
  • num_epochs: (1, 1)
  • max_steps: -1
  • sampling_strategy: oversampling
  • num_iterations: 5
  • body_learning_rate: (2e-05, 1e-05)
  • head_learning_rate: 0.01
  • loss: CosineSimilarityLoss
  • distance_metric: cosine_distance
  • margin: 0.25
  • end_to_end: False
  • use_amp: False
  • warmup_proportion: 0.1
  • seed: 42
  • eval_max_steps: -1
  • load_best_model_at_end: False

Training Results

Epoch Step Training Loss Validation Loss
0.0003 1 0.3075 -
0.0158 50 0.1854 -
0.0316 100 0.4431 -
0.0474 150 0.3251 -
0.0632 200 0.2486 -
0.0790 250 0.2371 -
0.0948 300 0.3149 -
0.1106 350 0.1397 -
0.1264 400 0.1131 -
0.1422 450 0.2388 -
0.1580 500 0.1256 -
0.1738 550 0.157 -
0.1896 600 0.3768 -
0.2054 650 0.022 -
0.2212 700 0.221 -
0.2370 750 0.122 -
0.2528 800 0.028 -
0.2686 850 0.102 -
0.2844 900 0.2231 -
0.3002 950 0.1853 -
0.3160 1000 0.2167 -
0.3318 1050 0.0054 -
0.3476 1100 0.027 -
0.3633 1150 0.0189 -
0.3791 1200 0.0033 -
0.3949 1250 0.2548 -
0.4107 1300 0.0043 -
0.4265 1350 0.0033 -
0.4423 1400 0.0012 -
0.4581 1450 0.1973 -
0.4739 1500 0.0006 -
0.4897 1550 0.001 -
0.5055 1600 0.0002 -
0.5213 1650 0.2304 -
0.5371 1700 0.0005 -
0.5529 1750 0.0025 -
0.5687 1800 0.0185 -
0.5845 1850 0.0023 -
0.6003 1900 0.185 -
0.6161 1950 0.0004 -
0.6319 2000 0.0003 -
0.6477 2050 0.0005 -
0.6635 2100 0.0126 -
0.6793 2150 0.0004 -
0.6951 2200 0.0103 -
0.7109 2250 0.0009 -
0.7267 2300 0.0019 -
0.7425 2350 0.0018 -
0.7583 2400 0.1837 -
0.7741 2450 0.002 -
0.7899 2500 0.0003 -
0.8057 2550 0.0006 -
0.8215 2600 0.2006 -
0.8373 2650 0.0003 -
0.8531 2700 0.0006 -
0.8689 2750 0.0003 -
0.8847 2800 0.0001 -
0.9005 2850 0.0002 -
0.9163 2900 0.0003 -
0.9321 2950 0.0002 -
0.9479 3000 0.0003 -
0.9637 3050 0.001 -
0.9795 3100 0.0002 -
0.9953 3150 0.0007 -
1.0 3165 - 0.2256

Framework Versions

  • Python: 3.10.13
  • SetFit: 1.0.3
  • Sentence Transformers: 3.0.1
  • spaCy: 3.7.5
  • Transformers: 4.36.2
  • PyTorch: 2.1.2
  • Datasets: 2.19.2
  • Tokenizers: 0.15.2

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}