gauneg
/

roberta-large-absa-ate-sentiment-lora-adapter

@@ -44,24 +44,67 @@ model = PeftModel.from_pretrained(base_model, 'gauneg/roberta-large-absa-ate-sen
 ```
 This model can be utilized in the following two methods:
-1. Making token level inference
-2. Using pipelines for end to end inference
 ## Making token level inference
 ```python
 # after loading base model and the adapter as shown in the previous snippet
 text_input = "Been here a few times and food has always been good but service really suffers when it gets crowded."
-tok_inputs = tokenizer(text_input, return_tensors="pt").to(device)
 y_pred = model(**tok_inputs) # predicting the logits
-y_pred_fin = y_pred.logits.argmax(dim=-1)[0] # selecting the most favoured labels for each token from the logits
-decoded_pred = [id2labels[logx.item()] for logx in y_pred_fin]
-tok_levl_pred = list(zip(tokenizer.convert_ids_to_tokens(tok_inputs['input_ids'][0]), decoded_pred))[1:-1]
 ```
 RESULTS in `tok_levl_pred` variable:
@@ -89,32 +132,78 @@ RESULTS in `tok_levl_pred` variable:
  ('.', 'O')]
 ```
-## Using end-to-end token classification pipeline
-```python
-# after loading base model and the adapter as shown in the previous snippet
-from transformers import pipeline
-ate_senti_pipeline = pipeline(task='ner',
-                  aggregation_strategy='simple',
-                  model=model,
-                  tokenizer=tokenizer)
-text_input = "Been here a few times and food has always been good but service really suffers when it gets crowded."
-ate_senti_pipeline(text_input)
-```
-OUTPUT
-```bash
-[{'entity_group': 'pos',
-  'score': 0.92310727,
-  'word': ' food',
-  'start': 26,
-  'end': 30},
- {'entity_group': 'neg',
-  'score': 0.90695626,
-  'word': ' service',
-  'start': 56,
-  'end': 63}]
-```

 ```
 This model can be utilized in the following two methods:
+1. Using pipelines for end to end inference
+2. Making token level inference
+## Using end-to-end token classification pipeline
+```python
+# after loading base model and the adapter as shown in the previous snippet
+from transformers import pipeline
+ate_senti_pipeline = pipeline(task='ner',
+                  aggregation_strategy='simple',
+                  model=model,
+                  tokenizer=tokenizer)
+text_input = "Been here a few times and food has always been good but service really suffers when it gets crowded."
+ate_senti_pipeline(text_input)
+```
+OUTPUT
+```bash
+[{'entity_group': 'pos',
+  'score': 0.92310727,
+  'word': ' food',
+  'start': 26,
+  'end': 30},
+ {'entity_group': 'neg',
+  'score': 0.90695626,
+  'word': ' service',
+  'start': 56,
+  'end': 63}]
+```
+# OR
 ## Making token level inference
 ```python
 # after loading base model and the adapter as shown in the previous snippet
+# making one prediction at a time (should be padded/batched and truncated for efficiency)
 text_input = "Been here a few times and food has always been good but service really suffers when it gets crowded."
+tok_inputs = tokenizer(text_input, return_tensors="pt")
 y_pred = model(**tok_inputs) # predicting the logits
+# since first and the last tokens are excluded (<s> and </s>) labels predicted against them
+# should be removed before decoding predicted labels
+y_pred_fin = y_pred.logits.argmax(dim=-1)[0][1:-1] # selecting the most favoured labels for each token from the logits
+decoded_pred = [id2lab[logx.item()] for logx in y_pred_fin]
+## displaying the input tokens with predictions and skipping <s> and </s> tokens at the beginning and the end respectively
+decoded_toks = tok_inputs['input_ids'][0][1:-1]
+tok_levl_pred = list(zip(tokenizer.convert_ids_to_tokens(decoded_toks), decoded_pred))
 ```
 RESULTS in `tok_levl_pred` variable:
  ('.', 'O')]
 ```
+# Evaluation on Benchmark Test Datasets
+The first evaluation is for token-extraction task without considering the polarity of the extracted tokens. The tokens expected to be extracted are aspect term tokens
+on which the sentiments have been expressed. (scores are expressed as micro-averages of B-I-O labels)
+# ATE (Aspect Term Extraction Only)
+| Test Dataset | Base Model | Fine-tuned Model | Precision | Recall | F1 Score |
+| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
+|hotel reviews (SemEval 2015)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|71.16|73.92|71.6|
+|hotel reviews (SemEval 2015)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|70.92|72.28|71.07|
+|hotel reviews (SemEval 2015)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|64.05|79.69|70.0|
+|hotel reviews (SemEval 2015)|FacebookAI/roberta-large|(this) [gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|66.29|72.78|68.92|
+| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
+|laptop reviews (SemEval 2014)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|70.58|61.52|64.21|
+|laptop reviews (SemEval 2014)|FacebookAI/roberta-large|(this) [gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|66.38|50.62|54.31|
+|laptop reviews (SemEval 2014)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|70.82|48.97|52.08|
+|laptop reviews (SemEval 2014)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|73.61|46.38|49.87|
+| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
+|MAMS-ATE (2019)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|81.07|79.66|80.35|
+|MAMS-ATE (2019)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|79.91|78.95|79.39|
+|MAMS-ATE (2019)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|74.46|84.5|78.75|
+|MAMS-ATE (2019)|FacebookAI/roberta-large|(this)[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|77.8|79.81|78.75|
+| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
+|restaurant reviews (SemEval 2014)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|88.59|87.0|87.45|
+|restaurant reviews (SemEval 2014)|FacebookAI/roberta-large|(this) [gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|92.26|82.95|86.57|
+|restaurant reviews (SemEval 2014)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|93.07|81.95|86.32|
+|restaurant reviews (SemEval 2014)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|92.94|81.71|86.01|
+| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
+|restaurant reviews (SemEval 2015)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|72.91|75.4|72.74|
+|restaurant reviews (SemEval 2015)|FacebookAI/roberta-large|(this) [gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|70.54|77.48|72.63|
+|restaurant reviews (SemEval 2015)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|68.32|79.84|72.28|
+|restaurant reviews (SemEval 2015)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|71.94|74.75|71.84|
+| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
+|restaurant reviews (SemEval 2016)|FacebookAI/roberta-large|(this) [gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|70.22|75.83|71.84|
+|restaurant reviews (SemEval 2016)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|71.54|73.38|71.2|
+|restaurant reviews (SemEval 2016)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|71.35|72.78|70.85|
+|restaurant reviews (SemEval 2016)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|66.68|77.97|70.79|
+# Aspect Sentiment Evaluation
+This evaluation considers token-extraction task with polarity of the extracted tokens. The tokens expected to be extracted are aspect term tokens
+on which the sentiments have been expressed along with the polarity of the sentiments. (scores are expressed as macro-averages)
+| Test Dataset | Base Model | Fine-tuned Model | Precision | Recall | F1 Score |
+| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
+|hotel reviews (SemEval 2015)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|51.92|65.55|54.94|
+|hotel reviews (SemEval 2015)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|54.62|53.65|54.08|
+|hotel reviews (SemEval 2015)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|55.43|56.53|54.03|
+|hotel reviews (SemEval 2015)|FacebookAI/roberta-large|(this) [gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|52.88|55.19|53.85|
+| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
+|laptop reviews (SemEval 2014)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|44.25|41.55|42.81|
+|laptop reviews (SemEval 2014)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|46.15|33.23|37.09|
+|laptop reviews (SemEval 2014)|FacebookAI/roberta-large|(this) [gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|41.7|34.38|36.93|
+|laptop reviews (SemEval 2014)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|44.98|31.87|35.67|
+| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
+|MAMS-ATE (2019)|FacebookAI/roberta-base|(this) [gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|72.06|72.98|72.49|
+|MAMS-ATE (2019)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|72.97|71.63|72.26|
+|MAMS-ATE (2019)|FacebookAI/roberta-large|(this) [gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|69.34|73.3|71.07|
+|MAMS-ATE (2019)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|65.74|75.11|69.77|
+| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
+|restaurant reviews (SemEval 2014)|FacebookAI/roberta-large|(this) [gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|61.15|58.46|59.74|
+|restaurant reviews (SemEval 2014)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|60.13|56.81|58.13|
+|restaurant reviews (SemEval 2014)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|56.79|59.3|57.93|
+|restaurant reviews (SemEval 2014)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|58.99|54.76|56.45|
+| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
+|restaurant reviews (SemEval 2015)|FacebookAI/roberta-large|(this) [gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|53.89|55.7|54.11|
+|restaurant reviews (SemEval 2015)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|54.36|55.38|53.6|
+|restaurant reviews (SemEval 2015)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|51.67|56.58|53.29|
+|restaurant reviews (SemEval 2015)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|54.55|53.68|53.12|
+| ------------ | ---------- | ---------------- | --------- | ------ | -------- |
+|restaurant reviews (SemEval 2016)|FacebookAI/roberta-large|(this) [gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|53.7|60.49|55.05|
+|restaurant reviews (SemEval 2016)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|52.31|54.58|52.33|
+|restaurant reviews (SemEval 2016)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|52.07|54.58|52.15|
+|restaurant reviews (SemEval 2016)|microsoft/deberta-v3-large|[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|49.07|56.5|51.25|