jbeno/electra-large-classifier-sentiment

ELECTRA Large Classifier for Sentiment Analysis

This is an ELECTRA large discriminator fine-tuned for sentiment analysis of reviews. It has a mean pooling layer and a classifier head (2 layers of 1024 dimension) with SwishGLU activation and dropout (0.3). It classifies text into three sentiment categories: 'negative' (0), 'neutral' (1), and 'positive' (2). It was fine-tuned on the Sentiment Merged dataset, which is a merge of Stanford Sentiment Treebank (SST-3), and DynaSent Rounds 1 and 2.

Labels

The model predicts the following labels:

• 0: negative
• 1: neutral
• 2: positive

How to Use

Install package

This model requires the classes in electra_classifier.py. You can download the file, or you can install the package from PyPI.

pip install electra-classifier

Load classes and model

# Install the package in a notebook
!pip install electra-classifier

# Import libraries
import torch
from transformers import AutoTokenizer
from electra_classifier import ElectraClassifier

# Load tokenizer and model
model_name = "jbeno/electra-large-classifier-sentiment"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = ElectraClassifier.from_pretrained(model_name)

# Set model to evaluation mode
model.eval()

# Run inference
text = "I love this restaurant!"
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    logits = model(**inputs)
    predicted_class_id = torch.argmax(logits, dim=1).item()
    predicted_label = model.config.id2label[predicted_class_id]
    print(f"Predicted label: {predicted_label}")

Requirements

• Python 3.7+
• PyTorch
• Transformers
• electra-classifier - Install with pip, or download electra_classifier.py

Training Details

Dataset

The model was trained on the Sentiment Merged dataset, which is a mix of Stanford Sentiment Treebank (SST-3), DynaSent Round 1, and DynaSent Round 2.

Code

The code used to train the model can be found on GitHub:

• jbeno/sentiment
• jbeno/electra-classifier

Research Paper

The research paper can be found here: ELECTRA and GPT-4o: Cost-Effective Partners for Sentiment Analysis

Performance Summary

• Merged Dataset
  • Macro Average F1: 82.36
  • Accuracy: 82.96
• DynaSent R1
  • Macro Average F1: 85.91
  • Accuracy: 85.83
• DynaSent R2
  • Macro Average F1: 76.29
  • Accuracy: 76.53
• SST-3
  • Macro Average F1: 70.90
  • Accuracy: 80.36

Model Architecture

• Base Model: ELECTRA large discriminator (google/electra-large-discriminator)
• Pooling Layer: Custom pooling layer supporting 'cls', 'mean', and 'max' pooling types.
• Classifier: Custom classifier with configurable hidden dimensions, number of layers, and dropout rate.
  • Activation Function: Custom SwishGLU activation function.

ElectraClassifier(
  (electra): ElectraModel(
    (embeddings): ElectraEmbeddings(
      (word_embeddings): Embedding(30522, 1024, padding_idx=0)
      (position_embeddings): Embedding(512, 1024)
      (token_type_embeddings): Embedding(2, 1024)
      (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)
      (dropout): Dropout(p=0.1, inplace=False)
    )
    (encoder): ElectraEncoder(
      (layer): ModuleList(
        (0-23): 24 x ElectraLayer(
          (attention): ElectraAttention(
            (self): ElectraSelfAttention(
              (query): Linear(in_features=1024, out_features=1024, bias=True)
              (key): Linear(in_features=1024, out_features=1024, bias=True)
              (value): Linear(in_features=1024, out_features=1024, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): ElectraSelfOutput(
              (dense): Linear(in_features=1024, out_features=1024, bias=True)
              (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): ElectraIntermediate(
            (dense): Linear(in_features=1024, out_features=4096, bias=True)
            (intermediate_act_fn): GELUActivation()
          )
          (output): ElectraOutput(
            (dense): Linear(in_features=4096, out_features=1024, bias=True)
            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
      )
    )
  )
  (custom_pooling): PoolingLayer()
  (classifier): Classifier(
    (layers): Sequential(
      (0): Linear(in_features=1024, out_features=1024, bias=True)
      (1): SwishGLU(
        (projection): Linear(in_features=1024, out_features=2048, bias=True)
        (activation): SiLU()
      )
      (2): Dropout(p=0.3, inplace=False)
      (3): Linear(in_features=1024, out_features=1024, bias=True)
      (4): SwishGLU(
        (projection): Linear(in_features=1024, out_features=2048, bias=True)
        (activation): SiLU()
      )
      (5): Dropout(p=0.3, inplace=False)
      (6): Linear(in_features=1024, out_features=3, bias=True)
    )
  )
)

Custom Model Components

SwishGLU Activation Function

The SwishGLU activation function combines the Swish activation with a Gated Linear Unit (GLU). It enhances the model's ability to capture complex patterns in the data.

class SwishGLU(nn.Module):
    def __init__(self, input_dim: int, output_dim: int):
        super(SwishGLU, self).__init__()
        self.projection = nn.Linear(input_dim, 2 * output_dim)
        self.activation = nn.SiLU()

    def forward(self, x):
        x_proj_gate = self.projection(x)
        projected, gate = x_proj_gate.tensor_split(2, dim=-1)
        return projected * self.activation(gate)

PoolingLayer

The PoolingLayer class allows you to choose between different pooling strategies:

• cls: Uses the representation of the [CLS] token.
• mean: Calculates the mean of the token embeddings.
• max: Takes the maximum value across token embeddings.

'mean' pooling was used in the fine-tuned model.

class PoolingLayer(nn.Module):
    def __init__(self, pooling_type='cls'):
        super().__init__()
        self.pooling_type = pooling_type

    def forward(self, last_hidden_state, attention_mask):
        if self.pooling_type == 'cls':
            return last_hidden_state[:, 0, :]
        elif self.pooling_type == 'mean':
            return (last_hidden_state * attention_mask.unsqueeze(-1)).sum(1) / attention_mask.sum(-1).unsqueeze(-1)
        elif self.pooling_type == 'max':
            return torch.max(last_hidden_state * attention_mask.unsqueeze(-1), dim=1)[0]
        else:
            raise ValueError(f"Unknown pooling method: {self.pooling_type}")

Classifier

The Classifier class is a customizable feed-forward neural network used for the final classification.

The fine-tuned model had:

• input_dim: 1024
• num_layers: 2
• hidden_dim: 1024
• hidden_activation: SwishGLU
• dropout_rate: 0.3
• n_classes: 3

class Classifier(nn.Module):
    def __init__(self, input_dim, hidden_dim, hidden_activation, num_layers, n_classes, dropout_rate=0.0):
        super().__init__()
        layers = []
        layers.append(nn.Linear(input_dim, hidden_dim))
        layers.append(hidden_activation)
        if dropout_rate > 0:
            layers.append(nn.Dropout(dropout_rate))

        for _ in range(num_layers - 1):
            layers.append(nn.Linear(hidden_dim, hidden_dim))
            layers.append(hidden_activation)
            if dropout_rate > 0:
                layers.append(nn.Dropout(dropout_rate))

        layers.append(nn.Linear(hidden_dim, n_classes))
        self.layers = nn.Sequential(*layers)

Model Configuration

The model's configuration (config.json) includes custom parameters:

• hidden_dim: Size of the hidden layers in the classifier.
• hidden_activation: Activation function used in the classifier ('SwishGLU').
• num_layers: Number of layers in the classifier.
• dropout_rate: Dropout rate used in the classifier.
• pooling: Pooling strategy used ('mean').

Performance by Dataset

Merged Dataset

Merged Dataset Classification Report

              precision    recall  f1-score   support

    negative   0.858503  0.843537  0.850954      2352
     neutral   0.747684  0.750137  0.748908      1829
    positive   0.864513  0.877395  0.870906      2349

    accuracy                       0.829556      6530
   macro avg   0.823567  0.823690  0.823590      6530
weighted avg   0.829626  0.829556  0.829549      6530

ROC AUC: 0.947247

Predicted  negative  neutral  positive
Actual                                
negative       1984      256       112
neutral         246     1372       211
positive         81      207      2061

Macro F1 Score: 0.82

DynaSent Round 1

DynaSent Round 1 Classification Report

              precision    recall  f1-score   support

    negative   0.913204  0.824167  0.866404      1200
     neutral   0.779433  0.915833  0.842146      1200
    positive   0.905149  0.835000  0.868661      1200

    accuracy                       0.858333      3600
   macro avg   0.865929  0.858333  0.859070      3600
weighted avg   0.865929  0.858333  0.859070      3600

ROC AUC: 0.963133

Predicted  negative  neutral  positive
Actual                                
negative        989      156        55
neutral          51     1099        50
positive         43      155      1002

Macro F1 Score: 0.86

DynaSent Round 2

DynaSent Round 2 Classification Report

              precision    recall  f1-score   support

    negative   0.764706  0.812500  0.787879       240
     neutral   0.814815  0.641667  0.717949       240
    positive   0.731884  0.841667  0.782946       240

    accuracy                       0.765278       720
   macro avg   0.770468  0.765278  0.762924       720
weighted avg   0.770468  0.765278  0.762924       720

ROC AUC: 0.927688

Predicted  negative  neutral  positive
Actual                                
negative        195       19        26
neutral          38      154        48
positive         22       16       202

Macro F1 Score: 0.76

Stanford Sentiment Treebank (SST-3)

SST-3 Classification Report

              precision    recall  f1-score   support

    negative   0.822199  0.877193  0.848806       912
     neutral   0.504237  0.305913  0.380800       389
    positive   0.856144  0.942794  0.897382       909

    accuracy                       0.803620      2210
   macro avg   0.727527  0.708633  0.708996      2210
weighted avg   0.780194  0.803620  0.786409      2210

ROC AUC: 0.904787

Predicted  negative  neutral  positive
Actual                                
negative        800       81        31
neutral         157      119       113
positive         16       36       857

Macro F1 Score: 0.71

License

This model is licensed under the MIT License.

Citation

If you use this model in your work, please consider citing it:

@misc{beno-2024-electra_base_classifier_sentiment,
  title={Electra Large Classifier for Sentiment Analysis},
  author={Jim Beno},
  year={2024},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/jbeno/electra-large-classifier-sentiment}},
}

Contact

For questions or comments, please open an issue on the repository or contact Jim Beno.

Acknowledgments

• The Hugging Face Transformers library for providing powerful tools for model development.
• The creators of the ELECTRA model for their foundational work.
• The authors of the datasets used: Stanford Sentiment Treebank, DynaSent.
• Stanford Engineering CGOE, Chris Potts, and the Course Facilitators of XCS224U