Upload 7 files

README.md

@@ -0,0 +1,165 @@
+---
+language: en
+tags:
+- emoberta
+- roberta
+license: mit
+datasets:
+- MELD
+- IEMOCAP
+---
+
+Check https://github.com/tae898/erc for the details
+
+[Watch a demo video!](https://youtu.be/qbr7fNd6J28)
+
+# Emotion Recognition in Coversation (ERC)
+
+[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/emoberta-speaker-aware-emotion-recognition-in/emotion-recognition-in-conversation-on)](https://paperswithcode.com/sota/emotion-recognition-in-conversation-on?p=emoberta-speaker-aware-emotion-recognition-in)
+[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/emoberta-speaker-aware-emotion-recognition-in/emotion-recognition-in-conversation-on-meld)](https://paperswithcode.com/sota/emotion-recognition-in-conversation-on-meld?p=emoberta-speaker-aware-emotion-recognition-in)
+
+At the moment, we only use the text modality to correctly classify the emotion of the utterances.The experiments were carried out on two datasets (i.e. MELD and IEMOCAP)
+
+## Prerequisites
+
+1. An x86-64 Unix or Unix-like machine
+1. Python 3.8 or higher
+1. Running in a virtual environment (e.g., conda, virtualenv, etc.) is highly recommended so that you don't mess up with the system python.
+1. [`multimodal-datasets` repo](https://github.com/tae898/multimodal-datasets) (submodule)
+1. pip install -r requirements.txt
+
+## EmoBERTa training
+
+First configure the hyper parameters and the dataset in `train-erc-text.yaml` and then,
+In this directory run the below commands. I recommend you to run this in a virtualenv.
+
+```sh
+python train-erc-text.py
+```
+
+This will subsequently call `train-erc-text-hp.py` and `train-erc-text-full.py`.
+
+## Results on the test split (weighted f1 scores)
+
+| Model    |                                 |   MELD    |  IEMOCAP  |
+| -------- | ------------------------------- | :-------: | :-------: |
+| EmoBERTa | No past and future utterances   |   63.46   |   56.09   |
+|          | Only past utterances            |   64.55   | **68.57** |
+|          | Only future utterances          |   64.23   |   66.56   |
+|          | Both past and future utterances | **65.61** |   67.42   |
+|          | → *without speaker names*       |   65.07   |   64.02   |
+
+Above numbers are the mean values of five random seed runs.
+
+If you want to see more training test details, check out `./results/`
+
+If you want to download the trained checkpoints and stuff, then [here](https://surfdrive.surf.nl/files/index.php/s/khREwk4MUI7MSnO/download) is where you can download them. It's a pretty big zip file.
+
+## Deployment
+
+### Huggingface
+
+We have released our models on huggingface:
+
+- [emoberta-base](https://huggingface.co/tae898/emoberta-base)
+- [emoberta-large](https://huggingface.co/tae898/emoberta-large)
+
+They are based on [RoBERTa-base](https://huggingface.co/roberta-base) and [RoBERTa-large](https://huggingface.co/roberta-large), respectively. They were trained on [both MELD and IEMOCAP datasets](utterance-ordered-MELD_IEMOCAP.json). Our deployed models are neither speaker-aware nor take previous utterances into account, meaning that it only classifies one utterance at a time without the speaker information (e.g., "I love you").
+
+### Flask app
+
+You can either run the Flask RESTful server app as a docker container or just as a python script.
+
+1. Running the app as a docker container **(recommended)**.
+
+   There are four images. Take what you need:
+
+   - `docker run -it --rm -p 10006:10006 tae898/emoberta-base`
+   - `docker run -it --rm -p 10006:10006 --gpus all tae898/emoberta-base-cuda`
+   - `docker run -it --rm -p 10006:10006 tae898/emoberta-large`
+   - `docker run -it --rm -p 10006:10006 --gpus all tae898/emoberta-large-cuda`
+
+1. Running the app in your python environment:
+
+   This method is less recommended than the docker one.
+
+   Run `pip install -r requirements-deploy.txt` first.<br>
+   The [`app.py`](app.py) is a flask RESTful server. The usage is below:
+
+   ```console
+   app.py [-h] [--host HOST] [--port PORT] [--device DEVICE] [--model-type MODEL_TYPE]
+   ```
+
+   For example:
+
+   ```sh
+   python app.py --host 0.0.0.0 --port 10006 --device cpu --model-type emoberta-base
+   ```
+
+### Client
+
+Once the app is running, you can send a text to the server. First install the necessary packages: `pip install -r requirements-client.txt`, and the run the [client.py](client.py). The usage is as below:
+
+```console
+client.py [-h] [--url-emoberta URL_EMOBERTA] --text TEXT
+```
+
+For example:
+
+```sh
+python client.py --text "Emotion recognition is so cool\!"
+```
+
+will give you:
+
+```json
+{
+    "neutral": 0.0049800905,
+    "joy": 0.96399665,
+    "surprise": 0.018937444,
+    "anger": 0.0071516023,
+    "sadness": 0.002021492,
+    "disgust": 0.001495996,
+    "fear": 0.0014167271
+}
+```
+
+## Troubleshooting
+
+The best way to find and solve your problems is to see in the github issue tab. If you can't find what you want, feel free to raise an issue. We are pretty responsive.
+
+## Contributing
+
+Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are **greatly appreciated**.
+
+1. Fork the Project
+1. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)
+1. Run `make style && quality` in the root repo directory, to ensure code quality.
+1. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)
+1. Push to the Branch (`git push origin feature/AmazingFeature`)
+1. Open a Pull Request
+
+## Cite our work
+
+Check out the [paper](https://arxiv.org/abs/2108.12009).
+
+```bibtex
+@misc{kim2021emoberta,
+      title={EmoBERTa: Speaker-Aware Emotion Recognition in Conversation with RoBERTa}, 
+      author={Taewoon Kim and Piek Vossen},
+      year={2021},
+      eprint={2108.12009},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```
+
+[![DOI](https://zenodo.org/badge/328375452.svg)](https://zenodo.org/badge/latestdoi/328375452)<br>
+
+## Authors
+
+- [Taewoon Kim](https://taewoonkim.com/)
+
+## License
+
+[MIT](https://choosealicense.com/licenses/mit/)

config.json

@@ -0,0 +1,46 @@
+{
+  "_name_or_path": "/home/tk/repos/erc/emoberta-large",
+  "architectures": [
+    "RobertaForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 1024,
+  "id2label": {
+    "0": "neutral",
+    "1": "joy",
+    "2": "surprise",
+    "3": "anger",
+    "4": "sadness",
+    "5": "disgust",
+    "6": "fear"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "label2id": {
+    "anger": 3,
+    "disgust": 5,
+    "fear": 6,
+    "joy": 1,
+    "neutral": 0,
+    "sadness": 4,
+    "surprise": 2
+  },
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "roberta",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "problem_type": "single_label_classification",
+  "torch_dtype": "float32",
+  "transformers_version": "4.16.2",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 50265
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a8cfc5507bf55efc68b682d2b2258bec556e9689d40ed387bb54996f2bb73b0e
+size 135

tokenizer_config.json

@@ -0,0 +1,16 @@
+{
+    "unk_token": "<unk>",
+    "bos_token": "<s>",
+    "eos_token": "</s>",
+    "add_prefix_space": false,
+    "errors": "replace",
+    "sep_token": "</s>",
+    "cls_token": "<s>",
+    "pad_token": "<pad>",
+    "mask_token": "<mask>",
+    "trim_offsets": true,
+    "model_max_length": 512,
+    "special_tokens_map_file": null,
+    "name_or_path": "roberta-large",
+    "tokenizer_class": "RobertaTokenizer"
+}

trainer_state.json

@@ -0,0 +1,118 @@
+{
+  "best_metric": 0.6281057228463983,
+  "best_model_checkpoint": "results/MELD_IEMOCAP/roberta-large/SEEDS/2022-03-14-19-06-40-speaker_mode-None-num_past_utterances-0-num_future_utterances-0-batch_size-16-seed-42/checkpoint-7086",
+  "epoch": 6.0,
+  "global_step": 7086,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 1.0,
+      "learning_rate": 1.1135900346584852e-06,
+      "loss": 1.6941,
+      "step": 1181
+    },
+    {
+      "epoch": 1.0,
+      "eval_f1_macro": 0.07791319292694036,
+      "eval_f1_micro": 0.35635123614663256,
+      "eval_f1_weighted": 0.19072939518738496,
+      "eval_loss": 1.5153617858886719,
+      "eval_runtime": 3.1794,
+      "eval_samples_per_second": 737.872,
+      "eval_steps_per_second": 23.275,
+      "step": 1181
+    },
+    {
+      "epoch": 2.0,
+      "learning_rate": 2.2338198736122084e-06,
+      "loss": 1.298,
+      "step": 2362
+    },
+    {
+      "epoch": 2.0,
+      "eval_f1_macro": 0.39258179738738447,
+      "eval_f1_micro": 0.5882352941176471,
+      "eval_f1_weighted": 0.570696903628535,
+      "eval_loss": 1.1774364709854126,
+      "eval_runtime": 3.1793,
+      "eval_samples_per_second": 737.903,
+      "eval_steps_per_second": 23.276,
+      "step": 2362
+    },
+    {
+      "epoch": 3.0,
+      "learning_rate": 3.3540497125659314e-06,
+      "loss": 1.0976,
+      "step": 3543
+    },
+    {
+      "epoch": 3.0,
+      "eval_f1_macro": 0.4195774350293237,
+      "eval_f1_micro": 0.6052855924978687,
+      "eval_f1_weighted": 0.5932857871930991,
+      "eval_loss": 1.0991432666778564,
+      "eval_runtime": 3.1785,
+      "eval_samples_per_second": 738.086,
+      "eval_steps_per_second": 23.281,
+      "step": 3543
+    },
+    {
+      "epoch": 4.0,
+      "learning_rate": 3.0822920081965484e-06,
+      "loss": 0.9869,
+      "step": 4724
+    },
+    {
+      "epoch": 4.0,
+      "eval_f1_macro": 0.4634417760566037,
+      "eval_f1_micro": 0.6287297527706734,
+      "eval_f1_weighted": 0.6211850773676282,
+      "eval_loss": 1.0642313957214355,
+      "eval_runtime": 3.1809,
+      "eval_samples_per_second": 737.537,
+      "eval_steps_per_second": 23.264,
+      "step": 4724
+    },
+    {
+      "epoch": 5.0,
+      "learning_rate": 2.8022345484581173e-06,
+      "loss": 0.8638,
+      "step": 5905
+    },
+    {
+      "epoch": 5.0,
+      "eval_f1_macro": 0.4789393979707725,
+      "eval_f1_micro": 0.6172208013640239,
+      "eval_f1_weighted": 0.6157912564989868,
+      "eval_loss": 1.0899823904037476,
+      "eval_runtime": 3.182,
+      "eval_samples_per_second": 737.269,
+      "eval_steps_per_second": 23.256,
+      "step": 5905
+    },
+    {
+      "epoch": 6.0,
+      "learning_rate": 2.522414224587374e-06,
+      "loss": 0.772,
+      "step": 7086
+    },
+    {
+      "epoch": 6.0,
+      "eval_f1_macro": 0.4957254043928597,
+      "eval_f1_micro": 0.6295822676896846,
+      "eval_f1_weighted": 0.6281057228463983,
+      "eval_loss": 1.120431661605835,
+      "eval_runtime": 3.1786,
+      "eval_samples_per_second": 738.072,
+      "eval_steps_per_second": 23.281,
+      "step": 7086
+    }
+  ],
+  "max_steps": 17715,
+  "num_train_epochs": 15,
+  "total_flos": 8224038236774502.0,
+  "trial_name": null,
+  "trial_params": null
+}

val-results.json

@@ -0,0 +1,10 @@
+{
+    "eval_loss": 1.120431661605835,
+    "eval_f1_weighted": 0.6281057228463983,
+    "eval_f1_micro": 0.6295822676896846,
+    "eval_f1_macro": 0.4957254043928597,
+    "eval_runtime": 3.2141,
+    "eval_samples_per_second": 729.912,
+    "eval_steps_per_second": 23.024,
+    "epoch": 15.0
+}