Spaces:
Sleeping
title: SemF1
tags:
- evaluate
- metric
sdk: gradio
sdk_version: 3.19.1
app_file: app.py
pinned: false
description: >-
SEM-F1 metric leverages the pre-trained contextual embeddings and evaluates
the model generated semantic overlap summary with the reference overlap
summary. It evaluates the semantic overlap summary at the sentence level and
computes precision, recall and F1 scores.
Refer to the paper `SEM-F1: an Automatic Way for Semantic Evaluation of
Multi-Narrative Overlap Summaries at Scale` for more details.
authors:
- user: nbansal
Metric Card for Sem-F1
Metric Description
Sem-F1 metric leverages the pre-trained contextual embeddings and evaluates the model generated semantic overlap summary with the reference overlap summary. It evaluates the semantic overlap summary at the sentence level and computes precision, recall and F1 scores.
How to Use
Before using this metric, you need to install the dependencies -
pip install -U evaluate sentence-transformers scikit-learn nltk
Python Usage
from evaluate import load
predictions = [
"I go to School. You are stupid.",
"I love adventure sports.",
]
references = [
"I go to School. You are stupid.",
"I love outdoor sports.",
]
metric = load("nbansal/semf1")
results = metric.compute(predictions=predictions, references=references)
for score in results:
print(f"Precision: {score.precision}, Recall: {score.recall}, F1: {score.f1}")
Sem-F1 takes 2 mandatory arguments:
predictions
- List of predictions. Format varies based ontokenize_sentences
andmulti_references
flags.references
: List of references. Format varies based ontokenize_sentences
andmulti_references
flags.
Sem-F1 also accepts multiple optional arguments:
model_type (str)
: Model to use for encoding sentences. Options: ['pv1' (paraphrase-distilroberta-base-v1), 'stsb' (stsb-roberta-large), 'use' (Universal Sentence Encoder) (Default)]. Furthermore, you can use any model on Huggingface/SentenceTransformer that is supported by SentenceTransformer such asall-mpnet-base-v2
orroberta-base
.tokenize_sentences (bool)
: Flag to indicate whether to tokenize the sentences in the input documents. Default: True.multi_references (bool)
: Flag to indicate whether multiple references are provided. Default: False.gpu (Union[bool, str, int, List[Union[str, int]]])
: Whether to use GPU, CPU or multiple-processes for computation.batch_size (int)
: Batch size for encoding. (Default: 32).verbose (bool)
: Flag to indicate verbose output. (Default: False).aggregate (bool)
: Flag which is used to automatically compute the mean precision, recall and f1 scores. (Default: False).
Refer to the inputs descriptions for more detailed usage as follows:
import evaluate
metric = evaluate.load("nbansal/semf1")
print(metric.inputs_description)
Output Values
List of Scores
dataclass corresponding to each sample -
precision: float
: Precision score, which ranges from 0.0 to 1.0.recall: List[float]
: Recall score corresponding to each referencef1: float
: F1 score (between precision and average recall).
Extensions
Currently, we have only implemented the 3 encoders* that we experimented with in our
paper. Furthermore, you can use any model on
Huggingface/SentenceTransformer that is supported by SentenceTransformer such as all-mpnet-base-v2
or roberta-base
.
If you want to use your own encoder model, either make sure that is supported by SentenceTransformer
. Or if it's a
completely new architecture, it can easily with extended for more models by extending the Encoder
base class (Refer to
encoder_models.py
file).
*
In out paper, we used the Tensorflow version
of the USE model, however, in our current implementation, we used PyTorch version.
Citation
@inproceedings{bansal-etal-2022-sem,
title = "{SEM}-F1: an Automatic Way for Semantic Evaluation of Multi-Narrative Overlap Summaries at Scale",
author = "Bansal, Naman and
Akter, Mousumi and
Karmaker Santu, Shubhra Kanti",
editor = "Goldberg, Yoav and
Kozareva, Zornitsa and
Zhang, Yue",
booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
month = dec,
year = "2022",
address = "Abu Dhabi, United Arab Emirates",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.emnlp-main.49",
doi = "10.18653/v1/2022.emnlp-main.49",
pages = "780--792",
abstract = "Recent work has introduced an important yet relatively under-explored NLP task called Semantic Overlap Summarization (SOS) that entails generating a summary from multiple alternative narratives which conveys the common information provided by those narratives. Previous work also published a benchmark dataset for this task by collecting 2,925 alternative narrative pairs from the web and manually annotating 411 different reference summaries by engaging human annotators. In this paper, we exclusively focus on the automated evaluation of the SOS task using the benchmark dataset. More specifically, we first use the popular ROUGE metric from text-summarization literature and conduct a systematic study to evaluate the SOS task. Our experiments discover that ROUGE is not suitable for this novel task and therefore, we propose a new sentence-level precision-recall style automated evaluation metric, called SEM-F1 (Semantic F1). It is inspired by the benefits of the sentence-wise annotation technique using overlap labels reported by the previous work. Our experiments show that the proposed SEM-F1 metric yields a higher correlation with human judgment and higher inter-rater agreement compared to the ROUGE metric.",
}