metadata

title: SemF1
tags:
  - evaluate
  - metric
sdk: gradio
sdk_version: 3.19.1
app_file: app.py
pinned: false
description: >-
  SEM-F1 metric leverages the pre-trained contextual embeddings and evaluates
  the model generated semantic overlap  summary with the reference overlap
  summary. It evaluates the semantic overlap summary at the sentence level and 
  computes precision, recall and F1 scores.

  Refer to the paper `SEM-F1: an Automatic Way for Semantic Evaluation of
  Multi-Narrative Overlap Summaries at Scale`  for more details.

Metric Card for SemF1

Metric Description

SEM-F1 metric leverages the pre-trained contextual embeddings and evaluates the model generated semantic overlap summary with the reference overlap summary. It evaluates the semantic overlap summary at the sentence level and computes precision, recall and F1 scores.

How to Use

SEM-F1 takes 2 mandatory arguments: predictions: (a list of system generated documents in the form of sentences i.e. List[List[str]]), references: (a list of ground-truth documents in the form of sentences i.e. List[List[str]])

from evaluate import load
predictions = [
    ["I go to School.", "You are stupid."],
    ["I love adventure sports."],
]
references = [
    ["I go to School.", "You are stupid."],
    ["I love adventure sports."],
]
metric = load("semf1")
results = metric.compute(predictions=predictions, references=references)

It also accepts multiple optional arguments: TODO: List optional arguments

Inputs

TODO: List all input arguments in the format below

input_field (type): Definition of input, with explanation if necessary. State any default value(s).

Output Values

BERTScore outputs a dictionary with the following values:

precision: The precision for each system summary, which ranges from 0.0 to 1.0.

recall: The recall for each system summary, which ranges from 0.0 to 1.0.

f1: The F1 score for each system summary, which ranges from 0.0 to 1.0.

Values from Popular Papers

Give examples, preferrably with links to leaderboards or publications, to papers that have reported this metric, along with the values they have reported.

Examples

Give code examples of the metric being used. Try to include examples that clear up any potential ambiguity left from the metric description above. If possible, provide a range of examples that show both typical and atypical results, as well as examples where a variety of input parameters are passed.

Limitations and Bias

Note any known limitations or biases that the metric has, with links and references if possible.

Citation

@inproceedings{bansal-etal-2022-sem,
    title = "{SEM}-F1: an Automatic Way for Semantic Evaluation of Multi-Narrative Overlap Summaries at Scale",
    author = "Bansal, Naman  and
      Akter, Mousumi  and
      Karmaker Santu, Shubhra Kanti",
    editor = "Goldberg, Yoav  and
      Kozareva, Zornitsa  and
      Zhang, Yue",
    booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, United Arab Emirates",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.emnlp-main.49",
    doi = "10.18653/v1/2022.emnlp-main.49",
    pages = "780--792",
    abstract = "Recent work has introduced an important yet relatively under-explored NLP task called Semantic Overlap Summarization (SOS) that entails generating a summary from multiple alternative narratives which conveys the common information provided by those narratives. Previous work also published a benchmark dataset for this task by collecting 2,925 alternative narrative pairs from the web and manually annotating 411 different reference summaries by engaging human annotators. In this paper, we exclusively focus on the automated evaluation of the SOS task using the benchmark dataset. More specifically, we first use the popular ROUGE metric from text-summarization literature and conduct a systematic study to evaluate the SOS task. Our experiments discover that ROUGE is not suitable for this novel task and therefore, we propose a new sentence-level precision-recall style automated evaluation metric, called SEM-F1 (Semantic F1). It is inspired by the benefits of the sentence-wise annotation technique using overlap labels reported by the previous work. Our experiments show that the proposed SEM-F1 metric yields a higher correlation with human judgment and higher inter-rater agreement compared to the ROUGE metric.",
}

Further References

TODO: Add links to the slides and video