Add necessary implementation

README.md

@@ -5,7 +5,7 @@ datasets:
 tags:
 - evaluate
 - metric
-description: "TODO: add a description here"
+description: "Word Error Rate (WER) metric with detailed error analysis capabilities for speech recognition evaluation"
 sdk: gradio
 sdk_version: 3.19.1
 app_file: app.py
@@ -14,37 +14,79 @@ pinned: false
 
 # Metric Card for ArgWER
 
-***Module Card Instructions:*** *Fill out the following subsections. Feel free to take a look at existing metric cards if you'd like examples.*
-
 ## Metric Description
-*Give a brief overview of this metric, including what task(s) it is usually used for, if any.*
+ArgWER is an enhanced version of the Word Error Rate (WER) metric used for evaluating speech recognition systems. While it calculates the standard WER score, it also provides detailed information about different types of errors (insertions, deletions, and substitutions) when requested. This makes it particularly useful for detailed analysis of speech recognition system performance.
 
 ## How to Use
-*Give general statement of how to use the metric*
+The metric can be loaded and used through the `evaluate` library:
 
-*Provide simplest possible example for using the metric*
+```python
+import evaluate
+wer = evaluate.load("EduardoPacheco/argwer")
+predictions = ["this is the prediction", "there is an other sample"]
+references = ["this is the reference", "there is another one"]
+wer_score = wer.compute(predictions=predictions, references=references)
+```
 
 ### Inputs
-*List all input arguments in the format below*
-- **input_field** *(type): Definition of input, with explanation if necessary. State any default value(s).*
+- **predictions** *(List[str])*: List of transcriptions to score from the speech recognition system.
+- **references** *(List[str])*: List of reference transcriptions for each speech input.
+- **detailed** *(bool, optional)*: Whether to return detailed error analysis. Defaults to False.
 
 ### Output Values
+The metric returns either a float value representing the WER score, or when `detailed=True`, a dictionary containing:
+- `wer`: Overall word error rate
+- `substitution_rate`: Rate of word substitutions
+- `deletion_rate`: Rate of word deletions
+- `insertion_rate`: Rate of word insertions
+- `num_substitutions`: Absolute number of substitutions
+- `num_deletions`: Absolute number of deletions
+- `num_insertions`: Absolute number of insertions
+- `num_hits`: Number of correct words
 
-*Explain what this metric outputs and provide an example of what the metric output looks like. Modules should return a dictionary with one or multiple key-value pairs, e.g. {"bleu" : 6.02}*
-
-*State the range of possible values that the metric's output can take, as well as what in that range is considered good. For example: "This metric can take on any value between 0 and 100, inclusive. Higher scores are better."*
+The WER score ranges from 0 to infinity, where:
+- 0 represents perfect transcription
+- Lower scores are better
+- Scores above 1 are possible due to insertions
 
 #### Values from Popular Papers
-*Give examples, preferrably with links to leaderboards or publications, to papers that have reported this metric, along with the values they have reported.*
+Word Error Rate is a standard metric in speech recognition. For example:
+- Modern speech recognition systems typically achieve WER scores between 0.02 (2%) to 0.15 (15%) on clean speech.
+- The exact values vary significantly based on factors like audio quality, accent, and background noise.
 
 ### Examples
-*Give code examples of the metric being used. Try to include examples that clear up any potential ambiguity left from the metric description above. If possible, provide a range of examples that show both typical and atypical results, as well as examples where a variety of input parameters are passed.*
+Basic usage:
+```python
+predictions = ["this is the prediction", "there is an other sample"]
+references = ["this is the reference", "there is another one"]
+wer = evaluate.load("EduardoPacheco/argwer")
+
+# Basic WER score
+wer_score = wer.compute(predictions=predictions, references=references)
+# Returns: 0.5
+
+# Detailed analysis
+detailed_scores = wer.compute(predictions=predictions, references=references, detailed=True)
+# Returns dictionary with detailed error analysis
+```
 
 ## Limitations and Bias
-*Note any known limitations or biases that the metric has, with links and references if possible.*
+- The metric treats all words equally, regardless of their importance in the sentence
+- It doesn't account for semantic similarity (e.g., synonyms are counted as errors)
+- The metric is sensitive to word order, which might not always reflect the actual quality of the transcription
+- Punctuation and capitalization can affect the scores if not properly normalized
 
 ## Citation
-*Cite the source where this metric was introduced.*
+```bibtex
+@inproceedings{inproceedings,
+    author = {Morris, Andrew and Maier, Viktoria and Green, Phil},
+    year = {2004},
+    month = {01},
+    pages = {},
+    title = {From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition.}
+}
+```
 
 ## Further References
-*Add any useful further references.*
+- [Word Error Rate on Wikipedia](https://en.wikipedia.org/wiki/Word_error_rate)
+- [JiWER Library](https://github.com/jitsi/jiwer/) - The underlying implementation used by this metric

argwer.py

@@ -11,85 +11,117 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-"""TODO: Add a description here."""
+"""This is the same as WER, but it also returns detailed information about the erros (insertions, deletions, substitutions)"""
 
 import evaluate
 import datasets
+from jiwer import compute_measures
 
 
-# TODO: Add BibTeX citation
 _CITATION = """\
-@InProceedings{huggingface:module,
-title = {A great new module},
-authors={huggingface, Inc.},
-year={2020}
+@inproceedings{inproceedings,
+    author = {Morris, Andrew and Maier, Viktoria and Green, Phil},
+    year = {2004},
+    month = {01},
+    pages = {},
+    title = {From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition.}
 }
 """
 
-# TODO: Add description of the module here
 _DESCRIPTION = """\
-This new module is designed to solve this great ML task and is crafted with a lot of care.
-"""
+Word error rate (WER) is a common metric of the performance of an automatic speech recognition system.
+
+The general difficulty of measuring performance lies in the fact that the recognized word sequence can have a different length from the reference word sequence (supposedly the correct one). The WER is derived from the Levenshtein distance, working at the word level instead of the phoneme level. The WER is a valuable tool for comparing different systems as well as for evaluating improvements within one system. This kind of measurement, however, provides no details on the nature of translation errors and further work is therefore required to identify the main source(s) of error and to focus any research effort.
+
+This problem is solved by first aligning the recognized word sequence with the reference (spoken) word sequence using dynamic string alignment. Examination of this issue is seen through a theory called the power law that states the correlation between perplexity and word error rate.
+
+Word error rate can then be computed as:
 
+WER = (S + D + I) / N = (S + D + I) / (S + D + C)
+
+where
+
+S is the number of substitutions,
+D is the number of deletions,
+I is the number of insertions,
+C is the number of correct words,
+N is the number of words in the reference (N=S+D+C).
+
+This value indicates the average number of errors per reference word. The lower the value, the better the
+performance of the ASR system with a WER of 0 being a perfect score.
+"""
 
-# TODO: Add description of the arguments of the module here
 _KWARGS_DESCRIPTION = """
-Calculates how good are predictions given some references, using certain scores
+Compute WER score of transcribed segments against references.
+
 Args:
-    predictions: list of predictions to score. Each predictions
-        should be a string with tokens separated by spaces.
-    references: list of reference for each prediction. Each
-        reference should be a string with tokens separated by spaces.
+    references: List of references for each speech input.
+    predictions: List of transcriptions to score.
+    detailed (bool, default=False): Whether to also return normalized substitutions, deletions and insertions.
+
 Returns:
-    accuracy: description of the first score,
-    another_score: description of the second score,
+    (float): the word error rate
+
 Examples:
-    Examples should be written in doctest format, and should illustrate how
-    to use the function.
 
-    >>> my_new_module = evaluate.load("my_new_module")
-    >>> results = my_new_module.compute(references=[0, 1], predictions=[0, 1])
-    >>> print(results)
-    {'accuracy': 1.0}
+    >>> predictions = ["this is the prediction", "there is an other sample"]
+    >>> references = ["this is the reference", "there is another one"]
+    >>> wer = evaluate.load("EduardoPacheco/argwer")
+    >>> wer_score = wer.compute(predictions=predictions, references=references)
+    >>> print(wer_score)
+    0.5
 """
 
-# TODO: Define external resources urls if needed
-BAD_WORDS_URL = "http://url/to/external/resource/bad_words.txt"
-
-
 @evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class ArgWER(evaluate.Metric):
     """TODO: Short description of my evaluation module."""
 
     def _info(self):
-        # TODO: Specifies the evaluate.EvaluationModuleInfo object
         return evaluate.MetricInfo(
-            # This is the description that will appear on the modules page.
-            module_type="metric",
             description=_DESCRIPTION,
             citation=_CITATION,
             inputs_description=_KWARGS_DESCRIPTION,
-            # This defines the format of each prediction and reference
-            features=datasets.Features({
-                'predictions': datasets.Value('int64'),
-                'references': datasets.Value('int64'),
-            }),
-            # Homepage of the module for documentation
-            homepage="http://module.homepage",
-            # Additional links to the codebase or references
-            codebase_urls=["http://github.com/path/to/codebase/of/new_module"],
-            reference_urls=["http://path.to.reference.url/new_module"]
+            features=datasets.Features(
+                {
+                    "predictions": datasets.Value("string", id="sequence"),
+                    "references": datasets.Value("string", id="sequence"),
+                }
+            ),
+            codebase_urls=["https://github.com/jitsi/jiwer/"],
+            reference_urls=[
+                "https://en.wikipedia.org/wiki/Word_error_rate",
+            ],
         )
 
     def _download_and_prepare(self, dl_manager):
         """Optional: download external resources useful to compute the scores"""
-        # TODO: Download external resources if needed
         pass
 
-    def _compute(self, predictions, references):
+    def _compute(self, predictions, references, detailed=False) -> float | dict[str, float]:
         """Returns the scores"""
-        # TODO: Compute the different scores of the module
-        accuracy = sum(i == j for i, j in zip(predictions, references)) / len(predictions)
-        return {
-            "accuracy": accuracy,
-        }
+        num_substitutions = 0
+        num_deletions = 0
+        num_insertions = 0
+        num_hits = 0
+        for prediction, reference in zip(predictions, references):
+            measures = compute_measures(reference, prediction)
+            num_substitutions += measures["substitutions"]
+            num_deletions += measures["deletions"]
+            num_insertions += measures["insertions"]
+            num_hits += measures["hits"]
+
+        total = num_substitutions + num_deletions + num_hits
+        incorrect = num_substitutions + num_deletions + num_insertions
+
+        if detailed:
+            return dict(
+                wer=incorrect / total,
+                substitution_rate=num_substitutions / total,
+                deletion_rate=num_deletions / total,
+                insertion_rate=num_insertions / total,
+                num_substitutions=num_substitutions,
+                num_deletions=num_deletions,
+                num_insertions=num_insertions,
+                num_hits=num_hits,
+            )
+        return incorrect / total