README.md

---
license: apache-2.0
---
# MetricX-23

*This is not an officially supported Google product.*

**GitHub repository: [https://github.com/google-research/metricx](https://github.com/google-research/metricx)**

This repository contains the MetricX-23 models,
a family of models for automatic evaluation of translations that were proposed
in the WMT'23 Metrics Shared Task submission
[MetricX-23: The Google Submission to the WMT 2023 Metrics Shared Task](https://aclanthology.org/2023.wmt-1.63/).
The models were trained in [T5X](https://github.com/google-research/t5x) and
then converted for use in PyTorch.

## Available Models
There are 6 models available on HuggingFace that vary in the number of
parameters and whether or not the model is reference-based or reference-free
(also known as quality estimation, or QE):

* [MetricX-23-XXL](https://huggingface.co/google/metricx-23-xxl-v2p0)
* [MetricX-23-XL](https://huggingface.co/google/metricx-23-xl-v2p0)
* [MetricX-23-Large](https://huggingface.co/google/metricx-23-large-v2p0)
* [MetricX-23-QE-XXL](https://huggingface.co/google/metricx-23-qe-xxl-v2p0)
* [MetricX-23-QE-XL](https://huggingface.co/google/metricx-23-qe-xl-v2p0)
* [MetricX-23-QE-Large](https://huggingface.co/google/metricx-23-qe-large-v2p0)

We recommend using the XXL model versions for the best agreement with human
judgments of translation quality, the Large versions for best speed, and the
XL for an intermediate use case.


## Changes to the WMT'23 Submission

These models available here are most similar to the primary submission to the WMT'23 Metrics
Shared Task. They are initialized with [mT5](https://aclanthology.org/2021.naacl-main.41/)
then fine-tuned on a combination of direct assessment and MQM data. However,
we made some changes that make these models different from the WMT'23 submissions.

First, the models are trained to regress the actual MQM score rather than a
normalized score between 0 and 1. **That means the output from the MetricX-23
models is a score in the range [0, 25] where lower is better (i.e., it predicts
an error score).**

Second, these models were trained with a larger variety of synthetic data that
makes them more robust to translation edge cases like over- and undertranslation,
described in more detail in the following section.

### Synthetic Data

In order for our MetricX models to learn to identify certain types of bad
translations that are not sufficiently (or at all) represented in the regular
training data, we created synthetic examples and mixed them in during training.
The synthetic training data was generated from the DA datasets ranging from
WMT15 to WMT21 (~ 43 language pairs). In most cases, the synthetic examples have
the candidate translation manipulated so as to turn it into a bad translation
with a specific issue commonly unrecognized by learned metrics.

The table below provides an overview of the various failure modes that we
considered, including brief descriptions of how we prepared the synthetic data
to address them.

| Failure mode | Synthetic example description |
| ----------- | ----------- |
| Undertranslation | Candidate translation with an arbitrary sentence removed (if multi-sentence); alternatively, candidate with a certain proportion of words removed from the end. |
| Overtranslation | Candidate translation duplicated (with space in between). |
| Fluent but unrelated translation | Arbitrary reference of a similar length from the dataset. |
| Gibberish | Text of a similar length as the reference, generated by sampling words from the reference translation vocabulary (built from all references in the data). |
| Missing punctuation | Reference translation with the end punctuation removed (11 punctuation symbols considered). |
| Latin instead of Chinese/Japanese or Hindi/Bengali punctuation | Candidate translation with the language-specific punctuation symbol at the end replaced with the Latin equivalent (e.g., "." instead of "。" or "।"); alternatively, the punctuation symbol is replaced with the Latin equivalent in the reference, keeping the correct one in the candidate. |
| Reference-matching translation | Reference translation copied as the candidate translation (unlike the rest of the synthetic data, these examples are meant to train the metric to predict a perfect score for candidates matching the reference). |

Examples from the first 4 categories were assigned a label corresponding to the
worst score on the given rating scale (e.g., 25 when mixed with MQM training
data), whereas the reference-matching translation examples are assigned the best
score (e.g., 0 when used with MQM data). The missing/incorrect punctuation
examples were labeled with a score slightly worse than perfect.

Note that some of the synthetic datasets are only meaningful in the
reference-based scenario, and we thus excluded them when training a QE variant
of MetricX. These are the Latin-vs-special punctuation and the
reference-matching translation examples.

Most of the synthetic training sets were created using stratified sampling
across target languages, taking 500 examples per target language. One exception
is the missing punctuation set, which used a stratified sample across different
punctuation symbols instead.

When training MetricX, a small proportion of the synthetic examples was mixed
with the regular training examples. During the first-stage fine-tuning on DA
data, each synthetic training set constituted between 0.1% and 1% of all
training examples, whereas in the second-stage fine-tuning on MQM data we used
an even smaller proportion, around 0.05%.

As for evaluating the effect of the synthetic training data on the model's
performance, the DEMETR challenge set - which we originally used to evaluate the
models submitted to the WMT23 Metrics Shared Task - was not adequate anymore. We
therefore created a new DEMETR-style test set based on the WMT22 DA data, with
examples constructed analogically to the synthetic training examples, as
described above. This test set helped us determine the right proportions of
synthetic data for fine-tuning in order to make MetricX robust for the failure
modes in consideration, without sacrificing the system- and segment-level
correlations with human ratings.

## Usage

The code for using MetricX models can be found at [https://github.com/google-research/metricx](https://github.com/google-research/metricx).
The repository contains example prediction scripts, described below.

The `metricx23/predict.py` script contains an example for how to run inference
on the models.

### Reference-Based
Example usage for a reference-based model:

```bash
python -m metricx23.predict \
  --tokenizer google/mt5-xl \
  --model_name_or_path google/metricx-23-xl-v2p0 \
  --max_input_length 1024 \
  --batch_size 1 \
  --input_file input.jsonl \
  --output_file output.jsonl
```

`input.jsonl` is expected to have 1 serialized JSON object per line with
`"reference"` and `"hypothesis"` fields. The output jsonl will be parallel
to `input.jsonl` but additionally contain a `"prediction"` field with the predicted score.

Note that the model was trained with a maximum input length of 1024 tokens, so
significantly increasing that value may lead to unpredictable behavior.

### Reference-Free
Example usage for a reference-free model:

```bash
python -m metricx23.predict \
  --tokenizer google/mt5-xl \
  --model_name_or_path google/metricx-23-qe-xl-v2p0 \
  --max_input_length 1024 \
  --batch_size 1 \
  --input_file input.jsonl \
  --output_file output.jsonl \
  --qe
```

`input.jsonl` is expected to have 1 serialized JSON object per line with
`"source"` and `"hypothesis"` fields. The output jsonl will be parallel
to `input.jsonl` but additionally contain a `"prediction"` field with the predicted score.


## Meta-Evaluation
The `metricx23/evaluate.py` script contains code to calculate various correlations
between the MetricX-23 scores and MQM ratings of translation quality using the
[MT Metrics Eval](https://github.com/google-research/mt-metrics-eval) library.

Example usage:

```bash
python -m metricx23.evaluate \
  --dataset wmt22 \
  --lp en-de \
  --input_file input.jsonl \
  --output_file output.json
```

`input.jsonl` is expected to have one JSON object serialized per line.
Each JSON object is expected to contain 4 fields:

* `"system_id"`: The name of the system that generated the translation.
* `"segment_id"`: The 0-based index of the corresponding segment in the MT
Metrics Eval data.
* `"label"`: The ground-truth translation quality score (with higher is better).
* `"prediction"`: The model predicted translation quality score (with lower is
better; the script negates the scores so higher is better).

The script will calculate the 4 agreement/correlations that were used in the
WMT'23 Shared Task. Below are the results for the MetricX-23 models on the
WMT'22 Metrics Shared Task data:

English-German:

| Model      | System-Level Accuracy | System-Level Pearson | Segment-Level Pearson | Segment-Level Pairwise Acc |
| ----------- | ----------- | ----------- | ----------- | ----------- |
| MetricX-23-XXL      | 0.795       | 0.835       | 0.546       | 0.619       |
| MetricX-23-XL   | 0.756        | 0.813       | 0.540       | 0.605       |
| MetricX-23-Large   | 0.769        | 0.759       | 0.507       | 0.595       |
| MetricX-23-QE-XXL   | 0.769        | 0.830       | 0.490       | 0.606       |
| MetricX-23-QE-XL   | 0.718        | 0.684       | 0.421       | 0.594       |
| MetricX-23-QE-Large   | 0.744        | 0.671       | 0.387       | 0.579       |

English-Russian:

| Model      | System-Level Accuracy | System-Level Pearson | Segment-Level Pearson | Segment-Level Pairwise Acc |
| ----------- | ----------- | ----------- | ----------- | ----------- |
| MetricX-23-XXL      | 0.905       | 0.943       | 0.477       | 0.609       |
| MetricX-23-XL   | 0.876        | 0.906       | 0.498       | 0.589       |
| MetricX-23-Large   | 0.876        | 0.841       | 0.474       | 0.569       |
| MetricX-23-QE-XXL   | 0.895        | 0.940       | 0.470       | 0.602       |
| MetricX-23-QE-XL   | 0.848        | 0.861       | 0.415       | 0.570       |
| MetricX-23-QE-Large   | 0.819        | 0.778       | 0.411       | 0.551       |

Chinese-English:

| Model      | System-Level Accuracy | System-Level Pearson | Segment-Level Pearson | Segment-Level Pairwise Acc |
| ----------- | ----------- | ----------- | ----------- | ----------- |
| MetricX-23-XXL      | 0.868       | 0.919       | 0.605       | 0.551       |
| MetricX-23-XL   | 0.868        | 0.924       | 0.584       | 0.543       |
| MetricX-23-Large   | 0.857        | 0.919       | 0.555       | 0.539       |
| MetricX-23-QE-XXL   | 0.857        | 0.928       | 0.573       | 0.544       |
| MetricX-23-QE-XL   | 0.802        | 0.879       | 0.546       | 0.529       |
| MetricX-23-QE-Large   | 0.758        | 0.904       | 0.522       | 0.529       |


The `metricx23/evaluate_wmt23.py` script re-calculates the average correlation
score that was used to rank submissions from the
[WMT'23 Shared Task](https://www2.statmt.org/wmt23/pdf/2023.wmt-1.51.pdf).

Example usage:

```bash
python -m metricx23.evaluate_wmt23 \
  --en_de predictions_ende.jsonl \
  --he_en predictions_heen.jsonl \
  --zh_en predictions_zhen.jsonl \
  --output_file output.json
```

Each of the 3 input files is expected to be in the same format as described
above. Each file should correspond to running inference on each of the language
pairs from the WMT'23 dataset.

The results for each of the models is the following:

| Model      | Average Correlation |
| ----------- | ----------- |
| MetricX-23-XXL      | 0.812       |
| MetricX-23-XL   | 0.813        |
| MetricX-23-Large   | 0.794        |
| MetricX-23-QE-XXL   | 0.797        |
| MetricX-23-QE-XL   | 0.767        |
| MetricX-23-QE-Large   | 0.762        |


## Citation
If you use MetricX-23 in your research, please cite the following publication:

```bibtex
@inproceedings{juraska-etal-2023-metricx,
    title = {{MetricX-23: The Google Submission to the WMT 2023 Metrics Shared Task}},
    author = "Juraska, Juraj  and
      Finkelstein, Mara  and
      Deutsch, Daniel  and
      Siddhant, Aditya  and
      Mirzazadeh, Mehdi  and
      Freitag, Markus",
    editor = "Koehn, Philipp  and
      Haddow, Barry  and
      Kocmi, Tom  and
      Monz, Christof",
    booktitle = "Proceedings of the Eighth Conference on Machine Translation",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.wmt-1.63",
    doi = "10.18653/v1/2023.wmt-1.63",
    pages = "756--767",
}
```