masakhane

/

africomet-qe-stl-1.1

like 2

Follow

Masakhane NLP 101

Hi, thanks for your question! Our QE model is designed for the reference-free translation quality estimation for African languages, such as English-Kikuyu translation. Could you provide details on how you ran the model and the domain you were testing on? Based on our evaluation in the Wikipedia domain, https://aclanthology.org/2024.wmt-1.36.pdf, the model achieves approximately 0.7 Pearson correlation and 0.47 Spearman ranking correlation with human evaluation.

I ran this code like this

from comet import download_model, load_from_checkpoint

model_path = download_model("masakhane/africomet-qe-stl-1.1")
model = load_from_checkpoint(model_path)

data = [
    {
        "src": "Uhoro waku",
        "mt": "Hi, how are you?",
    },
    {
        "src": "Arokire ira",
        "mt": "He came yesterday.",
    },
    {
        "src": "Wi mwega",
        "mt": "You good?",
    }
]
model_output = model.predict(data, batch_size=8, gpus=1)
print (model_output)

I was wondering how to specifically use kikuyu or swahili language.

I got this kind of results

Predicting DataLoader 0: 100%|██████████| 1/1 [00:00<00:00, 21.97it/s]

Prediction([('scores', [0.5098617076873779, 0.5110026001930237, 0.47440865635871887]), ('system_score', 0.49842432141304016)])

In AfriCOMET, specifying the source and target languages is not necessary. The model was trained using a unified pretrained multilingual encoder. You executed the model correctly. Regarding the scores, I briefly reviewed the source and translation you provided in your example. The relative ranking of the scores appears correct (the last example appears to be the worst translation among these three and the middle one would be the best one), though the absolute values may require further inspection. I would suggest to try to rescale the scores using some linear transformation to 0~1. We would investigate it in your case. Thank you!