opus-mt-tc-bible-big-mul-deu_eng_fra_por_spa

Table of Contents

• Model Details
• Uses
• Risks, Limitations and Biases
• How to Get Started With the Model
• Training
• Evaluation
• Citation Information
• Acknowledgements

Model Details

Neural machine translation model for translating from Multiple languages (mul) to unknown (deu+eng+fra+por+spa). Note that many of the listed languages will not be well supported by the model as the training data is very limited for the majority of the languages. Translation performance varies a lot and for a large number of language pairs it will not work at all.

This model is part of the OPUS-MT project, an effort to make neural machine translation models widely available and accessible for many languages in the world. All models are originally trained using the amazing framework of Marian NMT, an efficient NMT implementation written in pure C++. The models have been converted to pyTorch using the transformers library by huggingface. Training data is taken from OPUS and training pipelines use the procedures of OPUS-MT-train. Model Description:

• Developed by: Language Technology Research Group at the University of Helsinki
• Model Type: Translation (transformer-big)
• Release: 2024-05-30
• License: Apache-2.0
• Language(s):
  • Source Language(s): aai aar aau abi abk acd ace acf ach acm acn acr ade adj ady aeu aey afb afh afr agd agn agu ahk aia aka akh akl akp alj aln alp alq alt alz ame amh ami amk amu ang ann anp anv aoz apc apr apu ara arc arg arq arz asm aso ast atg atj atq aui auy ava avk avn avu awa awb awx aze azg azz bak bal bam ban bar bas bav bba bbo bbr bcl bcw bef beh bel bem ben bep bex bfa bfd bfo bgr bhl bho bhz bib bik bim bis biv bjr bjv bku bkv blh blt blz bmh bmk bmq bmu bmv bnp bod boj bom bos bov box bpr bps bpy bqc bqj bqp bre bru brx bss btd bth bto bts btt btx bua bud bug buk bul bus bvy bwq bwu byn bzd bzh bzj bzt caa cab cac cak cat cay cbk cce cco ceb ces cfm cgc cha che chf chm chq chr chu chv chy chz cjk cjo cjp cjv cjy ckb cko cle cme cmn cmo cmr cnh cni cnl cnr cnt cnw cok cop cor cos cot cpa cpu cre crh crn crs crx csb csk cso csy cta ctd ctp ctu cuc cui cuk cut cux cwe cwt cya cym czt daa dad dag dah dan ded deu dga dgi dig dik din diq div dje djk dng dni dnj dob dop drt dsb dsh dtp dty dug dws dww dyi dyo dyu dzo efi egl ell emi eng enm epo ess est eus ewe ext fai fal fao far fas fij fil fin fkv fon for fra frd frm fro frp frr fry fuc ful fur gag gah gaw gbm gcf gde gej gfk ghs gil gkn gla gle glg glk glv gnd gng gog gor gos got gqr grc grn gsw guc gud guh guj guo gur guw gux gvf gvl gwi gwr gym gyr hag hat hau haw hay hbo hbs hch heb heh her hif hig hil hin hla hlt hmn hne hnj hnn hns hoc hot hrv hrx hsb hsn hui hun hus hvn hwc hye hyw iba ibo icr ido ifa ifb ife ifk ifu ify ign igs iii ike iku ile ilo imo ina ind inh ino iou ipi ipk iri irk iry isl ita itv ium ixl izh izr jaa jac jak jam jav jbo jbu jdt jmc jpa jpn jun jvn kaa kab kac kal kam kan kao kas kat kau kaz kbd kbm kbp kdc kdj kdl kdn kea kek ken keo ker keu kew kez kgf kgk kha khm khz kia kik kin kir kjb kje kjh kjs kki kkj kle kma kmb kmg kmh kmo kmr kmu knc kne knj knk kno kog koi kok kom kon kpf kpg kpr kpv kpw kpz kqe kqf kqp kqw krc kri krj krl kru ksb ksh ksr ktb ktj kua kub kud kue kum kur kus kvn kwf kxc kxm kyc kyf kyg kyq kzf laa lac lad lah lao las lat lav law lbe lcm ldn lee lef lem leu lew lex lez lfn lgg lhu lia lid lif lij lim lin lip lit liv ljp lkt lld lln lme lmo lnd lob lok lon lou lrc lsi ltz lua luc lug luo lus lut luy lzz maa mad mag mah mai maj mak mal mam maq mar mau maw max maz mbb mbf mbt mcb mcp mcu mda mdf med mee meh mek men meq mfe mfh mfi mfk mfq mfy mgd mgm mgo mhi mhl mhx mhy mib mic mie mif mig mih mil mio mit mix miy miz mjc mkd mks mlg mlh mlp mlt mmo mmx mna mnb mnf mnh mni mnr mnw moa mog moh mol mon mop mor mos mox mpg mpm mpt mpx mqb mqj mri mrj mrw msa msm mta muh mux muy mva mvp mvv mwc mwl mwm mwv mww mxb mxt mya myb myk myu myv myw myx mzk mzm mzn mzw mzz naf nak nap nas nau nav nbl nca nch ncj ncl ncu nde ndo nds ndz neb nep new nfr ngt ngu nhe nhg nhi nhn nhu nhw nhx nhy nia nif nii nij nim nin niu njm nlc nld nlv nmz nnb nnh nno nnw nob nog non nop nor not nou nov npi npl npy nqo nsn nso nss nst nsu ntm ntp ntr nuj nus nuy nwb nwi nya nyf nyn nyo nyy nzi oar obo oci ofs oji oku okv old omw ood opm ori orm orv osp oss ota ote otk otm otn otq ozm pab pad pag pai pal pam pan pao pap pau pbi pbl pcd pck pcm pdc pes pfl phn pib pih pio pis pkb pli pls plt plw pmf pms pmy pne pnt poe poh pol por pot ppk ppl prf prg prs ptp ptu pus pwg pww quc qya rai rap rav rej rhg rif rim rmy roh rom ron rop rro rue rug run rup rus rwo sab sag sah san sas sat sba sbd sbl scn sco sda sdh seh ses sgb sgs sgw sgz shi shk shn shs shy sig sil sin sjn skr sld slk sll slv sma sme smk sml smn smo sna snc snd snp snw som sot soy spa spl spp sps sqi srd srm srn srp srq ssd ssw ssx stn stp stq sue suk sun sur sus suz swa swc swe swg swh swp sxb sxn syc syl syr szb szl tab tac tah taj tam taq tat tbc tbl tbo tbz tcs tcy tel tem teo ter tet tfr tgk tgl tgo tgp tha thk thv tig tik tim tir tkl tlb tlf tlh tlj tlx tly tmc tmh tmr tmw toh toi toj ton tpa tpi tpm tpw tpz trc trn trq trs trv tsn tso tsw ttc tte ttr tts tuc tuf tuk tum tur tvl twb twi twu txa tyj tyv tzh tzj tzl tzm tzo ubr ubu udm udu uig ukr umb urd usa usp uvl uzb vag vec ven vep vie viv vls vmw vmy vol vot vro vun wae waj wal wap war wbm wbp wed wln wmt wmw wnc wnu wob wol wsk wuu wuv xal xcl xed xho xmf xog xon xrb xsb xsi xsm xsr xtd xtm xuo yal yam yaq yaz yby ycl ycn yid yli yml yon yor yua yue yut yuw zam zap zea zgh zha zia zlm zom zsm zul zyp zza
  • Target Language(s): deu eng fra por spa
  • Valid Target Language Labels: >>deu<< >>eng<< >>fra<< >>por<< >>spa<< >>xxx<<
• Original Model: opusTCv20230926max50+bt+jhubc_transformer-big_2024-05-30.zip
• Resources for more information:
  • OPUS-MT dashboard
  • OPUS-MT-train GitHub Repo
  • More information about MarianNMT models in the transformers library
  • Tatoeba Translation Challenge
  • HPLT bilingual data v1 (as part of the Tatoeba Translation Challenge dataset)
  • A massively parallel Bible corpus

This is a multilingual translation model with multiple target languages. A sentence initial language token is required in the form of >>id<< (id = valid target language ID), e.g. >>deu<<

Uses

This model can be used for translation and text-to-text generation.

Risks, Limitations and Biases

CONTENT WARNING: Readers should be aware that the model is trained on various public data sets that may contain content that is disturbing, offensive, and can propagate historical and current stereotypes.

Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021) and Bender et al. (2021)).

Also note that many of the listed languages will not be well supported by the model as the training data is very limited for the majority of the languages. Translation performance varies a lot and for a large number of language pairs it will not work at all.

How to Get Started With the Model

A short example code:

from transformers import MarianMTModel, MarianTokenizer

src_text = [
    ">>deu<< Replace this with text in an accepted source language.",
    ">>spa<< This is the second sentence."
]

model_name = "pytorch-models/opus-mt-tc-bible-big-mul-deu_eng_fra_por_spa"
tokenizer = MarianTokenizer.from_pretrained(model_name)
model = MarianMTModel.from_pretrained(model_name)
translated = model.generate(**tokenizer(src_text, return_tensors="pt", padding=True))

for t in translated:
    print( tokenizer.decode(t, skip_special_tokens=True) )

You can also use OPUS-MT models with the transformers pipelines, for example:

from transformers import pipeline
pipe = pipeline("translation", model="Helsinki-NLP/opus-mt-tc-bible-big-mul-deu_eng_fra_por_spa")
print(pipe(">>deu<< Replace this with text in an accepted source language."))

Training

• Data: opusTCv20230926max50+bt+jhubc (source)
• Pre-processing: SentencePiece (spm32k,spm32k)
• Model Type: transformer-big
• Original MarianNMT Model: opusTCv20230926max50+bt+jhubc_transformer-big_2024-05-30.zip
• Training Scripts: GitHub Repo

Evaluation

• Model scores at the OPUS-MT dashboard
• test set translations: opusTCv20230926max50+bt+jhubc_transformer-big_2024-05-29.test.txt
• test set scores: opusTCv20230926max50+bt+jhubc_transformer-big_2024-05-29.eval.txt
• benchmark results: benchmark_results.txt
• benchmark output: benchmark_translations.zip

langpair	testset	chr-F	BLEU	#sent	#words
afr-deu	tatoeba-test-v2021-08-07	0.65092	44.5	1583	9105
afr-eng	tatoeba-test-v2021-08-07	0.71045	57.7	1374	9622
afr-spa	tatoeba-test-v2021-08-07	0.70761	54.9	448	2783
ara-deu	tatoeba-test-v2021-08-07	0.57255	36.6	1209	8371
ara-eng	tatoeba-test-v2021-08-07	0.54904	36.6	10305	76975
ara-fra	tatoeba-test-v2021-08-07	0.51444	33.3	1569	11066
ara-spa	tatoeba-test-v2021-08-07	0.59275	40.4	1511	9708
awa-eng	tatoeba-test-v2021-08-07	0.54336	35.1	279	1335
bel-deu	tatoeba-test-v2021-08-07	0.61601	39.7	551	4182
bel-eng	tatoeba-test-v2021-08-07	0.58808	39.9	2500	18571
bel-fra	tatoeba-test-v2021-08-07	0.64213	42.6	283	2005
bel-spa	tatoeba-test-v2021-08-07	0.61926	40.5	205	1412
ben-eng	tatoeba-test-v2021-08-07	0.59355	40.4	2500	13978
bos_Latn-eng	tatoeba-test-v2021-08-07	0.74642	59.3	301	1826
bul-deu	tatoeba-test-v2021-08-07	0.64537	45.7	314	2224
bul-eng	tatoeba-test-v2021-08-07	0.66875	51.4	10000	71872
bul-fra	tatoeba-test-v2021-08-07	0.67507	50.2	446	3669
bul-spa	tatoeba-test-v2021-08-07	0.60580	43.0	286	1783
cat-deu	tatoeba-test-v2021-08-07	0.59831	38.7	723	5676
cat-eng	tatoeba-test-v2021-08-07	0.67167	51.0	1631	12627
cat-fra	tatoeba-test-v2021-08-07	0.68397	50.9	700	5664
cat-por	tatoeba-test-v2021-08-07	0.72419	54.3	747	6119
cat-spa	tatoeba-test-v2021-08-07	0.82715	70.1	1534	12094
ceb-eng	tatoeba-test-v2021-08-07	0.37649	22.2	378	2293
ces-deu	tatoeba-test-v2021-08-07	0.64331	45.2	3490	27155
ces-eng	tatoeba-test-v2021-08-07	0.65217	48.9	13824	105010
ces-fra	tatoeba-test-v2021-08-07	0.68572	51.3	438	3346
ces-spa	tatoeba-test-v2021-08-07	0.66549	48.1	1807	12716
cmn_Hans-deu	tatoeba-test-v2021-08-07	0.41434	17.3	2609	20805
cmn_Hans-eng	tatoeba-test-v2021-08-07	0.43629	22.9	4450	38093
cmn_Hans-fra	tatoeba-test-v2021-08-07	0.43664	21.9	6825	59962
cmn_Hans-por	tatoeba-test-v2021-08-07	0.44057	21.7	638	4580
cmn_Hans-spa	tatoeba-test-v2021-08-07	0.50740	27.4	1142	7523
cmn_Hant-deu	tatoeba-test-v2021-08-07	0.41129	18.9	1242	8527
cmn_Hant-eng	tatoeba-test-v2021-08-07	0.42200	21.8	4476	35172
cmn_Hant-fra	tatoeba-test-v2021-08-07	0.42720	22.4	2132	16599
cmn_Hant-por	tatoeba-test-v2021-08-07	0.44075	21.2	386	2372
cmn_Hant-spa	tatoeba-test-v2021-08-07	0.46395	23.8	1471	9530
cym-eng	tatoeba-test-v2021-08-07	0.56219	37.3	818	5563
dan-deu	tatoeba-test-v2021-08-07	0.68986	48.8	9998	76055
dan-eng	tatoeba-test-v2021-08-07	0.72855	58.6	10795	79684
dan-fra	tatoeba-test-v2021-08-07	0.74245	58.8	1731	11882
dan-por	tatoeba-test-v2021-08-07	0.75509	57.8	873	5360
dan-spa	tatoeba-test-v2021-08-07	0.69396	50.7	5000	35528
deu-deu	tatoeba-test-v2021-08-07	0.59000	35.3	2500	20806
deu-eng	tatoeba-test-v2021-08-07	0.62522	44.3	17565	149462
deu-fra	tatoeba-test-v2021-08-07	0.64153	45.5	12418	102721
deu-por	tatoeba-test-v2021-08-07	0.60440	39.4	10000	81482
deu-spa	tatoeba-test-v2021-08-07	0.64608	44.8	10521	82570
dsb-deu	tatoeba-test-v2021-08-07	0.49361	26.8	640	4469
ell-deu	tatoeba-test-v2021-08-07	0.65559	47.0	2500	17025
ell-eng	tatoeba-test-v2021-08-07	0.72452	57.0	10899	68682
ell-fra	tatoeba-test-v2021-08-07	0.71519	55.8	1506	9726
ell-por	tatoeba-test-v2021-08-07	0.65702	44.2	885	5196
ell-spa	tatoeba-test-v2021-08-07	0.61398	43.7	1829	10828
eng-deu	tatoeba-test-v2021-08-07	0.58678	36.1	17565	151568
eng-eng	tatoeba-test-v2021-08-07	0.82914	68.7	12062	115106
eng-fra	tatoeba-test-v2021-08-07	0.65063	46.1	12681	106378
eng-por	tatoeba-test-v2021-08-07	0.68859	49.2	13222	105265
eng-spa	tatoeba-test-v2021-08-07	0.68920	50.5	16583	134710
epo-deu	tatoeba-test-v2021-08-07	0.59014	37.4	13439	133281
epo-eng	tatoeba-test-v2021-08-07	0.63122	46.3	17675	142283
epo-fra	tatoeba-test-v2021-08-07	0.58149	37.6	12168	103156
epo-por	tatoeba-test-v2021-08-07	0.60324	39.2	10809	98394
epo-spa	tatoeba-test-v2021-08-07	0.62955	42.6	10735	83284
est-deu	tatoeba-test-v2021-08-07	0.64025	45.7	244	1611
est-eng	tatoeba-test-v2021-08-07	0.65633	49.9	1359	8811
eus-deu	tatoeba-test-v2021-08-07	0.57833	35.8	456	3315
eus-eng	tatoeba-test-v2021-08-07	0.61860	45.0	1060	8157
eus-spa	tatoeba-test-v2021-08-07	0.59847	39.9	1850	12469
fao-eng	tatoeba-test-v2021-08-07	0.51390	36.2	294	1984
fas-deu	tatoeba-test-v2021-08-07	0.54134	29.5	3185	25590
fas-eng	tatoeba-test-v2021-08-07	0.55065	35.9	3762	31480
fas-fra	tatoeba-test-v2021-08-07	0.54990	33.8	376	3377
fin-deu	tatoeba-test-v2021-08-07	0.59382	39.6	2647	19163
fin-eng	tatoeba-test-v2021-08-07	0.62477	45.3	10690	80552
fin-fra	tatoeba-test-v2021-08-07	0.61527	43.2	1920	12193
fin-por	tatoeba-test-v2021-08-07	0.69569	51.1	477	3021
fin-spa	tatoeba-test-v2021-08-07	0.62879	44.2	2513	16912
fra-deu	tatoeba-test-v2021-08-07	0.64320	43.5	12418	100545
fra-eng	tatoeba-test-v2021-08-07	0.68541	52.3	12681	101754
fra-fra	tatoeba-test-v2021-08-07	0.62951	42.7	1000	7757
fra-por	tatoeba-test-v2021-08-07	0.67912	48.2	10518	77650
fra-spa	tatoeba-test-v2021-08-07	0.69174	50.8	10294	78406
fry-eng	tatoeba-test-v2021-08-07	0.53679	34.9	220	1573
gle-eng	tatoeba-test-v2021-08-07	0.55023	38.2	1913	11190
glg-eng	tatoeba-test-v2021-08-07	0.66283	49.8	1015	8421
glg-por	tatoeba-test-v2021-08-07	0.75766	59.9	433	3105
glg-spa	tatoeba-test-v2021-08-07	0.76978	63.8	2121	17443
gos-deu	tatoeba-test-v2021-08-07	0.42288	21.0	207	1168
gos-eng	tatoeba-test-v2021-08-07	0.36783	21.2	1154	5635
gsw-eng	tatoeba-test-v2021-08-07	0.43570	28.2	205	990
hbs-deu	tatoeba-test-v2021-08-07	0.65914	46.6	1959	15559
hbs-eng	tatoeba-test-v2021-08-07	0.66115	51.3	10017	68934
hbs-fra	tatoeba-test-v2021-08-07	0.64444	43.4	474	3370
hbs-spa	tatoeba-test-v2021-08-07	0.67758	50.4	607	3766
heb-deu	tatoeba-test-v2021-08-07	0.59478	38.2	3090	25101
heb-eng	tatoeba-test-v2021-08-07	0.59663	43.2	10519	77427
heb-fra	tatoeba-test-v2021-08-07	0.59598	41.1	3281	26123
heb-por	tatoeba-test-v2021-08-07	0.59893	39.0	719	5335
heb-spa	tatoeba-test-v2021-08-07	0.63095	43.6	1849	14213
hin-eng	tatoeba-test-v2021-08-07	0.61435	43.6	5000	33943
hrv-deu	tatoeba-test-v2021-08-07	0.64724	46.3	782	5734
hrv-eng	tatoeba-test-v2021-08-07	0.69236	53.5	1480	10620
hrv-fra	tatoeba-test-v2021-08-07	0.61205	40.0	258	1943
hrv-spa	tatoeba-test-v2021-08-07	0.64826	48.2	254	1702
hsb-deu	tatoeba-test-v2021-08-07	0.51359	29.4	666	4818
hun-deu	tatoeba-test-v2021-08-07	0.55409	32.7	15342	127344
hun-eng	tatoeba-test-v2021-08-07	0.58597	40.3	13037	94699
hun-fra	tatoeba-test-v2021-08-07	0.61238	43.8	2494	16914
hun-por	tatoeba-test-v2021-08-07	0.59369	37.7	2500	16563
hun-spa	tatoeba-test-v2021-08-07	0.62248	42.0	2500	16670
hye-eng	tatoeba-test-v2021-08-07	0.45721	22.0	1121	5066
ido_Latn-eng	tatoeba-test-v2021-08-07	0.49142	31.1	1959	14065
ilo-eng	tatoeba-test-v2021-08-07	0.48258	30.9	1093	7384
ind-deu	tatoeba-test-v2021-08-07	0.58984	37.5	497	3881
ind-eng	tatoeba-test-v2021-08-07	0.61331	43.6	4289	31355
ind-fra	tatoeba-test-v2021-08-07	0.59252	39.7	905	6994
isl-deu	tatoeba-test-v2021-08-07	0.59711	39.7	969	6279
isl-eng	tatoeba-test-v2021-08-07	0.58680	41.5	2503	19788
isl-spa	tatoeba-test-v2021-08-07	0.59365	40.3	238	1229
ita-deu	tatoeba-test-v2021-08-07	0.63984	43.4	10094	79762
ita-eng	tatoeba-test-v2021-08-07	0.76391	64.1	17320	119214
ita-fra	tatoeba-test-v2021-08-07	0.73609	58.8	10091	66377
ita-por	tatoeba-test-v2021-08-07	0.71299	51.9	3066	25668
ita-spa	tatoeba-test-v2021-08-07	0.75113	59.2	5000	34937
jpn-deu	tatoeba-test-v2021-08-07	0.42487	18.3	11428	97987
jpn-eng	tatoeba-test-v2021-08-07	0.43223	20.8	13862	123562
jpn-fra	tatoeba-test-v2021-08-07	0.43219	21.0	10169	88857
jpn-por	tatoeba-test-v2021-08-07	0.43496	21.0	1940	15560
jpn-spa	tatoeba-test-v2021-08-07	0.45502	22.1	10742	82888
kat-eng	tatoeba-test-v2021-08-07	0.54012	34.9	984	5994
kaz-eng	tatoeba-test-v2021-08-07	0.49920	29.7	403	2692
kaz_Cyrl-eng	tatoeba-test-v2021-08-07	0.50415	30.2	396	2645
kur_Latn-deu	tatoeba-test-v2021-08-07	3.099	0.0	223	1323
kur_Latn-eng	tatoeba-test-v2021-08-07	2.189	0.0	290	1708
lat-spa	tatoeba-test-v2021-08-07	0.40919	20.1	3129	34036
lav-eng	tatoeba-test-v2021-08-07	0.64727	48.0	1631	11213
lfn_Latn-deu	tatoeba-test-v2021-08-07	0.53889	31.3	291	2258
lfn_Latn-eng	tatoeba-test-v2021-08-07	0.52686	32.4	2450	16627
lfn_Latn-fra	tatoeba-test-v2021-08-07	0.54701	33.0	361	3011
lfn_Latn-por	tatoeba-test-v2021-08-07	0.53719	29.9	1725	22993
mal-eng	tatoeba-test-v2021-08-07	0.51324	28.0	802	5558
mar-eng	tatoeba-test-v2021-08-07	0.60262	41.2	10396	67527
mkd-eng	tatoeba-test-v2021-08-07	0.62702	48.1	10010	65667
mkd-spa	tatoeba-test-v2021-08-07	0.70753	54.3	217	1121
mlt-eng	tatoeba-test-v2021-08-07	0.55253	37.6	203	1165
msa-deu	tatoeba-test-v2021-08-07	0.58593	37.1	544	4260
msa-eng	tatoeba-test-v2021-08-07	0.60501	42.6	5000	37033
nld-deu	tatoeba-test-v2021-08-07	0.69163	50.4	10218	74131
nor-deu	tatoeba-test-v2021-08-07	0.65791	45.1	3651	34575
oci-fra	tatoeba-test-v2021-08-07	0.45312	23.7	806	6302
pcd-fra	tatoeba-test-v2021-08-07	0.40669	15.3	266	1677
pes-eng	tatoeba-test-v2021-08-07	0.55345	36.1	3757	31411
pol-deu	tatoeba-test-v2021-08-07	0.63490	43.0	5000	37421
pol-eng	tatoeba-test-v2021-08-07	0.64737	48.1	10099	75766
pol-fra	tatoeba-test-v2021-08-07	0.62800	44.2	3087	24257
pol-por	tatoeba-test-v2021-08-07	0.61151	39.8	705	5063
pol-spa	tatoeba-test-v2021-08-07	0.62972	43.6	2544	18113
por-deu	tatoeba-test-v2021-08-07	0.64062	42.9	10000	81246
por-eng	tatoeba-test-v2021-08-07	0.73166	58.9	13222	105351
por-fra	tatoeba-test-v2021-08-07	0.72391	55.5	10518	80459
por-por	tatoeba-test-v2021-08-07	0.73726	54.4	2500	19220
por-spa	tatoeba-test-v2021-08-07	0.77104	61.2	10947	87335
ron-deu	tatoeba-test-v2021-08-07	0.64398	45.5	1141	7893
ron-eng	tatoeba-test-v2021-08-07	0.68856	53.2	5508	40717
ron-fra	tatoeba-test-v2021-08-07	0.67584	49.9	1925	13347
ron-por	tatoeba-test-v2021-08-07	0.71321	51.9	681	4593
ron-spa	tatoeba-test-v2021-08-07	0.71088	54.1	1959	12679
run-eng	tatoeba-test-v2021-08-07	0.42256	28.3	1703	10041
run-fra	tatoeba-test-v2021-08-07	0.38934	22.8	1274	7479
rus-deu	tatoeba-test-v2021-08-07	0.64030	43.6	12800	98842
rus-eng	tatoeba-test-v2021-08-07	0.65436	49.2	19425	147872
rus-fra	tatoeba-test-v2021-08-07	0.63112	44.8	11490	80579
rus-por	tatoeba-test-v2021-08-07	0.61328	40.1	10000	74713
rus-spa	tatoeba-test-v2021-08-07	0.64612	45.2	10506	75246
slv-deu	tatoeba-test-v2021-08-07	0.67667	50.1	492	3003
slv-eng	tatoeba-test-v2021-08-07	0.66956	50.9	2495	16940
slv-fra	tatoeba-test-v2021-08-07	0.57163	39.5	448	3792
afr-fra	flores101-devtest	0.57150	30.7	1012	28343
afr-por	flores101-devtest	0.56505	29.7	1012	26519
afr-spa	flores101-devtest	0.47939	19.3	1012	29199
amh-spa	flores101-devtest	2.986	0.0	1012	29199
ara-eng	flores101-devtest	0.55947	28.2	1012	24721
ara-fra	flores101-devtest	0.52333	24.4	1012	28343
bel-deu	flores101-devtest	0.42452	9.4	1012	25094
bel-eng	flores101-devtest	0.45285	12.7	1012	24721
bel-por	flores101-devtest	0.43909	12.2	1012	26519
ben-deu	flores101-devtest	0.41198	11.7	1012	25094
ben-eng	flores101-devtest	0.48599	17.6	1012	24721
bul-por	flores101-devtest	0.55794	27.4	1012	26519
cat-fra	flores101-devtest	0.60255	34.4	1012	28343
ceb-deu	flores101-devtest	0.44474	15.8	1012	25094
ceb-eng	flores101-devtest	0.56418	31.5	1012	24721
ceb-fra	flores101-devtest	0.47733	21.7	1012	28343
ceb-por	flores101-devtest	0.48276	22.3	1012	26519
ceb-spa	flores101-devtest	0.42858	16.1	1012	29199
ces-eng	flores101-devtest	0.59579	31.5	1012	24721
ces-spa	flores101-devtest	0.47675	18.8	1012	29199
cym-eng	flores101-devtest	0.53043	28.7	1012	24721
cym-fra	flores101-devtest	0.44739	18.6	1012	28343
cym-por	flores101-devtest	0.42913	17.3	1012	26519
dan-eng	flores101-devtest	0.67095	41.8	1012	24721
dan-por	flores101-devtest	0.58416	31.4	1012	26519
dan-spa	flores101-devtest	0.49829	21.2	1012	29199
deu-fra	flores101-devtest	0.58079	30.8	1012	28343
eng-spa	flores101-devtest	0.52948	24.2	1012	29199
est-por	flores101-devtest	0.49186	21.5	1012	26519
est-spa	flores101-devtest	7.401	0.0	1012	29199
fas-fra	flores101-devtest	0.50503	22.5	1012	28343
fas-spa	flores101-devtest	0.44071	15.6	1012	29199
fra-spa	flores101-devtest	0.51076	22.4	1012	29199
glg-spa	flores101-devtest	0.52622	23.6	1012	29199
heb-fra	flores101-devtest	0.51604	24.7	1012	28343
heb-por	flores101-devtest	0.50443	23.4	1012	26519
hin-eng	flores101-devtest	0.55255	27.2	1012	24721
hin-fra	flores101-devtest	0.49272	21.3	1012	28343
hrv-deu	flores101-devtest	0.52262	22.1	1012	25094
hun-deu	flores101-devtest	0.48932	18.8	1012	25094
hun-eng	flores101-devtest	0.55055	26.3	1012	24721
hun-fra	flores101-devtest	0.52136	23.9	1012	28343
hun-por	flores101-devtest	0.50885	22.8	1012	26519
hun-spa	flores101-devtest	0.45354	17.3	1012	29199
hye-deu	flores101-devtest	0.41076	12.9	1012	25094
hye-eng	flores101-devtest	0.46030	17.7	1012	24721
ind-deu	flores101-devtest	0.50105	19.8	1012	25094
ind-spa	flores101-devtest	0.47114	18.1	1012	29199
acm-deu	flores200-devtest	0.45619	14.6	1012	25094
acm-eng	flores200-devtest	0.52366	23.3	1012	24721
acm-fra	flores200-devtest	0.49554	21.1	1012	28343
acm-por	flores200-devtest	0.48961	20.0	1012	26519
acm-spa	flores200-devtest	0.44080	15.6	1012	29199
afr-deu	flores200-devtest	0.54906	25.2	1012	25094
afr-eng	flores200-devtest	0.70517	48.9	1012	24721
afr-fra	flores200-devtest	0.58448	32.2	1012	28343
afr-por	flores200-devtest	0.58153	31.6	1012	26519
afr-spa	flores200-devtest	0.48878	20.3	1012	29199
apc-deu	flores200-devtest	0.43490	13.1	1012	25094
apc-eng	flores200-devtest	0.50478	21.4	1012	24721
apc-fra	flores200-devtest	0.48076	20.1	1012	28343
apc-por	flores200-devtest	0.47020	19.0	1012	26519
apc-spa	flores200-devtest	0.42945	14.8	1012	29199
arz-deu	flores200-devtest	0.42474	11.3	1012	25094
arz-eng	flores200-devtest	0.47931	18.3	1012	24721
arz-fra	flores200-devtest	0.46238	17.4	1012	28343
arz-por	flores200-devtest	0.44887	15.9	1012	26519
arz-spa	flores200-devtest	0.41615	13.2	1012	29199
ast-deu	flores200-devtest	0.48985	19.4	1012	25094
ast-eng	flores200-devtest	0.57515	31.5	1012	24721
ast-fra	flores200-devtest	0.54322	28.3	1012	28343
ast-por	flores200-devtest	0.54963	28.5	1012	26519
ast-spa	flores200-devtest	0.48973	20.6	1012	29199
awa-deu	flores200-devtest	0.43494	12.9	1012	25094
awa-eng	flores200-devtest	0.51602	21.6	1012	24721
awa-fra	flores200-devtest	0.46758	18.5	1012	28343
awa-por	flores200-devtest	0.45451	16.7	1012	26519
awa-spa	flores200-devtest	0.41150	13.1	1012	29199
bak-eng	flores200-devtest	0.40644	13.0	1012	24721
ban-eng	flores200-devtest	0.42334	15.8	1012	24721
bel-deu	flores200-devtest	0.43902	10.4	1012	25094
bel-eng	flores200-devtest	0.46940	13.8	1012	24721
bel-fra	flores200-devtest	0.46990	14.9	1012	28343
bel-por	flores200-devtest	0.45316	13.5	1012	26519
bel-spa	flores200-devtest	0.42921	12.5	1012	29199
ben-deu	flores200-devtest	0.42924	12.9	1012	25094
ben-eng	flores200-devtest	0.50522	19.7	1012	24721
ben-fra	flores200-devtest	0.46063	17.9	1012	28343
ben-por	flores200-devtest	0.45038	16.8	1012	26519
ben-spa	flores200-devtest	0.40539	12.9	1012	29199
bho-eng	flores200-devtest	0.45196	15.3	1012	24721
bho-fra	flores200-devtest	0.40604	13.0	1012	28343
bho-por	flores200-devtest	0.40225	12.3	1012	26519
bul-deu	flores200-devtest	0.54127	23.5	1012	25094
bul-eng	flores200-devtest	0.62424	34.4	1012	24721
bul-fra	flores200-devtest	0.58328	30.7	1012	28343
bul-por	flores200-devtest	5.337	0.0	1012	26519
bul-spa	flores200-devtest	0.49353	20.4	1012	29199
cat-deu	flores200-devtest	0.54433	24.6	1012	25094
cat-eng	flores200-devtest	0.65595	39.7	1012	24721
cat-fra	flores200-devtest	0.60898	35.2	1012	28343
cat-por	flores200-devtest	0.61141	35.1	1012	26519
cat-spa	flores200-devtest	0.52769	24.0	1012	29199
ceb-deu	flores200-devtest	0.47268	18.2	1012	25094
ceb-eng	flores200-devtest	0.60119	35.0	1012	24721
ceb-fra	flores200-devtest	0.50664	24.1	1012	28343
ceb-por	flores200-devtest	0.51634	25.5	1012	26519
ceb-spa	flores200-devtest	0.45011	17.6	1012	29199
ces-deu	flores200-devtest	0.54106	24.0	1012	25094
ces-eng	flores200-devtest	0.61006	33.3	1012	24721
ces-fra	flores200-devtest	0.57112	29.8	1012	28343
ces-por	flores200-devtest	0.55542	27.3	1012	26519
ces-spa	flores200-devtest	0.48656	19.7	1012	29199
crh-eng	flores200-devtest	0.43370	15.6	1012	24721
crh-fra	flores200-devtest	0.42209	15.5	1012	28343
crh-por	flores200-devtest	0.41198	14.3	1012	26519
cym-deu	flores200-devtest	0.43731	15.2	1012	25094
cym-eng	flores200-devtest	0.57737	33.6	1012	24721
cym-fra	flores200-devtest	0.48258	22.0	1012	28343
cym-por	flores200-devtest	0.46295	20.3	1012	26519
cym-spa	flores200-devtest	0.41890	15.0	1012	29199
dan-deu	flores200-devtest	0.57400	27.6	1012	25094
dan-eng	flores200-devtest	0.68190	43.4	1012	24721
dan-fra	flores200-devtest	0.60529	34.2	1012	28343
dan-por	flores200-devtest	0.59291	32.6	1012	26519
dan-spa	flores200-devtest	0.50557	21.9	1012	29199
deu-eng	flores200-devtest	0.63939	37.5	1012	24721
deu-fra	flores200-devtest	0.59224	32.1	1012	28343
deu-por	flores200-devtest	0.57294	29.9	1012	26519
deu-spa	flores200-devtest	0.49911	21.2	1012	29199
ell-deu	flores200-devtest	0.49105	19.1	1012	25094
ell-eng	flores200-devtest	0.56033	27.8	1012	24721
ell-fra	flores200-devtest	0.53870	26.6	1012	28343
ell-por	flores200-devtest	0.52260	24.4	1012	26519
ell-spa	flores200-devtest	0.46740	18.7	1012	29199
eng-deu	flores200-devtest	0.60520	32.4	1012	25094
eng-fra	flores200-devtest	0.66872	43.5	1012	28343
eng-por	flores200-devtest	0.67743	44.1	1012	26519
eng-spa	flores200-devtest	0.53557	24.9	1012	29199
epo-deu	flores200-devtest	0.53318	24.2	1012	25094
epo-eng	flores200-devtest	0.62715	37.7	1012	24721
epo-fra	flores200-devtest	0.57884	31.6	1012	28343
epo-por	flores200-devtest	0.56714	30.2	1012	26519
epo-spa	flores200-devtest	0.49035	20.5	1012	29199
est-deu	flores200-devtest	0.50626	20.4	1012	25094
est-eng	flores200-devtest	0.57279	29.3	1012	24721
est-fra	flores200-devtest	0.53780	26.3	1012	28343
est-por	flores200-devtest	0.51310	23.4	1012	26519
est-spa	flores200-devtest	0.46627	18.2	1012	29199
eus-deu	flores200-devtest	0.44830	14.7	1012	25094
eus-eng	flores200-devtest	0.51096	22.0	1012	24721
eus-fra	flores200-devtest	0.49651	21.8	1012	28343
eus-por	flores200-devtest	0.48740	20.7	1012	26519
eus-spa	flores200-devtest	0.44865	16.5	1012	29199
fao-eng	flores200-devtest	0.43250	19.1	1012	24721
fin-deu	flores200-devtest	0.49527	19.2	1012	25094
fin-eng	flores200-devtest	0.55423	27.0	1012	24721
fin-fra	flores200-devtest	0.52835	25.0	1012	28343
fin-por	flores200-devtest	0.51183	23.0	1012	26519
fin-spa	flores200-devtest	0.45946	17.3	1012	29199
fra-deu	flores200-devtest	0.55138	24.9	1012	25094
fra-eng	flores200-devtest	0.64752	39.0	1012	24721
fra-por	flores200-devtest	0.60213	33.6	1012	26519
fra-spa	flores200-devtest	0.51450	22.8	1012	29199
fur-deu	flores200-devtest	0.45446	14.7	1012	25094
fur-eng	flores200-devtest	0.53564	25.1	1012	24721
fur-fra	flores200-devtest	0.51427	23.1	1012	28343
fur-por	flores200-devtest	0.50812	22.2	1012	26519
fur-spa	flores200-devtest	0.44720	16.2	1012	29199
gle-deu	flores200-devtest	0.42946	12.9	1012	25094
gle-eng	flores200-devtest	0.51262	23.5	1012	24721
gle-fra	flores200-devtest	0.46495	19.1	1012	28343
gle-por	flores200-devtest	0.45996	17.9	1012	26519
gle-spa	flores200-devtest	0.41473	14.0	1012	29199
glg-deu	flores200-devtest	0.53798	23.1	1012	25094
glg-eng	flores200-devtest	0.63363	35.7	1012	24721
glg-fra	flores200-devtest	0.60004	33.4	1012	28343
glg-por	flores200-devtest	0.59372	31.7	1012	26519
glg-spa	flores200-devtest	0.52788	23.8	1012	29199
guj-eng	flores200-devtest	0.47897	17.8	1012	24721
guj-fra	flores200-devtest	0.40053	13.7	1012	28343
hat-deu	flores200-devtest	0.46100	16.4	1012	25094
hat-eng	flores200-devtest	0.55513	28.4	1012	24721
hat-fra	flores200-devtest	0.57504	30.2	1012	28343
hat-por	flores200-devtest	0.51068	23.4	1012	26519
hat-spa	flores200-devtest	0.44874	15.7	1012	29199
heb-deu	flores200-devtest	0.49571	20.0	1012	25094
heb-eng	flores200-devtest	0.57539	31.5	1012	24721
heb-fra	flores200-devtest	0.53710	27.4	1012	28343
heb-por	flores200-devtest	0.52607	25.6	1012	26519
heb-spa	flores200-devtest	0.45847	18.0	1012	29199
hin-deu	flores200-devtest	0.47254	17.3	1012	25094
hin-eng	flores200-devtest	0.57180	29.2	1012	24721
hin-fra	flores200-devtest	0.50629	22.7	1012	28343
hin-por	flores200-devtest	0.50034	22.1	1012	26519
hin-spa	flores200-devtest	0.43766	15.4	1012	29199
hne-deu	flores200-devtest	0.43825	13.4	1012	25094
hne-eng	flores200-devtest	0.54087	25.1	1012	24721
hne-fra	flores200-devtest	0.47247	19.6	1012	28343
hne-por	flores200-devtest	0.46857	18.8	1012	26519
hne-spa	flores200-devtest	0.41174	13.3	1012	29199
hrv-deu	flores200-devtest	0.53335	23.4	1012	25094
hrv-eng	flores200-devtest	0.59962	32.5	1012	24721
hrv-fra	flores200-devtest	0.57021	29.8	1012	28343
hrv-por	flores200-devtest	0.55294	27.7	1012	26519
hrv-spa	flores200-devtest	0.48816	20.1	1012	29199
hun-deu	flores200-devtest	0.50156	19.7	1012	25094
hun-eng	flores200-devtest	0.56407	27.8	1012	24721
hun-fra	flores200-devtest	0.53635	25.6	1012	28343
hun-por	flores200-devtest	0.51983	24.0	1012	26519
hun-spa	flores200-devtest	0.46453	18.1	1012	29199
hye-deu	flores200-devtest	0.43223	14.1	1012	25094
hye-eng	flores200-devtest	0.48108	18.4	1012	24721
hye-fra	flores200-devtest	0.46422	19.7	1012	28343
hye-por	flores200-devtest	0.45328	17.7	1012	26519
hye-spa	flores200-devtest	0.40969	14.4	1012	29199
ilo-deu	flores200-devtest	0.43664	14.9	1012	25094
ilo-eng	flores200-devtest	0.54456	28.3	1012	24721
ilo-fra	flores200-devtest	0.46765	20.0	1012	28343
ilo-por	flores200-devtest	0.43883	18.4	1012	26519
ilo-spa	flores200-devtest	0.41178	14.8	1012	29199
ind-deu	flores200-devtest	0.51349	21.0	1012	25094
ind-eng	flores200-devtest	0.63177	36.0	1012	24721
ind-fra	flores200-devtest	0.55816	27.9	1012	28343
ind-por	flores200-devtest	0.55437	27.4	1012	26519
ind-spa	flores200-devtest	0.47702	18.5	1012	29199
isl-deu	flores200-devtest	0.44626	17.1	1012	25094
isl-eng	flores200-devtest	0.50201	24.4	1012	24721
isl-fra	flores200-devtest	0.47250	20.9	1012	28343
isl-por	flores200-devtest	0.45749	19.6	1012	26519
isl-spa	flores200-devtest	0.41715	15.0	1012	29199
ita-deu	flores200-devtest	0.52234	20.3	1012	25094
ita-eng	flores200-devtest	0.59001	28.7	1012	24721
ita-fra	flores200-devtest	0.58465	29.6	1012	28343
ita-por	flores200-devtest	0.56491	26.5	1012	26519
ita-spa	flores200-devtest	0.51136	21.8	1012	29199
jav-deu	flores200-devtest	0.41271	13.2	1012	25094
jav-eng	flores200-devtest	0.51285	25.7	1012	24721
jav-fra	flores200-devtest	0.44758	18.2	1012	28343
jav-por	flores200-devtest	0.44290	18.2	1012	26519
jav-spa	flores200-devtest	0.40022	13.3	1012	29199
jpn-deu	flores200-devtest	0.40420	9.8	1012	25094
jpn-eng	flores200-devtest	0.44981	14.7	1012	24721
jpn-fra	flores200-devtest	0.42825	14.5	1012	28343
jpn-por	flores200-devtest	0.41558	13.1	1012	26519
kan-eng	flores200-devtest	0.42126	12.6	1012	24721
kat-eng	flores200-devtest	0.41805	11.0	1012	24721
kat-fra	flores200-devtest	0.41761	13.8	1012	28343
kat-por	flores200-devtest	0.40563	11.6	1012	26519
kaz-deu	flores200-devtest	0.40259	11.5	1012	25094
kaz-eng	flores200-devtest	0.47058	18.6	1012	24721
kaz-fra	flores200-devtest	0.43576	17.0	1012	28343
kaz-por	flores200-devtest	0.42895	16.0	1012	26519
kaz-spa	flores200-devtest	0.40349	13.2	1012	29199
kea-deu	flores200-devtest	0.44701	15.5	1012	25094
kea-eng	flores200-devtest	0.55673	29.0	1012	24721
kea-fra	flores200-devtest	0.50910	23.0	1012	28343
kea-por	flores200-devtest	0.54023	25.9	1012	26519
kea-spa	flores200-devtest	0.44788	16.5	1012	29199
lij-deu	flores200-devtest	0.45413	14.9	1012	25094
lij-eng	flores200-devtest	0.54269	25.4	1012	24721
lij-fra	flores200-devtest	0.51381	23.4	1012	28343
lij-por	flores200-devtest	0.50523	22.0	1012	26519
lij-spa	flores200-devtest	0.44284	16.3	1012	29199
lim-deu	flores200-devtest	0.42001	12.5	1012	25094
lim-eng	flores200-devtest	0.47493	20.5	1012	24721
lim-fra	flores200-devtest	0.41951	15.8	1012	28343
lim-por	flores200-devtest	0.41365	15.2	1012	26519
mag-fra	flores200-devtest	0.47212	19.2	1012	28343
mai-eng	flores200-devtest	2.920	0.0	1012	24721
mal-eng	flores200-devtest	0.41587	11.2	1012	24721
mar-deu	flores200-devtest	0.41360	12.0	1012	25094
mar-eng	flores200-devtest	0.49778	20.4	1012	24721
mar-fra	flores200-devtest	0.44433	16.4	1012	28343
mar-por	flores200-devtest	0.41099	14.1	1012	26519
mkd-deu	flores200-devtest	0.53803	24.4	1012	25094
mkd-eng	flores200-devtest	0.62358	35.5	1012	24721
mkd-fra	flores200-devtest	0.57363	30.7	1012	28343
mkd-por	flores200-devtest	0.56317	29.3	1012	26519
mkd-spa	flores200-devtest	0.49183	20.7	1012	29199
mlt-deu	flores200-devtest	0.53283	23.4	1012	25094
mlt-eng	flores200-devtest	0.68020	43.0	1012	24721
mlt-fra	flores200-devtest	0.59058	32.9	1012	28343
mlt-por	flores200-devtest	0.58768	31.4	1012	26519
mlt-spa	flores200-devtest	0.49482	20.7	1012	29199
nld-eng	flores200-devtest	0.56888	27.9	1012	24721
nno-eng	flores200-devtest	0.65106	39.8	1012	24721
nno-fra	flores200-devtest	0.57584	31.3	1012	28343
nno-por	flores200-devtest	4.158	0.0	1012	26519
nob-eng	flores200-devtest	0.64640	39.4	1012	24721
nob-por	flores200-devtest	0.56479	28.9	1012	26519
nob-spa	flores200-devtest	0.48859	19.8	1012	29199
npi-deu	flores200-devtest	0.41007	11.8	1012	25094
npi-eng	flores200-devtest	0.49418	20.6	1012	24721
nso-eng	flores200-devtest	0.40320	17.2	1012	24721
oci-deu	flores200-devtest	0.53449	23.1	1012	25094
oci-spa	flores200-devtest	0.50129	20.2	1012	29199
pan-eng	flores200-devtest	0.44813	15.1	1012	24721
pan-fra	flores200-devtest	0.40807	14.4	1012	28343
pap-deu	flores200-devtest	0.49116	19.3	1012	25094
pap-eng	flores200-devtest	0.63771	37.6	1012	24721
pap-fra	flores200-devtest	0.54728	27.3	1012	28343
pap-por	flores200-devtest	0.55832	29.0	1012	26519
pap-spa	flores200-devtest	0.47334	19.0	1012	29199
pes-deu	flores200-devtest	0.47708	17.9	1012	25094
pes-eng	flores200-devtest	0.55400	27.4	1012	24721
pes-fra	flores200-devtest	0.51858	24.3	1012	28343
pes-por	flores200-devtest	0.50867	22.8	1012	26519
pes-spa	flores200-devtest	0.44988	16.3	1012	29199
plt-eng	flores200-devtest	0.42666	16.5	1012	24721
plt-fra	flores200-devtest	0.40604	15.3	1012	28343
pol-deu	flores200-devtest	0.48933	17.7	1012	25094
pol-eng	flores200-devtest	0.53490	23.3	1012	24721
pol-fra	flores200-devtest	0.53036	24.5	1012	28343
pol-por	flores200-devtest	0.50576	21.8	1012	26519
pol-spa	flores200-devtest	0.46596	17.7	1012	29199
por-deu	flores200-devtest	0.55612	25.9	1012	25094
por-eng	flores200-devtest	0.68137	43.6	1012	24721
por-fra	flores200-devtest	0.62509	37.3	1012	28343
por-spa	flores200-devtest	0.52540	24.1	1012	29199
prs-deu	flores200-devtest	0.46660	17.6	1012	25094
prs-eng	flores200-devtest	0.54932	28.1	1012	24721
prs-fra	flores200-devtest	0.50825	23.5	1012	28343
prs-por	flores200-devtest	0.50179	23.3	1012	26519
prs-spa	flores200-devtest	0.44000	16.1	1012	29199
ron-deu	flores200-devtest	0.54879	24.7	1012	25094
ron-eng	flores200-devtest	0.64557	37.2	1012	24721
ron-fra	flores200-devtest	0.61341	34.7	1012	28343
ron-por	flores200-devtest	0.60040	32.8	1012	26519
ron-spa	flores200-devtest	0.51475	22.8	1012	29199
rus-deu	flores200-devtest	0.51905	20.8	1012	25094
rus-eng	flores200-devtest	0.57662	28.8	1012	24721
rus-fra	flores200-devtest	0.55678	27.9	1012	28343
rus-por	flores200-devtest	0.53912	25.5	1012	26519
rus-spa	flores200-devtest	0.47828	19.2	1012	29199
sat-deu	flores200-devtest	6.870	0.0	1012	25094
slv-eng	flores200-devtest	0.57057	28.2	1012	24721
slv-fra	flores200-devtest	0.55067	27.4	1012	28343
slv-por	flores200-devtest	0.53086	25.1	1012	26519
slv-spa	flores200-devtest	0.47518	18.7	1012	29199
ces-eng	generaltest2022	0.60672	34.6	1448	30675
deu-eng	generaltest2022	0.52754	26.9	1984	37634
deu-fra	generaltest2022	0.55856	30.8	1984	38276
eng-deu	generaltest2022	0.57350	28.6	2037	38914
fra-deu	generaltest2022	0.60353	34.7	2006	37696
rus-eng	generaltest2022	0.60801	34.6	2016	38529
ces-deu	multi30k_test_2016_flickr	0.51274	20.8	1000	12106
ces-eng	multi30k_test_2016_flickr	0.52823	27.9	1000	12955
ces-fra	multi30k_test_2016_flickr	0.52744	24.4	1000	13505
deu-eng	multi30k_test_2016_flickr	0.56595	35.4	1000	12955
deu-fra	multi30k_test_2016_flickr	0.57493	31.5	1000	13505
eng-deu	multi30k_test_2016_flickr	0.59379	29.4	1000	12106
eng-fra	multi30k_test_2016_flickr	0.64500	40.5	1000	13505
fra-deu	multi30k_test_2016_flickr	0.56056	26.2	1000	12106
fra-eng	multi30k_test_2016_flickr	0.60673	40.2	1000	12955
deu-eng	multi30k_test_2017_flickr	0.58069	34.8	1000	11374
deu-fra	multi30k_test_2017_flickr	0.56846	30.1	1000	12118
eng-deu	multi30k_test_2017_flickr	0.58669	29.4	1000	10755
eng-fra	multi30k_test_2017_flickr	0.65169	42.2	1000	12118
fra-deu	multi30k_test_2017_flickr	0.54803	24.4	1000	10755
fra-eng	multi30k_test_2017_flickr	0.63521	43.2	1000	11374
deu-eng	multi30k_test_2017_mscoco	0.50914	27.9	461	5231
deu-fra	multi30k_test_2017_mscoco	0.54647	28.3	461	5484
eng-deu	multi30k_test_2017_mscoco	0.53295	24.1	461	5158
eng-fra	multi30k_test_2017_mscoco	0.67467	46.2	461	5484
fra-deu	multi30k_test_2017_mscoco	0.51160	21.8	461	5158
fra-eng	multi30k_test_2017_mscoco	0.61818	41.7	461	5231
ces-deu	multi30k_test_2018_flickr	0.47659	18.5	1071	13703
ces-eng	multi30k_test_2018_flickr	0.52268	27.2	1071	14689
ces-fra	multi30k_test_2018_flickr	0.49188	21.8	1071	15867
deu-eng	multi30k_test_2018_flickr	0.54633	31.1	1071	14689
deu-fra	multi30k_test_2018_flickr	0.52709	25.7	1071	15867
eng-deu	multi30k_test_2018_flickr	0.55437	25.8	1071	13703
eng-fra	multi30k_test_2018_flickr	0.60698	35.3	1071	15867
fra-deu	multi30k_test_2018_flickr	0.51119	20.4	1071	13703
fra-eng	multi30k_test_2018_flickr	0.59774	37.8	1071	14689
eng-fra	newsdiscusstest2015	0.60132	33.5	1500	27975
fra-eng	newsdiscusstest2015	0.58599	35.3	1500	26982
ces-deu	newssyscomb2009	0.50450	19.5	502	11271
ces-eng	newssyscomb2009	0.53905	26.4	502	11818
ces-fra	newssyscomb2009	0.54939	26.1	502	12331
ces-spa	newssyscomb2009	0.54720	27.3	502	12503
deu-eng	newssyscomb2009	0.53558	26.9	502	11818
deu-fra	newssyscomb2009	0.53583	24.5	502	12331
deu-spa	newssyscomb2009	0.53401	25.7	502	12503
eng-deu	newssyscomb2009	0.51555	20.1	502	11271
eng-fra	newssyscomb2009	0.56789	27.8	502	12331
eng-spa	newssyscomb2009	0.56968	29.8	502	12503
fra-deu	newssyscomb2009	0.50943	20.6	502	11271
fra-eng	newssyscomb2009	0.55832	29.2	502	11818
fra-spa	newssyscomb2009	0.58793	32.7	502	12503
hun-deu	newssyscomb2009	0.46007	14.7	502	11271
hun-eng	newssyscomb2009	0.48677	20.4	502	11818
hun-fra	newssyscomb2009	0.48681	18.9	502	12331
hun-spa	newssyscomb2009	0.49098	20.9	502	12503
ita-deu	newssyscomb2009	0.51720	20.8	502	11271
ita-eng	newssyscomb2009	0.57787	31.8	502	11818
ita-fra	newssyscomb2009	0.59854	31.5	502	12331
ita-spa	newssyscomb2009	0.59911	34.1	502	12503
ces-deu	newstest2008	0.50184	19.2	2051	47447
ces-eng	newstest2008	0.50980	22.6	2051	49380
ces-fra	newstest2008	0.52405	23.4	2051	52685
ces-spa	newstest2008	0.52270	24.3	2051	52586
deu-eng	newstest2008	0.52210	24.4	2051	49380
deu-fra	newstest2008	0.52589	23.9	2051	52685
deu-spa	newstest2008	0.51902	23.8	2051	52586
eng-deu	newstest2008	0.50792	20.7	2051	47447
eng-fra	newstest2008	0.53657	24.8	2051	52685
eng-spa	newstest2008	0.54883	27.6	2051	52586
fra-deu	newstest2008	0.50380	20.0	2051	47447
fra-eng	newstest2008	0.52690	24.6	2051	49380
fra-spa	newstest2008	0.56865	30.9	2051	52586
hun-deu	newstest2008	0.45945	15.1	2051	47447
hun-eng	newstest2008	0.46583	17.4	2051	49380
hun-fra	newstest2008	0.47705	18.5	2051	52685
hun-spa	newstest2008	0.47837	19.5	2051	52586
ces-deu	newstest2009	0.50212	20.1	2525	62816
ces-eng	newstest2009	0.52672	25.0	2525	65399
ces-fra	newstest2009	0.52509	23.6	2525	69263
ces-spa	newstest2009	0.52698	24.5	2525	68111
deu-eng	newstest2009	0.51657	24.0	2525	65399
deu-fra	newstest2009	0.51700	23.2	2525	69263
deu-spa	newstest2009	0.51476	23.3	2525	68111
eng-deu	newstest2009	0.50867	19.7	2525	62816
eng-fra	newstest2009	0.55505	27.0	2525	69263
eng-spa	newstest2009	0.56130	28.0	2525	68111
fra-deu	newstest2009	0.50213	19.9	2525	62816
fra-eng	newstest2009	0.54762	27.9	2525	65399
fra-spa	newstest2009	0.57098	30.2	2525	68111
hun-deu	newstest2009	0.45859	15.0	2525	62816
hun-eng	newstest2009	0.47458	19.1	2525	65399
hun-fra	newstest2009	0.47259	18.4	2525	69263
hun-spa	newstest2009	0.47918	19.5	2525	68111
ita-deu	newstest2009	0.50313	19.9	2525	62816
ita-eng	newstest2009	0.57010	30.9	2525	65399
ita-fra	newstest2009	0.57497	29.8	2525	69263
ita-spa	newstest2009	0.58142	31.8	2525	68111
ces-deu	newstest2010	0.51231	20.9	2489	61503
ces-eng	newstest2010	0.54084	26.2	2489	61711
ces-fra	newstest2010	0.53352	24.8	2489	66022
ces-spa	newstest2010	0.55395	28.3	2489	65480
deu-eng	newstest2010	0.55776	27.8	2489	61711
deu-fra	newstest2010	0.55086	26.4	2489	66022
deu-spa	newstest2010	0.56549	29.3	2489	65480
eng-deu	newstest2010	0.52287	22.8	2489	61503
eng-fra	newstest2010	0.57485	29.9	2489	66022
eng-spa	newstest2010	0.60005	33.8	2489	65480
fra-deu	newstest2010	0.51053	20.9	2489	61503
fra-eng	newstest2010	0.56909	29.5	2489	61711
fra-spa	newstest2010	0.60389	34.8	2489	65480
ces-deu	newstest2011	0.49767	20.0	3003	72981
ces-eng	newstest2011	0.53124	25.2	3003	74681
ces-fra	newstest2011	0.53716	26.0	3003	80626
ces-spa	newstest2011	0.54005	27.9	3003	79476
deu-eng	newstest2011	0.52573	24.1	3003	74681
deu-fra	newstest2011	0.53006	24.4	3003	80626
deu-spa	newstest2011	0.53884	27.2	3003	79476
eng-deu	newstest2011	0.50516	20.6	3003	72981
eng-fra	newstest2011	0.58992	31.6	3003	80626
eng-spa	newstest2011	0.60182	36.0	3003	79476
fra-deu	newstest2011	0.50177	20.3	3003	72981
fra-eng	newstest2011	0.57307	30.2	3003	74681
fra-spa	newstest2011	0.60832	36.8	3003	79476
ces-deu	newstest2012	0.49618	20.2	3003	72886
ces-eng	newstest2012	0.52837	24.8	3003	72812
ces-fra	newstest2012	0.53314	24.9	3003	78011
ces-spa	newstest2012	0.53704	27.1	3003	79006
deu-eng	newstest2012	0.53682	25.7	3003	72812
deu-fra	newstest2012	0.53422	24.9	3003	78011
deu-spa	newstest2012	0.54192	27.9	3003	79006
eng-deu	newstest2012	0.50517	21.1	3003	72886
eng-fra	newstest2012	0.57322	29.6	3003	78011
eng-spa	newstest2012	0.60322	35.9	3003	79006
fra-deu	newstest2012	0.49972	20.8	3003	72886
fra-eng	newstest2012	0.57028	30.2	3003	72812
fra-spa	newstest2012	0.60309	36.2	3003	79006
rus-deu	newstest2012	0.45798	16.1	3003	72886
rus-eng	newstest2012	0.58723	32.9	3003	72812
rus-fra	newstest2012	0.50740	22.2	3003	78011
rus-spa	newstest2012	0.51641	25.1	3003	79006
ces-deu	newstest2013	0.51978	22.6	3000	63737
ces-eng	newstest2013	0.55207	28.2	3000	64505
ces-fra	newstest2013	0.53236	26.0	3000	70037
ces-spa	newstest2013	0.54234	28.1	3000	70528
deu-eng	newstest2013	0.55601	28.7	3000	64505
deu-fra	newstest2013	0.53764	27.1	3000	70037
deu-spa	newstest2013	0.54225	28.4	3000	70528
eng-deu	newstest2013	0.53039	24.6	3000	63737
eng-fra	newstest2013	0.56938	30.9	3000	70037
eng-spa	newstest2013	0.57944	32.7	3000	70528
fra-deu	newstest2013	0.51437	22.1	3000	63737
fra-eng	newstest2013	0.57158	31.4	3000	64505
fra-spa	newstest2013	0.57480	32.5	3000	70528
rus-deu	newstest2013	0.49914	20.5	3000	63737
rus-eng	newstest2013	0.53711	26.6	3000	64505
rus-fra	newstest2013	0.52616	25.2	3000	70037
rus-spa	newstest2013	0.53637	27.8	3000	70528
ces-eng	newstest2014	0.58425	30.3	3003	68065
deu-eng	newstest2014	0.56105	28.8	3003	67337
eng-deu	newstest2014	0.54995	23.6	3003	62688
eng-fra	newstest2014	0.62054	35.7	3003	77306
fra-eng	newstest2014	0.60603	33.8	3003	70708
hin-eng	newstest2014	0.52740	22.1	2507	55571
rus-eng	newstest2014	0.59871	32.9	3003	69210
ces-eng	newstest2015	0.53850	27.5	2656	53569
deu-eng	newstest2015	0.56659	30.2	2169	46443
eng-deu	newstest2015	0.56263	27.3	2169	44260
fin-eng	newstest2015	0.50789	22.9	1370	27270
rus-eng	newstest2015	0.56533	29.7	2818	64428
ces-eng	newstest2016	0.56310	29.3	2999	64670
deu-eng	newstest2016	0.61288	35.7	2999	64119
eng-deu	newstest2016	0.60235	32.9	2999	62669
fin-eng	newstest2016	0.53305	25.4	3000	62945
ron-eng	newstest2016	0.60502	34.6	1999	47562
rus-eng	newstest2016	0.56287	29.4	2998	69278
ces-eng	newstest2017	0.53171	25.8	3005	61721
deu-eng	newstest2017	0.57941	31.8	3004	64399
eng-deu	newstest2017	0.55674	26.1	3004	61287
fin-eng	newstest2017	0.54579	27.1	3002	61846
lav-eng	newstest2017	0.49180	19.0	2001	47511
rus-eng	newstest2017	0.59103	33.2	3001	69025
ces-eng	newstest2018	0.54210	26.7	2983	63495
deu-eng	newstest2018	0.62725	38.0	2998	67012
eng-deu	newstest2018	0.63963	38.2	2998	64276
est-eng	newstest2018	0.53620	25.7	2000	45405
fin-eng	newstest2018	0.48070	20.0	3000	62325
rus-eng	newstest2018	0.55683	28.4	3000	71291
ces-deu	newstest2019	0.51243	21.3	1997	48746
deu-eng	newstest2019	0.59331	34.2	2000	39227
deu-fra	newstest2019	0.59123	31.1	1701	42509
eng-deu	newstest2019	0.61248	35.4	1997	48746
fin-eng	newstest2019	0.51773	24.3	1996	36215
fra-deu	newstest2019	0.56890	24.2	1701	36446
guj-eng	newstest2019	0.45948	17.1	1016	17757
kaz-eng	newstest2019	0.42380	15.5	1000	20452
rus-eng	newstest2019	0.59517	34.0	2000	42642
deu-eng	newstest2020	0.52067	26.3	785	38220
deu-fra	newstest2020	0.57389	29.4	1619	36890
eng-deu	newstest2020	0.54791	26.0	1418	52383
fra-deu	newstest2020	0.57243	23.9	1619	30265
pol-eng	newstest2020	0.54194	25.4	1001	21755
rus-eng	newstest2020	0.59454	30.1	991	20217
ces-eng	newstest2021	0.53322	23.8	1000	22056
deu-eng	newstest2021	0.56154	28.2	1000	20180
deu-fra	newstest2021	0.55568	27.8	1000	23757
eng-deu	newstest2021	0.53503	22.9	1002	27970
fra-deu	newstest2021	0.59753	31.7	1026	26077
isl-eng	newstest2021	0.49479	25.0	1000	22529
rus-eng	newstest2021	0.58195	31.1	1000	21228
deu-eng	newstestALL2020	0.52067	26.3	785	38220
eng-deu	newstestALL2020	0.54791	26.0	1418	52383
rus-eng	newstestALL2020	0.59454	30.1	991	20217
fin-eng	newstestB2017	0.51538	23.6	3002	61846
deu-eng	newstestB2020	0.51904	26.0	785	37696
eng-deu	newstestB2020	0.53738	25.4	1418	53092
rus-eng	newstestB2020	0.59950	31.6	991	20423
afr-deu	ntrex128	0.52108	22.8	1997	48761
afr-eng	ntrex128	0.68008	45.8	1997	47673
afr-fra	ntrex128	0.52929	25.0	1997	53481
afr-por	ntrex128	0.52564	25.3	1997	51631
afr-spa	ntrex128	0.54997	29.4	1997	54107
aze_Latn-deu	ntrex128	0.41299	11.9	1997	48761
aze_Latn-eng	ntrex128	0.47426	17.6	1997	47673
aze_Latn-fra	ntrex128	0.43101	15.3	1997	53481
aze_Latn-por	ntrex128	0.42046	13.6	1997	51631
aze_Latn-spa	ntrex128	0.44574	17.0	1997	54107
bel-deu	ntrex128	0.44908	15.1	1997	48761
bel-eng	ntrex128	0.51751	23.4	1997	47673
bel-fra	ntrex128	0.46436	18.2	1997	53481
bel-por	ntrex128	0.46030	17.7	1997	51631
bel-spa	ntrex128	0.47946	20.8	1997	54107
ben-deu	ntrex128	0.40415	10.8	1997	48761
ben-eng	ntrex128	0.48275	19.5	1997	47673
ben-fra	ntrex128	0.41637	14.2	1997	53481
ben-por	ntrex128	0.40736	13.3	1997	51631
ben-spa	ntrex128	0.43341	16.3	1997	54107
bul-deu	ntrex128	0.49598	18.7	1997	48761
bul-eng	ntrex128	0.56767	27.9	1997	47673
bul-fra	ntrex128	0.51709	23.2	1997	53481
bul-por	ntrex128	0.50800	22.3	1997	51631
bul-spa	ntrex128	0.53009	26.3	1997	54107
cat-deu	ntrex128	0.50784	20.6	1997	48761
cat-eng	ntrex128	0.59903	31.7	1997	47673
cat-fra	ntrex128	0.54239	25.9	1997	53481
cat-por	ntrex128	0.55184	27.4	1997	51631
cat-spa	ntrex128	0.60628	34.9	1997	54107
ces-deu	ntrex128	0.50833	20.4	1997	48761
ces-eng	ntrex128	0.58185	29.9	1997	47673
ces-fra	ntrex128	0.52110	23.9	1997	53481
ces-por	ntrex128	0.50726	22.2	1997	51631
ces-spa	ntrex128	0.53301	26.2	1997	54107
cym-deu	ntrex128	0.41859	14.4	1997	48761
cym-eng	ntrex128	0.52770	26.7	1997	47673
cym-fra	ntrex128	0.44433	18.5	1997	53481
cym-por	ntrex128	0.42658	16.4	1997	51631
cym-spa	ntrex128	0.45942	21.0	1997	54107
dan-deu	ntrex128	0.51512	22.0	1997	48761
dan-eng	ntrex128	0.60259	34.9	1997	47673
dan-fra	ntrex128	0.51370	23.6	1997	53481
dan-por	ntrex128	0.51423	24.6	1997	51631
dan-spa	ntrex128	0.53528	27.6	1997	54107
deu-eng	ntrex128	0.58787	30.4	1997	47673
deu-fra	ntrex128	0.52764	25.0	1997	53481
deu-por	ntrex128	0.52135	23.6	1997	51631
deu-spa	ntrex128	0.54183	27.6	1997	54107
ell-deu	ntrex128	0.48157	18.4	1997	48761
ell-eng	ntrex128	0.56388	29.6	1997	47673
ell-fra	ntrex128	0.50285	22.3	1997	53481
ell-por	ntrex128	0.50226	23.5	1997	51631
ell-spa	ntrex128	0.52469	26.7	1997	54107
eng-deu	ntrex128	0.55255	26.3	1997	48761
eng-fra	ntrex128	0.58111	31.5	1997	53481
eng-por	ntrex128	0.57731	31.0	1997	51631
eng-spa	ntrex128	0.60903	36.0	1997	54107
est-deu	ntrex128	0.47614	17.7	1997	48761
est-eng	ntrex128	0.54290	25.3	1997	47673
est-fra	ntrex128	0.49258	21.4	1997	53481
est-por	ntrex128	0.47893	19.4	1997	51631
est-spa	ntrex128	0.50947	23.7	1997	54107
eus-deu	ntrex128	0.42792	13.4	1997	48761
eus-eng	ntrex128	0.49242	20.7	1997	47673
eus-fra	ntrex128	0.45518	17.8	1997	53481
eus-por	ntrex128	0.45450	17.7	1997	51631
eus-spa	ntrex128	0.48737	21.7	1997	54107
fao-eng	ntrex128	0.45401	21.6	1997	47673
fao-spa	ntrex128	0.40129	16.2	1997	54107
fas-deu	ntrex128	0.44078	13.4	1997	48761
fas-eng	ntrex128	0.50749	20.9	1997	47673
fas-fra	ntrex128	0.46234	17.7	1997	53481
fas-por	ntrex128	0.44990	16.5	1997	51631
fas-spa	ntrex128	0.47415	19.8	1997	54107
fil-deu	ntrex128	0.45874	17.0	1997	48761
fil-eng	ntrex128	0.59882	35.9	1997	47673
fil-fra	ntrex128	0.48281	21.2	1997	53481
fil-por	ntrex128	0.47948	21.3	1997	51631
fil-spa	ntrex128	0.50214	24.7	1997	54107
fin-deu	ntrex128	0.46233	15.8	1997	48761
fin-eng	ntrex128	0.51653	22.3	1997	47673
fin-fra	ntrex128	0.47519	19.3	1997	53481
fin-por	ntrex128	0.47136	18.2	1997	51631
fin-spa	ntrex128	0.49105	21.8	1997	54107
fra-deu	ntrex128	0.49992	20.2	1997	48761
fra-eng	ntrex128	0.58050	30.3	1997	47673
fra-por	ntrex128	0.52964	24.6	1997	51631
fra-spa	ntrex128	0.55325	28.9	1997	54107
gle-deu	ntrex128	0.40356	12.3	1997	48761
gle-eng	ntrex128	0.48545	20.9	1997	47673
gle-fra	ntrex128	0.42065	14.8	1997	53481
gle-por	ntrex128	0.41363	14.2	1997	51631
gle-spa	ntrex128	0.43570	17.2	1997	54107
glg-deu	ntrex128	0.50745	20.3	1997	48761
glg-eng	ntrex128	0.61075	34.2	1997	47673
glg-fra	ntrex128	0.53952	25.8	1997	53481
glg-por	ntrex128	0.55664	28.2	1997	51631
glg-spa	ntrex128	0.61214	36.1	1997	54107
guj-eng	ntrex128	0.44942	15.5	1997	47673
hau-eng	ntrex128	0.40202	16.7	1997	47673
heb-deu	ntrex128	0.45797	16.2	1997	48761
heb-eng	ntrex128	0.53174	24.9	1997	47673
heb-fra	ntrex128	0.47719	20.0	1997	53481
heb-por	ntrex128	0.47161	19.4	1997	51631
heb-spa	ntrex128	0.49348	23.0	1997	54107
hin-deu	ntrex128	0.42959	12.2	1997	48761
hin-eng	ntrex128	0.53396	23.6	1997	47673
hin-fra	ntrex128	0.44793	16.2	1997	53481
hin-por	ntrex128	0.43466	14.7	1997	51631
hin-spa	ntrex128	0.46019	18.6	1997	54107
hrv-deu	ntrex128	0.51370	20.4	1997	48761
hrv-eng	ntrex128	0.59129	30.9	1997	47673
hrv-fra	ntrex128	0.53330	25.5	1997	53481
hrv-por	ntrex128	0.52485	24.2	1997	51631
hrv-spa	ntrex128	0.54681	27.8	1997	54107
hun-deu	ntrex128	0.45105	14.7	1997	48761
hun-eng	ntrex128	0.49083	19.0	1997	47673
hun-fra	ntrex128	0.46131	18.3	1997	53481
hun-por	ntrex128	0.45430	16.9	1997	51631
hun-spa	ntrex128	0.47514	20.5	1997	54107
hye-eng	ntrex128	0.42795	14.1	1997	47673
hye-spa	ntrex128	0.40709	14.7	1997	54107
ind-deu	ntrex128	0.48605	17.9	1997	48761
ind-eng	ntrex128	0.61861	34.6	1997	47673
ind-fra	ntrex128	0.50741	22.5	1997	53481
ind-por	ntrex128	0.50279	22.2	1997	51631
ind-spa	ntrex128	0.52849	26.3	1997	54107
isl-deu	ntrex128	0.44960	16.6	1997	48761
isl-eng	ntrex128	0.50794	23.6	1997	47673
isl-fra	ntrex128	0.46395	19.1	1997	53481
isl-por	ntrex128	0.45094	17.7	1997	51631
isl-spa	ntrex128	0.47760	22.0	1997	54107
ita-deu	ntrex128	0.51269	21.3	1997	48761
ita-eng	ntrex128	0.61055	34.4	1997	47673
ita-fra	ntrex128	0.55388	27.6	1997	53481
ita-por	ntrex128	0.55618	28.0	1997	51631
ita-spa	ntrex128	0.57218	30.9	1997	54107
jpn-eng	ntrex128	0.42510	13.8	1997	47673
kat-eng	ntrex128	0.40279	12.7	1997	47673
kaz-eng	ntrex128	0.41891	13.7	1997	47673
kaz-spa	ntrex128	0.40670	13.8	1997	54107
lav-deu	ntrex128	0.46085	15.0	1997	48761
lav-eng	ntrex128	0.52552	21.7	1997	47673
lav-fra	ntrex128	0.47464	18.1	1997	53481
lav-por	ntrex128	0.46459	16.4	1997	51631
lav-spa	ntrex128	0.48650	20.4	1997	54107
ltz-eng	ntrex128	0.54302	27.8	1997	47673
ltz-por	ntrex128	0.45079	18.2	1997	51631
mar-eng	ntrex128	0.47392	18.3	1997	47673
mar-spa	ntrex128	0.42040	14.9	1997	54107
mkd-deu	ntrex128	0.50258	19.6	1997	48761
mkd-eng	ntrex128	0.59926	32.3	1997	47673
mkd-fra	ntrex128	0.52273	24.1	1997	53481
mkd-por	ntrex128	0.51528	23.2	1997	51631
mkd-spa	ntrex128	0.54270	27.6	1997	54107
mlg-eng	ntrex128	0.41059	14.3	1997	47673
mlt-deu	ntrex128	0.48680	19.6	1997	48761
mlt-eng	ntrex128	0.63462	37.6	1997	47673
mlt-fra	ntrex128	0.52138	24.8	1997	53481
mlt-por	ntrex128	0.51943	24.6	1997	51631
mlt-spa	ntrex128	0.53967	28.0	1997	54107
msa-deu	ntrex128	0.45669	16.7	1997	48761
msa-eng	ntrex128	0.57773	31.9	1997	47673
msa-fra	ntrex128	0.47740	20.1	1997	53481
nld-fra	ntrex128	0.53426	25.0	1997	53481
nld-spa	ntrex128	0.55158	29.3	1997	54107
nno-por	ntrex128	0.50504	23.1	1997	51631
nno-spa	ntrex128	0.53069	27.0	1997	54107
nob-fra	ntrex128	0.53221	25.8	1997	53481
nob-por	ntrex128	0.52513	24.9	1997	51631
pan-eng	ntrex128	0.42669	13.5	1997	47673
pol-deu	ntrex128	0.48613	18.0	1997	48761
pol-eng	ntrex128	0.53276	23.3	1997	47673
pol-fra	ntrex128	0.50446	22.0	1997	53481
pol-por	ntrex128	0.49314	20.4	1997	51631
pol-spa	ntrex128	0.51313	23.8	1997	54107
por-deu	ntrex128	0.51862	21.4	1997	48761
por-eng	ntrex128	0.61838	34.9	1997	47673
por-fra	ntrex128	0.54956	27.0	1997	53481
por-spa	ntrex128	0.58363	32.8	1997	54107
prs-deu	ntrex128	0.42183	12.0	1997	48761
prs-eng	ntrex128	0.49772	21.2	1997	47673
prs-fra	ntrex128	0.43811	15.7	1997	53481
prs-por	ntrex128	0.43047	14.9	1997	51631
prs-spa	ntrex128	0.45246	18.0	1997	54107
ron-deu	ntrex128	0.50587	19.6	1997	48761
ron-eng	ntrex128	0.58832	29.5	1997	47673
ron-fra	ntrex128	0.53785	24.7	1997	53481
ron-por	ntrex128	0.53776	25.5	1997	51631
ron-spa	ntrex128	0.56115	29.8	1997	54107
rus-deu	ntrex128	0.47395	16.1	1997	48761
rus-eng	ntrex128	0.52728	22.8	1997	47673
rus-fra	ntrex128	0.49638	20.6	1997	53481
rus-por	ntrex128	0.47749	18.9	1997	51631
rus-spa	ntrex128	0.50193	22.8	1997	54107
slv-deu	ntrex128	0.49861	19.5	1997	48761
slv-eng	ntrex128	0.55850	26.9	1997	47673
slv-fra	ntrex128	0.51117	22.9	1997	53481
slv-por	ntrex128	0.50114	21.5	1997	51631
ara-eng	tico19-test	0.54754	27.3	2100	56323
ara-fra	tico19-test	0.47603	20.9	2100	64661
ara-por	tico19-test	0.51671	25.6	2100	62729
ara-spa	tico19-test	0.53399	27.7	2100	66563
ben-eng	tico19-test	0.48344	15.8	2100	56824
ben-fra	tico19-test	0.41452	16.3	2100	64661
ben-por	tico19-test	0.43394	12.7	2100	62729
ben-spa	tico19-test	0.45544	15.9	2100	66563
eng-fra	tico19-test	0.60115	35.6	2100	64661
eng-por	tico19-test	0.69271	45.3	2100	62729
eng-spa	tico19-test	0.69653	47.2	2100	66563
fas-eng	tico19-test	0.54574	27.5	2100	56315
fas-fra	tico19-test	0.47294	20.9	2100	64661
fas-por	tico19-test	0.51476	24.7	2100	62729
fas-spa	tico19-test	0.52553	27.2	2100	66563
fra-eng	tico19-test	0.58589	34.6	2100	56323
fra-por	tico19-test	0.55536	30.8	2100	62729
fra-spa	tico19-test	0.56734	33.6	2100	66563
hin-eng	tico19-test	0.59076	30.9	2100	56323
hin-fra	tico19-test	0.47808	22.0	2100	64661
hin-por	tico19-test	0.52503	26.7	2100	62729
hin-spa	tico19-test	0.53813	27.8	2100	66563
ind-eng	tico19-test	0.66479	39.1	2100	56315
ind-fra	tico19-test	0.53716	27.3	2100	64661
ind-por	tico19-test	0.59409	32.7	2100	62729
ind-spa	tico19-test	0.60176	34.1	2100	66563
mar-eng	tico19-test	0.47258	18.6	2100	56315
mar-fra	tico19-test	0.40156	14.6	2100	64661
mar-por	tico19-test	0.41281	13.5	2100	62729
mar-spa	tico19-test	0.44960	19.8	2100	66563
msa-fra	tico19-test	0.52071	26.0	2100	64661
msa-por	tico19-test	0.58553	32.2	2100	62729
por-eng	tico19-test	0.70548	46.4	2100	56315
por-fra	tico19-test	0.57355	32.0	2100	64661
por-spa	tico19-test	0.66116	42.3	2100	66563
prs-eng	tico19-test	0.51514	22.6	2100	56824
prs-fra	tico19-test	0.44725	17.9	2100	64661
prs-por	tico19-test	0.48683	22.1	2100	62729
prs-spa	tico19-test	0.50656	24.9	2100	66563
pus-eng	tico19-test	0.41078	14.1	2100	56315
pus-spa	tico19-test	0.40026	16.1	2100	66563
rus-fra	tico19-test	0.49722	23.0	2100	64661
rus-por	tico19-test	0.53902	26.0	2100	62729

Citation Information

• Publications: Democratizing neural machine translation with OPUS-MT and OPUS-MT – Building open translation services for the World and The Tatoeba Translation Challenge – Realistic Data Sets for Low Resource and Multilingual MT (Please, cite if you use this model.)

@article{tiedemann2023democratizing,
  title={Democratizing neural machine translation with {OPUS-MT}},
  author={Tiedemann, J{\"o}rg and Aulamo, Mikko and Bakshandaeva, Daria and Boggia, Michele and Gr{\"o}nroos, Stig-Arne and Nieminen, Tommi and Raganato, Alessandro and Scherrer, Yves and Vazquez, Raul and Virpioja, Sami},
  journal={Language Resources and Evaluation},
  number={58},
  pages={713--755},
  year={2023},
  publisher={Springer Nature},
  issn={1574-0218},
  doi={10.1007/s10579-023-09704-w}
}

@inproceedings{tiedemann-thottingal-2020-opus,
    title = "{OPUS}-{MT} {--} Building open translation services for the World",
    author = {Tiedemann, J{\"o}rg  and Thottingal, Santhosh},
    booktitle = "Proceedings of the 22nd Annual Conference of the European Association for Machine Translation",
    month = nov,
    year = "2020",
    address = "Lisboa, Portugal",
    publisher = "European Association for Machine Translation",
    url = "https://aclanthology.org/2020.eamt-1.61",
    pages = "479--480",
}

@inproceedings{tiedemann-2020-tatoeba,
    title = "The Tatoeba Translation Challenge {--} Realistic Data Sets for Low Resource and Multilingual {MT}",
    author = {Tiedemann, J{\"o}rg},
    booktitle = "Proceedings of the Fifth Conference on Machine Translation",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2020.wmt-1.139",
    pages = "1174--1182",
}

Acknowledgements

The work is supported by the HPLT project, funded by the European Union’s Horizon Europe research and innovation programme under grant agreement No 101070350. We are also grateful for the generous computational resources and IT infrastructure provided by CSC -- IT Center for Science, Finland, and the EuroHPC supercomputer LUMI.

Model conversion info

• transformers version: 4.45.1
• OPUS-MT git hash: 0882077
• port time: Wed Oct 9 18:24:15 EEST 2024
• port machine: LM0-400-22516.local