Update README.md
Browse files
README.md
CHANGED
@@ -438,6 +438,161 @@ Below are the evaluation results on the Flores+200 devtest set, compared against
|
|
438 |
| SalamandraTA-7b-base | 24.71 | 62.33 | 52.96 | **0.85** | **0.84** | 0.73 | 1.06 | 1.37 |
|
439 |
|
440 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
441 |
|
442 |
## Ethical Considerations and Limitations
|
443 |
|
|
|
438 |
| SalamandraTA-7b-base | 24.71 | 62.33 | 52.96 | **0.85** | **0.84** | 0.73 | 1.06 | 1.37 |
|
439 |
|
440 |
|
441 |
+
<details>
|
442 |
+
<summary>Click to show full table ES-XX Flores+devtest</summary>
|
443 |
+
|
444 |
+
| | Source | Target | Bleu↑ | Ter↓ | ChrF↑ | Comet↑ | Comet-kiwi↑ | Bleurt↑ | MetricX↓ | MetricX-QE↓ |
|
445 |
+
|:---------------------------------|:---------|:---------|-------:|------:|-------:|--------:|-------------:|---------:|----------:|-------------:|
|
446 |
+
| SalamandraTA-7b-base | es | uk | 19.15 | 74.57 | 49.57 | 0.88 | 0.85 | 0.76 | 1.33 | 0.78 |
|
447 |
+
| MADLAD400-7B | es | uk | 19.19 | 74.61 | 49.87 | 0.89 | 0.86 | 0.77 | 1.2 | 0.75 |
|
448 |
+
| SalamandraTA-7b-instruct | es | uk | 20.42 | 70.67 | 49.85 | 0.89 | 0.85 | 0.77 | 1.18 | 0.85 |
|
449 |
+
| | | | | | | | | | | |
|
450 |
+
| MADLAD400-7B | es | sv | 26.55 | 63.25 | 57.28 | 0.89 | 0.86 | 0.8 | 0.68 | 0.69 |
|
451 |
+
| SalamandraTA-7b-base | es | sv | 25.87 | 64.49 | 56.08 | 0.88 | 0.85 | 0.79 | 0.87 | 1 |
|
452 |
+
| SalamandraTA-7b-instruct | es | sv | 27.54 | 60.34 | 56.72 | 0.89 | 0.85 | 0.8 | 0.64 | 0.72 |
|
453 |
+
| | | | | | | | | | | |
|
454 |
+
| MADLAD400-7B | es | sr | 22.48 | 66.61 | 52.22 | 0.88 | 0.86 | 0.83 | 1.14 | 1.25 |
|
455 |
+
| SalamandraTA-7b-instruct | es | sr | 22.06 | 66.38 | 51.32 | 0.88 | 0.86 | 0.82 | 1.16 | 0.95 |
|
456 |
+
| SalamandraTA-7b-base | es | sr | 21.02 | 69.04 | 51 | 0.87 | 0.86 | 0.81 | 1.46 | 1 |
|
457 |
+
| | | | | | | | | | | |
|
458 |
+
| SalamandraTA-7b-instruct | es | sl | 22.24 | 68.56 | 51.13 | 0.89 | 0.84 | 0.84 | 0.76 | 0.96 |
|
459 |
+
| MADLAD400-7B | es | sl | 22.84 | 69.52 | 52.85 | 0.89 | 0.85 | 0.85 | 0.64 | 0.67 |
|
460 |
+
| SalamandraTA-7b-base | es | sl | 20.86 | 72.13 | 51.11 | 0.88 | 0.85 | 0.83 | 0.99 | 0.83 |
|
461 |
+
| | | | | | | | | | | |
|
462 |
+
| SalamandraTA-7b-instruct | es | sk | 21.96 | 67.95 | 50.76 | 0.9 | 0.85 | 0.81 | 0.78 | 0.88 |
|
463 |
+
| MADLAD400-7B | es | sk | 22.52 | 69.42 | 52.29 | 0.9 | 0.86 | 0.83 | 0.66 | 0.72 |
|
464 |
+
| SalamandraTA-7b-base | es | sk | 20.96 | 71.63 | 50.87 | 0.89 | 0.86 | 0.8 | 0.98 | 0.82 |
|
465 |
+
| | | | | | | | | | | |
|
466 |
+
| SalamandraTA-7b-base | es | ru | 21.07 | 72.57 | 50.84 | 0.88 | 0.84 | 0.75 | 1.3 | 0.84 |
|
467 |
+
| MADLAD400-7B | es | ru | 21.41 | 72.23 | 52.16 | 0.89 | 0.85 | 0.76 | 1.07 | 0.74 |
|
468 |
+
| SalamandraTA-7b-instruct | es | ru | 21.95 | 68.71 | 51.26 | 0.89 | 0.83 | 0.76 | 1.1 | 0.9 |
|
469 |
+
| | | | | | | | | | | |
|
470 |
+
| SalamandraTA-7b-base | es | ro | 25.38 | 65.94 | 54.6 | 0.88 | 0.86 | 0.8 | 1 | 0.85 |
|
471 |
+
| SalamandraTA-7b-instruct | es | ro | 27.71 | 62.01 | 54.8 | 0.89 | 0.85 | 0.81 | 0.87 | 0.97 |
|
472 |
+
| MADLAD400-7B | es | ro | 26.65 | 64.11 | 55.68 | 0.89 | 0.87 | 0.82 | 0.76 | 0.78 |
|
473 |
+
| | | | | | | | | | | |
|
474 |
+
| SalamandraTA-7b-base | es | pt | 26.23 | 63.59 | 56.16 | 0.88 | 0.85 | 0.74 | 0.9 | 1.11 |
|
475 |
+
| MADLAD400-7B | es | pt | 27.11 | 62.5 | 56.95 | 0.88 | 0.85 | 0.74 | 0.84 | 1.01 |
|
476 |
+
| SalamandraTA-7b-instruct | es | pt | 32.05 | 56.01 | 58.85 | 0.88 | 0.84 | 0.75 | 0.77 | 0.99 |
|
477 |
+
| | | | | | | | | | | |
|
478 |
+
| SalamandraTA-7b-base | es | pl | 16.98 | 77.3 | 47.99 | 0.89 | 0.85 | 0.79 | 0.99 | 0.81 |
|
479 |
+
| MADLAD400-7B | es | pl | 18.07 | 76.69 | 49.32 | 0.89 | 0.85 | 0.81 | 0.84 | 0.8 |
|
480 |
+
| SalamandraTA-7b-instruct | es | pl | 18.6 | 73.56 | 48.14 | 0.9 | 0.84 | 0.81 | 0.71 | 0.68 |
|
481 |
+
| | | | | | | | | | | |
|
482 |
+
| SalamandraTA-7b-instruct | es | oc | 17.65 | 71.75 | 49.05 | 0.69 | 0.6 | 0.45 | 2.59 | 3.19 |
|
483 |
+
| MADLAD400-7B | es | oc | 17.02 | 73.48 | 47.74 | 0.66 | 0.55 | 0.4 | 3.87 | 5.55 |
|
484 |
+
| SalamandraTA-7b-base | es | oc | 16.06 | 75.17 | 47.47 | 0.69 | 0.62 | 0.45 | 2.66 | 3.55 |
|
485 |
+
| | | | | | | | | | | |
|
486 |
+
| SalamandraTA-7b-base | es | nn | 18.87 | 76.97 | 49.2 | 0.86 | 0.83 | 0.69 | 1.12 | 1.3 |
|
487 |
+
| MADLAD400-7B | es | nn | 21.69 | 67.33 | 51.71 | 0.87 | 0.84 | 0.7 | 0.84 | 1.06 |
|
488 |
+
| SalamandraTA-7b-instruct | es | nn | 20.13 | 68.82 | 49.91 | 0.86 | 0.83 | 0.69 | 1.03 | 1.03 |
|
489 |
+
| | | | | | | | | | | |
|
490 |
+
| MADLAD400-7B | es | nl | 20.94 | 70.16 | 53.83 | 0.86 | 0.86 | 0.77 | 0.77 | 0.71 |
|
491 |
+
| SalamandraTA-7b-instruct | es | nl | 21.63 | 66.87 | 53.23 | 0.86 | 0.84 | 0.76 | 0.78 | 0.84 |
|
492 |
+
| SalamandraTA-7b-base | es | nl | 20.12 | 71.76 | 53.1 | 0.85 | 0.85 | 0.75 | 0.95 | 0.82 |
|
493 |
+
| | | | | | | | | | | |
|
494 |
+
| SalamandraTA-7b-base | es | mt | 17.26 | 73.99 | 55.22 | 0.66 | 0.65 | 0.26 | 2.12 | 1.74 |
|
495 |
+
| SalamandraTA-7b-instruct | es | mt | 20.66 | 65.35 | 56.07 | 0.67 | 0.62 | 0.3 | 1.97 | 1.93 |
|
496 |
+
| MADLAD400-7B | es | mt | 20.36 | 67.41 | 56.96 | 0.67 | 0.65 | 0.29 | 1.67 | 1.47 |
|
497 |
+
| | | | | | | | | | | |
|
498 |
+
| SalamandraTA-7b-base | es | lv | 20.51 | 72.13 | 52.25 | 0.88 | 0.81 | 0.81 | 1.49 | 0.93 |
|
499 |
+
| MADLAD400-7B | es | lv | 22.24 | 70.18 | 54.08 | 0.89 | 0.82 | 0.84 | 1.12 | 0.89 |
|
500 |
+
| SalamandraTA-7b-instruct | es | lv | 21.57 | 69.41 | 52.32 | 0.89 | 0.8 | 0.84 | 1.16 | 0.98 |
|
501 |
+
| | | | | | | | | | | |
|
502 |
+
| MADLAD400-7B | es | lt | 19.38 | 74.98 | 53.39 | 0.89 | 0.83 | 0.85 | 1.07 | 1.22 |
|
503 |
+
| SalamandraTA-7b-base | es | lt | 17.87 | 75.34 | 51.84 | 0.88 | 0.82 | 0.83 | 1.37 | 0.91 |
|
504 |
+
| SalamandraTA-7b-instruct | es | lt | 19.26 | 73.59 | 51.18 | 0.89 | 0.81 | 0.84 | 0.96 | 0.8 |
|
505 |
+
| | | | | | | | | | | |
|
506 |
+
| SalamandraTA-7b-base | es | it | 23.91 | 67.25 | 54.32 | 0.88 | 0.86 | 0.77 | 0.93 | 0.92 |
|
507 |
+
| MADLAD400-7B | es | it | 24.97 | 66.18 | 54.93 | 0.88 | 0.86 | 0.78 | 0.87 | 0.81 |
|
508 |
+
| SalamandraTA-7b-instruct | es | it | 26.42 | 63.18 | 54.89 | 0.88 | 0.85 | 0.78 | 0.77 | 0.87 |
|
509 |
+
| | | | | | | | | | | |
|
510 |
+
| MADLAD400-7B | es | hu | 18.84 | 72.31 | 52.62 | 0.88 | 0.85 | 0.9 | 0.72 | 0.79 |
|
511 |
+
| SalamandraTA-7b-instruct | es | hu | 18.86 | 70.36 | 51.18 | 0.88 | 0.84 | 0.89 | 0.8 | 0.79 |
|
512 |
+
| SalamandraTA-7b-base | es | hu | 18.14 | 73.22 | 51.26 | 0.87 | 0.84 | 0.89 | 0.97 | 0.65 |
|
513 |
+
| | | | | | | | | | | |
|
514 |
+
| SalamandraTA-7b-base | es | hr | 20.05 | 71.73 | 52.09 | 0.89 | 0.86 | 0.83 | 1.89 | 1.6 |
|
515 |
+
| MADLAD400-7B | es | hr | 21.07 | 70.6 | 53.07 | 0.9 | 0.86 | 0.84 | 1.59 | 1.71 |
|
516 |
+
| SalamandraTA-7b-instruct | es | hr | 22.14 | 66.85 | 52.72 | 0.9 | 0.85 | 0.83 | 1.59 | 1.58 |
|
517 |
+
| | | | | | | | | | | |
|
518 |
+
| SalamandraTA-7b-base | es | gl | 22.28 | 66.67 | 52.86 | 0.87 | 0.84 | 0.67 | 1.13 | 1.81 |
|
519 |
+
| MADLAD400-7B | es | gl | 21.98 | 66.74 | 53.1 | 0.87 | 0.84 | 0.67 | 0.94 | 1.42 |
|
520 |
+
| SalamandraTA-7b-instruct | es | gl | 28.27 | 58.19 | 56.35 | 0.88 | 0.85 | 0.7 | 0.79 | 0.97 |
|
521 |
+
| | | | | | | | | | | |
|
522 |
+
| SalamandraTA-7b-instruct | es | ga | 20.07 | 70.37 | 49.64 | 0.77 | 0.65 | 0.6 | 1.93 | 1.7 |
|
523 |
+
| SalamandraTA-7b-base | es | ga | 20.73 | 71.47 | 50.62 | 0.76 | 0.67 | 0.6 | 1.87 | 1.48 |
|
524 |
+
| MADLAD400-7B | es | ga | 21.5 | 70.22 | 51.6 | 0.77 | 0.67 | 0.61 | 1.89 | 1.61 |
|
525 |
+
| | | | | | | | | | | |
|
526 |
+
| SalamandraTA-7b-instruct | es | fr | 33.83 | 54.94 | 60.21 | 0.86 | 0.85 | 0.74 | 0.87 | 1.21 |
|
527 |
+
| SalamandraTA-7b-base | es | fr | 30.8 | 59.03 | 59.38 | 0.86 | 0.86 | 0.73 | 0.92 | 1.06 |
|
528 |
+
| MADLAD400-7B | es | fr | 30.68 | 59.12 | 59.76 | 0.86 | 0.86 | 0.73 | 0.88 | 1.04 |
|
529 |
+
| | | | | | | | | | | |
|
530 |
+
| SalamandraTA-7b-base | es | fi | 15.84 | 76.23 | 52.44 | 0.89 | 0.82 | 0.82 | 1.16 | 0.79 |
|
531 |
+
| MADLAD400-7B | es | fi | 18.49 | 74.35 | 55.35 | 0.91 | 0.84 | 0.85 | 0.71 | 0.54 |
|
532 |
+
| SalamandraTA-7b-instruct | es | fi | 18.36 | 70.94 | 53.7 | 0.91 | 0.83 | 0.85 | 0.61 | 0.5 |
|
533 |
+
| | | | | | | | | | | |
|
534 |
+
| SalamandraTA-7b-base | es | eu | 14.72 | 78.76 | 54.44 | 0.86 | 0.85 | 0.79 | 0.95 | 0.76 |
|
535 |
+
| SalamandraTA-7b-instruct | es | eu | 16.52 | 75.42 | 54.27 | 0.86 | 0.84 | 0.8 | 0.73 | 0.6 |
|
536 |
+
| MADLAD400-7B | es | eu | 13.24 | 84.15 | 54.54 | 0.87 | 0.86 | 0.81 | 0.76 | 0.63 |
|
537 |
+
| | | | | | | | | | | |
|
538 |
+
| SalamandraTA-7b-base | es | et | 17.98 | 74.39 | 52.84 | 0.89 | 0.83 | 0.81 | 1.01 | 0.83 |
|
539 |
+
| SalamandraTA-7b-instruct | es | et | 18.91 | 70.78 | 52.93 | 0.9 | 0.83 | 0.82 | 0.69 | 0.77 |
|
540 |
+
| MADLAD400-7B | es | et | 17.21 | 78.52 | 52.85 | 0.89 | 0.83 | 0.82 | 0.92 | 1.54 |
|
541 |
+
| | | | | | | | | | | |
|
542 |
+
| SalamandraTA-7b-base | es | en | 32.37 | 55.92 | 61.52 | 0.87 | 0.86 | 0.78 | 1.26 | 1.41 |
|
543 |
+
| MADLAD400-7B | es | en | 31.45 | 56.44 | 61.37 | 0.88 | 0.86 | 0.77 | 1.23 | 1.05 |
|
544 |
+
| SalamandraTA-7b-instruct | es | en | 37.22 | 49.72 | 63.39 | 0.88 | 0.86 | 0.78 | 1.09 | 1.07 |
|
545 |
+
| | | | | | | | | | | |
|
546 |
+
| SalamandraTA-7b-base | es | el | 20.14 | 70.61 | 48.13 | 0.88 | 0.84 | 0.74 | 1.37 | 0.91 |
|
547 |
+
| MADLAD400-7B | es | el | 21.44 | 68.53 | 49.72 | 0.88 | 0.84 | 0.75 | 1.24 | 0.97 |
|
548 |
+
| SalamandraTA-7b-instruct | es | el | 20.63 | 67.07 | 48.29 | 0.88 | 0.84 | 0.74 | 1.14 | 0.79 |
|
549 |
+
| | | | | | | | | | | |
|
550 |
+
| SalamandraTA-7b-instruct | es | de | 25.67 | 63.66 | 55.75 | 0.85 | 0.84 | 0.77 | 0.68 | 0.95 |
|
551 |
+
| MADLAD400-7B | es | de | 25.67 | 65.67 | 56.68 | 0.85 | 0.85 | 0.77 | 0.68 | 0.8 |
|
552 |
+
| SalamandraTA-7b-base | es | de | 24.69 | 67.29 | 55.79 | 0.85 | 0.84 | 0.76 | 0.75 | 1.02 |
|
553 |
+
| | | | | | | | | | | |
|
554 |
+
| SalamandraTA-7b-instruct | es | da | 29.24 | 59.71 | 57.14 | 0.89 | 0.84 | 0.77 | 0.72 | 0.76 |
|
555 |
+
| MADLAD400-7B | es | da | 28.59 | 62.21 | 57.84 | 0.89 | 0.85 | 0.77 | 0.68 | 0.75 |
|
556 |
+
| SalamandraTA-7b-base | es | da | 27.26 | 63.93 | 56.75 | 0.88 | 0.84 | 0.76 | 0.97 | 1.03 |
|
557 |
+
| | | | | | | | | | | |
|
558 |
+
| SalamandraTA-7b-instruct | es | cy | 25.35 | 63.81 | 54.64 | 0.83 | 0.76 | 0.48 | 1.85 | 1.65 |
|
559 |
+
| SalamandraTA-7b-base | es | cy | 24.94 | 65.32 | 55.22 | 0.82 | 0.77 | 0.48 | 1.72 | 1.39 |
|
560 |
+
| MADLAD400-7B | es | cy | 23.64 | 67.17 | 53.03 | 0.83 | 0.78 | 0.47 | 1.83 | 1.63 |
|
561 |
+
| | | | | | | | | | | |
|
562 |
+
| MADLAD400-7B | es | cs | 21.87 | 69.43 | 51.43 | 0.91 | 0.87 | 0.82 | 0.7 | 0.76 |
|
563 |
+
| SalamandraTA-7b-instruct | es | cs | 21.76 | 67.89 | 50 | 0.9 | 0.86 | 0.81 | 0.68 | 0.71 |
|
564 |
+
| SalamandraTA-7b-base | es | cs | 20.42 | 71.58 | 49.78 | 0.9 | 0.86 | 0.8 | 0.95 | 0.86 |
|
565 |
+
| | | | | | | | | | | |
|
566 |
+
| SalamandraTA-7b-base | es | ca | 23.31 | 66.87 | 54.09 | 0.86 | 0.85 | 0.72 | 1.13 | 1.33 |
|
567 |
+
| SalamandraTA-7b-instruct | es | ca | 31.15 | 57.76 | 57.49 | 0.87 | 0.84 | 0.75 | 0.92 | 1.18 |
|
568 |
+
| MADLAD400-7B | es | ca | 23.1 | 66.98 | 54.12 | 0.86 | 0.85 | 0.73 | 1.1 | 1.19 |
|
569 |
+
| | | | | | | | | | | |
|
570 |
+
| SalamandraTA-7b-instruct | es | bg | 27.47 | 62.02 | 56.35 | 0.9 | 0.86 | 0.87 | 1 | 0.86 |
|
571 |
+
| MADLAD400-7B | es | bg | 27.2 | 64.02 | 57.46 | 0.9 | 0.87 | 0.87 | 1.09 | 0.92 |
|
572 |
+
| SalamandraTA-7b-base | es | bg | 26.41 | 65.63 | 56.51 | 0.89 | 0.87 | 0.86 | 1.13 | 0.85 |
|
573 |
+
|
574 |
+
</details>
|
575 |
+
|
576 |
+
|
577 |
+
### Catalan
|
578 |
+
|
579 |
+
| | Bleu↑ | Ter↓ | ChrF↑ | Comet↑ | Comet-kiwi↑ | Bleurt↑ | MetricX↓ | MetricX-QE↓ |
|
580 |
+
|:---------------------------------|-------:|------:|-------:|--------:|-------------:|---------:|----------:|-------------:|
|
581 |
+
| **CA-XX** | | | | | | | | |
|
582 |
+
| MADLAD400-7B | **29.37** | 59.01 | **58.47** | **0.87** | **0.81** | **0.77** | **1.08** | 1.31 |
|
583 |
+
| SalamandraTA-7b-instruct | 29.23 | **58.32** | 57.76 | **0.87** | **0.81** | **0.77** | **1.08** | **1.22** |
|
584 |
+
| SalamandraTA-7b-base | 29.06 | 59.32 | 58 | **0.87** | **0.81** | 0.76 | 1.23 | 1.28 |
|
585 |
+
| **XX-CA** | | | | | | | | |
|
586 |
+
| SalamandraTA-7b-instruct | **33.64** | **54.49** | 59.03 | **0.86** | 0.8 | **0.75** | **1.07** | **1.6** |
|
587 |
+
| MADLAD400-7B | 33.02 | 55.01 | 59.38 | **0.86** | **0.81** | **0.75** | 1.18 | 1.79 |
|
588 |
+
| SalamandraTA-7b-base | 32.75 | 55.78 | **59.42** | **0.86** | **0.81** | **0.75** | 1.17 | 1.63 |
|
589 |
+
|
590 |
+
<details>
|
591 |
+
<summary>Click to show full table CA-XX Flores+devtest</summary>
|
592 |
+
|
593 |
+
|
594 |
+
</details>
|
595 |
+
|
596 |
|
597 |
## Ethical Considerations and Limitations
|
598 |
|