Home / Evaluation

Evaluation method

Every score on this site comes from one screening run on 2026-08-04. Here is exactly what it measured and what it did not.

Translation pairs

Test set: FLORES-200 dev split, the first 48 sentences per pair. FLORES is professionally translated and is not the corpus these models were trained on (OPUS / Tatoeba), so it does not reward memorisation.
Metrics: chrF++ (character n-grams plus word bigrams, the WMT default) and BLEU, computed with sacrebleu.
Bands, by chrF++: Excellent ≥ 60 (30 pairs), Good 45–60 (221), Usable 30–45 (284), Limited 15–30 (238), Not recommended < 15 (15). Median 36.79.
Scored: 788 of 1,660 pair models.

Why 872 pairs have no score

Reason	Models
FLORES-200 has no test set for the language (most of the long tail)	818
CTranslate2 (HPLT) models: measured, but that harness was found broken, so scores are withheld	25
Not part of the screening run	25
Same-language models (a score would only measure copying)	4

Language verification

53 unscored models passed a GlotLID language-identification check on their output: the model writes the language it claims. That is evidence of identity, not of quality, so it is shown as a separate "language-verified" badge and never mixed into the bands.

Caveats

48 sentences is a screening sample. Small differences between pairs (a point or two of chrF++) are noise.
FLORES-200 is news and encyclopedic prose. It says little about chat, menus, medicine or law.
No human evaluation was performed.
chrF++ is not "accuracy". A 60 does not mean 60% correct.
Numbers for the three multilingual flagships come from their model cards (FLORES-200 devtest, 1,012 sentences, spBLEU/chrF) and were not re-measured here.
Speech-to-text models are not benchmarked here yet.

What is next

Re-score every pair on the full FLORES-200 devtest split (1,012 sentences), fix the CTranslate2 harness and publish the HPLT scores, and add word-error-rate results for the speech models. Scores will be published either way, including the weak ones.