WiLI-2018
WiLI-2018, the Wikipedia language identification benchmark dataset, contains 235000 paragraphs of 235 languages. The dataset is balanced and a train-test split is provided.
-
The WiLI benchmark dataset for written language identification
language identification
nan
Martin Thoma
This paper describes the WiLI-2018 benchmark
dataset for monolingual written natural language identification.
WiLI-2018 is a publicly available,1
free of charge dataset of
short text extracts from Wikipedia. It contains 1000 paragraphs
of 235 languages, totaling in 235 000 paragraphs. WiLI is a
classification dataset: Given an unknown paragraph written in
one dominant language, it has to be decided which language it
is.
