Language Identification - a it-at-m Collection

it-at-m 's Collections

Language Identification

Language Identification

updated Jan 20

Models to evaluate for en/de language identification in context of DLF/RAG Pipelines

facebook/fasttext-language-identification

Text Classification • Updated Jun 9, 2023 • 598k • • 219

Note SOTA framework from Facebook, but NonCommercial License for the model
cis-lmu/glotlid

Text Classification • Updated Oct 26, 2024 • 54.7k • 59

Note Also uses Facebook framework, model is Apache-2.0, built by LMU
papluca/xlm-roberta-base-language-detection

Text Classification • Updated Dec 28, 2023 • 5.38M • • 315

Note Most used after facebook, uses normal transformers library
protectai/xlm-roberta-base-language-detection-onnx

Text Classification • Updated Mar 25, 2024 • 21.9k • 5

Note ONNX optimisation of model above, works quite well