README.md

---
license: mit
datasets:
- mlengineer-ai/jomleh
language:
- fa
metrics:
- perplexity
tags:
- kneser-ney
- n-gram
- kenlm
---

# KenLM models for Farsi

This repository contains trained KenLM models for Farsi (Persian) language trained on the Jomleh
dataset. Among all the use cases for the language models like KenLM, the models provided here are
very useful for ASR (automatic speech recognition) task. They can be used along with CTC to select
the more likely sequence of tokens extracted from spectogram.

The models in this repository are KenLM arpa files turned into binary. KenLM supports two types of
binary formats: probing and trie. The models provided here are of the probing format. KenLM claims
that they are faster but with bigger memory footprint.

There are a total 36 different KenLM models that you can find here. Unless you are doing some
research, you won't be needing all of them. If that's the case, I suggest downloading the ones you
need and not the whole repository. As the total size of files is larger than half a TB.

# Sample code how to use the models

Unfortunately, I could not find an easy way to integrate the Python code that loads the models
using Huggingface library. These are the steps that you have to take if you want to use any of the
models provided here:

1. Install KenLM package:

```
pip install https://github.com/kpu/kenlm/archive/master.zip
```

2. Install the SentencePiece for the tokenization:
```
pip install sentencepiece
```

3. Download the model that you are interested in from this repository along the Python code
`model.py`. Keep the model in the `files` folder with the `model.py` by it (just like the file
structure in the repository). Don't forget to download the SentencePiece files as well. For
instance, if you were interested in 32000 vocabulary size tokenizer, 5-gram model with maximum
pruning, these are the files you'll need:
```
model.py
files/jomleh-sp-32000.model
files/jomleh-sp-32000.vocab
files/jomleh-sp-32000-o5-prune01111.probing
```

4. Write your own code to instantiate a model and use it:

```
```

# What are the different models provided here

There a total of 36 models in this repository and while all of the are trained on Jomleh daatset,
which is a Farsi dataset, there differences among them. Namely:

1. Different vocabulary sizes: For research purposes, I trained on 6 different vocabulary sizes.
Of course, the vocabulary size is a hyperparameter for the tokenizer (SentencePiece here) but
once you have a new tokenizer, it will result in a new model. The different vocabulary sizes used
here are: 2000, 4000, 8000, 16000, 32000, and 57218 tokens. For most use cases, ethier 32000 or
57218 token vocabulary size should be the best option.