Update README.md

README.md

@@ -32,6 +32,27 @@ This model is Hybrid CTC/Attention model with pre-trained HuBERT encoder.
 
 For evaluation, the metrics are CER and WER. before WER evaluation, transcriptions were re-tokenized using newmm tokenizer in [PyThaiNLP](https://github.com/PyThaiNLP/pythainlp)
 
+In this reposirity, we also provide the vocabulary for building the newmm tokenizer using this script:
+
+```python
+from pythainlp import Tokenizer
+
+def get_tokenizer(vocab):
+
+    custom_vocab = set(vocab)
+    custom_tokenizer = Tokenizer(custom_vocab, engine='newmm')
+    return custom_tokenizer
+
+with open(<vocab_path>,'r',encoding='utf-8') as f:
+        vocab = []
+        for line in f.readlines():
+            vocab.append(line.strip())
+
+custom_tokenizer = get_tokenizer(vocab)
+
+tokenized_sentence_list = custom_tokenizer.word_tokenize(<your_sentence>)
+```
+
 |Micro CER|Macro CER|Survival CER|E-commerce WER|Micro WER|Macro WER|Survival WER|E-commerce WER|
 |---|---|---|---|---|---|---|---|
 |5.35|5.65|6.29|5.02|7.53|8.73|11.38|6.09|