Update README.md

README.md

@@ -18,11 +18,46 @@ should probably proofread and complete it, then remove this comment. -->
 
 # TinySatirik-m
 
-This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
+This model is a fine-tuned version of [](https://huggingface.co/) on an [anekdots](https://huggingface.co/datasets/igorktech/anekdots) dataset.
+
+## Tokenizer
+
+To utilize the model, install the [special tokenizer](https://github.com/Koziev/character-tokenizer):
+
+```bash
+pip install git+https://github.com/Koziev/character-tokenizer
+```
+
+In addition to recognizing Cyrillic characters and punctuation, this tokenizer is aware of special tokens such as ```<s>```, ```</s>```, ```<pad>```, and ```<unk>```.
+
+As this is a non-standard tokenizer for transformers, load it not via ```transformers.AutoTokenizer.from_pretrained```, but somewhat like this:
+
+```python
+import charactertokenizer
+
+...
+tokenizer = charactertokenizer.CharacterTokenizer.from_pretrained('igorktech/CharPicoSatirik-m')
+```
+
+To observe tokenization, use this code snippet:
+
+```python
+prompt = '<s>Hello World\n'
+encoded_prompt = tokenizer.encode(prompt, return_tensors='pt')
+print('Tokenized prompt:', ' | '.join(tokenizer.decode([t]) for t in encoded_prompt[0]))
+```
+
+You will see a list of tokens separated by the ```|``` symbol:
+
+```
+Tokenized prompt: <s> | H | e | l | l | o |   | W | o | r | l | d | 
+```
+
+Tokenizer created by [Koziev](https://github.com/Koziev).
 
 ## Model description
 
-More information needed
+Llama2 architecture based.
 
 ## Intended uses & limitations