Update README.md

README.md

@@ -10,8 +10,45 @@ tags:
 - gemma
 ---
 
-Number of tokens in google/gemma-2-9b: 256000
-Number of tokens in rishiraj/gemma-2-9b-bn: 392402
+This repository extends the `google/gemma-2-9b` tokenizer by training it on Bengali text.
 
-Why fewer tokens than English?
-While Bengali is very expressive and flexible, it hasn't undergone as much global influence as English in terms of absorbing new words from many different languages.
+## Token Information
+
+| Tokenizer                         | Number of Tokens |
+|------------------------------------|------------------|
+| `google/gemma-2-9b`                | 256,000          |
+| `rishiraj/gemma-2-9b-bn`           | 392,402          |
+
+### Why Fewer Tokens for Bengali?
+While Bengali is very expressive and flexible, it hasn't undergone as much global influence as English in terms of absorbing new words from many different languages.
+
+## Tokenizer Comparison
+
+**Text:**
+```text
+আমি একজন ভালো ছেলে এবং আমি ফুটবল খেলতে পছন্দ করি
+```
+
+| Tokenizer                  | Output                                                                                                               |
+|----------------------------|----------------------------------------------------------------------------------------------------------------------|
+| `gemma_tokenizer`           | ['আ', 'মি', '▁এক', 'জন', '▁ভ', 'াল', 'ো', '▁', 'ছে', 'লে', '▁এবং', '▁আম', 'ি', '▁ফ', 'ু', 'ট', 'ব', 'ল', '▁খ', 'েল', 'তে', '▁প', 'ছ', 'ন্দ', '▁কর', 'ি'] |
+| `our_tokenizer`             | ['আমি', '▁একজন', '▁ভালো', '▁ছেলে', '▁এবং', '▁আমি', '▁ফুটবল', '▁খেলতে', '▁পছন্দ', '▁করি']                                                      |
+
+## Usage
+
+1. Install dependencies:
+   ```bash
+   pip install transformers
+   ```
+
+2. Load and use the tokenizer:
+   ```python
+   from transformers import AutoTokenizer
+   tokenizer = AutoTokenizer.from_pretrained("rishiraj/gemma-2-9b-bn")
+   tokens = tokenizer.tokenize("আমি একজন ভালো ছেলে এবং আমি ফুটবল খেলতে পছন্দ করি")
+   print(tokens)
+   ```
+
+## Conclusion
+
+The original `gemma_tokenizer` splits many Bengali words into subword components, leading to inefficiency and loss of meaning. Our extended Bengali tokenizer better preserves word integrity, tokenizing more effectively with fewer splits, ensuring more meaningful representation of the text.