add stats

README.md

@@ -23,4 +23,29 @@ print(tokens)
 
 number_of_tokens = len(enc['input_ids'])
 print("Number of tokens:", number_of_tokens)
-```
+```
+
+## Computing number of tokens
+
+The following values can be used to approximate the number of tokens given the number input characters:
+```
+approx_number_of_tokens = len(input_text) / ratio
+```
+
+E.g. for English, `approx_number_of_tokens = len(input_text) / 4.8`.
+
+| Language | Avg. characters per token |
+| --- | :---: |
+| ar | 3.6 |
+| de | 4.6 |
+| en | 4.8 |
+| es | 4.6 |
+| fr | 4.4 |
+| hi | 3.8 |
+| it | 4.5 |
+| ja | 1.3 |
+| ko | 2.0 |
+| zh | 1.1 |
+
+
+These values have been computed on the first 10,000 paragraphs from [Wikipedia](https://huggingface.co/datasets/Cohere/wikipedia-22-12). For other dataset, these values might change.