---
license: gpl-3.0
datasets:
- Mxode/IndustryCorpus-Subset-zh-en
---

# **Bilingual Tokenizer**


A portion of the data from the [IndustryCorpus-Subset-zh-en](https://huggingface.co/datasets/Mxode/IndustryCorpus-Subset-zh-en) dataset was used for training.

This dataset consists of **Chinese and English bilingual text**.

10,000 samples were taken from the untrained portion to test the compression rate of the tokenizer.

Compression rate formula:

$$
\text{Compression rate} = \frac{\text{length after tokenization}}{\text{character length of the original corpus}}
$$

Here is the test result:

|                            Model                             | Tokenizer Size | Compression Rate |
| :----------------------------------------------------------: | :------------: | :------------: |
| [deepseek-llm-7b-base](https://huggingface.co/deepseek-ai/deepseek-llm-7b-base) |     100015     |     36.63%     |
| [deepseek-coder-33b-base](https://huggingface.co/deepseek-ai/deepseek-coder-33b-base) |     32022      |     41.75%     |
|   [gemma-2-27b](https://huggingface.co/google/gemma-2-27b)   |     256000     |     37.75%     |
|      [glm-4-9b](https://huggingface.co/THUDM/glm-4-9b)       |     151343     |     34.26%     |
| [internlm2_5-7b-chat](https://huggingface.co/internlm/internlm2_5-7b-chat) |     92550      |     35.15%     |
| [Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf) |     32000      |     63.33%     |
| [Meta-Llama-3.1-8B](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B) |     128256     |     41.48%     |
| [Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3) |     32768      |     52.43%     |
| [Phi-3.5-mini-instruct](https://huggingface.co/microsoft/Phi-3.5-mini-instruct) |     32011      |     63.29%     |
| [Qwen2-7B-Instruct](https://huggingface.co/Qwen/Qwen2-7B-Instruct) |     151646     |     35.91%     |
|     [Yi-1.5-9B](https://huggingface.co/01-ai/Yi-1.5-9B)      |     63992      |     36.86%     |
|                    BilingualTokenizer-1K                     |      1000      |     75.61%     |
|                    BilingualTokenizer-2K                     |      2000      |     62.26%     |
|                    BilingualTokenizer-4K                     |      4000      |     52.81%     |
|                    BilingualTokenizer-8K                     |      8000      |     45.92%     |
|                    BilingualTokenizer-16K                    |     16000      |     40.94%     |