Create README.md

README.md

@@ -1,130 +1,218 @@
-# ⏳ tiktoken
+---
+language:
+- fr
+- it
+- de
+- es
+- en
+license: apache-2.0
+inference:
+  parameters:
+    temperature: 0.5
+widget:
+- messages:
+  - role: user
+    content: What is your favorite condiment?
+---
+# Model Card for Mixtral-8x7B
+
+### Tokenization with `mistral-common`
+
+```py
+from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
+from mistral_common.protocol.instruct.messages import UserMessage
+from mistral_common.protocol.instruct.request import ChatCompletionRequest
+ 
+mistral_models_path = "MISTRAL_MODELS_PATH"
+ 
+tokenizer = MistralTokenizer.v1()
+ 
+completion_request = ChatCompletionRequest(messages=[UserMessage(content="Explain Machine Learning to me in a nutshell.")])
+ 
+tokens = tokenizer.encode_chat_completion(completion_request).tokens
+```
+ 
+## Inference with `mistral_inference`
+ 
+ ```py
+from mistral_inference.model import Transformer
+from mistral_inference.generate import generate
+ 
+model = Transformer.from_folder(mistral_models_path)
+out_tokens, _ = generate([tokens], model, max_tokens=64, temperature=0.0, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)
+
+result = tokenizer.decode(out_tokens[0])
+
+print(result)
+```
 
-tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with
-OpenAI's models.
+## Inference with hugging face `transformers`
+ 
+```py
+from transformers import AutoModelForCausalLM
+ 
+model = AutoModelForCausalLM.from_pretrained("mistralai/Mixtral-8x7B-Instruct-v0.1")
+model.to("cuda")
+ 
+generated_ids = model.generate(tokens, max_new_tokens=1000, do_sample=True)
+
+# decode with mistral tokenizer
+result = tokenizer.decode(generated_ids[0].tolist())
+print(result)
+```
 
-```python
-import tiktoken
-enc = tiktoken.get_encoding("cl100k_base")
-assert enc.decode(enc.encode("hello world")) == "hello world"
+> [!TIP]
+> PRs to correct the transformers tokenizer so that it gives 1-to-1 the same results as the mistral-common reference implementation are very welcome!
+     
+        
+---
+The Mixtral-8x7B Large Language Model (LLM) is a pretrained generative Sparse Mixture of Experts. The Mixtral-8x7B outperforms Llama 2 70B on most benchmarks we tested.
 
-# To get the tokeniser corresponding to a specific model in the OpenAI API:
-enc = tiktoken.encoding_for_model("gpt-4")
-```
+For full details of this model please read our [release blog post](https://mistral.ai/news/mixtral-of-experts/).
+
+## Warning
+This repo contains weights that are compatible with [vLLM](https://github.com/vllm-project/vllm) serving of the model as well as Hugging Face [transformers](https://github.com/huggingface/transformers) library. It is based on the original Mixtral [torrent release](magnet:?xt=urn:btih:5546272da9065eddeb6fcd7ffddeef5b75be79a7&dn=mixtral-8x7b-32kseqlen&tr=udp%3A%2F%http://2Fopentracker.i2p.rocks%3A6969%2Fannounce&tr=http%3A%2F%http://2Ftracker.openbittorrent.com%3A80%2Fannounce), but the file format and parameter names are different. Please note that model cannot (yet) be instantiated with HF.
+
+## Instruction format
 
-The open source version of `tiktoken` can be installed from PyPI:
+This format must be strictly respected, otherwise the model will generate sub-optimal outputs.
+
+The template used to build a prompt for the Instruct model is defined as follows:
 ```
-pip install tiktoken
+<s> [INST] Instruction [/INST] Model answer</s> [INST] Follow-up instruction [/INST]
 ```
+Note that `<s>` and `</s>` are special tokens for beginning of string (BOS) and end of string (EOS) while [INST] and [/INST] are regular strings.
 
-The tokeniser API is documented in `tiktoken/core.py`.
+As reference, here is the pseudo-code used to tokenize instructions during fine-tuning:
+```python
+def tokenize(text):
+    return tok.encode(text, add_special_tokens=False)
+
+[BOS_ID] + 
+tokenize("[INST]") + tokenize(USER_MESSAGE_1) + tokenize("[/INST]") +
+tokenize(BOT_MESSAGE_1) + [EOS_ID] +
+…
+tokenize("[INST]") + tokenize(USER_MESSAGE_N) + tokenize("[/INST]") +
+tokenize(BOT_MESSAGE_N) + [EOS_ID]
+```
 
-Example code using `tiktoken` can be found in the
-[OpenAI Cookbook](https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb).
+In the pseudo-code above, note that the `tokenize` method should not add a BOS or EOS token automatically, but should add a prefix space. 
 
+In the Transformers library, one can use [chat templates](https://huggingface.co/docs/transformers/main/en/chat_templating) which make sure the right format is applied.
 
-## Performance
+## Run the model
 
-`tiktoken` is between 3-6x faster than a comparable open source tokeniser:
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
 
-![image](https://raw.githubusercontent.com/openai/tiktoken/main/perf.svg)
+model_id = "mistralai/Mixtral-8x7B-Instruct-v0.1"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
 
-Performance measured on 1GB of text using the GPT-2 tokeniser, using `GPT2TokenizerFast` from
-`tokenizers==0.13.2`, `transformers==4.24.0` and `tiktoken==0.2.0`.
+model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
 
+messages = [
+    {"role": "user", "content": "What is your favourite condiment?"},
+    {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
+    {"role": "user", "content": "Do you have mayonnaise recipes?"}
+]
 
-## Getting help
+inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")
 
-Please post questions in the [issue tracker](https://github.com/openai/tiktoken/issues).
+outputs = model.generate(inputs, max_new_tokens=20)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
 
-If you work at OpenAI, make sure to check the internal documentation or feel free to contact
-@shantanu.
+By default, transformers will load the model in full precision. Therefore you might be interested to further reduce down the memory requirements to run the model through the optimizations we offer in HF ecosystem:
 
-## What is BPE anyway?
+### In half-precision
 
-Language models don't see text like you and I, instead they see a sequence of numbers (known as tokens).
-Byte pair encoding (BPE) is a way of converting text into tokens. It has a couple desirable
-properties:
-1) It's reversible and lossless, so you can convert tokens back into the original text
-2) It works on arbitrary text, even text that is not in the tokeniser's training data
-3) It compresses the text: the token sequence is shorter than the bytes corresponding to the
-   original text. On average, in practice, each token corresponds to about 4 bytes.
-4) It attempts to let the model see common subwords. For instance, "ing" is a common subword in
-   English, so BPE encodings will often split "encoding" into tokens like "encod" and "ing"
-   (instead of e.g. "enc" and "oding"). Because the model will then see the "ing" token again and
-   again in different contexts, it helps models generalise and better understand grammar.
+Note `float16` precision only works on GPU devices
 
-`tiktoken` contains an educational submodule that is friendlier if you want to learn more about
-the details of BPE, including code that helps visualise the BPE procedure:
-```python
-from cognitivecomputations/dolphin-2.9-llama3-70b import *
+<details>
+<summary> Click to expand </summary>
+
+```diff
++ import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
 
-# Train a BPE tokeniser on a small amount of text
-enc = train_simple_encoding(cognitivecomputations/dolphin-2.9-llama3-70b)
+model_id = "mistralai/Mixtral-8x7B-Instruct-v0.1"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
 
-# Visualise how the GPT-4 encoder encodes text
-enc = SimpleBytePairEncoding.from_tiktoken("cl100k_base")
-enc.encode("hello world e")
++ model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")
+
+messages = [
+    {"role": "user", "content": "What is your favourite condiment?"},
+    {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
+    {"role": "user", "content": "Do you have mayonnaise recipes?"}
+]
+
+input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")
+
+outputs = model.generate(input_ids, max_new_tokens=20)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
 ```
+</details>
 
+### Lower precision using (8-bit & 4-bit) using `bitsandbytes`
 
-## Extending tiktoken
+<details>
+<summary> Click to expand </summary>
 
-You may wish to extend `tiktoken` to support new encodings. There are two ways to do this.
+```diff
++ import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
 
+model_id = "mistralai/Mixtral-8x7B-Instruct-v0.1"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
 
-**Create your `Encoding` object exactly the way you want and simply pass it around.**
++ model = AutoModelForCausalLM.from_pretrained(model_id, load_in_4bit=True, device_map="auto")
 
-```python
-cl100k_base = tiktoken.get_encoding("cl100k_base")
-
-# In production, load the arguments directly instead of accessing private attributes
-# See openai_public.py for examples of arguments for specific encodings
-enc = tiktoken.Encoding(
-    # If you're changing the set of special tokens, make sure to use a different name
-    # It should be clear from the name what behaviour to expect.
-    name="cl100k_im",
-    pat_str=cl100k_base._pat_str,
-    mergeable_ranks=cl100k_base._mergeable_ranks,
-    special_tokens={
-        **cl100k_base._special_tokens,
-        "<|im_start|>": 100264,
-        "<|im_end|>": 100265,
-    }
-)
+text = "Hello my name is"
+messages = [
+    {"role": "user", "content": "What is your favourite condiment?"},
+    {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
+    {"role": "user", "content": "Do you have mayonnaise recipes?"}
+]
+
+input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")
+
+outputs = model.generate(input_ids, max_new_tokens=20)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
 ```
+</details>
 
-**Use the `tiktoken_ext` plugin mechanism to register your `Encoding` objects with `tiktoken`.**
+### Load the model with Flash Attention 2
 
-This is only useful if you need `tiktoken.get_encoding` to find your encoding, otherwise prefer
-option 1.
+<details>
+<summary> Click to expand </summary>
 
-To do this, you'll need to create a namespace package under `tiktoken_ext`.
+```diff
++ import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
 
-Layout your project like this, making sure to omit the `tiktoken_ext/__init__.py` file:
-```
-my_tiktoken_extension
-├── tiktoken_ext
-│   └── my_encodings.py
-└── setup.py
-```
+model_id = "mistralai/Mixtral-8x7B-Instruct-v0.1"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
 
-`my_encodings.py` should be a module that contains a variable named `ENCODING_CONSTRUCTORS`.
-This is a dictionary from an encoding name to a function that takes no arguments and returns
-arguments that can be passed to `tiktoken.Encoding` to construct that encoding. For an example, see
-`tiktoken_ext/openai_public.py`. For precise details, see `tiktoken/registry.py`.
++ model = AutoModelForCausalLM.from_pretrained(model_id, use_flash_attention_2=True, device_map="auto")
 
-Your `setup.py` should look something like this:
-```python
-from setuptools import setup, find_namespace_packages
-
-setup(
-    name="my_tiktoken_extension",
-    packages=find_namespace_packages(include=['tiktoken_ext*']),
-    install_requires=["tiktoken"],
-    ...
-)
+messages = [
+    {"role": "user", "content": "What is your favourite condiment?"},
+    {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
+    {"role": "user", "content": "Do you have mayonnaise recipes?"}
+]
+
+input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")
+
+outputs = model.generate(input_ids, max_new_tokens=20)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
 ```
+</details>
+
+## Limitations
+
+The Mixtral-8x7B Instruct model is a quick demonstration that the base model can be easily fine-tuned to achieve compelling performance. 
+It does not have any moderation mechanisms. We're looking forward to engaging with the community on ways to
+make the model finely respect guardrails, allowing for deployment in environments requiring moderated outputs.
 
-Then simply `pip install ./my_tiktoken_extension` and you should be able to use your
-custom encodings! Make sure **not** to use an editable install.
+# The Mistral AI Team
+Eduardo ruiz