|
--- |
|
base_model: mistralai/Mixtral-8x22B-Instruct-v0.1 |
|
model_creator: mistralai |
|
quantized_by: jartine |
|
license: apache-2.0 |
|
prompt_template: | |
|
[INST] {{prompt}} [/INST] |
|
tags: |
|
- llamafile |
|
language: |
|
- en |
|
--- |
|
|
|
# Mixtral 8x22B Instruct v0.1 - llamafile |
|
|
|
This repository contains executable weights (which we call |
|
[llamafiles](https://github.com/Mozilla-Ocho/llamafile)) that run on |
|
Linux, MacOS, Windows, FreeBSD, OpenBSD, and NetBSD for AMD64 and ARM64. |
|
|
|
- Model creator: [Mistral AI](https://mistral.ai/) |
|
- Original model: [mistralai/Mixtral-8x22B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1) |
|
|
|
## Quickstart |
|
|
|
Assuming your system has at least 128GB of RAM, you can try running the |
|
following command which download, concatenate, and execute the model. |
|
|
|
``` |
|
( curl -L https://huggingface.co/jartine/Mixtral-8x22B-Instruct-v0.1-llamafile/resolve/main/Mixtral-8x22B-Instruct-v0.1.Q4_0.llamafile.cat0 |
|
curl -L https://huggingface.co/jartine/Mixtral-8x22B-Instruct-v0.1-llamafile/resolve/main/Mixtral-8x22B-Instruct-v0.1.Q4_0.llamafile.cat1 |
|
) > Mixtral-8x22B-Instruct-v0.1.Q4_0.llamafile |
|
chmod +x Mixtral-8x22B-Instruct-v0.1.Q4_0.llamafile |
|
./Mixtral-8x22B-Instruct-v0.1.Q4_0.llamafile --help # view manual |
|
./Mixtral-8x22B-Instruct-v0.1.Q4_0.llamafile # launch web gui + oai api |
|
./Mixtral-8x22B-Instruct-v0.1.Q4_0.llamafile -p ... # cli interface (scriptable) |
|
``` |
|
|
|
Alternatively, you may download an official `llamafile` executable from |
|
Mozilla Ocho on GitHub, in which case you can use the Mixtral llamafiles |
|
as a simple weights data file. |
|
|
|
``` |
|
llamafile -m Mixtral-8x22B-Instruct-v0.1.Q4_0.llamafile ... |
|
``` |
|
|
|
For further information, please see the [llamafile |
|
README](https://github.com/mozilla-ocho/llamafile/). |
|
|
|
Having **trouble?** See the ["Gotchas" |
|
section](https://github.com/mozilla-ocho/llamafile/?tab=readme-ov-file#gotchas-and-troubleshooting) |
|
of the README. |
|
|
|
## Prompting |
|
|
|
Prompt template: |
|
|
|
``` |
|
[INST] {{prompt}} [/INST] |
|
``` |
|
|
|
Command template: |
|
|
|
``` |
|
./Mixtral-8x22B-Instruct-v0.1.Q4_0.llamafile -p "[INST]{{prompt}}[/INST]" |
|
``` |
|
|
|
## About llamafile |
|
|
|
llamafile is a new format introduced by Mozilla Ocho on Nov 20th 2023. |
|
It uses Cosmopolitan Libc to turn LLM weights into runnable llama.cpp |
|
binaries that run on the stock installs of six OSes for both ARM64 and |
|
AMD64. |
|
|
|
In addition to being executables, llamafiles are also zip archives. Each |
|
llamafile contains a GGUF file, which you can extract using the `unzip` |
|
command. If you want to change or add files to your llamafiles, then the |
|
`zipalign` command (distributed on the llamafile github) should be used |
|
instead of the traditional `zip` command. |
|
|
|
## About Upload Limits |
|
|
|
Files which exceed the Hugging Face 50GB upload limit have a .cat𝑋 |
|
extension. You need to use the `cat` command locally to turn them back |
|
into a single file, using the same order. |
|
|
|
## About Quantization Formats (General Advice) |
|
|
|
Your choice of quantization format depends on three things: |
|
|
|
1. Will it fit in RAM or VRAM? |
|
2. Is your use case reading (e.g. summarization) or writing (e.g. chatbot)? |
|
3. llamafiles bigger than 4.30 GB are hard to run on Windows (see [gotchas](https://github.com/mozilla-ocho/llamafile/?tab=readme-ov-file#gotchas-and-troubleshooting)) |
|
|
|
Good quants for writing (prediction speed) are Q5\_K\_M, and Q4\_0. Text |
|
generation is bounded by memory speed, so smaller quants help, but they |
|
cause the LLM to hallucinate more. However that doesn't mean they can't |
|
think correctly. A highly degraded quant like `Q2_K` may not make a |
|
great encyclopedia, but it's still capable of logical reasoning and |
|
the emergent capabilities LLMs exhibit. |
|
|
|
Good quants for reading (evaluation speed) are BF16, F16, Q8\_0, and |
|
Q4\_0 (ordered from fastest to slowest). Prompt evaluation is bounded by |
|
flop count, which means perf can be improved through software |
|
engineering alone, e.g. BLAS algorithms, in which case quantization |
|
starts hurting more than it helps, since it competes for CPU resources |
|
and makes it harder for the compiler to parallelize instructions. You |
|
want to ideally use the simplest smallest floating point format that's |
|
natively implemented by your hardware. In most cases, that's BF16 or |
|
FP16. However, llamafile is able to still offer respectable tinyBLAS |
|
speedups for llama.cpp's simplest quants: Q8\_0 and Q4\_0. |
|
|
|
## Hardware Choices (Mixtral 8x22B Specific) |
|
|
|
This model is very large. Even at Q2 quantization, it's still well-over |
|
twice as large the highest tier NVIDIA gaming GPUs. llamafile supports |
|
splitting models over multiple GPUs (for NVIDIA only currently) if you |
|
have such a system. The easiest way to have one, if you don't, is to pay |
|
a few bucks an hour to rent a 4x RTX 4090 rig off vast.ai. |
|
|
|
Mac Studio is a good option for running this model locally. An M2 Ultra |
|
desktop from Apple is affordable and has 128GB of unified RAM+VRAM. If |
|
you have one, then llamafile will use your Metal GPU. Try starting out |
|
with the `Q4_0` quantization level. |
|
|
|
Another good option for running large, large language models locally and |
|
fully under your control is to just use CPU inference. We developed new |
|
tensor multiplication kernels on the llamafile project specifically to |
|
speed up "mixture of experts" LLMs like Mixtral. On a AMD Threadripper |
|
Pro 7995WX with 256GB of 5200 MT/s RAM, llamafile v0.8 runs Mixtral |
|
8x22B Q4\_0 on Linux at 98 tokens per second for evaluation, and it |
|
predicts 9.44 tokens per second. |
|
|
|
--- |
|
|
|
# Model Card for Mixtral-8x22B-Instruct-v0.1 |
|
The Mixtral-8x22B-Instruct-v0.1 Large Language Model (LLM) is an instruct fine-tuned version of the [Mixtral-8x22B-v0.1](https://huggingface.co/mistralai/Mixtral-8x22B-v0.1). |
|
|
|
## Run the model |
|
```python |
|
from transformers import AutoModelForCausalLM |
|
from mistral_common.protocol.instruct.messages import ( |
|
AssistantMessage, |
|
UserMessage, |
|
) |
|
from mistral_common.protocol.instruct.tool_calls import ( |
|
Tool, |
|
Function, |
|
) |
|
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer |
|
from mistral_common.tokens.instruct.normalize import ChatCompletionRequest |
|
|
|
device = "cuda" # the device to load the model onto |
|
|
|
tokenizer_v3 = MistralTokenizer.v3() |
|
|
|
mistral_query = ChatCompletionRequest( |
|
tools=[ |
|
Tool( |
|
function=Function( |
|
name="get_current_weather", |
|
description="Get the current weather", |
|
parameters={ |
|
"type": "object", |
|
"properties": { |
|
"location": { |
|
"type": "string", |
|
"description": "The city and state, e.g. San Francisco, CA", |
|
}, |
|
"format": { |
|
"type": "string", |
|
"enum": ["celsius", "fahrenheit"], |
|
"description": "The temperature unit to use. Infer this from the users location.", |
|
}, |
|
}, |
|
"required": ["location", "format"], |
|
}, |
|
) |
|
) |
|
], |
|
messages=[ |
|
UserMessage(content="What's the weather like today in Paris"), |
|
], |
|
model="test", |
|
) |
|
|
|
encodeds = tokenizer_v3.encode_chat_completion(mistral_query).tokens |
|
model = AutoModelForCausalLM.from_pretrained("mistralai/Mixtral-8x22B-Instruct-v0.1") |
|
model_inputs = encodeds.to(device) |
|
model.to(device) |
|
|
|
generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True) |
|
sp_tokenizer = tokenizer_v3.instruct_tokenizer.tokenizer |
|
decoded = sp_tokenizer.decode(generated_ids[0]) |
|
print(decoded) |
|
``` |
|
|
|
# Instruct tokenizer |
|
The HuggingFace tokenizer included in this release should match our own. To compare: |
|
`pip install mistral-common` |
|
|
|
```py |
|
from mistral_common.protocol.instruct.messages import ( |
|
AssistantMessage, |
|
UserMessage, |
|
) |
|
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer |
|
from mistral_common.tokens.instruct.normalize import ChatCompletionRequest |
|
|
|
from transformers import AutoTokenizer |
|
|
|
tokenizer_v3 = MistralTokenizer.v3() |
|
|
|
mistral_query = ChatCompletionRequest( |
|
messages=[ |
|
UserMessage(content="How many experts ?"), |
|
AssistantMessage(content="8"), |
|
UserMessage(content="How big ?"), |
|
AssistantMessage(content="22B"), |
|
UserMessage(content="Noice 🎉 !"), |
|
], |
|
model="test", |
|
) |
|
hf_messages = mistral_query.model_dump()['messages'] |
|
|
|
tokenized_mistral = tokenizer_v3.encode_chat_completion(mistral_query).tokens |
|
|
|
tokenizer_hf = AutoTokenizer.from_pretrained('mistralai/Mixtral-8x22B-Instruct-v0.1') |
|
tokenized_hf = tokenizer_hf.apply_chat_template(hf_messages, tokenize=True) |
|
|
|
assert tokenized_hf == tokenized_mistral |
|
``` |
|
|
|
# Function calling and special tokens |
|
This tokenizer includes more special tokens, related to function calling : |
|
- [TOOL_CALLS] |
|
- [AVAILABLE_TOOLS] |
|
- [/AVAILABLE_TOOLS] |
|
- [TOOL_RESULTS] |
|
- [/TOOL_RESULTS] |
|
|
|
If you want to use this model with function calling, please be sure to apply it similarly to what is done in our [SentencePieceTokenizerV3](https://github.com/mistralai/mistral-common/blob/main/src/mistral_common/tokens/tokenizers/sentencepiece.py#L299). |
|
|
|
# The Mistral AI Team |
|
Albert Jiang, Alexandre Sablayrolles, Alexis Tacnet, Antoine Roux, |
|
Arthur Mensch, Audrey Herblin-Stoop, Baptiste Bout, Baudouin de Monicault, |
|
Blanche Savary, Bam4d, Caroline Feldman, Devendra Singh Chaplot, |
|
Diego de las Casas, Eleonore Arcelin, Emma Bou Hanna, Etienne Metzger, |
|
Gianna Lengyel, Guillaume Bour, Guillaume Lample, Harizo Rajaona, |
|
Jean-Malo Delignon, Jia Li, Justus Murke, Louis Martin, Louis Ternon, |
|
Lucile Saulnier, Lélio Renard Lavaud, Margaret Jennings, Marie Pellat, |
|
Marie Torelli, Marie-Anne Lachaux, Nicolas Schuhl, Patrick von Platen, |
|
Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, |
|
Thibaut Lavril, Timothée Lacroix, Théophile Gervet, Thomas Wang, |
|
Valera Nemychnikova, William El Sayed, William Marshall |
|
|
