RachidAR/Phi-3-mini-4k-instruct-Q6_K-GGUF (June 2024 Update)

This model was converted to GGUF format from microsoft/Phi-3-mini-4k-instruct using llama.cpp via the ggml.ai's GGUF-my-repo space. Refer to the original model card for more details on the model.

Release Notes

This is an update over the original instruction-tuned Phi-3-mini release based on valuable customer feedback. The model used additional post-training data leading to substantial gains on instruction following and structure output. We also improve multi-turn conversation quality, explicitly support <|system|> tag, and significantly improve reasoning capability. We believe most use cases will benefit from this release, but we encourage users to test in their particular AI applications. We appreciate the enthusiastic adoption of the Phi-3 model family, and continue to welcome all feedback from the community.

The table below highlights improvements on instruction following, structure output, and reasoning of the new release on publich and internal benchmark datasets.

Benchmarks	Original	June 2024 Update
Instruction Extra Hard	5.7	6.0
Instruction Hard	4.9	5.1
Instructions Challenge	24.6	42.3
JSON Structure Output	11.5	52.3
XML Structure Output	14.4	49.8
GPQA	23.7	30.6
MMLU	68.8	70.9
Average	21.9	36.7

Chat Format

Given the nature of the training data, the Phi-3 Mini-4K-Instruct model is best suited for prompts using the chat format as follows. You can provide the prompt as a question with a generic template as follow:

<|system|>
You are a helpful assistant.<|end|>
<|user|>
Question?<|end|>
<|assistant|>

For example:

<|system|>
You are a helpful assistant.<|end|>
<|user|>
How to explain Internet for a medieval knight?<|end|>
<|assistant|>

where the model generates the text after <|assistant|> . In case of few-shots prompt, the prompt can be formatted as the following:

<|system|>
You are a helpful travel assistant.<|end|>
<|user|>
I am going to Paris, what should I see?<|end|>
<|assistant|>
Paris, the capital of France, is known for its stunning architecture, art museums, historical landmarks, and romantic atmosphere. Here are some of the top attractions to see in Paris:\n\n1. The Eiffel Tower: The iconic Eiffel Tower is one of the most recognizable landmarks in the world and offers breathtaking views of the city.\n2. The Louvre Museum: The Louvre is one of the world's largest and most famous museums, housing an impressive collection of art and artifacts, including the Mona Lisa.\n3. Notre-Dame Cathedral: This beautiful cathedral is one of the most famous landmarks in Paris and is known for its Gothic architecture and stunning stained glass windows.\n\nThese are just a few of the many attractions that Paris has to offer. With so much to see and do, it's no wonder that Paris is one of the most popular tourist destinations in the world."<|end|>
<|user|>
What is so great about #1?<|end|>
<|assistant|>

Use with llama.cpp

Install llama.cpp through brew (works on Mac and Linux)

brew install llama.cpp

Invoke the llama.cpp server or the CLI.

CLI:

llama-cli --hf-repo RachidAR/Phi-3-mini-4k-instruct-Q6_K-GGUF --hf-file phi-3-mini-4k-instruct-q6_k.gguf -p "The meaning to life and the universe is"

Server:

llama-server --hf-repo RachidAR/Phi-3-mini-4k-instruct-Q6_K-GGUF --hf-file phi-3-mini-4k-instruct-q6_k.gguf -c 2048

Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.

git clone https://github.com/ggerganov/llama.cpp

Step 2: Move into the llama.cpp folder and build it with LLAMA_CURL=1 flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).

cd llama.cpp && LLAMA_CURL=1 make

Step 3: Run inference through the main binary.

./llama-cli --hf-repo RachidAR/Phi-3-mini-4k-instruct-Q6_K-GGUF --hf-file phi-3-mini-4k-instruct-q6_k.gguf -p "The meaning to life and the universe is"

./llama-server --hf-repo RachidAR/Phi-3-mini-4k-instruct-Q6_K-GGUF --hf-file phi-3-mini-4k-instruct-q6_k.gguf -c 2048