Mixtral-8x7b-Instruct-v0.1-int4-ov

Model creator: Mistral AI
Original model: Mixtral 8X7B Instruct v0.1

Description

This is Mixtral-8x7b-Instruct-v0.1 model converted to OpenVINO Intermediate Representation (IR) format with INT4 compressed weights using NNCF.

Compatibility

This provided IR is compatible with openvino starting with 2024.0.0 version and optimum-intel 1.16.0

Usage

Install required packages

To install the required components for using Optimum Intel integration with the OpenVINO backend, do:

pip install optimum[openvino]

Run model inference

from transformers import AutoTokenizer
from optimum.intel.openvino import OVModelForCausalLM

model_id = "mistralai/Mixtral-8x7B-Instruct-v0.1"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = OVModelForCausalLM.from_pretrained(model_id)


messages = [
    {"role": "user", "content": "What is your favourite condiment?"},
    {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
    {"role": "user", "content": "Do you have mayonnaise recipes?"}
]

inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")

outputs = model.generate(inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

For more examples and possible optimizations please refer OpenVINO Large Language Model Inference Guide

Limitations

Please check original model card for model usage limitations