--- tags: - w8a8 - int8 - vllm license: apache-2.0 license_link: https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/apache-2.0.md language: - en base_model: ibm-granite/granite-3.1-8b-instruct library_name: transformers --- # granite-3.1-8b-instruct-quantized.w8a8 ## Model Overview - **Model Architecture:** granite-3.1-8b-instruct - **Input:** Text - **Output:** Text - **Model Optimizations:** - **Weight quantization:** INT8 - **Activation quantization:** INT8 - **Release Date:** 1/8/2025 - **Version:** 1.0 - **Model Developers:** Neural Magic Quantized version of [ibm-granite/granite-3.1-8b-instruct](https://huggingface.co/ibm-granite/granite-3.1-8b-instruct). It achieves an average score of 70.26 on the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) benchmark (version 1), whereas the unquantized model achieves 70.30. ### Model Optimizations This model was obtained by quantizing the weights and activations of [ibm-granite/granite-3.1-8b-instruct](https://huggingface.co/ibm-granite/granite-3.1-8b-instruct) to INT8 data type, ready for inference with vLLM >= 0.5.2. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%. Only the weights and activations of the linear operators within transformers blocks are quantized. ## Deployment ### Use with vLLM This model can be deployed efficiently using the [vLLM](https://docs.vllm.ai/en/latest/) backend, as shown in the example below. ```python from transformers import AutoTokenizer from vllm import LLM, SamplingParams max_model_len, tp_size = 4096, 1 model_name = "neuralmagic-ent/granite-3.1-8b-instruct-quantized.w8a8" tokenizer = AutoTokenizer.from_pretrained(model_name) llm = LLM(model=model_name, tensor_parallel_size=tp_size, max_model_len=max_model_len, trust_remote_code=True) sampling_params = SamplingParams(temperature=0.3, max_tokens=256, stop_token_ids=[tokenizer.eos_token_id]) messages_list = [ [{"role": "user", "content": "Who are you? Please respond in pirate speak!"}], ] prompt_token_ids = [tokenizer.apply_chat_template(messages, add_generation_prompt=True) for messages in messages_list] outputs = llm.generate(prompt_token_ids=prompt_token_ids, sampling_params=sampling_params) generated_text = [output.outputs[0].text for output in outputs] print(generated_text) ``` vLLM also supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details. ## Creation This model was created with [llm-compressor](https://github.com/vllm-project/llm-compressor) by running the code snippet below. ```bash python quantize.py --model_path ibm-granite/granite-3.1-8b-instruct --quant_path "output_dir/granite-3.1-8b-instruct-quantized.w8a8" --calib_size 3072 --dampening_frac 0.1 --observer mse ``` ```python from datasets import load_dataset from transformers import AutoTokenizer, AutoModelForCausalLM from llmcompressor.modifiers.quantization import GPTQModifier from llmcompressor.modifiers.smoothquant import SmoothQuantModifier from llmcompressor.transformers import oneshot, apply import argparse from compressed_tensors.quantization import QuantizationScheme, QuantizationArgs, QuantizationType, QuantizationStrategy parser = argparse.ArgumentParser() parser.add_argument('--model_path', type=str) parser.add_argument('--quant_path', type=str) parser.add_argument('--calib_size', type=int, default=256) parser.add_argument('--dampening_frac', type=float, default=0.1) parser.add_argument('--observer', type=str, default="minmax") args = parser.parse_args() model = AutoModelForCausalLM.from_pretrained( args.model_path, device_map="auto", torch_dtype="auto", use_cache=False, trust_remote_code=True, ) tokenizer = AutoTokenizer.from_pretrained(args.model_path) NUM_CALIBRATION_SAMPLES = args.calib_size DATASET_ID = "neuralmagic/LLM_compression_calibration" DATASET_SPLIT = "train" ds = load_dataset(DATASET_ID, split=DATASET_SPLIT) ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES)) def preprocess(example): concat_txt = example["instruction"] + "\n" + example["output"] return {"text": concat_txt} ds = ds.map(preprocess) def tokenize(sample): return tokenizer( sample["text"], padding=False, truncation=False, add_special_tokens=True, ) ds = ds.map(tokenize, remove_columns=ds.column_names) ignore=["lm_head"] mappings=[ [["re:.*q_proj", "re:.*k_proj", "re:.*v_proj"], "re:.*input_layernorm"], [["re:.*gate_proj", "re:.*up_proj"], "re:.*post_attention_layernorm"], [["re:.*down_proj"], "re:.*up_proj"] ] recipe = [ SmoothQuantModifier(smoothing_strength=0.8, ignore=ignore, mappings=mappings), GPTQModifier( targets=["Linear"], ignore=["lm_head"], scheme="W8A8", dampening_frac=args.dampening_frac, observer=args.observer, ) ] oneshot( model=model, dataset=ds, recipe=recipe, num_calibration_samples=args.calib_size, max_seq_length=8196, ) # Save to disk compressed. model.save_pretrained(SAVE_DIR, save_compressed=True) tokenizer.save_pretrained(SAVE_DIR) ``` ## Evaluation The model was evaluated on OpenLLM Leaderboard [V1](https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard) and on [HumanEval](https://github.com/neuralmagic/evalplus), using the following commands: OpenLLM Leaderboard V1: ``` lm_eval \ --model vllm \ --model_args pretrained="neuralmagic-ent/granite-3.1-8b-instruct-quantized.w8a8",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=1,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True \ --tasks openllm \ --write_out \ --batch_size auto \ --output_path output_dir \ --show_config ``` #### HumanEval ##### Generation ``` python3 codegen/generate.py \ --model neuralmagic-ent/granite-3.1-8b-instruct-quantized.w8a8 \ --bs 16 \ --temperature 0.2 \ --n_samples 50 \ --root "." \ --dataset humaneval ``` ##### Sanitization ``` python3 evalplus/sanitize.py \ humaneval/neuralmagic-ent--granite-3.1-8b-instruct-quantized.w8a8_vllm_temp_0.2 ``` ##### Evaluation ``` evalplus.evaluate \ --dataset humaneval \ --samples humaneval/neuralmagic-ent--granite-3.1-8b-instruct-quantized.w8a8_vllm_temp_0.2-sanitized ``` ### Accuracy #### OpenLLM Leaderboard V1 evaluation scores | Metric | ibm-granite/granite-3.1-8b-instruct | neuralmagic-ent/granite-3.1-8b-instruct-quantized.w8a8 | |-----------------------------------------|:---------------------------------:|:-------------------------------------------:| | ARC-Challenge (Acc-Norm, 25-shot) | 66.81 | 67.06 | | GSM8K (Strict-Match, 5-shot) | 64.52 | 65.66 | | HellaSwag (Acc-Norm, 10-shot) | 84.18 | 83.93 | | MMLU (Acc, 5-shot) | 65.52 | 65.03 | | TruthfulQA (MC2, 0-shot) | 60.57 | 60.02 | | Winogrande (Acc, 5-shot) | 80.19 | 79.87 | | **Average Score** | **70.30** | **70.26** | | **Recovery** | **100.00** | **99.95** | #### OpenLLM Leaderboard V2 evaluation scores | Metric | ibm-granite/granite-3.1-8b-instruct | neuralmagic-ent/granite-3.1-8b-instruct-quantized.w8a8 | |-----------------------------------------|:---------------------------------:|:-------------------------------------------:| | IFEval (Inst Level Strict Acc, 0-shot)| 74.01 | 73.50 | | BBH (Acc-Norm, 3-shot) | 53.19 | 52.59 | | Math-Hard (Exact-Match, 4-shot) | 14.77 | 15.73 | | GPQA (Acc-Norm, 0-shot) | 31.76 | 30.62 | | MUSR (Acc-Norm, 0-shot) | 46.01 | 44.30 | | MMLU-Pro (Acc, 5-shot) | 35.81 | 35.41 | | **Average Score** | **42.61** | **42.03** | | **Recovery** | **100.00** | **98.64** | #### HumanEval pass@1 scores | Metric | ibm-granite/granite-3.1-8b-instruct | neuralmagic-ent/granite-3.1-8b-instruct-quantized.w8a8 | |-----------------------------------------|:---------------------------------:|:-------------------------------------------:| | HumanEval Pass@1 | 71.00 | 70.50 | ## Inference Performance This model achieves up to 1.6x speedup in single-stream deployment and up to 1.7x speedup in multi-stream asynchronous deployment, depending on hardware and use-case scenario. The following performance benchmarks were conducted with [vLLM](https://docs.vllm.ai/en/latest/) version 0.6.6.post1, and [GuideLLM](https://github.com/neuralmagic/guidellm). ### Single-stream performance (measured with vLLM version 0.6.6.post1)
Latency (s)
GPU class Model Speedup Code Completion
prefill: 256 tokens
decode: 1024 tokens		 Docstring Generation
prefill: 768 tokens
decode: 128 tokens		 Code Fixing
prefill: 1024 tokens
decode: 1024 tokens		 RAG
prefill: 1024 tokens
decode: 128 tokens		 Instruction Following
prefill: 256 tokens
decode: 128 tokens		 Multi-turn Chat
prefill: 512 tokens
decode: 256 tokens		 Large Summarization
prefill: 4096 tokens
decode: 512 tokens
A5000 granite-3.1-8b-instruct 28.3 3.7 28.8 3.8 3.6 7.2 15.7
granite-3.1-8b-instruct-quantized.w8a8
(this model)		 1.60 17.7 2.3 18.0 2.4 2.2 4.5 10.0
granite-3.1-8b-instruct-quantized.w4a16 2.61 10.3 1.5 10.7 1.5 1.3 2.7 6.6
A6000 granite-3.1-8b-instruct 25.8 3.4 26.2 3.4 3.3 6.5 14.2
granite-3.1-8b-instruct-quantized.w8a8
(this model)		 1.50 17.4 2.3 16.9 2.2 2.2 4.4 9.8
granite-3.1-8b-instruct-quantized.w4a16 2.48 10.0 1.4 10.4 1.5 1.3 2.5 6.2
A100 granite-3.1-8b-instruct 13.6 1.8 13.7 1.8 1.7 3.4 7.3
granite-3.1-8b-instruct-quantized.w8a8
(this model)		 1.31 10.4 1.3 10.5 1.4 1.3 2.6 5.6
granite-3.1-8b-instruct-quantized.w4a16 1.80 7.3 1.0 7.4 1.0 0.9 1.9 4.3
### Multi-stream asynchronous performance (measured with vLLM version 0.6.6.post1)
Maximum Throughput (Queries per Second)
GPU class Model Speedup Code Completion
prefill: 256 tokens
decode: 1024 tokens		 Docstring Generation
prefill: 768 tokens
decode: 128 tokens		 Code Fixing
prefill: 1024 tokens
decode: 1024 tokens		 RAG
prefill: 1024 tokens
decode: 128 tokens		 Instruction Following
prefill: 256 tokens
decode: 128 tokens		 Multi-turn Chat
prefill: 512 tokens
decode: 256 tokens		 Large Summarization
prefill: 4096 tokens
decode: 512 tokens
A5000 granite-3.1-8b-instruct 0.8 3.1 0.4 2.5 6.7 2.7 0.3
granite-3.1-8b-instruct-quantized.w8a8
(this model)		 1.71 1.3 5.2 0.9 4.0 10.5 4.4 0.5
granite-3.1-8b-instruct-quantized.w4a16 1.46 1.3 3.9 0.8 2.9 8.2 3.6 0.5
A6000 granite-3.1-8b-instruct 1.3 5.1 0.9 4.0 0.3 4.3 0.6
granite-3.1-8b-instruct-quantized.w8a8
(this model)		 1.39 1.8 7.0 1.3 5.6 14.0 6.3 0.8
granite-3.1-8b-instruct-quantized.w4a16 1.09 1.9 4.8 1.0 3.8 10.0 5.0 0.6
A100 granite-3.1-8b-instruct 3.1 10.7 2.1 8.5 20.6 9.6 1.4
granite-3.1-8b-instruct-quantized.w8a8
(this model)		 1.23 3.8 14.2 2.1 11.4 25.9 12.1 1.7
granite-3.1-8b-instruct-quantized.w4a16 0.96 3.4 9.0 2.6 7.2 18.0 8.8 1.3