--- tags: - gptq - 4bit - int4 - gptqmodel - modelcloud - llama-3.1 - 70b - instruct license: llama3.1 --- This model has been quantized using [GPTQModel](https://github.com/ModelCloud/GPTQModel). - **bits**: 4 - **group_size**: 128 - **desc_act**: true - **static_groups**: false - **sym**: true - **lm_head**: false - **damp_percent**: 0.0025 - **true_sequential**: true - **model_name_or_path**: "" - **model_file_base_name**: "model" - **quant_method**: "gptq" - **checkpoint_format**: "gptq" - **meta**： - **quantizer**: "gptqmodel:0.9.9-dev0" ## Example: ```python from transformers import AutoTokenizer from gptqmodel import GPTQModel model_name = "ModelCloud/Meta-Llama-3.1-70B-Instruct-gptq-4bit" prompt = [{"role": "user", "content": "I am in Shanghai, preparing to visit the natural history museum. Can you tell me the best way to"}] tokenizer = AutoTokenizer.from_pretrained(model_name) model = GPTQModel.from_quantized(model_name) input_tensor = tokenizer.apply_chat_template(prompt, add_generation_prompt=True, return_tensors="pt") outputs = model.generate(input_ids=input_tensor.to(model.device), max_new_tokens=100) result = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True) print(result) ``` ## lm-eval benchmark: ``` | Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr| |---------------------------------------|------:|------|-----:|----------|---|-----:|---|-----:| |arc_challenge | 1|none | 0|acc |↑ |0.6186|± |0.0142| | | |none | 0|acc_norm |↑ |0.6297|± |0.0141| |arc_easy | 1|none | 0|acc |↑ |0.8628|± |0.0071| | | |none | 0|acc_norm |↑ |0.8338|± |0.0076| |boolq | 2|none | 0|acc |↑ |0.8761|± |0.0058| |hellaswag | 1|none | 0|acc |↑ |0.6463|± |0.0048| | | |none | 0|acc_norm |↑ |0.8389|± |0.0037| |lambada_openai | 1|none | 0|acc |↑ |0.7561|± |0.0060| | | |none | 0|perplexity|↓ |3.0311|± |0.0639| |mmlu | 1|none | |acc |↑ |0.8100|± |0.0032| | - humanities | 1|none | |acc |↑ |0.7981|± |0.0057| | - formal_logic | 0|none | 0|acc |↑ |0.6349|± |0.0431| | - high_school_european_history | 0|none | 0|acc |↑ |0.8545|± |0.0275| | - high_school_us_history | 0|none | 0|acc |↑ |0.9412|± |0.0165| | - high_school_world_history | 0|none | 0|acc |↑ |0.9198|± |0.0177| | - international_law | 0|none | 0|acc |↑ |0.9008|± |0.0273| | - jurisprudence | 0|none | 0|acc |↑ |0.8796|± |0.0315| | - logical_fallacies | 0|none | 0|acc |↑ |0.8650|± |0.0268| | - moral_disputes | 0|none | 0|acc |↑ |0.8266|± |0.0204| | - moral_scenarios | 0|none | 0|acc |↑ |0.8559|± |0.0117| | - philosophy | 0|none | 0|acc |↑ |0.8360|± |0.0210| | - prehistory | 0|none | 0|acc |↑ |0.8827|± |0.0179| | - professional_law | 0|none | 0|acc |↑ |0.6675|± |0.0120| | - world_religions | 0|none | 0|acc |↑ |0.9181|± |0.0210| | - other | 1|none | |acc |↑ |0.8304|± |0.0064| | - business_ethics | 0|none | 0|acc |↑ |0.7900|± |0.0409| | - clinical_knowledge | 0|none | 0|acc |↑ |0.8566|± |0.0216| | - college_medicine | 0|none | 0|acc |↑ |0.7630|± |0.0324| | - global_facts | 0|none | 0|acc |↑ |0.5800|± |0.0496| | - human_aging | 0|none | 0|acc |↑ |0.8206|± |0.0257| | - management | 0|none | 0|acc |↑ |0.8835|± |0.0318| | - marketing | 0|none | 0|acc |↑ |0.9231|± |0.0175| | - medical_genetics | 0|none | 0|acc |↑ |0.9400|± |0.0239| | - miscellaneous | 0|none | 0|acc |↑ |0.9144|± |0.0100| | - nutrition | 0|none | 0|acc |↑ |0.8660|± |0.0195| | - professional_accounting | 0|none | 0|acc |↑ |0.6454|± |0.0285| | - professional_medicine | 0|none | 0|acc |↑ |0.8971|± |0.0185| | - virology | 0|none | 0|acc |↑ |0.5602|± |0.0386| | - social sciences | 1|none | |acc |↑ |0.8736|± |0.0059| | - econometrics | 0|none | 0|acc |↑ |0.7018|± |0.0430| | - high_school_geography | 0|none | 0|acc |↑ |0.9242|± |0.0189| | - high_school_government_and_politics| 0|none | 0|acc |↑ |0.9741|± |0.0115| | - high_school_macroeconomics | 0|none | 0|acc |↑ |0.8410|± |0.0185| | - high_school_microeconomics | 0|none | 0|acc |↑ |0.8992|± |0.0196| | - high_school_psychology | 0|none | 0|acc |↑ |0.9229|± |0.0114| | - human_sexuality | 0|none | 0|acc |↑ |0.8779|± |0.0287| | - professional_psychology | 0|none | 0|acc |↑ |0.8497|± |0.0145| | - public_relations | 0|none | 0|acc |↑ |0.7273|± |0.0427| | - security_studies | 0|none | 0|acc |↑ |0.8163|± |0.0248| | - sociology | 0|none | 0|acc |↑ |0.9154|± |0.0197| | - us_foreign_policy | 0|none | 0|acc |↑ |0.9300|± |0.0256| | - stem | 1|none | |acc |↑ |0.7456|± |0.0075| | - abstract_algebra | 0|none | 0|acc |↑ |0.6300|± |0.0485| | - anatomy | 0|none | 0|acc |↑ |0.7926|± |0.0350| | - astronomy | 0|none | 0|acc |↑ |0.8947|± |0.0250| | - college_biology | 0|none | 0|acc |↑ |0.9444|± |0.0192| | - college_chemistry | 0|none | 0|acc |↑ |0.5800|± |0.0496| | - college_computer_science | 0|none | 0|acc |↑ |0.6700|± |0.0473| | - college_mathematics | 0|none | 0|acc |↑ |0.5400|± |0.0501| | - college_physics | 0|none | 0|acc |↑ |0.6275|± |0.0481| | - computer_security | 0|none | 0|acc |↑ |0.8200|± |0.0386| | - conceptual_physics | 0|none | 0|acc |↑ |0.7830|± |0.0269| | - electrical_engineering | 0|none | 0|acc |↑ |0.7862|± |0.0342| | - elementary_mathematics | 0|none | 0|acc |↑ |0.7593|± |0.0220| | - high_school_biology | 0|none | 0|acc |↑ |0.9194|± |0.0155| | - high_school_chemistry | 0|none | 0|acc |↑ |0.7143|± |0.0318| | - high_school_computer_science | 0|none | 0|acc |↑ |0.9200|± |0.0273| | - high_school_mathematics | 0|none | 0|acc |↑ |0.5185|± |0.0305| | - high_school_physics | 0|none | 0|acc |↑ |0.6556|± |0.0388| | - high_school_statistics | 0|none | 0|acc |↑ |0.7361|± |0.0301| | - machine_learning | 0|none | 0|acc |↑ |0.7054|± |0.0433| |openbookqa | 1|none | 0|acc |↑ |0.3660|± |0.0216| | | |none | 0|acc_norm |↑ |0.4620|± |0.0223| |piqa | 1|none | 0|acc |↑ |0.8264|± |0.0088| | | |none | 0|acc_norm |↑ |0.8319|± |0.0087| |rte | 1|none | 0|acc |↑ |0.7184|± |0.0271| |truthfulqa_mc1 | 2|none | 0|acc |↑ |0.3917|± |0.0171| |winogrande | 1|none | 0|acc |↑ |0.7924|± |0.0114| | Groups |Version|Filter|n-shot|Metric| |Value | |Stderr| |------------------|------:|------|------|------|---|-----:|---|-----:| |mmlu | 1|none | |acc |↑ |0.8100|± |0.0032| | - humanities | 1|none | |acc |↑ |0.7981|± |0.0057| | - other | 1|none | |acc |↑ |0.8304|± |0.0064| | - social sciences| 1|none | |acc |↑ |0.8736|± |0.0059| | - stem | 1|none | |acc |↑ |0.7456|± |0.0075| ```