--- tags: - gptq - 4bit - int4 - gptqmodel - modelcloud - llama-3.1 - 405b - instruct license: llama3.1 --- This model has been quantized using [GPTQModel](https://github.com/ModelCloud/GPTQModel). - **bits**: 4 - **group_size**: 128 - **desc_act**: true - **static_groups**: false - **sym**: true - **lm_head**: false - **damp_percent**: 0.01 - **true_sequential**: true - **model_name_or_path**: "" - **model_file_base_name**: "model" - **quant_method**: "gptq" - **checkpoint_format**: "gptq" - **meta**： - **quantizer**: "gptqmodel:0.9.9-dev0" ## Example: ```python from transformers import AutoTokenizer from gptqmodel import GPTQModel model_name = "ModelCloud/Meta-Llama-3.1-405B-Instruct-gptq-4bit" prompt = [{"role": "user", "content": "I am in Shanghai, preparing to visit the natural history museum. Can you tell me the best way to"}] tokenizer = AutoTokenizer.from_pretrained(model_name) model = GPTQModel.from_quantized(model_name) input_tensor = tokenizer.apply_chat_template(prompt, add_generation_prompt=True, return_tensors="pt") outputs = model.generate(input_ids=input_tensor.to(model.device), max_new_tokens=100) result = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True) print(result) ``` ## lm-eval benchmark ``` | Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr| |---------------------------------------|------:|------|-----:|----------|---|-----:|---|-----:| |arc_challenge | 1|none | 0|acc |↑ |0.5990|± |0.0143| | | |none | 0|acc_norm |↑ |0.6425|± |0.0140| |arc_easy | 1|none | 0|acc |↑ |0.8645|± |0.0070| | | |none | 0|acc_norm |↑ |0.8359|± |0.0076| |boolq | 2|none | 0|acc |↑ |0.8820|± |0.0056| |hellaswag | 1|none | 0|acc |↑ |0.6560|± |0.0047| | | |none | 0|acc_norm |↑ |0.8446|± |0.0036| |lambada_openai | 1|none | 0|acc |↑ |0.7252|± |0.0062| | | |none | 0|perplexity|↓ |3.5576|± |0.0880| |mmlu | 1|none | |acc |↑ |0.8245|± |0.0031| | - humanities | 1|none | |acc |↑ |0.7892|± |0.0058| | - formal_logic | 0|none | 0|acc |↑ |0.6349|± |0.0431| | - high_school_european_history | 0|none | 0|acc |↑ |0.8667|± |0.0265| | - high_school_us_history | 0|none | 0|acc |↑ |0.9314|± |0.0177| | - high_school_world_history | 0|none | 0|acc |↑ |0.9367|± |0.0158| | - international_law | 0|none | 0|acc |↑ |0.9091|± |0.0262| | - jurisprudence | 0|none | 0|acc |↑ |0.8796|± |0.0315| | - logical_fallacies | 0|none | 0|acc |↑ |0.8834|± |0.0252| | - moral_disputes | 0|none | 0|acc |↑ |0.8295|± |0.0202| | - moral_scenarios | 0|none | 0|acc |↑ |0.7888|± |0.0137| | - philosophy | 0|none | 0|acc |↑ |0.8521|± |0.0202| | - prehistory | 0|none | 0|acc |↑ |0.8735|± |0.0185| | - professional_law | 0|none | 0|acc |↑ |0.6760|± |0.0120| | - world_religions | 0|none | 0|acc |↑ |0.8830|± |0.0246| | - other | 1|none | |acc |↑ |0.8539|± |0.0060| | - business_ethics | 0|none | 0|acc |↑ |0.8100|± |0.0394| | - clinical_knowledge | 0|none | 0|acc |↑ |0.8679|± |0.0208| | - college_medicine | 0|none | 0|acc |↑ |0.7688|± |0.0321| | - global_facts | 0|none | 0|acc |↑ |0.7000|± |0.0461| | - human_aging | 0|none | 0|acc |↑ |0.8341|± |0.0250| | - management | 0|none | 0|acc |↑ |0.8932|± |0.0306| | - marketing | 0|none | 0|acc |↑ |0.9444|± |0.0150| | - medical_genetics | 0|none | 0|acc |↑ |0.9300|± |0.0256| | - miscellaneous | 0|none | 0|acc |↑ |0.9425|± |0.0083| | - nutrition | 0|none | 0|acc |↑ |0.8987|± |0.0173| | - professional_accounting | 0|none | 0|acc |↑ |0.6773|± |0.0279| | - professional_medicine | 0|none | 0|acc |↑ |0.9228|± |0.0162| | - virology | 0|none | 0|acc |↑ |0.5542|± |0.0387| | - social sciences | 1|none | |acc |↑ |0.8833|± |0.0057| | - econometrics | 0|none | 0|acc |↑ |0.7193|± |0.0423| | - high_school_geography | 0|none | 0|acc |↑ |0.9394|± |0.0170| | - high_school_government_and_politics| 0|none | 0|acc |↑ |0.9741|± |0.0115| | - high_school_macroeconomics | 0|none | 0|acc |↑ |0.8615|± |0.0175| | - high_school_microeconomics | 0|none | 0|acc |↑ |0.9412|± |0.0153| | - high_school_psychology | 0|none | 0|acc |↑ |0.9321|± |0.0108| | - human_sexuality | 0|none | 0|acc |↑ |0.8550|± |0.0309| | - professional_psychology | 0|none | 0|acc |↑ |0.8497|± |0.0145| | - public_relations | 0|none | 0|acc |↑ |0.7636|± |0.0407| | - security_studies | 0|none | 0|acc |↑ |0.8163|± |0.0248| | - sociology | 0|none | 0|acc |↑ |0.9204|± |0.0191| | - us_foreign_policy | 0|none | 0|acc |↑ |0.9300|± |0.0256| | - stem | 1|none | |acc |↑ |0.7907|± |0.0070| | - abstract_algebra | 0|none | 0|acc |↑ |0.5800|± |0.0496| | - anatomy | 0|none | 0|acc |↑ |0.8296|± |0.0325| | - astronomy | 0|none | 0|acc |↑ |0.9145|± |0.0228| | - college_biology | 0|none | 0|acc |↑ |0.9236|± |0.0222| | - college_chemistry | 0|none | 0|acc |↑ |0.5800|± |0.0496| | - college_computer_science | 0|none | 0|acc |↑ |0.7300|± |0.0446| | - college_mathematics | 0|none | 0|acc |↑ |0.5800|± |0.0496| | - college_physics | 0|none | 0|acc |↑ |0.7157|± |0.0449| | - computer_security | 0|none | 0|acc |↑ |0.8000|± |0.0402| | - conceptual_physics | 0|none | 0|acc |↑ |0.8383|± |0.0241| | - electrical_engineering | 0|none | 0|acc |↑ |0.7931|± |0.0338| | - elementary_mathematics | 0|none | 0|acc |↑ |0.8730|± |0.0171| | - high_school_biology | 0|none | 0|acc |↑ |0.9161|± |0.0158| | - high_school_chemistry | 0|none | 0|acc |↑ |0.7685|± |0.0297| | - high_school_computer_science | 0|none | 0|acc |↑ |0.9600|± |0.0197| | - high_school_mathematics | 0|none | 0|acc |↑ |0.6556|± |0.0290| | - high_school_physics | 0|none | 0|acc |↑ |0.7086|± |0.0371| | - high_school_statistics | 0|none | 0|acc |↑ |0.7778|± |0.0284| | - machine_learning | 0|none | 0|acc |↑ |0.7054|± |0.0433| |openbookqa | 1|none | 0|acc |↑ |0.3300|± |0.0210| | | |none | 0|acc_norm |↑ |0.4420|± |0.0222| |piqa | 1|none | 0|acc |↑ |0.8188|± |0.0090| | | |none | 0|acc_norm |↑ |0.8308|± |0.0087| |rte | 1|none | 0|acc |↑ |0.7220|± |0.0270| |truthfulqa_mc1 | 2|none | 0|acc |↑ |0.4333|± |0.0173| |winogrande | 1|none | 0|acc |↑ |0.7656|± |0.0119| | Groups |Version|Filter|n-shot|Metric| |Value | |Stderr| |------------------|------:|------|------|------|---|-----:|---|-----:| |mmlu | 1|none | |acc |↑ |0.8245|± |0.0031| | - humanities | 1|none | |acc |↑ |0.7892|± |0.0058| | - other | 1|none | |acc |↑ |0.8539|± |0.0060| | - social sciences| 1|none | |acc |↑ |0.8833|± |0.0057| | - stem | 1|none | |acc |↑ |0.7907|± |0.0070| ```