Can't open this model anymore
Not sure why, im pretty sure I was opening it with latest GPTQ before Triton.
Traceback (most recent call last):
File “/home/perplexity/text-generation-webui/server.py”, line 67, in load_model_wrapper
shared.model, shared.tokenizer = load_model(shared.model_name)
File “/home/perplexity/text-generation-webui/modules/models.py”, line 159, in load_model
model = load_quantized(model_name)
File “/home/perplexity/text-generation-webui/modules/GPTQ_loader.py”, line 178, in load_quantized
model = load_quant(str(path_to_model), str(pt_path), shared.args.wbits, shared.args.groupsize, kernel_switch_threshold=threshold)
File “/home/perplexity/text-generation-webui/modules/GPTQ_loader.py”, line 84, in _load_quant
model.load_state_dict(safe_load(checkpoint), strict=False)
File “/home/perplexity/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 2041, in load_state_dict
raise RuntimeError(‘Error(s) in loading state_dict for {}:\n\t{}’.format(
RuntimeError: Error(s) in loading state_dict for LlamaForCausalLM:
size mismatch for model.layers.0.self_attn.k_proj.qzeros: copying a param with shape torch.Size([52, 832]) from checkpoint, the shape in current model is torch.Size([1, 832]).
size mismatch for model.layers.0.self_attn.k_proj.scales: copying a param with shape