Unable to load Llama 3.1 to Text-Genration WebUI

#45
by keeeeesz - opened

Hi,

every time I try to load this model is oobabooga the following error appears. Do you know what could be the solution to this problem?

Thank you
Kamil

Traceback (most recent call last):

File "/workspace/text-generation-webui/modules/ui_model_menu.py", line 232, in load_model_wrapper

shared.model, shared.tokenizer = load_model(selected_model, loader)

                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/workspace/text-generation-webui/modules/models.py", line 94, in load_model

output = load_func_maploader

     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/workspace/text-generation-webui/modules/models.py", line 152, in huggingface_loader

config = AutoConfig.from_pretrained(path_to_model, trust_remote_code=shared.args.trust_remote_code)

     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/workspace/text-generation-webui/installer_files/env/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 989, in from_pretrained

return config_class.from_dict(config_dict, **unused_kwargs)

   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/workspace/text-generation-webui/installer_files/env/lib/python3.11/site-packages/transformers/configuration_utils.py", line 772, in from_dict

config = cls(**config_dict)

     ^^^^^^^^^^^^^^^^^^

File "/workspace/text-generation-webui/installer_files/env/lib/python3.11/site-packages/transformers/models/llama/configuration_llama.py", line 161, in init

self._rope_scaling_validation()
File "/workspace/text-generation-webui/installer_files/env/lib/python3.11/site-packages/transformers/models/llama/configuration_llama.py", line 182, in _rope_scaling_validation

raise ValueError(
ValueError: rope_scaling must be a dictionary with two fields, type and factor, got {'factor': 8.0, 'low_freq_factor': 1.0, 'high_freq_factor': 4.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'}

Hi there! Please make sure to update to the latest transformers version. Please let us know if that doesn't fix the issue.

try changing the inference method to transformers or xformershf. let me know if it resolved your problem. thank you

Thank you! Now everything is working :)

Sign up or log in to comment