tiiuae

/

falcon-7b-instruct

like 915

Follow

Technology Innovation Institute 917

Same here - also on a MB M1 Pro

Perhaps you need to install the latest versions of libraries from github. There is an example how to install them. Probably you need not all of them, but maybe useful

pip install -q -U bitsandbytes
pip install -q -U git+https://github.com/huggingface/transformers.git 
pip install -q -U git+https://github.com/huggingface/peft.git
pip install -q -U git+https://github.com/huggingface/accelerate.git
pip install -q einops

@CleverShovel Thanks, I did this, but it didn't solve it. Oddly enough though the error just changed a bit:

│ transformers/pipelines/base.py:278 in infer_framework_load_model │
│ │
│ 275 │ │ │ │ continue │
│ 276 │ │ │
│ 277 │ │ if isinstance(model, str): │
│ ❱ 278 │ │ │ raise ValueError(f"Could not load model {model} with any │
│ 279 │ │
│ 280 │ framework = infer_framework(model.class) │
│ 281 │ return framework, model │
╰──────────────────────────────────────────────────────────────────────────────╯
ValueError: Could not load model tiiuae/falcon-7b-instruct with any of the
following classes: (<class
'transformers.models.auto.modeling_auto.AutoModelForCausalLM'>,).

I also tried running it on Colab and get the same issue:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in <cell line: 8>:8                                                                              │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/pipelines/__init__.py:788 in pipeline       │
│                                                                                                  │
│   785 │   # Forced if framework already defined, inferred if it's None                           │
│   786 │   # Will load the correct model if possible                                              │
│   787 │   model_classes = {"tf": targeted_task["tf"], "pt": targeted_task["pt"]}                 │
│ ❱ 788 │   framework, model = infer_framework_load_model(                                         │
│   789 │   │   model,                                                                             │
│   790 │   │   model_classes=model_classes,                                                       │
│   791 │   │   config=config,                                                                     │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/pipelines/base.py:278 in                    │
│ infer_framework_load_model                                                                       │
│                                                                                                  │
│    275 │   │   │   │   continue                                                                  │
│    276 │   │                                                                                     │
│    277 │   │   if isinstance(model, str):                                                        │
│ ❱  278 │   │   │   raise ValueError(f"Could not load model {model} with any of the following cl  │
│    279 │                                                                                         │
│    280 │   framework = infer_framework(model.__class__)                                          │
│    281 │   return framework, model                                                               │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Could not load model tiiuae/falcon-7b-instruct with any of the following classes: (<class 
'transformers.models.auto.modeling_auto.AutoModelForCausalLM'>, <class 
'transformers.models.auto.modeling_tf_auto.TFAutoModelForCausalLM'>).

@aviadatlas Yeah, it is strange, maybe some problem with transformers.pipeline.
I managed to launch it on Colab, here is notebook. I used sharded version of this model otherwise it can't be loaded in free tier Colab.

I'm having the same exact issue as OP.

It would help if there were a requirements.txt or environment.yaml or dockerfile so we are not just guessing when it comes to library versioning, especially in such a fast-moving space as LLMs. Without version numbers, the dependencies are almost guaranteed to not be in sync with pip / conda (even if you can figure out what dependencies are needed by getting pip errors over and over).

I got this to work by running pip install -U transformers. I'd used conda to install transformers originally.

I'm still getting the same error. Does anybody have a solution for this?

Me too!

For me it was a memory error, the way I solved this was by exporting TRANSFORMERS_CACHE to a different location

Can someone who it is working for please share an conda env.yaml or requirements.txt file reproducing their exact config so that we can try to use that setup to load the model and run it? This would be really helpful! :)

Getting the same error. Hard to work with this stuff without versions of dependencies.

Hello Thanks for your help all, I Get the below error now, can anyone please help ?

Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████| 2/2 [01:01<00:00, 30.80s/it]
Traceback (most recent call last):
File "/Users/clustered/Python Application/02. Falcon_Model/Falcon-Application.py", line 10, in
tokenizer = AutoTokenizer.from_pretrained(model)
File "/Users/clustered/miniconda3/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 643, in from_pretrained
tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs)
File "/Users/clustered/miniconda3/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 487, in get_tokenizer_config
resolved_config_file = cached_file(
File "/Users/clustered/miniconda3/lib/python3.10/site-packages/transformers/utils/hub.py", line 417, in cached_file
resolved_file = hf_hub_download(
File "/Users/clustered/miniconda3/lib/python3.10/site-packages/huggingface_hub-0.14.1-py3.8.egg/huggingface_hub/utils/_validators.py", line 112, in _inner_fn
File "/Users/clustered/miniconda3/lib/python3.10/site-packages/huggingface_hub-0.14.1-py3.8.egg/huggingface_hub/utils/_validators.py", line 166, in validate_repo_id
huggingface_hub.utils.validators.HFValidationError: Repo id must use alphanumeric chars or '-', '', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: 'RWForCausalLM(
(transformer): RWModel(
(word_embeddings): Embedding(65024, 4544)
(h): ModuleList(
(0-31): 32 x DecoderLayer(
(input_layernorm): LayerNorm((4544,), eps=1e-05, elementwise_affine=True)
(self_attention): Attention(
(maybe_rotary): RotaryEmbedding()
(query_key_value): Linear(in_features=4544, out_features=4672, bias=False)
(dense): Linear(in_features=4544, out_features=4544, bias=False)
(attention_dropout): Dropout(p=0.0, inplace=False)
)
(mlp): MLP(
(dense_h_to_4h): Linear(in_features=4544, out_features=18176, bias=False)
(act): GELU(approximate='none')
(dense_4h_to_h): Linear(in_features=18176, out_features=4544, bias=False)
)
)
)
(ln_f): LayerNorm((4544,), eps=1e-05, elementwise_affine=True)
)
(lm_head): Linear(in_features=4544, out_features=65024, bias=False)

Can anyone please help 9n this ? :)

Anyone?

So I managed to make the sample code work on Colab at least. I just needed to have the GPU enabled via "Runtime -> Change runtime type"

It seems like the issue here has been solved -- but feel free to also check-out this great blogpost from HuggingFace on using Falcon.