facebook/MobileLLM-1B · AttributeError: 'bool' object has no attribute 'vocab

Oct 31

I can't test in lm-eval:

Traceback (most recent call last):
  File "venv/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "lm-evaluation-harness/lm_eval/__main__.py", line 369, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "lm-evaluation-harness/lm_eval/utils.py", line 346, in _wrapper
    return fn(*args, **kwargs)
  File "lm-evaluation-harness/lm_eval/evaluator.py", line 192, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "lm-evaluation-harness/lm_eval/api/model.py", line 148, in create_from_arg_string
    return cls(**args, **args2)
  File "lm-evaluation-harness/lm_eval/models/huggingface.py", line 254, in __init__
    self.vocab_size = self.tokenizer.vocab_size
AttributeError: 'bool' object has no attribute 'vocab_size'

mikezhaocs

AI at Meta org Oct 31

There's a typo in model card. Please use the following command to load tokenizer:
tokenizer=AutoTokenizer.from_pretrained("facebook/MobileLLM-1B", use_fast=False)

Alternatively, you can use lm-eval cli directly (for example arc_easy task):
lm_eval --model hf --model_args pretrained=facebook/MobileLLM-1B,trust_remote_code=True,use_fast_tokenizer=False --tasks arc_easy

zechunliu changed discussion status to closed Nov 1