Model Loader?
Trying to load this in Ooobabooga text gen webui but none of the loaders seem to know what to do with it. What model loader and parameters should I use? Thanks! Hoping to get better response time with a quantized, smaller version.
It's ExllamaV2 as the name suggests "exl2" is short for that type of quantized models. Not entirely sure why Ooba doesn't always choose the right loader automatically.
Thanks! It did choose that, but I got the following error when trying to load it. I'll reload the webui and see if that does it.
22:54:52-323063 INFO Loading "mpasila_Moistral-11B-v1-exl2-4bpw"
22:54:52-326053 ERROR Failed to load the model.
Traceback (most recent call last):
File "B:\ai_art\text-generation-webui-main\modules\ui_model_menu.py", line 245, in load_model_wrapper
shared.model, shared.tokenizer = load_model(selected_model, loader)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "B:\ai_art\text-generation-webui-main\modules\models.py", line 87, in load_model
output = load_func_maploader
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "B:\ai_art\text-generation-webui-main\modules\models.py", line 378, in ExLlamav2_HF_loader
from modules.exllamav2_hf import Exllamav2HF
File "B:\ai_art\text-generation-webui-main\modules\exllamav2_hf.py", line 7, in
from exllamav2 import (
ImportError: cannot import name 'ExLlamaV2Cache_Q4' from 'exllamav2' (B:\ai_art\text-generation-webui-main\installer_files\env\Lib\site-packages\exllamav2_init_.py)
Make sure your text-generation-webui is up to date since it runs just fine on my machine. I also tested it on Runpod (using Jupyter Notebook since TheBloke's template is outdated) and Colab and it appears to be working just fine on both services.
If even after updating it doesn't work you may need to reinstall Ooba.