Problem loading model in oobabooga/text-generation-webui
Hi all
Despite even changing my ubuntu swap file file and setting restrictions in the webui - I consistently get an error loading this model in text-generation-webui.
Relevant info below - any ideas. Before I did the swap changes - it just died with a "Killed" message.
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.54.03 Driver Version: 535.54.03 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 3080 Off | 00000000:01:00.0 On | N/A |
| 0% 45C P8 19W / 350W | 1929MiB / 12288MiB | 6% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
2023-08-07 21:27:46 INFO:Loading TheBloke_Wizard-Vicuna-30B-Uncensored-GPTQ...
2023-08-07 21:27:46 INFO:The AutoGPTQ params are: {'model_basename': 'Wizard-Vicuna-30B-Uncensored-GPTQ-4bit--1g.act.order', 'device': 'cuda:0', 'use_triton': False, 'inject_fused_attention': False, 'inject_fused_mlp': True, 'use_safetensors': True, 'trust_remote_code': False, 'max_memory': {0: '3100MiB', 'cpu': '4000MiB'}, 'quantize_config': None, 'use_cuda_fp16': True}
2023-08-07 21:27:52 ERROR:Failed to load the model.
anaconda3/envs/textgen/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 1218, in load_checkpoint_in_model
raise ValueError(
ValueError: At least one of the model submodule will be offloaded to disk, please pass along an offload_folder.