Running an rtx3060 with 12GBvram - managed to get this model working on method in link in description
hope this helps someone. I just followed the instructions, then moved this model / files to the new models folder.
Start-webui.bat file:
@echo off
@echo Starting the web UI...
cd /D "%~dp0"
set MAMBA_ROOT_PREFIX=%cd%\installer_files\mamba
set INSTALL_ENV_DIR=%cd%\installer_files\env
if not exist "%MAMBA_ROOT_PREFIX%\condabin\micromamba.bat" (
call "%MAMBA_ROOT_PREFIX%\micromamba.exe" shell hook >nul 2>&1
)
call "%MAMBA_ROOT_PREFIX%\condabin\micromamba.bat" activate "%INSTALL_ENV_DIR%" || ( echo MicroMamba hook not found. && goto end )
cd text-generation-webui
call python server.py --auto-devices --cai-chat --threads 8 --wbits 4 --groupsize 128
:end
pause
Also adjusted the HDD virtual memory setting to be managed by applications
Hmmm I just downloaded this model and loaded it with the oobabooga UI - no changes to any settings / configuration file. I see 9GB memory used while loading the model and it goes upto 11.2 GB during inference. (same GPU)
Wow that was a huge help, been struggling to get a working environment !
You can easily run any models using this method.