upgrade to new default pytorch version and instructions

exllamav2 scripts/auto-exl2-upload/INSTRUCTIONS.txt

@@ -1,20 +1,26 @@
 For NVIDIA cards install the CUDA toolkit
 
-Nvidia Maxwell or higher
-https://developer.nvidia.com/cuda-12-1-0-download-archive
+Recommended for all exllama features:
+https://developer.nvidia.com/cuda-12-4-0-download-archive
 
-Nvidia Kepler or higher
+Only for compatibily, doesn't support all features:
 https://developer.nvidia.com/cuda-11-8-0-download-archive
 
-Restart your computer after installing the CUDA toolkit to make sure the PATH is set correctly.
+Paths should overall lead to the version of cuda you're going to install with
+Paths examples with 12.4:
+CUDA_HOME: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4
+CUDA_PATH: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4
+CUDA_PATH_V12_4: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4
 
-Visual Studio with desktop development for C++ is required.
+(FOR WINDOWS ONLY) Visual Studio with desktop development for C++ is required. 2019 works fine for me.
 https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=community&rel=16&utm_medium=microsoft&utm_campaign=download+from+relnotes&utm_content=vs2019ga+button
 install the desktop development for C++ workload
 
+you might need cl.exe in path, for me it looks like this: "c:\program files (x86)\microsoft visual studio\2019\community\vc\tools\msvc\14.29.30133\bin\hostx64\x64"
+
 Make sure to have git and wget installed on your system.
 
-For Linux, install the build tools from your package manager.
+For Linux, you need gcc so you'll need to install the build tools from your package manager.
 For example, on Ubuntu use: sudo apt-get install build-essential
 
 This may work with AMD cards but only on linux and possibly WSL2. I can't guarantee that it will work on AMD cards, I personally don't have one to test with. You may need to install stuff before starting. https://rocm.docs.amd.com/projects/install-on-linux/en/latest/tutorial/quick-start.html
@@ -28,13 +34,14 @@ First setup your environment by using either windows.bat or linux.sh. If you wan
 After setup is complete then you'll have a file called start-quant. Use this to run the quant script.
 
 Make sure that your storage space is 3x the amount of the model's size. To mesure this, take the number of billion parameters and mutliply by two, afterwards mutliply by 3 and that's the recommended storage. There's a chance you may get away with 2.5x the size as well.
-Make sure to also have a lot of RAM depending on the model. Have noticed gemma to use a lot.
+Make sure to also have a lot of RAM depending on the model. Have noticed gemma to use a lot. If the quantizing crashes without a cuda out of memory error then it most likely ran out of system memory.
 
 If you close the terminal or the terminal crashes, check the last BPW it was on and enter the remaining quants you wanted. It should be able to pick up where it left off. Don't type the BPW of completed quants as it will start from the beginning. You may also use ctrl + c to pause at any time during the quant process.
 
-To add more options to the quantization process, you can add them to line 199. All options: https://github.com/turboderp/exllamav2/blob/master/doc/convert.md
+To add more options to the quantization process, you can add them to line 206. All options: https://github.com/turboderp/exllamav2/blob/master/doc/convert.md
 
-Things may break in the future as it downloads the latest version of all the dependencies which may either change names or how they work. If something breaks, please open a discussion at https://huggingface.co/Anthonyg5005/hf-scripts/discussions
+Things may break in the future as it downloads the latest version of all the dependencies which may either change names or how they work. If something breaks, try redownloading the latest version, and if that doesn't work then please open a discussion at https://huggingface.co/Anthonyg5005/hf-scripts/discussions
+for faster/easier support, join the exllama discord server and go to community-projects where you can find "easy exl2 quantizing" https://discord.gg/NSFwVuCjRq
 
 
 Credit to turboderp for creating exllamav2 and the exl2 quantization method.
@@ -44,6 +51,4 @@ Credit to oobabooga the original download and safetensors scripts.
 https://github.com/oobabooga
 
 Credit to Lucain Pouget for maintaining huggingface-hub.
-https://github.com/Wauplin
-
-Only tested with CUDA 12.1 on Windows 11 and WSL2 Ubuntu 24.04
+https://github.com/Wauplin

exllamav2 scripts/auto-exl2-upload/auto-exl2-upload.zip

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4d5c346d6dd344f29c196b33bdb8e2d07cf8d70e1e9135e1f2bfab8550bcba4e
-size 8717
+oid sha256:0282faa2e9e5f59ad353505dcac6c7fa581faead8da3e5d94584e636a5f8f647
+size 8497

exllamav2 scripts/auto-exl2-upload/linux-setup.sh

@@ -39,20 +39,11 @@ fi
 # if CUDA version 12 install pytorch for 12.1, else if CUDA 11 install pytorch for 11.8. If ROCm, install pytorch for ROCm 5.7
 read -p "Please enter your GPU compute version, CUDA 11/12 or AMD ROCm (11, 12, rocm): " pytorch_version
 
-# ask to install flash attention
-echo "Flash attention is a feature that could fix overflow issues on some more broken models, however, it will increase install time by a few hours."
-read -p "Would you like to install flash-attention? (rarely needed and optional) (y/n) " flash_attention
-if [ "$flash_attention" != "y" ] && [ "$flash_attention" != "n" ]; then
-    echo "Invalid input. Please enter y or n."
-    read -p "Press enter to continue"
-    exit
-fi
-
 if [ "$pytorch_version" = "11" ]; then
     echo "Installing PyTorch for CUDA 11.8"
     venv/bin/python -m pip install torch --index-url https://download.pytorch.org/whl/cu118 --upgrade
 elif [ "$pytorch_version" = "12" ]; then
-    echo "Installing PyTorch for CUDA 12.1"
+    echo "Installing PyTorch for CUDA 12.4"
     venv/bin/python -m pip install torch
 elif [ "$pytorch_version" = "rocm" ]; then
     echo "Installing PyTorch for AMD ROCm 5.7"
@@ -100,15 +91,6 @@ echo "#!/bin/bash" > enter-venv.sh
 echo "bash --init-file venv/bin/activate" >> enter-venv.sh
 chmod +x enter-venv.sh
 
-if [ "$flash_attention" = "y" ]; then
-    echo "Going to attempt to install flash attention but it isn't required."
-    echo "You may close now if you'd like and continue without flash attention."
-    read -p "Press enter to continue and install flash attention"
-    echo "Get some popcorn and watch a movie, this will take a while."
-    echo "Installing flash-attn..."
-    venv/bin/python -m pip install git+https://github.com/Dao-AILab/flash-attention.git
-fi
-
 echo "If you use ctrl+c to stop, you may need to also use 'pkill python' to stop running scripts."
 echo "Environment setup complete. run start-quant.sh to start the quantization process."
 read -p "Press enter to exit"

exllamav2 scripts/auto-exl2-upload/windows-setup.bat

@@ -40,23 +40,16 @@ if not "%exllamav2_version%"=="stable" if not "%exllamav2_version%"=="dev" (
 REM if CUDA version 12 install pytorch for 12.1, else if CUDA 11 install pytorch for 11.8
 echo CUDA compilers:
 where nvcc
+echo CUDA_HOME:
+echo %CUDA_HOME%
 set /p cuda_version="Please enter your CUDA version (11 or 12): "
 
-REM ask to install flash attention
-echo Flash attention is a feature that could fix overflow issues on some more broken models, however, it will increase install time by a few hours.
-set /p flash_attention="Would you like to install flash-attention? (rarely needed and optional) (y/n) "
-if not "%flash_attention%"=="y" if not "%flash_attention%"=="n" (
-    echo Invalid input. Please enter y or n.
-    pause
-    exit
-)
-
 if "%cuda_version%"=="11" (
     echo Installing PyTorch for CUDA 11.8...
     venv\scripts\python.exe -m pip install torch --index-url https://download.pytorch.org/whl/cu118 --upgrade
 ) else if "%cuda_version%"=="12" (
-    echo Installing PyTorch for CUDA 12.1...
-    venv\scripts\python.exe -m pip install torch --index-url https://download.pytorch.org/whl/cu121 --upgrade
+    echo Installing PyTorch for CUDA 12.4...
+    venv\scripts\python.exe -m pip install torch --index-url https://download.pytorch.org/whl/cu124 --upgrade
 ) else (
     echo Invalid CUDA version. Please enter 11 or 12.
     pause
@@ -99,15 +92,6 @@ REM create enter-venv.bat
 echo @echo off > enter-venv.bat
 echo cmd /k call venv\scripts\activate.bat >> enter-venv.bat
 
-if "%flash_attention%"=="y" (
-    echo Going to attempt to install flash attention but it isn't required.
-    echo You may close now if you'd like and continue without flash attention.
-    pause
-    echo Get some popcorn and watch a movie. This will take a while.
-    echo Installing flash-attn...
-    venv\scripts\python.exe -m pip install git+https://github.com/Dao-AILab/flash-attention.git
-)
-
 powershell -c (New-Object Media.SoundPlayer "C:\Windows\Media\tada.wav").PlaySync();
 echo Environment setup complete. run start-quant.bat to start the quantization process.
 pause

exllamav2 scripts/exl2-multi-quant-local/INSTRUCTIONS.txt

@@ -1,20 +1,26 @@
 For NVIDIA cards install the CUDA toolkit
 
-Nvidia Maxwell or higher
-https://developer.nvidia.com/cuda-12-1-0-download-archive
+Recommended for all exllama features:
+https://developer.nvidia.com/cuda-12-4-0-download-archive
 
-Nvidia Kepler or higher
+Only for compatibily, doesn't support all features:
 https://developer.nvidia.com/cuda-11-8-0-download-archive
 
-Restart your computer after installing the CUDA toolkit to make sure the PATH is set correctly.
+Paths should overall lead to the version of cuda you're going to install with
+Paths examples with 12.4:
+CUDA_HOME: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4
+CUDA_PATH: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4
+CUDA_PATH_V12_4: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4
 
-Visual Studio with desktop development for C++ is required.
+(FOR WINDOWS ONLY) Visual Studio with desktop development for C++ is required. 2019 works fine for me.
 https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=community&rel=16&utm_medium=microsoft&utm_campaign=download+from+relnotes&utm_content=vs2019ga+button
 install the desktop development for C++ workload
 
-make sure to have git and wget installed on your system.
+you might need cl.exe in path, for me it looks like this: "c:\program files (x86)\microsoft visual studio\2019\community\vc\tools\msvc\14.29.30133\bin\hostx64\x64"
 
-For Linux, install the build tools from your package manager.
+Make sure to have git and wget installed on your system.
+
+For Linux, you need gcc so you'll need to install the build tools from your package manager.
 For example, on Ubuntu use: sudo apt-get install build-essential
 
 This may work with AMD cards but only on linux and possibly WSL2. I can't guarantee that it will work on AMD cards, I personally don't have one to test with. You may need to install stuff before starting. https://rocm.docs.amd.com/projects/install-on-linux/en/latest/tutorial/quick-start.html
@@ -27,14 +33,15 @@ First setup your environment by using either windows.bat or linux.sh. If you wan
 
 After setup is complete then you'll have a file called start-quant. Use this to run the quant script.
 
-Make sure that your storage space is 3x the amount of the model's size plus 1 more time per quant. To mesure this, take the number of billion parameters and mutliply by two, afterwards mutliply by 3 and that's the recommended storage. There's a chance you may get away with 2.5x the size as well.
-Make sure to also have a lot of RAM depending on the model. Have noticed gemma to use a lot.
+Make sure that your storage space is 3x the amount of the model's size. To mesure this, take the number of billion parameters and mutliply by two, afterwards mutliply by 3 and that's the recommended storage. There's a chance you may get away with 2.5x the size as well.
+Make sure to also have a lot of RAM depending on the model. Have noticed gemma to use a lot. If the quantizing crashes without a cuda out of memory error then it most likely ran out of system memory.
 
 If you close the terminal or the terminal crashes, check the last BPW it was on and enter the remaining quants you wanted. It should be able to pick up where it left off. Don't type the BPW of completed quants as it will start from the beginning. You may also use ctrl + c to pause at any time during the quant process.
 
-To add more options to the quantization process, you can add them to line 145. All options: https://github.com/turboderp/exllamav2/blob/master/doc/convert.md
+To add more options to the quantization process, you can add them to line 153. All options: https://github.com/turboderp/exllamav2/blob/master/doc/convert.md
 
-Things may break in the future as it downloads the latest version of all the dependencies which may either change names or how they work. If something breaks, please open a discussion at https://huggingface.co/Anthonyg5005/hf-scripts/discussions
+Things may break in the future as it downloads the latest version of all the dependencies which may either change names or how they work. If something breaks, try redownloading the latest version, and if that doesn't work then please open a discussion at https://huggingface.co/Anthonyg5005/hf-scripts/discussions
+for faster/easier support, join the exllama discord server and go to community-projects where you can find "easy exl2 quantizing" https://discord.gg/NSFwVuCjRq
 
 
 Credit to turboderp for creating exllamav2 and the exl2 quantization method.
@@ -44,6 +51,4 @@ Credit to oobabooga the original download and safetensors scripts.
 https://github.com/oobabooga
 
 Credit to Lucain Pouget for maintaining huggingface-hub.
-https://github.com/Wauplin
-
-Only tested with CUDA 12.1 on Windows 11 and WSL2 Ubuntu 24.04
+https://github.com/Wauplin

exllamav2 scripts/exl2-multi-quant-local/exl2-multi-quant-local.zip

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bd45a8c909552d9e4f1f8b2069d210e91a2ce0fd8043b45dcffef764dab370ba
-size 7513
+oid sha256:e5c9eef354fe8bc983870772b2fbfa47af7b0dc5d8e8dbb5b10fae8f5e9ba8cb
+size 7280

exllamav2 scripts/exl2-multi-quant-local/linux-setup.sh

@@ -39,20 +39,11 @@ fi
 # if CUDA version 12 install pytorch for 12.1, else if CUDA 11 install pytorch for 11.8. If ROCm, install pytorch for ROCm 5.7
 read -p "Please enter your GPU compute version, CUDA 11/12 or AMD ROCm (11, 12, rocm): " pytorch_version
 
-# ask to install flash attention
-echo "Flash attention is a feature that could fix overflow issues on some more broken models, however, it will increase install time by a few hours."
-read -p "Would you like to install flash-attention? (rarely needed and optional) (y/n) " flash_attention
-if [ "$flash_attention" != "y" ] && [ "$flash_attention" != "n" ]; then
-    echo "Invalid input. Please enter y or n."
-    read -p "Press enter to continue"
-    exit
-fi
-
 if [ "$pytorch_version" = "11" ]; then
     echo "Installing PyTorch for CUDA 11.8"
     venv/bin/python -m pip install torch --index-url https://download.pytorch.org/whl/cu118 --upgrade
 elif [ "$pytorch_version" = "12" ]; then
-    echo "Installing PyTorch for CUDA 12.1"
+    echo "Installing PyTorch for CUDA 12.4"
     venv/bin/python -m pip install torch
 elif [ "$pytorch_version" = "rocm" ]; then
     echo "Installing PyTorch for AMD ROCm 5.7"
@@ -100,15 +91,6 @@ echo "#!/bin/bash" > enter-venv.sh
 echo "bash --init-file venv/bin/activate" >> enter-venv.sh
 chmod +x enter-venv.sh
 
-if [ "$flash_attention" = "y" ]; then
-    echo "Going to attempt to install flash attention but it isn't required."
-    echo "You may close now if you'd like and continue without flash attention."
-    read -p "Press enter to continue and install flash attention"
-    echo "Get some popcorn and watch a movie, this will take a while."
-    echo "Installing flash-attn..."
-    venv/bin/python -m pip install git+https://github.com/Dao-AILab/flash-attention.git
-fi
-
 echo "If you use ctrl+c to stop, you may need to also use 'pkill python' to stop running scripts."
 echo "Environment setup complete. run start-quant.sh to start the quantization process."
 read -p "Press enter to exit"

exllamav2 scripts/exl2-multi-quant-local/windows-setup.bat

@@ -40,23 +40,16 @@ if not "%exllamav2_version%"=="stable" if not "%exllamav2_version%"=="dev" (
 REM if CUDA version 12 install pytorch for 12.1, else if CUDA 11 install pytorch for 11.8
 echo CUDA compilers:
 where nvcc
+echo CUDA_HOME:
+echo %CUDA_HOME%
 set /p cuda_version="Please enter your CUDA version (11 or 12): "
 
-REM ask to install flash attention
-echo Flash attention is a feature that could fix overflow issues on some more broken models, however, it will increase install time by a few hours.
-set /p flash_attention="Would you like to install flash-attention? (rarely needed and optional) (y/n) "
-if not "%flash_attention%"=="y" if not "%flash_attention%"=="n" (
-    echo Invalid input. Please enter y or n.
-    pause
-    exit
-)
-
 if "%cuda_version%"=="11" (
     echo Installing PyTorch for CUDA 11.8...
     venv\scripts\python.exe -m pip install torch --index-url https://download.pytorch.org/whl/cu118 --upgrade
 ) else if "%cuda_version%"=="12" (
-    echo Installing PyTorch for CUDA 12.1...
-    venv\scripts\python.exe -m pip install torch --index-url https://download.pytorch.org/whl/cu121 --upgrade
+    echo Installing PyTorch for CUDA 12.4...
+    venv\scripts\python.exe -m pip install torch --index-url https://download.pytorch.org/whl/cu124 --upgrade
 ) else (
     echo Invalid CUDA version. Please enter 11 or 12.
     pause
@@ -99,15 +92,6 @@ REM create enter-venv.bat
 echo @echo off > enter-venv.bat
 echo cmd /k call venv\scripts\activate.bat >> enter-venv.bat
 
-if "%flash_attention%"=="y" (
-    echo Going to attempt to install flash attention but it isn't required.
-    echo You may close now if you'd like and continue without flash attention.
-    pause
-    echo Get some popcorn and watch a movie. This will take a while.
-    echo Installing flash-attn...
-    venv\scripts\python.exe -m pip install git+https://github.com/Dao-AILab/flash-attention.git
-)
-
 powershell -c (New-Object Media.SoundPlayer "C:\Windows\Media\tada.wav").PlaySync();
 echo Environment setup complete. run start-quant.bat to start the quantization process.
 pause