Hugging Face's logo

xmadai
/
Llama-3.1-8B-Instruct-xMADai-INT4

Text Generation
Transformers
llama
conversational
Inference Endpoints
4-bit precision
gptq
Model card Files Community
Oscar Wu commited on
Commit
94b73f6
•
1 Parent(s): 1b33db0

Updated README

Browse files
Files changed (1) hide show
  1. README.md +7 -2
README.md CHANGED
@@ -32,12 +32,17 @@ This repository contains [`meta-llama/Llama-3.1-8B-Instruct`](https://huggingfac
32
 
33
  Loading the model checkpoint of this xMADified model requires less than 6 GiB of VRAM. Hence it can be efficiently run on a 8 GB GPU.
34
 
35
- **Package prerequisites**: Run the following commands to install the required packages.
 
 
36
  ```bash
37
- pip install torch==2.4.0 transformers accelerate optimum
 
38
  pip install -vvv --no-build-isolation "git+https://github.com/PanQiWei/AutoGPTQ.git@v0.7.1"
39
  ```
40
 
 
 
41
  **Sample Inference Code**
42
 
43
  ```python
 
32
 
33
  Loading the model checkpoint of this xMADified model requires less than 6 GiB of VRAM. Hence it can be efficiently run on a 8 GB GPU.
34
 
35
+ **Package prerequisites**:
36
+
37
+ 1. Run the following *commands to install the required packages.
38
  ```bash
39
+ pip install torch==2.4.0 # Run following if you have CUDA version 11.8: pip install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu118
40
+ pip install transformers accelerate optimum
41
  pip install -vvv --no-build-isolation "git+https://github.com/PanQiWei/AutoGPTQ.git@v0.7.1"
42
  ```
43
 
44
+
45
+
46
  **Sample Inference Code**
47
 
48
  ```python