Hugging Face's logo

Spaces:
dorkai
/
text-generation-webui-main
Runtime error

App Files Community
text-generation-webui-main
File size: 3,323 Bytes 
6a4546d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
 
You have now entered a hidden corner of the internet.

A confusing yet intriguing realm of paradoxes and contradictions.

A place where you will find out that what you thought you knew, you in fact didn't know, and what you didn't know was in front of you all along.

![](https://i.pinimg.com/originals/6e/e2/7b/6ee27bad351d3aca470d80f1033ba9c6.jpg)

*In other words, here I will document little-known facts about this web UI that I could not find another place for in the wiki.*

#### You can train LoRAs in CPU mode

Load the web UI with

```
python server.py --cpu
```

and start training the LoRA from the training tab as usual.

#### 8-bit mode works with CPU offloading

```
python server.py --load-in-8bit --gpu-memory 4000MiB
```

#### `--pre_layer`, and not `--gpu-memory`, is the right way to do CPU offloading with 4-bit models

```
python server.py --wbits 4 --groupsize 128 --pre_layer 20
```

#### Models can be loaded in 32-bit, 16-bit, 8-bit, and 4-bit modes

```
python server.py --cpu
python server.py
python server.py --load-in-8bit
python server.py --wbits 4
```

#### The web UI works with any version of GPTQ-for-LLaMa

Including the up to date triton and cuda branches. But you have to delete the `repositories/GPTQ-for-LLaMa` folder and reinstall the new one every time:

```
cd text-generation-webui/repositories
rm -r GPTQ-for-LLaMa
pip uninstall quant-cuda
git clone https://github.com/oobabooga/GPTQ-for-LLaMa -b cuda # or any other repository and branch
cd GPTQ-for-LLaMa
python setup_cuda.py install
```

#### Instruction-following templates are represented as chat characters

https://github.com/oobabooga/text-generation-webui/tree/main/characters/instruction-following

#### The right way to run Alpaca, Open Assistant, Vicuna, etc is Instruct mode, not normal chat mode

Otherwise the prompt will not be formatted correctly.

1. Start the web UI with

```
python server.py --chat
```

2. Click on the "instruct" option under "Chat modes"

3. Select the correct template in the hidden dropdown menu that will become visible. 

#### Notebook mode is best mode

Ascended individuals have realized that notebook mode is the superset of chat mode and can do chats with ultimate flexibility, including group chats, editing replies, starting a new bot reply in a given way, and impersonating.

#### RWKV is a RNN 

Most models are transformers, but not RWKV, which is a RNN. It's a great model.

#### `--gpu-memory` is not a hard limit on the GPU memory

It is simply a parameter that is passed to the `accelerate` library while loading the model. More memory will be allocated during generation. That's why this parameter has to be set to less than your total GPU memory.

#### Contrastive search perhaps the best preset

But it uses a ton of VRAM.

#### You can check the sha256sum of downloaded models with the download script

```
python download-model.py facebook/galactica-125m --check
```

#### The download script continues interrupted downloads by default

It doesn't start over.

#### You can download models with multiple threads

```
python download-model.py facebook/galactica-125m --threads 8
```

#### LoRAs work in 4-bit mode

You need to follow [these instructions](GPTQ-models-(4-bit-mode).md#using-loras-in-4-bit-mode) and then start the web UI with the `--monkey-patch` flag.