TheBloke commited on
Commit
08b0731
1 Parent(s): 94e9f5b

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +49 -69
README.md CHANGED
@@ -1,6 +1,12 @@
1
  ---
2
  inference: false
3
  license: other
 
 
 
 
 
 
4
  ---
5
 
6
  <!-- header start -->
@@ -19,15 +25,25 @@ license: other
19
 
20
  # Eric Hartford's Wizard Vicuna 30B Uncensored merged with Kaio Ken's SuperHOT 8K GPTQ
21
 
22
- These files are GPTQ 4bit model files for [Eric Hartford's Wizard Vicuna 30B Uncensored merged with Kaio Ken's SuperHOT 8K](https://huggingface.co/kaiokendev/superhot-30b-8k-no-rlhf-test).
23
 
24
  It is the result of quantising to 4bit using [GPTQ-for-LLaMa](https://github.com/qwopqwop200/GPTQ-for-LLaMa).
25
 
 
 
 
 
 
 
 
 
 
 
26
  ## Repositories available
27
 
28
  * [4-bit GPTQ models for GPU inference](https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Superhot-8K-GPTQ)
29
- * [2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference](https://huggingface.co/none)
30
- * [Unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/ehartford/Wizard-Vicuna-30B-Uncensored)
31
 
32
  ## How to easily download and use this model in text-generation-webui
33
 
@@ -37,70 +53,17 @@ Please make sure you're using the latest version of text-generation-webui
37
  2. Under **Download custom model or LoRA**, enter `TheBloke/Wizard-Vicuna-30B-Superhot-8K-GPTQ`.
38
  3. Click **Download**.
39
  4. The model will start downloading. Once it's finished it will say "Done"
40
- 5. In the top left, click the refresh icon next to **Model**.
41
- 6. In the **Model** dropdown, choose the model you just downloaded: `Wizard-Vicuna-30B-Superhot-8K-GPTQ`
42
- 7. The model will automatically load, and is now ready for use!
43
- 8. If you want any custom settings, set them and then click **Save settings for this model** followed by **Reload the Model** in the top right.
44
- * Note that you do not need to and should not set manual GPTQ parameters any more. These are set automatically from the file `quantize_config.json`.
45
- 9. Once you're ready, click the **Text Generation tab** and enter a prompt to get started!
46
-
47
- ## How to use this GPTQ model from Python code
48
-
49
- First make sure you have [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ) installed:
50
-
51
- `pip install auto-gptq`
52
-
53
- Then try the following example code:
54
-
55
- ```python
56
- from transformers import AutoTokenizer, pipeline, logging
57
- from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
58
- import argparse
59
-
60
- model_name_or_path = "TheBloke/Wizard-Vicuna-30B-Superhot-8K-GPTQ"
61
- model_basename = "wizard-vicuna-30b-superhot-8k-GPTQ-4bit--1g.act.order"
62
-
63
- use_triton = False
64
-
65
- tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
66
-
67
- model = AutoGPTQForCausalLM.from_quantized(model_name_or_path,
68
- model_basename=model_basename,
69
- use_safetensors=True,
70
- trust_remote_code=False,
71
- device="cuda:0",
72
- use_triton=use_triton,
73
- quantize_config=None)
74
 
75
- # Note: check the prompt template is correct for this model.
76
- prompt = "Tell me about AI"
77
- prompt_template=f'''USER: {prompt}
78
- ASSISTANT:'''
79
 
80
- print("\n\n*** Generate:")
81
-
82
- input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()
83
- output = model.generate(inputs=input_ids, temperature=0.7, max_new_tokens=512)
84
- print(tokenizer.decode(output[0]))
85
-
86
- # Inference can also be done using transformers' pipeline
87
-
88
- # Prevent printing spurious transformers error when using pipeline with AutoGPTQ
89
- logging.set_verbosity(logging.CRITICAL)
90
-
91
- print("*** Pipeline:")
92
- pipe = pipeline(
93
- "text-generation",
94
- model=model,
95
- tokenizer=tokenizer,
96
- max_new_tokens=512,
97
- temperature=0.7,
98
- top_p=0.95,
99
- repetition_penalty=1.15
100
- )
101
-
102
- print(pipe(prompt_template)[0]['generated_text'])
103
- ```
104
 
105
  ## Provided files
106
 
@@ -111,9 +74,9 @@ This will work with AutoGPTQ, ExLlama, and CUDA versions of GPTQ-for-LLaMa. Ther
111
  It was created without group_size to lower VRAM requirements, and with --act-order (desc_act) to boost inference accuracy as much as possible.
112
 
113
  * `wizard-vicuna-30b-superhot-8k-GPTQ-4bit--1g.act.order.safetensors`
114
- * Works with AutoGPTQ in CUDA or Triton modes.
115
- * LLaMa models also work with [ExLlama](https://github.com/turboderp/exllama}, which usually provides much higher performance, and uses less VRAM, than AutoGPTQ.
116
- * Works with GPTQ-for-LLaMa in CUDA mode. May have issues with GPTQ-for-LLaMa Triton mode.
117
  * Works with text-generation-webui, including one-click-installers.
118
  * Parameters: Groupsize = -1. Act Order / desc_act = True.
119
 
@@ -145,7 +108,7 @@ Thank you to all my generous patrons and donaters!
145
 
146
  <!-- footer end -->
147
 
148
- # Original model card: Eric Hartford's Wizard Vicuna 30B Uncensored merged with Kaio Ken's SuperHOT 8K
149
 
150
 
151
  ### SuperHOT Prototype 2 w/ 8K Context
@@ -177,3 +140,20 @@ I trained the LoRA with the following configuration:
177
  - weight decay of 0.1
178
  - AdamW beta1 of 0.9 and beta2 0.99, epsilon of 1e-5
179
  - Trained on 4-bit base model
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  inference: false
3
  license: other
4
+ datasets:
5
+ - ehartford/wizard_vicuna_70k_unfiltered
6
+ language:
7
+ - en
8
+ tags:
9
+ - uncensored
10
  ---
11
 
12
  <!-- header start -->
 
25
 
26
  # Eric Hartford's Wizard Vicuna 30B Uncensored merged with Kaio Ken's SuperHOT 8K GPTQ
27
 
28
+ These files are GPTQ 4bit model files for [Eric Hartford's Wizard Vicuna 30B Uncensored](https://huggingface.co/ehartford/Wizard-Vicuna-30B-Uncensored) merged with [Kaio Ken's SuperHOT 8K](https://huggingface.co/kaiokendev/superhot-30b-8k-no-rlhf-test).
29
 
30
  It is the result of quantising to 4bit using [GPTQ-for-LLaMa](https://github.com/qwopqwop200/GPTQ-for-LLaMa).
31
 
32
+ **This is an experimental new GPTQ which offers up to 8K context size**
33
+
34
+ The increased context is currently only tested to work with [ExLlama](https://github.com/turboderp/exllama), via the latest release of [text-generation-webui](https://github.com/oobabooga/text-generation-webui).
35
+
36
+ Please read carefully below to see how to use it.
37
+
38
+ **NOTE**: Using the full 8K context will exceed 24GB VRAM.
39
+
40
+ GGML versions are not yet provided, as there is not yet support for SuperHOT in llama.cpp. This is being investigated and will hopefully come soon.
41
+
42
  ## Repositories available
43
 
44
  * [4-bit GPTQ models for GPU inference](https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Superhot-8K-GPTQ)
45
+ * [Unquantised SuperHOT fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Superhot-8K-GPTQ)
46
+ * [Eric's base unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/ehartford/Wizard-Vicuna-30B-Uncensored)
47
 
48
  ## How to easily download and use this model in text-generation-webui
49
 
 
53
  2. Under **Download custom model or LoRA**, enter `TheBloke/Wizard-Vicuna-30B-Superhot-8K-GPTQ`.
54
  3. Click **Download**.
55
  4. The model will start downloading. Once it's finished it will say "Done"
56
+ 5. Untick **Autoload the model**
57
+ 6. In the top left, click the refresh icon next to **Model**.
58
+ 7. In the **Model** dropdown, choose the model you just downloaded: `Wizard-Vicuna-30B-Superhot-8K-GPTQ`
59
+ 8. To use the increased context, set the **Loader** to **ExLlama**, set **max_seq_len** to 8192 or 4096, and set **compress_pos_emb** to **4** for 8192 context, or to **2** for 4096 context.
60
+ 9. Now click **Save Settings** followed by **Reload**
61
+ 10. The model will automatically load, and is now ready for use!
62
+ 11. Once you're ready, click the **Text Generation tab** and enter a prompt to get started!
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
63
 
64
+ ## How to use this GPTQ model from Python code - TBC
 
 
 
65
 
66
+ Using this model with increased context from Python code is currently untested, so this section is removed for now.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
67
 
68
  ## Provided files
69
 
 
74
  It was created without group_size to lower VRAM requirements, and with --act-order (desc_act) to boost inference accuracy as much as possible.
75
 
76
  * `wizard-vicuna-30b-superhot-8k-GPTQ-4bit--1g.act.order.safetensors`
77
+ * Works for use with ExLlama with increased context (4096 or 8192)
78
+ * Works with AutoGPTQ in Python code, including with increased context, if `trust_remote_code=True` is set.
79
+ * Should work with GPTQ-for-LLaMa in CUDA mode, but unknown if increased context works - TBC. May have issues with GPTQ-for-LLaMa Triton mode.
80
  * Works with text-generation-webui, including one-click-installers.
81
  * Parameters: Groupsize = -1. Act Order / desc_act = True.
82
 
 
108
 
109
  <!-- footer end -->
110
 
111
+ # Original model card: Kaio Ken's SuperHOT 8K
112
 
113
 
114
  ### SuperHOT Prototype 2 w/ 8K Context
 
140
  - weight decay of 0.1
141
  - AdamW beta1 of 0.9 and beta2 0.99, epsilon of 1e-5
142
  - Trained on 4-bit base model
143
+
144
+ # Original model card: Eric Hartford's Wizard Vicuna 30B Uncensored
145
+
146
+
147
+ This is [wizard-vicuna-13b](https://huggingface.co/junelee/wizard-vicuna-13b) trained with a subset of the dataset - responses that contained alignment / moralizing were removed. The intent is to train a WizardLM that doesn't have alignment built-in, so that alignment (of any sort) can be added separately with for example with a RLHF LoRA.
148
+
149
+ Shout out to the open source AI/ML community, and everyone who helped me out.
150
+
151
+ Note:
152
+
153
+ An uncensored model has no guardrails.
154
+
155
+ You are responsible for anything you do with the model, just as you are responsible for anything you do with any dangerous object such as a knife, gun, lighter, or car.
156
+
157
+ Publishing anything this model generates is the same as publishing it yourself.
158
+
159
+ You are responsible for the content you publish, and you cannot blame the model any more than you can blame the knife, gun, lighter, or car for what you do with it.