Update README.md

README.md

@@ -1,6 +1,12 @@
 ---
 inference: false
 license: other
+datasets:
+- ehartford/wizard_vicuna_70k_unfiltered
+language:
+- en
+tags:
+- uncensored
 ---
 
 <!-- header start -->
@@ -19,17 +25,27 @@ license: other
 
 # Eric Hartford's Wizard Vicuna 30B Uncensored merged with Kaio Ken's SuperHOT 8K GPTQ
 
-These files are GPTQ 4bit model files for [Eric Hartford's Wizard Vicuna 30B Uncensored merged with Kaio Ken's SuperHOT 8K](https://huggingface.co/kaiokendev/superhot-30b-8k-no-rlhf-test).
+These files are GPTQ 4bit model files for [Eric Hartford's Wizard Vicuna 30B Uncensored](https://huggingface.co/ehartford/Wizard-Vicuna-30B-Uncensored) merged with [Kaio Ken's SuperHOT 8K](https://huggingface.co/kaiokendev/superhot-30b-8k-no-rlhf-test).
 
 It is the result of quantising to 4bit using [GPTQ-for-LLaMa](https://github.com/qwopqwop200/GPTQ-for-LLaMa).
 
+**This is an experimental new GPTQ which offers up to 8K context size**
+
+The increased context is currently only tested to work with [ExLlama](https://github.com/turboderp/exllama), via the latest release of [text-generation-webui](https://github.com/oobabooga/text-generation-webui).
+
+Please read carefully below to see how to use it.
+
+**NOTE**: Using the full 8K context will exceed 24GB VRAM.
+
+GGML versions are not yet provided, as there is not yet support for SuperHOT in llama.cpp. This is being investigated and will hopefully come soon.
+
 ## Repositories available
 
 * [4-bit GPTQ models for GPU inference](https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Superhot-8K-GPTQ)
-* [2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference](https://huggingface.co/none)
-* [Unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/ehartford/Wizard-Vicuna-30B-Uncensored)
+* [Unquantised SuperHOT fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Superhot-8K-GPTQ)
+* [Eric's base unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/ehartford/Wizard-Vicuna-30B-Uncensored)
 
-## How to easily download and use this model in text-generation-webui
+## How to easily download and use this model in text-generation-webui with ExLlama
 
 Please make sure you're using the latest version of text-generation-webui
 
@@ -37,20 +53,25 @@ Please make sure you're using the latest version of text-generation-webui
 2. Under **Download custom model or LoRA**, enter `TheBloke/Wizard-Vicuna-30B-Superhot-8K-GPTQ`.
 3. Click **Download**.
 4. The model will start downloading. Once it's finished it will say "Done"
-5. In the top left, click the refresh icon next to **Model**.
-6. In the **Model** dropdown, choose the model you just downloaded: `Wizard-Vicuna-30B-Superhot-8K-GPTQ`
-7. The model will automatically load, and is now ready for use!
-8. If you want any custom settings, set them and then click **Save settings for this model** followed by **Reload the Model** in the top right.
-  * Note that you do not need to and should not set manual GPTQ parameters any more. These are set automatically from the file `quantize_config.json`.
-9. Once you're ready, click the **Text Generation tab** and enter a prompt to get started!
+5. Untick **Autoload the model**
+6. In the top left, click the refresh icon next to **Model**.
+7. In the **Model** dropdown, choose the model you just downloaded: `Wizard-Vicuna-30B-Superhot-8K-GPTQ`
+8. To use the increased context, set the **Loader** to **ExLlama**, set **max_seq_len** to 8192 or 4096, and set **compress_pos_emb** to **4** for 8192 context, or to **2** for 4096 context.
+9. Now click **Save Settings** followed by **Reload**
+10. The model will automatically load, and is now ready for use!
+11. Once you're ready, click the **Text Generation tab** and enter a prompt to get started!
 
-## How to use this GPTQ model from Python code
+## How to use this GPTQ model from Python code with AutoGPTQ
 
-First make sure you have [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ) installed:
+First make sure you have AutoGPTQ and Einops installed:
 
-`pip install auto-gptq`
+```
+pip3 install einops auto-gptq
+```
+
+Then run the following code. Note that in order to get this to work, `config.json` has been hardcoded to a sequence length of 8192.
 
-Then try the following example code:
+If you want to try 4096 instead to reduce VRAM usage, please manually edit `config.json` to set `max_position_embeddings` to the value you want.
 
 ```python
 from transformers import AutoTokenizer, pipeline, logging
@@ -67,11 +88,13 @@ tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
 model = AutoGPTQForCausalLM.from_quantized(model_name_or_path,
         model_basename=model_basename,
         use_safetensors=True,
-        trust_remote_code=False,
-        device="cuda:0",
+        trust_remote_code=True,
+        device_map='auto',
         use_triton=use_triton,
         quantize_config=None)
 
+model.seqlen = 8192
+
 # Note: check the prompt template is correct for this model.
 prompt = "Tell me about AI"
 prompt_template=f'''USER: {prompt}
@@ -102,6 +125,12 @@ pipe = pipeline(
 print(pipe(prompt_template)[0]['generated_text'])
 ```
 
+## Using other UIs: monkey patch
+
+Provided in the repo is `llama_rope_scaled_monkey_patch.py`, written by @kaiokendev.
+
+It can be theoretically be added to any Python UI or custom code to enable the same result as `trust_remote_code=True`.  I have not tested this, and it should be superseded by using `trust_remote_code=True`, but I include it for completeness and for interest.
+
 ## Provided files
 
 **wizard-vicuna-30b-superhot-8k-GPTQ-4bit--1g.act.order.safetensors**
@@ -111,9 +140,9 @@ This will work with AutoGPTQ, ExLlama, and CUDA versions of GPTQ-for-LLaMa. Ther
 It was created without group_size to lower VRAM requirements, and with --act-order (desc_act) to boost inference accuracy as much as possible.
 
 * `wizard-vicuna-30b-superhot-8k-GPTQ-4bit--1g.act.order.safetensors`
-  * Works with AutoGPTQ in CUDA or Triton modes.
-  * LLaMa models also work with [ExLlama](https://github.com/turboderp/exllama}, which usually provides much higher performance, and uses less VRAM, than AutoGPTQ.
-  * Works with GPTQ-for-LLaMa in CUDA mode.  May have issues with GPTQ-for-LLaMa Triton mode.
+  * Works for use with ExLlama with increased context (4096 or 8192)
+  * Works with AutoGPTQ in Python code, including with increased context, if `trust_remote_code=True` is set.
+  * Should work with GPTQ-for-LLaMa in CUDA mode, but unknown if increased context works - TBC.  May have issues with GPTQ-for-LLaMa Triton mode.
   * Works with text-generation-webui, including one-click-installers.
   * Parameters: Groupsize = -1. Act Order / desc_act = True.
 
@@ -145,7 +174,7 @@ Thank you to all my generous patrons and donaters!
 
 <!-- footer end -->
 
-# Original model card: Eric Hartford's Wizard Vicuna 30B Uncensored merged with Kaio Ken's SuperHOT 8K
+# Original model card: Kaio Ken's SuperHOT 8K
 
 
 ### SuperHOT Prototype 2 w/ 8K Context
@@ -177,3 +206,20 @@ I trained the LoRA with the following configuration:
 - weight decay of 0.1
 - AdamW beta1 of 0.9 and beta2 0.99, epsilon of 1e-5
 - Trained on 4-bit base model
+
+# Original model card: Eric Hartford's Wizard Vicuna 30B Uncensored
+
+
+This is [wizard-vicuna-13b](https://huggingface.co/junelee/wizard-vicuna-13b) trained with a subset of the dataset - responses that contained alignment / moralizing were removed. The intent is to train a WizardLM that doesn't have alignment built-in, so that alignment (of any sort) can be added separately with for example with a RLHF LoRA.
+
+Shout out to the open source AI/ML community, and everyone who helped me out.
+
+Note:  
+
+An uncensored model has no guardrails.  
+
+You are responsible for anything you do with the model, just as you are responsible for anything you do with any dangerous object such as a knife, gun, lighter, or car.
+
+Publishing anything this model generates is the same as publishing it yourself.
+
+You are responsible for the content you publish, and you cannot blame the model any more than you can blame the knife, gun, lighter, or car for what you do with it.