upload model

README.md

@@ -1,3 +1,115 @@
 ---
 license: llama2
+train: false
+inference: false
+pipeline_tag: text-generation
 ---
+
+## Llama-2-13b-chat-hf-4bit_g64-HQQ
+This is a version of the LLama-2-13B-chat-hf model quantized to 4-bit via Half-Quadratic Quantization (HQQ): https://mobiusml.github.io/hqq_blog/
+
+### Basic Usage
+To run the model, install the HQQ library from https://github.com/mobiusml/hqq and use it as follows:
+``` Python
+from hqq.models.llama_hf import LlamaHQQ
+import transformers
+
+model_id = 'mobiuslabsgmbh/Llama-2-13b-chat-hf-4bit_g64-HQQ'
+#Load the tokenizer
+tokenizer = transformers.AutoTokenizer.from_pretrained(model_id)
+#Load the model
+model = LlamaHQQ.from_quantized(model_id)
+```
+
+### Basic Chat Example
+``` Python
+import transformers
+from hqq.models.llama_hf import LlamaHQQ
+
+model_id  = 'mobiuslabsgmbh/Llama-2-13b-chat-hf-4bit_g64-HQQ'
+#Load the tokenizer
+tokenizer = transformers.AutoTokenizer.from_pretrained(model_id)
+#Load the model
+model     = LlamaHQQ.from_quantized(model_id)
+
+##########################################################################################################
+from threading import Thread
+
+from sys import stdout
+def print_flush(data):
+	stdout.write("\r" + data)
+	stdout.flush()
+
+#Adapted from https://huggingface.co/spaces/huggingface-projects/llama-2-7b-chat/blob/main/app.py
+def process_conversation(chat):
+	system_prompt = chat['system_prompt']
+	chat_history  = chat['chat_history']
+	message       = chat['message']
+
+	conversation = []
+	if system_prompt:
+		conversation.append({"role": "system", "content": system_prompt})
+	for user, assistant in chat_history:
+		conversation.extend([{"role": "user", "content": user}, {"role": "assistant", "content": assistant}])
+	conversation.append({"role": "user", "content": message})
+
+	return tokenizer.apply_chat_template(conversation, return_tensors="pt").to('cuda')
+
+def chat_processor(chat, max_new_tokens=100, do_sample=True):
+	tokenizer.use_default_system_prompt = False
+	streamer = transformers.TextIteratorStreamer(tokenizer, timeout=10.0, skip_prompt=True, skip_special_tokens=True)
+
+	generate_params = dict(
+		{"input_ids": process_conversation(chat)},
+		streamer=streamer,
+		max_new_tokens=max_new_tokens,
+		do_sample=do_sample,
+		top_p=0.90,
+		top_k=50,
+		temperature= 0.6,
+		num_beams=1,
+		repetition_penalty=1.2,
+	)
+
+	t = Thread(target=model.generate, kwargs=generate_params)
+	t.start()
+
+	outputs = []
+	for text in streamer:
+		outputs.append(text)
+		print_flush("".join(outputs))
+
+	return outputs
+
+###################################################################################################
+
+outputs = chat_processor({'system_prompt':"You are a helpful assistant.",
+						'chat_history':[],
+						'message':"How can I build a car?"
+						}, 
+						 max_new_tokens=1000, do_sample=False)
+```
+
+<b>Output</b>:
+<p>
+ Wow, that's an exciting project! Building a car from scratch can be a challenging but rewarding experience. Here are some general steps you might consider as you embark on this journey:
+
+1. Define your goals and requirements: What kind of car do you want to build? Do you have any specific performance or design preferences? How many passengers will the car seat? Answers to these questions will help guide your decisions throughout the process.
+2. Choose a platform: Decide on the type of vehicle you want to build (e.g., sedan, SUV, sports car) and select a suitable platform (chassis and body style). This will influence the choice of components and materials for the rest of the build.
+3. Design and prototype: Sketch out your ideas and create a detailed set of blueprints or computer-aided designs (CADs). Build a scale model or mockup to test fit and form before committing to final plans.
+4. Source materials and components: Purchase or fabricate the necessary parts, including the frame, suspension, steering system, brakes, engine, transmission, and electrical components. You may need to customize certain pieces or find specialized suppliers.
+5. Assemble the chassis: Start by building the frame and attaching the suspension, steering, and brake systems. Ensure everything is properly aligned and balanced.
+6. Install the powertrain: Select and install the appropriate engine and transmission, considering factors like power output, fuel efficiency, and compatibility with your chosen platform.
+7. Add the bodywork: Install the body panels, paying close attention to gaps, fits, and finishes. Use rustproofing methods to protect the metal components.
+8. Integrate the electrical and electronics: Install the wiring harness, batteries, and other essential electrical components. Consider adding advanced features like infotainment systems or driver assistance technologies.
+9. Test and refine: Once the major components are in place, perform thorough tests to ensure proper functioning and durability. Make any necessary adjustments or upgrades based on your testing results.
+10. Register and insure your creation: Depending on where you live, you may need to register your DIY car with local authorities and obtain insurance coverage. Be sure to research the laws and regulations in your area regarding homemade vehicles.
+
+Please note that building a car from scratch can be a complex and time-consuming process, requiring significant expertise, tools, and resources. It's important to approach this project carefully and thoughtfully, taking into account safety considerations, legal requirements, and your own skills and limitations. If you're new to car construction, it may be helpful to seek guidance from experienced enthusiasts or professionals in the field. Good luck with your project!
+
+----------------------------------------------------------------------------------------------------------------------------------
+</p>
+
+*Limitations*: <br>
+-Only supports single GPU runtime.<br>
+-Not compatible with HuggingFace's PEFT.<br>

config.json

@@ -0,0 +1,27 @@
+{
+  "_name_or_path": "meta-llama/Llama-2-13b-chat-hf",
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 13824,
+  "max_position_embeddings": 4096,
+  "model_type": "llama",
+  "num_attention_heads": 40,
+  "num_hidden_layers": 40,
+  "num_key_value_heads": 40,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float16",
+  "transformers_version": "4.35.2",
+  "use_cache": true,
+  "vocab_size": 32000
+}

generation_config.json

@@ -0,0 +1,10 @@
+{
+  "bos_token_id": 1,
+  "do_sample": true,
+  "eos_token_id": 2,
+  "pad_token_id": 0,
+  "temperature": 0.6,
+  "max_length": 4096,
+  "top_p": 0.9,
+  "transformers_version": "4.31.0.dev0"
+}

qmodel.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:24a78e1992a55f25970826768c489475aa7594c812bbb79c2614ad44370555a2
+size 7595286109

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
+size 499723

tokenizer_config.json

@@ -0,0 +1,36 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "bos_token": {
+    "__type": "AddedToken",
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "chat_template": "{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% set system_message = false %}{% endif %}{% for message in loop_messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if loop.index0 == 0 and system_message != false %}{% set content = '<<SYS>>\\n' + system_message + '\\n<</SYS>>\\n\\n' + message['content'] %}{% else %}{% set content = message['content'] %}{% endif %}{% if message['role'] == 'user' %}{{ bos_token + '[INST] ' + content.strip() + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ ' '  + content.strip() + ' ' + eos_token }}{% endif %}{% endfor %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": {
+    "__type": "AddedToken",
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "legacy": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": null,
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": {
+    "__type": "AddedToken",
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}