TheBloke commited on
Commit
7dcc315
1 Parent(s): d80728a

Upload README.md

Browse files
Files changed (1) hide show
  1. README.md +567 -0
README.md ADDED
@@ -0,0 +1,567 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: PAIXAI/Astrid-Mistral-7B
3
+ inference: false
4
+ language:
5
+ - en
6
+ library_name: transformers
7
+ license: apache-2.0
8
+ model_creator: PAIX
9
+ model_name: Astrid Mistral 7B
10
+ model_type: mistral
11
+ prompt_template: '<|im_start|>system
12
+
13
+ {system_message}<|im_end|>
14
+
15
+ <|im_start|>user
16
+
17
+ {prompt}<|im_end|>
18
+
19
+ <|im_start|>assistant
20
+
21
+ '
22
+ quantized_by: TheBloke
23
+ tags:
24
+ - gpt
25
+ - llm
26
+ - large language model
27
+ - PAIX.Cloud
28
+ thumbnail: https://static.wixstatic.com/media/bdee4e_8aa5cefc86024bc88f7e20e3e19d9ff3~mv2.png/v1/fill/w_192%2Ch_192%2Clg_1%2Cusm_0.66_1.00_0.01/bdee4e_8aa5cefc86024bc88f7e20e3e19d9ff3~mv2.png
29
+ ---
30
+ <!-- markdownlint-disable MD041 -->
31
+
32
+ <!-- header start -->
33
+ <!-- 200823 -->
34
+ <div style="width: auto; margin-left: auto; margin-right: auto">
35
+ <img src="https://i.imgur.com/EBdldam.jpg" alt="TheBlokeAI" style="width: 100%; min-width: 400px; display: block; margin: auto;">
36
+ </div>
37
+ <div style="display: flex; justify-content: space-between; width: 100%;">
38
+ <div style="display: flex; flex-direction: column; align-items: flex-start;">
39
+ <p style="margin-top: 0.5em; margin-bottom: 0em;"><a href="https://discord.gg/theblokeai">Chat & support: TheBloke's Discord server</a></p>
40
+ </div>
41
+ <div style="display: flex; flex-direction: column; align-items: flex-end;">
42
+ <p style="margin-top: 0.5em; margin-bottom: 0em;"><a href="https://www.patreon.com/TheBlokeAI">Want to contribute? TheBloke's Patreon page</a></p>
43
+ </div>
44
+ </div>
45
+ <div style="text-align:center; margin-top: 0em; margin-bottom: 0em"><p style="margin-top: 0.25em; margin-bottom: 0em;">TheBloke's LLM work is generously supported by a grant from <a href="https://a16z.com">andreessen horowitz (a16z)</a></p></div>
46
+ <hr style="margin-top: 1.0em; margin-bottom: 1.0em;">
47
+ <!-- header end -->
48
+
49
+ # Astrid Mistral 7B - AWQ
50
+ - Model creator: [PAIX](https://huggingface.co/PAIXAI)
51
+ - Original model: [Astrid Mistral 7B](https://huggingface.co/PAIXAI/Astrid-Mistral-7B)
52
+
53
+ <!-- description start -->
54
+ ## Description
55
+
56
+ This repo contains AWQ model files for [PAIX's Astrid Mistral 7B](https://huggingface.co/PAIXAI/Astrid-Mistral-7B).
57
+
58
+ These files were quantised using hardware kindly provided by [Massed Compute](https://massedcompute.com/).
59
+
60
+
61
+ ### About AWQ
62
+
63
+ AWQ is an efficient, accurate and blazing-fast low-bit weight quantization method, currently supporting 4-bit quantization. Compared to GPTQ, it offers faster Transformers-based inference with equivalent or better quality compared to the most commonly used GPTQ settings.
64
+
65
+ It is supported by:
66
+
67
+ - [Text Generation Webui](https://github.com/oobabooga/text-generation-webui) - using Loader: AutoAWQ
68
+ - [vLLM](https://github.com/vllm-project/vllm) - Llama and Mistral models only
69
+ - [Hugging Face Text Generation Inference (TGI)](https://github.com/huggingface/text-generation-inference)
70
+ - [Transformers](https://huggingface.co/docs/transformers) version 4.35.0 and later, from any code or client that supports Transformers
71
+ - [AutoAWQ](https://github.com/casper-hansen/AutoAWQ) - for use from Python code
72
+
73
+ <!-- description end -->
74
+ <!-- repositories-available start -->
75
+ ## Repositories available
76
+
77
+ * [AWQ model(s) for GPU inference.](https://huggingface.co/TheBloke/Astrid-Mistral-7B-AWQ)
78
+ * [GPTQ models for GPU inference, with multiple quantisation parameter options.](https://huggingface.co/TheBloke/Astrid-Mistral-7B-GPTQ)
79
+ * [2, 3, 4, 5, 6 and 8-bit GGUF models for CPU+GPU inference](https://huggingface.co/TheBloke/Astrid-Mistral-7B-GGUF)
80
+ * [PAIX's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/PAIXAI/Astrid-Mistral-7B)
81
+ <!-- repositories-available end -->
82
+
83
+ <!-- prompt-template start -->
84
+ ## Prompt template: ChatML
85
+
86
+ ```
87
+ <|im_start|>system
88
+ {system_message}<|im_end|>
89
+ <|im_start|>user
90
+ {prompt}<|im_end|>
91
+ <|im_start|>assistant
92
+
93
+ ```
94
+
95
+ <!-- prompt-template end -->
96
+
97
+
98
+ <!-- README_AWQ.md-provided-files start -->
99
+ ## Provided files, and AWQ parameters
100
+
101
+ I currently release 128g GEMM models only. The addition of group_size 32 models, and GEMV kernel models, is being actively considered.
102
+
103
+ Models are released as sharded safetensors files.
104
+
105
+ | Branch | Bits | GS | AWQ Dataset | Seq Len | Size |
106
+ | ------ | ---- | -- | ----------- | ------- | ---- |
107
+ | [main](https://huggingface.co/TheBloke/Astrid-Mistral-7B-AWQ/tree/main) | 4 | 128 | [VMware Open Instruct](https://huggingface.co/datasets/VMware/open-instruct/viewer/) | 4096 | 4.15 GB
108
+
109
+ <!-- README_AWQ.md-provided-files end -->
110
+
111
+ <!-- README_AWQ.md-text-generation-webui start -->
112
+ ## How to easily download and use this model in [text-generation-webui](https://github.com/oobabooga/text-generation-webui)
113
+
114
+ Please make sure you're using the latest version of [text-generation-webui](https://github.com/oobabooga/text-generation-webui).
115
+
116
+ It is strongly recommended to use the text-generation-webui one-click-installers unless you're sure you know how to make a manual install.
117
+
118
+ 1. Click the **Model tab**.
119
+ 2. Under **Download custom model or LoRA**, enter `TheBloke/Astrid-Mistral-7B-AWQ`.
120
+ 3. Click **Download**.
121
+ 4. The model will start downloading. Once it's finished it will say "Done".
122
+ 5. In the top left, click the refresh icon next to **Model**.
123
+ 6. In the **Model** dropdown, choose the model you just downloaded: `Astrid-Mistral-7B-AWQ`
124
+ 7. Select **Loader: AutoAWQ**.
125
+ 8. Click Load, and the model will load and is now ready for use.
126
+ 9. If you want any custom settings, set them and then click **Save settings for this model** followed by **Reload the Model** in the top right.
127
+ 10. Once you're ready, click the **Text Generation** tab and enter a prompt to get started!
128
+ <!-- README_AWQ.md-text-generation-webui end -->
129
+
130
+ <!-- README_AWQ.md-use-from-vllm start -->
131
+ ## Multi-user inference server: vLLM
132
+
133
+ Documentation on installing and using vLLM [can be found here](https://vllm.readthedocs.io/en/latest/).
134
+
135
+ - Please ensure you are using vLLM version 0.2 or later.
136
+ - When using vLLM as a server, pass the `--quantization awq` parameter.
137
+
138
+ For example:
139
+
140
+ ```shell
141
+ python3 -m vllm.entrypoints.api_server --model TheBloke/Astrid-Mistral-7B-AWQ --quantization awq --dtype auto
142
+ ```
143
+
144
+ - When using vLLM from Python code, again set `quantization=awq`.
145
+
146
+ For example:
147
+
148
+ ```python
149
+ from vllm import LLM, SamplingParams
150
+
151
+ prompts = [
152
+ "Tell me about AI",
153
+ "Write a story about llamas",
154
+ "What is 291 - 150?",
155
+ "How much wood would a woodchuck chuck if a woodchuck could chuck wood?",
156
+ ]
157
+ prompt_template=f'''<|im_start|>system
158
+ {system_message}<|im_end|>
159
+ <|im_start|>user
160
+ {prompt}<|im_end|>
161
+ <|im_start|>assistant
162
+ '''
163
+
164
+ prompts = [prompt_template.format(prompt=prompt) for prompt in prompts]
165
+
166
+ sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
167
+
168
+ llm = LLM(model="TheBloke/Astrid-Mistral-7B-AWQ", quantization="awq", dtype="auto")
169
+
170
+ outputs = llm.generate(prompts, sampling_params)
171
+
172
+ # Print the outputs.
173
+ for output in outputs:
174
+ prompt = output.prompt
175
+ generated_text = output.outputs[0].text
176
+ print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
177
+ ```
178
+ <!-- README_AWQ.md-use-from-vllm start -->
179
+
180
+ <!-- README_AWQ.md-use-from-tgi start -->
181
+ ## Multi-user inference server: Hugging Face Text Generation Inference (TGI)
182
+
183
+ Use TGI version 1.1.0 or later. The official Docker container is: `ghcr.io/huggingface/text-generation-inference:1.1.0`
184
+
185
+ Example Docker parameters:
186
+
187
+ ```shell
188
+ --model-id TheBloke/Astrid-Mistral-7B-AWQ --port 3000 --quantize awq --max-input-length 3696 --max-total-tokens 4096 --max-batch-prefill-tokens 4096
189
+ ```
190
+
191
+ Example Python code for interfacing with TGI (requires [huggingface-hub](https://github.com/huggingface/huggingface_hub) 0.17.0 or later):
192
+
193
+ ```shell
194
+ pip3 install huggingface-hub
195
+ ```
196
+
197
+ ```python
198
+ from huggingface_hub import InferenceClient
199
+
200
+ endpoint_url = "https://your-endpoint-url-here"
201
+
202
+ prompt = "Tell me about AI"
203
+ prompt_template=f'''<|im_start|>system
204
+ {system_message}<|im_end|>
205
+ <|im_start|>user
206
+ {prompt}<|im_end|>
207
+ <|im_start|>assistant
208
+ '''
209
+
210
+ client = InferenceClient(endpoint_url)
211
+ response = client.text_generation(prompt,
212
+ max_new_tokens=128,
213
+ do_sample=True,
214
+ temperature=0.7,
215
+ top_p=0.95,
216
+ top_k=40,
217
+ repetition_penalty=1.1)
218
+
219
+ print(f"Model output: ", response)
220
+ ```
221
+ <!-- README_AWQ.md-use-from-tgi end -->
222
+
223
+ <!-- README_AWQ.md-use-from-python start -->
224
+ ## Inference from Python code using Transformers
225
+
226
+ ### Install the necessary packages
227
+
228
+ - Requires: [Transformers](https://huggingface.co/docs/transformers) 4.35.0 or later.
229
+ - Requires: [AutoAWQ](https://github.com/casper-hansen/AutoAWQ) 0.1.6 or later.
230
+
231
+ ```shell
232
+ pip3 install --upgrade "autoawq>=0.1.6" "transformers>=4.35.0"
233
+ ```
234
+
235
+ Note that if you are using PyTorch 2.0.1, the above AutoAWQ command will automatically upgrade you to PyTorch 2.1.0.
236
+
237
+ If you are using CUDA 11.8 and wish to continue using PyTorch 2.0.1, instead run this command:
238
+
239
+ ```shell
240
+ pip3 install https://github.com/casper-hansen/AutoAWQ/releases/download/v0.1.6/autoawq-0.1.6+cu118-cp310-cp310-linux_x86_64.whl
241
+ ```
242
+
243
+ If you have problems installing [AutoAWQ](https://github.com/casper-hansen/AutoAWQ) using the pre-built wheels, install it from source instead:
244
+
245
+ ```shell
246
+ pip3 uninstall -y autoawq
247
+ git clone https://github.com/casper-hansen/AutoAWQ
248
+ cd AutoAWQ
249
+ pip3 install .
250
+ ```
251
+
252
+ ### Transformers example code (requires Transformers 4.35.0 and later)
253
+
254
+ ```python
255
+ from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
256
+
257
+ model_name_or_path = "TheBloke/Astrid-Mistral-7B-AWQ"
258
+
259
+ tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
260
+ model = AutoModelForCausalLM.from_pretrained(
261
+ model_name_or_path,
262
+ low_cpu_mem_usage=True,
263
+ device_map="cuda:0"
264
+ )
265
+
266
+ # Using the text streamer to stream output one token at a time
267
+ streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
268
+
269
+ prompt = "Tell me about AI"
270
+ prompt_template=f'''<|im_start|>system
271
+ {system_message}<|im_end|>
272
+ <|im_start|>user
273
+ {prompt}<|im_end|>
274
+ <|im_start|>assistant
275
+ '''
276
+
277
+ # Convert prompt to tokens
278
+ tokens = tokenizer(
279
+ prompt_template,
280
+ return_tensors='pt'
281
+ ).input_ids.cuda()
282
+
283
+ generation_params = {
284
+ "do_sample": True,
285
+ "temperature": 0.7,
286
+ "top_p": 0.95,
287
+ "top_k": 40,
288
+ "max_new_tokens": 512,
289
+ "repetition_penalty": 1.1
290
+ }
291
+
292
+ # Generate streamed output, visible one token at a time
293
+ generation_output = model.generate(
294
+ tokens,
295
+ streamer=streamer,
296
+ **generation_params
297
+ )
298
+
299
+ # Generation without a streamer, which will include the prompt in the output
300
+ generation_output = model.generate(
301
+ tokens,
302
+ **generation_params
303
+ )
304
+
305
+ # Get the tokens from the output, decode them, print them
306
+ token_output = generation_output[0]
307
+ text_output = tokenizer.decode(token_output)
308
+ print("model.generate output: ", text_output)
309
+
310
+ # Inference is also possible via Transformers' pipeline
311
+ from transformers import pipeline
312
+
313
+ pipe = pipeline(
314
+ "text-generation",
315
+ model=model,
316
+ tokenizer=tokenizer,
317
+ **generation_params
318
+ )
319
+
320
+ pipe_output = pipe(prompt_template)[0]['generated_text']
321
+ print("pipeline output: ", pipe_output)
322
+
323
+ ```
324
+ <!-- README_AWQ.md-use-from-python end -->
325
+
326
+ <!-- README_AWQ.md-compatibility start -->
327
+ ## Compatibility
328
+
329
+ The files provided are tested to work with:
330
+
331
+ - [text-generation-webui](https://github.com/oobabooga/text-generation-webui) using `Loader: AutoAWQ`.
332
+ - [vLLM](https://github.com/vllm-project/vllm) version 0.2.0 and later.
333
+ - [Hugging Face Text Generation Inference (TGI)](https://github.com/huggingface/text-generation-inference) version 1.1.0 and later.
334
+ - [Transformers](https://huggingface.co/docs/transformers) version 4.35.0 and later.
335
+ - [AutoAWQ](https://github.com/casper-hansen/AutoAWQ) version 0.1.1 and later.
336
+
337
+ <!-- README_AWQ.md-compatibility end -->
338
+
339
+ <!-- footer start -->
340
+ <!-- 200823 -->
341
+ ## Discord
342
+
343
+ For further support, and discussions on these models and AI in general, join us at:
344
+
345
+ [TheBloke AI's Discord server](https://discord.gg/theblokeai)
346
+
347
+ ## Thanks, and how to contribute
348
+
349
+ Thanks to the [chirper.ai](https://chirper.ai) team!
350
+
351
+ Thanks to Clay from [gpus.llm-utils.org](llm-utils)!
352
+
353
+ I've had a lot of people ask if they can contribute. I enjoy providing models and helping people, and would love to be able to spend even more time doing it, as well as expanding into new projects like fine tuning/training.
354
+
355
+ If you're able and willing to contribute it will be most gratefully received and will help me to keep providing more models, and to start work on new AI projects.
356
+
357
+ Donaters will get priority support on any and all AI/LLM/model questions and requests, access to a private Discord room, plus other benefits.
358
+
359
+ * Patreon: https://patreon.com/TheBlokeAI
360
+ * Ko-Fi: https://ko-fi.com/TheBlokeAI
361
+
362
+ **Special thanks to**: Aemon Algiz.
363
+
364
+ **Patreon special mentions**: Brandon Frisco, LangChain4j, Spiking Neurons AB, transmissions 11, Joseph William Delisle, Nitin Borwankar, Willem Michiel, Michael Dempsey, vamX, Jeffrey Morgan, zynix, jjj, Omer Bin Jawed, Sean Connelly, jinyuan sun, Jeromy Smith, Shadi, Pawan Osman, Chadd, Elijah Stavena, Illia Dulskyi, Sebastain Graf, Stephen Murray, terasurfer, Edmond Seymore, Celu Ramasamy, Mandus, Alex, biorpg, Ajan Kanaga, Clay Pascal, Raven Klaugh, 阿明, K, ya boyyy, usrbinkat, Alicia Loh, John Villwock, ReadyPlayerEmma, Chris Smitley, Cap'n Zoog, fincy, GodLy, S_X, sidney chen, Cory Kujawski, OG, Mano Prime, AzureBlack, Pieter, Kalila, Spencer Kim, Tom X Nguyen, Stanislav Ovsiannikov, Michael Levine, Andrey, Trailburnt, Vadim, Enrico Ros, Talal Aujan, Brandon Phillips, Jack West, Eugene Pentland, Michael Davis, Will Dee, webtim, Jonathan Leane, Alps Aficionado, Rooh Singh, Tiffany J. Kim, theTransient, Luke @flexchar, Elle, Caitlyn Gatomon, Ari Malik, subjectnull, Johann-Peter Hartmann, Trenton Dambrowitz, Imad Khwaja, Asp the Wyvern, Emad Mostaque, Rainer Wilmers, Alexandros Triantafyllidis, Nicholas, Pedro Madruga, SuperWojo, Harry Royden McLaughlin, James Bentley, Olakabola, David Ziegler, Ai Maven, Jeff Scroggin, Nikolai Manek, Deo Leter, Matthew Berman, Fen Risland, Ken Nordquist, Manuel Alberto Morcote, Luke Pendergrass, TL, Fred von Graf, Randy H, Dan Guido, NimbleBox.ai, Vitor Caleffi, Gabriel Tamborski, knownsqashed, Lone Striker, Erik Bjäreholt, John Detwiler, Leonard Tan, Iucharbius
365
+
366
+
367
+ Thank you to all my generous patrons and donaters!
368
+
369
+ And thank you again to a16z for their generous grant.
370
+
371
+ <!-- footer end -->
372
+
373
+ # Original model card: PAIX's Astrid Mistral 7B
374
+
375
+ # Model Card
376
+ ## Summary
377
+
378
+ - Base model: [mistralai/Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1)
379
+
380
+ This model, Astrid-7B-Assistant is a Mistral-7B base model for causal language modeling, designed to generate human-like text.
381
+ It's part of our mission to make AI technology accessible to everyone, focusing on personalization, data privacy, and transparent AI governance.
382
+ Trained in English, it's a versatile tool for a variety of applications.
383
+ This model is one of the many models available on our platform, and we currently have a 1B and 7B open-source model.
384
+
385
+ This model was trained by [PAIX.Cloud](https://www.paix.cloud/).
386
+ - Wait list: [Wait List](https://www.paix.cloud/join-waitlist)
387
+
388
+
389
+ ## Usage
390
+
391
+ To use the model with the `transformers` library on a machine with GPUs, first make sure you have the `transformers` library installed.
392
+
393
+ ```bash
394
+ pip install transformers==4.34.0
395
+ ```
396
+
397
+ Also make sure you are providing your huggingface token to the pipeline if the model is lying in a private repo.
398
+ - Either leave `token=True` in the `pipeline` and login to hugginface_hub by running
399
+ ```python
400
+ import huggingface_hub
401
+ huggingface_hub.login(<ACCES_TOKEN>)
402
+ ```
403
+ - Or directly pass your <ACCES_TOKEN> to `token` in the `pipeline`
404
+
405
+ ```python
406
+ from transformers import pipeline
407
+
408
+ generate_text = pipeline(
409
+ model="PAIXAI/Astrid-Mistral-7B",
410
+ torch_dtype="auto",
411
+ trust_remote_code=True,
412
+ use_fast=True,
413
+ device_map={"": "cuda:0"},
414
+ token=True,
415
+ )
416
+
417
+ res = generate_text(
418
+ "Why is drinking water so healthy?",
419
+ min_new_tokens=2,
420
+ max_new_tokens=256,
421
+ do_sample=False,
422
+ num_beams=1,
423
+ temperature=float(0.3),
424
+ repetition_penalty=float(1.2),
425
+ renormalize_logits=True
426
+ )
427
+ print(res[0]["generated_text"])
428
+ ```
429
+
430
+ You can print a sample prompt after the preprocessing step to see how it is feed to the tokenizer:
431
+
432
+ ```python
433
+ print(generate_text.preprocess("Why is drinking water so healthy?")["prompt_text"])
434
+ ```
435
+
436
+ ```bash
437
+ <|prompt|>Why is drinking water so healthy?<|im_end|><|answer|>
438
+ ```
439
+
440
+ Alternatively, you can download [h2oai_pipeline.py](h2oai_pipeline.py), store it alongside your notebook, and construct the pipeline yourself from the loaded model and tokenizer. If the model and the tokenizer are fully supported in the `transformers` package, this will allow you to set `trust_remote_code=False`.
441
+
442
+ ```python
443
+ from h2oai_pipeline import H2OTextGenerationPipeline
444
+ from transformers import AutoModelForCausalLM, AutoTokenizer
445
+
446
+ tokenizer = AutoTokenizer.from_pretrained(
447
+ "PAIXAI/Astrid-Mistral-7B",
448
+ use_fast=True,
449
+ padding_side="left",
450
+ trust_remote_code=True,
451
+ )
452
+ model = AutoModelForCausalLM.from_pretrained(
453
+ "PAIXAI/Astrid-Mistral-7B",
454
+ torch_dtype="auto",
455
+ device_map={"": "cuda:0"},
456
+ trust_remote_code=True,
457
+ )
458
+ generate_text = H2OTextGenerationPipeline(model=model, tokenizer=tokenizer)
459
+
460
+ res = generate_text(
461
+ "Why is drinking water so healthy?",
462
+ min_new_tokens=2,
463
+ max_new_tokens=256,
464
+ do_sample=False,
465
+ num_beams=1,
466
+ temperature=float(0.3),
467
+ repetition_penalty=float(1.2),
468
+ renormalize_logits=True
469
+ )
470
+ print(res[0]["generated_text"])
471
+ ```
472
+
473
+
474
+ You may also construct the pipeline from the loaded model and tokenizer yourself and consider the preprocessing steps:
475
+
476
+ ```python
477
+ from transformers import AutoModelForCausalLM, AutoTokenizer
478
+
479
+ model_name = "PAIXAI/Astrid-Mistral-7B" # either local folder or huggingface model name
480
+ # Important: The prompt needs to be in the same format the model was trained with.
481
+ # You can find an example prompt in the experiment logs.
482
+ prompt = "<|prompt|>How are you?<|im_end|><|answer|>"
483
+
484
+ tokenizer = AutoTokenizer.from_pretrained(
485
+ model_name,
486
+ use_fast=True,
487
+ trust_remote_code=True,
488
+ )
489
+ model = AutoModelForCausalLM.from_pretrained(
490
+ model_name,
491
+ torch_dtype="auto",
492
+ device_map={"": "cuda:0"},
493
+ trust_remote_code=True,
494
+ )
495
+ model.cuda().eval()
496
+ inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to("cuda")
497
+
498
+ # generate configuration can be modified to your needs
499
+ tokens = model.generate(
500
+ input_ids=inputs["input_ids"],
501
+ attention_mask=inputs["attention_mask"],
502
+ min_new_tokens=2,
503
+ max_new_tokens=256,
504
+ do_sample=False,
505
+ num_beams=1,
506
+ temperature=float(0.3),
507
+ repetition_penalty=float(1.2),
508
+ renormalize_logits=True
509
+ )[0]
510
+
511
+ tokens = tokens[inputs["input_ids"].shape[1]:]
512
+ answer = tokenizer.decode(tokens, skip_special_tokens=True)
513
+ print(answer)
514
+ ```
515
+
516
+ ## Quantization and sharding
517
+
518
+ You can load the models using quantization by specifying ```load_in_8bit=True``` or ```load_in_4bit=True```. Also, sharding on multiple GPUs is possible by setting ```device_map=auto```.
519
+
520
+ ## Model Architecture
521
+
522
+ ```
523
+ MistralForCausalLM(
524
+ (model): MistralModel(
525
+ (embed_tokens): Embedding(32002, 4096, padding_idx=0)
526
+ (layers): ModuleList(
527
+ (0-31): 32 x MistralDecoderLayer(
528
+ (self_attn): MistralAttention(
529
+ (q_proj): Linear(in_features=4096, out_features=4096, bias=False)
530
+ (k_proj): Linear(in_features=4096, out_features=1024, bias=False)
531
+ (v_proj): Linear(in_features=4096, out_features=1024, bias=False)
532
+ (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
533
+ (rotary_emb): MistralRotaryEmbedding()
534
+ )
535
+ (mlp): MistralMLP(
536
+ (gate_proj): Linear(in_features=4096, out_features=14336, bias=False)
537
+ (up_proj): Linear(in_features=4096, out_features=14336, bias=False)
538
+ (down_proj): Linear(in_features=14336, out_features=4096, bias=False)
539
+ (act_fn): SiLUActivation()
540
+ )
541
+ (input_layernorm): MistralRMSNorm()
542
+ (post_attention_layernorm): MistralRMSNorm()
543
+ )
544
+ )
545
+ (norm): MistralRMSNorm()
546
+ )
547
+ (lm_head): Linear(in_features=4096, out_features=32002, bias=False)
548
+ )
549
+ ```
550
+
551
+ ## Model Configuration
552
+
553
+ This model was trained using H2O LLM Studio and with the configuration in [cfg.yaml](cfg.yaml). Visit [H2O LLM Studio](https://github.com/h2oai/h2o-llmstudio) to learn how to train your own large language models.
554
+
555
+
556
+ ## Disclaimer
557
+
558
+ Please read this disclaimer carefully before using the large language model provided in this repository. Your use of the model signifies your agreement to the following terms and conditions.
559
+
560
+ - Biases and Offensiveness: The large language model is trained on a diverse range of internet text data, which may contain biased, racist, offensive, or otherwise inappropriate content. By using this model, you acknowledge and accept that the generated content may sometimes exhibit biases or produce content that is offensive or inappropriate. The developers of this repository do not endorse, support, or promote any such content or viewpoints.
561
+ - Limitations: The large language model is an AI-based tool and not a human. It may produce incorrect, nonsensical, or irrelevant responses. It is the user's responsibility to critically evaluate the generated content and use it at their discretion.
562
+ - Use at Your Own Risk: Users of this large language model must assume full responsibility for any consequences that may arise from their use of the tool. The developers and contributors of this repository shall not be held liable for any damages, losses, or harm resulting from the use or misuse of the provided model.
563
+ - Ethical Considerations: Users are encouraged to use the large language model responsibly and ethically. By using this model, you agree not to use it for purposes that promote hate speech, discrimination, harassment, or any form of illegal or harmful activities.
564
+ - Reporting Issues: If you encounter any biased, offensive, or otherwise inappropriate content generated by the large language model, please report it to the repository maintainers through the provided channels. Your feedback will help improve the model and mitigate potential issues.
565
+ - Changes to this Disclaimer: The developers of this repository reserve the right to modify or update this disclaimer at any time without prior notice. It is the user's responsibility to periodically review the disclaimer to stay informed about any changes.
566
+
567
+ By using the large language model provided in this repository, you agree to accept and comply with the terms and conditions outlined in this disclaimer. If you do not agree with any part of this disclaimer, you should refrain from using the model and any content generated by it.