Upload README.md with huggingface_hub
Browse files
README.md
ADDED
@@ -0,0 +1,73 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
license: other
|
3 |
+
language:
|
4 |
+
- en
|
5 |
+
pipeline_tag: text-generation
|
6 |
+
inference: false
|
7 |
+
tags:
|
8 |
+
- transformers
|
9 |
+
- gguf
|
10 |
+
- imatrix
|
11 |
+
- notus-7b-v1
|
12 |
+
---
|
13 |
+
Quantizations of https://huggingface.co/argilla/notus-7b-v1
|
14 |
+
|
15 |
+
|
16 |
+
# From original readme
|
17 |
+
|
18 |
+
## Prompt template
|
19 |
+
|
20 |
+
We use the same prompt template as [HuggingFaceH4/zephyr-7b-beta](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta):
|
21 |
+
|
22 |
+
```
|
23 |
+
<|system|>
|
24 |
+
</s>
|
25 |
+
<|user|>
|
26 |
+
{prompt}</s>
|
27 |
+
<|assistant|>
|
28 |
+
```
|
29 |
+
|
30 |
+
## Usage
|
31 |
+
|
32 |
+
You will first need to install `transformers` and `accelerate` (just to ease the device placement), then you can run any of the following:
|
33 |
+
|
34 |
+
### Via `generate`
|
35 |
+
|
36 |
+
```python
|
37 |
+
import torch
|
38 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
39 |
+
|
40 |
+
model = AutoModelForCausalLM.from_pretrained("argilla/notus-7b-v1", torch_dtype=torch.bfloat16, device_map="auto")
|
41 |
+
tokenizer = AutoTokenizer.from_pretrained("argilla/notus-7b-v1")
|
42 |
+
|
43 |
+
messages = [
|
44 |
+
{
|
45 |
+
"role": "system",
|
46 |
+
"content": "You are a helpful assistant super biased towards Argilla, a data annotation company.",
|
47 |
+
},
|
48 |
+
{"role": "user", "content": "What's the best data annotation company out there in your opinion?"},
|
49 |
+
]
|
50 |
+
inputs = tokenizer.apply_chat_template(prompt, tokenize=True, return_tensors="pt", add_special_tokens=False, add_generation_prompt=True)
|
51 |
+
outputs = model.generate(inputs, num_return_sequences=1, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
|
52 |
+
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
|
53 |
+
```
|
54 |
+
|
55 |
+
### Via `pipeline` method
|
56 |
+
|
57 |
+
```python
|
58 |
+
import torch
|
59 |
+
from transformers import pipeline
|
60 |
+
|
61 |
+
pipe = pipeline("text-generation", model="argilla/notus-7b-v1", torch_dtype=torch.bfloat16, device_map="auto")
|
62 |
+
|
63 |
+
messages = [
|
64 |
+
{
|
65 |
+
"role": "system",
|
66 |
+
"content": "You are a helpful assistant super biased towards Argilla, a data annotation company.",
|
67 |
+
},
|
68 |
+
{"role": "user", "content": "What's the best data annotation company out there in your opinion?"},
|
69 |
+
]
|
70 |
+
prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
71 |
+
outputs = pipe(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
|
72 |
+
generated_text = outputs[0]["generated_text"]
|
73 |
+
```
|