Commit
•
efd0231
1
Parent(s):
0bea04a
Update README.md
Browse files
README.md
CHANGED
@@ -56,6 +56,34 @@ _NOTE_: `bitsandbytes` library is only needed for 8 bit version. Otherwise, it's
|
|
56 |
|
57 |
### Inference on a big GPU
|
58 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
59 |
### Inference on a small GPU (Consumer Hardware/Free Colab)
|
60 |
|
61 |
## Known Issues
|
|
|
56 |
|
57 |
### Inference on a big GPU
|
58 |
|
59 |
+
If you have a big enough GPU like an A100 in your posession, this code is for you.
|
60 |
+
|
61 |
+
```python
|
62 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig
|
63 |
+
import torch
|
64 |
+
|
65 |
+
model_name_or_id = "MaralGPT/Maral-7B-alpha-1"
|
66 |
+
|
67 |
+
model = AutoModelForCausalLM.from_pretrained(model_name_or_id, torch_dtype=torch.float16, device_map="auto")
|
68 |
+
tokenizer = AutoTokenizer.from_pretrained(model_name_or_id)
|
69 |
+
|
70 |
+
prompt = "در سال ۱۹۹۶ چه کسی رییس جمهور آمریکا بود؟"
|
71 |
+
prompt = f"### Human:{prompt}\n### Assistant:"
|
72 |
+
|
73 |
+
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
|
74 |
+
|
75 |
+
generation_config = GenerationConfig(
|
76 |
+
do_sample=True,
|
77 |
+
top_k=1,
|
78 |
+
temperature=0.5,
|
79 |
+
max_new_tokens=300,
|
80 |
+
pad_token_id=tokenizer.eos_token_id
|
81 |
+
)
|
82 |
+
|
83 |
+
outputs = model.generate(**inputs, generation_config=generation_config)
|
84 |
+
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
85 |
+
```
|
86 |
+
|
87 |
### Inference on a small GPU (Consumer Hardware/Free Colab)
|
88 |
|
89 |
## Known Issues
|