Create README.md
Browse files
README.md
ADDED
@@ -0,0 +1,95 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
license: creativeml-openrail-m
|
3 |
+
language:
|
4 |
+
- en
|
5 |
+
tags:
|
6 |
+
- LLM
|
7 |
+
- tensorRT
|
8 |
+
- chatGLM
|
9 |
+
---
|
10 |
+
## Model Card for lyraChatGLM
|
11 |
+
|
12 |
+
lyraChatGLM is currently the **fastest chatGLM-6B** available, as far as we know, it is also the **fisrt accelerated version of chatGLM-6B**.
|
13 |
+
|
14 |
+
The inference speed of lyraChatGLM is **10x** faster than the original version, and we're still working to improve the performance.
|
15 |
+
|
16 |
+
Among its main features are:
|
17 |
+
|
18 |
+
- weights: original ChatGLM-6B weights released by THUDM.
|
19 |
+
- device: lyraChatGLM is mainly based on FasterTransformer compiled for SM=80 (A100, for example).
|
20 |
+
|
21 |
+
## Speed
|
22 |
+
|
23 |
+
### test environment
|
24 |
+
|
25 |
+
- device: Nvidia A100 40G
|
26 |
+
- img size: 512x512
|
27 |
+
- percision:fp16
|
28 |
+
- steps: 30
|
29 |
+
- solver: LMSD
|
30 |
+
|
31 |
+
### text2img
|
32 |
+
|
33 |
+
|
34 |
+
|
35 |
+
|
36 |
+
## Model Sources
|
37 |
+
|
38 |
+
- **Repository:** [https://huggingface.co/THUDM/chatglm-6b]
|
39 |
+
|
40 |
+
## Uses
|
41 |
+
|
42 |
+
```python
|
43 |
+
from faster_chat_glm import GLM6B, FasterChatGLM
|
44 |
+
|
45 |
+
# kernel for chat model.
|
46 |
+
kernel = GLM6B(plan_path=plan_path,
|
47 |
+
batch_size=BATCH_SIZE,
|
48 |
+
num_beams=1,
|
49 |
+
use_cache=USE_CACHE,
|
50 |
+
num_heads=32,
|
51 |
+
emb_size_per_heads=128,
|
52 |
+
decoder_layers=28,
|
53 |
+
vocab_size=150528,
|
54 |
+
max_seq_len=MAX_OUT_LEN)
|
55 |
+
|
56 |
+
chat = FasterChatGLM(model_dir=chatglm6b_dir, kernel=kernel).half().cuda()
|
57 |
+
|
58 |
+
# generate
|
59 |
+
sample_output = chat.generate(inputs=input_ids, max_length=MAX_OUT_LEN)
|
60 |
+
|
61 |
+
```
|
62 |
+
## Demo output
|
63 |
+
|
64 |
+
### text2img
|
65 |
+
![text2img_demo](./output/text2img_demo.jpg)
|
66 |
+
|
67 |
+
### img2img
|
68 |
+
|
69 |
+
![text2img_demo](./output/img2img_input.jpg)
|
70 |
+
|
71 |
+
![text2img_demo](./output/img2img_demo.jpg)
|
72 |
+
|
73 |
+
|
74 |
+
|
75 |
+
## Environment
|
76 |
+
|
77 |
+
- hardware: Nvidia Ampere architecture (A100) or compatable
|
78 |
+
- docker image avaible: https://hub.docker.com/r/bigmoyan/lyra_aigc/tags
|
79 |
+
```
|
80 |
+
docker pull bigmoyan/lyra_aigc:v0.1
|
81 |
+
```
|
82 |
+
|
83 |
+
## Citation
|
84 |
+
``` bibtex
|
85 |
+
@Misc{lyraChatGLM2023,
|
86 |
+
author = {Kangjian Wu, Zhengtao Wang, Bin Wu},
|
87 |
+
title = {lyaraChatGLM: Accelerating chatGLM by 10x+},
|
88 |
+
howpublished = {\url{https://huggingface.co/TMElyralab/lyraChatGLM}},
|
89 |
+
year = {2023}
|
90 |
+
}
|
91 |
+
```
|
92 |
+
|
93 |
+
## Report bug
|
94 |
+
- start a discussion to report any bugs!--> https://huggingface.co/TMElyralab/lyraChatGLM/discussions
|
95 |
+
- report bug with a `[bug]` mark in the title.
|