bigmoyan commited on
Commit
d1850c3
1 Parent(s): f268989

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +95 -0
README.md ADDED
@@ -0,0 +1,95 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: creativeml-openrail-m
3
+ language:
4
+ - en
5
+ tags:
6
+ - LLM
7
+ - tensorRT
8
+ - chatGLM
9
+ ---
10
+ ## Model Card for lyraChatGLM
11
+
12
+ lyraChatGLM is currently the **fastest chatGLM-6B** available, as far as we know, it is also the **fisrt accelerated version of chatGLM-6B**.
13
+
14
+ The inference speed of lyraChatGLM is **10x** faster than the original version, and we're still working to improve the performance.
15
+
16
+ Among its main features are:
17
+
18
+ - weights: original ChatGLM-6B weights released by THUDM.
19
+ - device: lyraChatGLM is mainly based on FasterTransformer compiled for SM=80 (A100, for example).
20
+
21
+ ## Speed
22
+
23
+ ### test environment
24
+
25
+ - device: Nvidia A100 40G
26
+ - img size: 512x512
27
+ - percision:fp16
28
+ - steps: 30
29
+ - solver: LMSD
30
+
31
+ ### text2img
32
+
33
+
34
+
35
+
36
+ ## Model Sources
37
+
38
+ - **Repository:** [https://huggingface.co/THUDM/chatglm-6b]
39
+
40
+ ## Uses
41
+
42
+ ```python
43
+ from faster_chat_glm import GLM6B, FasterChatGLM
44
+
45
+ # kernel for chat model.
46
+ kernel = GLM6B(plan_path=plan_path,
47
+ batch_size=BATCH_SIZE,
48
+ num_beams=1,
49
+ use_cache=USE_CACHE,
50
+ num_heads=32,
51
+ emb_size_per_heads=128,
52
+ decoder_layers=28,
53
+ vocab_size=150528,
54
+ max_seq_len=MAX_OUT_LEN)
55
+
56
+ chat = FasterChatGLM(model_dir=chatglm6b_dir, kernel=kernel).half().cuda()
57
+
58
+ # generate
59
+ sample_output = chat.generate(inputs=input_ids, max_length=MAX_OUT_LEN)
60
+
61
+ ```
62
+ ## Demo output
63
+
64
+ ### text2img
65
+ ![text2img_demo](./output/text2img_demo.jpg)
66
+
67
+ ### img2img
68
+
69
+ ![text2img_demo](./output/img2img_input.jpg)
70
+
71
+ ![text2img_demo](./output/img2img_demo.jpg)
72
+
73
+
74
+
75
+ ## Environment
76
+
77
+ - hardware: Nvidia Ampere architecture (A100) or compatable
78
+ - docker image avaible: https://hub.docker.com/r/bigmoyan/lyra_aigc/tags
79
+ ```
80
+ docker pull bigmoyan/lyra_aigc:v0.1
81
+ ```
82
+
83
+ ## Citation
84
+ ``` bibtex
85
+ @Misc{lyraChatGLM2023,
86
+ author = {Kangjian Wu, Zhengtao Wang, Bin Wu},
87
+ title = {lyaraChatGLM: Accelerating chatGLM by 10x+},
88
+ howpublished = {\url{https://huggingface.co/TMElyralab/lyraChatGLM}},
89
+ year = {2023}
90
+ }
91
+ ```
92
+
93
+ ## Report bug
94
+ - start a discussion to report any bugs!--> https://huggingface.co/TMElyralab/lyraChatGLM/discussions
95
+ - report bug with a `[bug]` mark in the title.