Converted INT8/INT4 files for fastllm with baichuan2-13b-chat
Directly download from Baidu Netdisk:
Link：https://pan.baidu.com/s/1Xsiif_1VzDyWFei1u5oJcA Code：wxbo
Updated time: 2023/09/11
baichuan2-13b-chat-int8.flm:
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:05:00.0 Off | N/A |
| 31% 36C P8 28W / 250W | 15420MiB / 22528MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
from fastllm_pytools import llm
model = llm.model("baichuan2-13b-chat-int8.flm")
for response in model.stream_response("介绍一下南京"):
print(response, flush = True, end = "")
Note: please use the lastest version of FastLLM (no later than 2023/09/11 main branch)