Spaces:
Paused
Paused
makaveli10
commited on
Commit
•
95bcc6a
1
Parent(s):
5d7959c
update README for phi support
Browse files
README.md
CHANGED
@@ -50,6 +50,27 @@ python build.py --model_dir teknium/OpenHermes-2.5-Mistral-7B \
|
|
50 |
--max_batch_size 1
|
51 |
```
|
52 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
53 |
## Run WhisperBot
|
54 |
- Clone this repo and install requirements.
|
55 |
```bash
|
@@ -60,12 +81,24 @@ apt install ffmpeg portaudio19-dev -y
|
|
60 |
pip install -r requirements.txt
|
61 |
```
|
62 |
|
|
|
63 |
- Take the folder path for Whisper TensorRT model, folder_path and tokenizer_path for Mistral TensorRT from the build phase. If a huggingface model is used to build mistral then just use the huggingface repo name as the tokenizer path.
|
64 |
```bash
|
65 |
-
python3 main.py --
|
|
|
66 |
--mistral_tensorrt_path /root/TensorRT-LLM/examples/llama/tmp/mistral/7B/trt_engines/fp16/1-gpu/ \
|
67 |
--mistral_tokenizer_path teknium/OpenHermes-2.5-Mistral-7B
|
68 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
69 |
- On the client side clone the repo, install the requirements and execute `run_client.py`
|
70 |
```bash
|
71 |
cd WhisperBot
|
|
|
50 |
--max_batch_size 1
|
51 |
```
|
52 |
|
53 |
+
### Build Phi TensorRT Engine
|
54 |
+
Note: Phi is only available in main branch and hasnt been released yet. So, make sure to build TensorRT-LLM from main branch.
|
55 |
+
- Change working dir to [phi example dir](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/phi) in TensorRT-LLM folder.
|
56 |
+
```bash
|
57 |
+
cd TensorRT-LLM/examples/phi
|
58 |
+
```
|
59 |
+
- Build phi TensorRT engine
|
60 |
+
```bash
|
61 |
+
git lfs install
|
62 |
+
git clone https://huggingface.co/microsoft/phi-2
|
63 |
+
python3 build.py --dtype=float16 \
|
64 |
+
--log_level=verbose \
|
65 |
+
--use_gpt_attention_plugin float16 \
|
66 |
+
--use_gemm_plugin float16 \
|
67 |
+
--max_batch_size=16 \
|
68 |
+
--max_input_len=1024 \
|
69 |
+
--max_output_len=1024 \
|
70 |
+
--output_dir=phi_engine \
|
71 |
+
--model_dir=phi-2>&1 | tee build.log
|
72 |
+
```
|
73 |
+
|
74 |
## Run WhisperBot
|
75 |
- Clone this repo and install requirements.
|
76 |
```bash
|
|
|
81 |
pip install -r requirements.txt
|
82 |
```
|
83 |
|
84 |
+
### Whisper + Mistral
|
85 |
- Take the folder path for Whisper TensorRT model, folder_path and tokenizer_path for Mistral TensorRT from the build phase. If a huggingface model is used to build mistral then just use the huggingface repo name as the tokenizer path.
|
86 |
```bash
|
87 |
+
python3 main.py --mistral
|
88 |
+
--whisper_tensorrt_path /root/TensorRT-LLM/examples/whisper/whisper_small_en \
|
89 |
--mistral_tensorrt_path /root/TensorRT-LLM/examples/llama/tmp/mistral/7B/trt_engines/fp16/1-gpu/ \
|
90 |
--mistral_tokenizer_path teknium/OpenHermes-2.5-Mistral-7B
|
91 |
```
|
92 |
+
|
93 |
+
### Whisper + Phi
|
94 |
+
- Take the folder path for Whisper TensorRT model, folder_path and tokenizer_path for Phi TensorRT from the build phase. If a huggingface model is used to build phi then just use the huggingface repo name as the tokenizer path.
|
95 |
+
```bash
|
96 |
+
python3 main.py --phi
|
97 |
+
--whisper_tensorrt_path /root/TensorRT-LLM/examples/whisper/whisper_small_en \
|
98 |
+
--phi_tensorrt_path /root/TensorRT-LLM/examples/phi/phi_engine \
|
99 |
+
--phi_tokenizer_path /root/TensorRT-LLM/examples/phi/phi-2
|
100 |
+
```
|
101 |
+
|
102 |
- On the client side clone the repo, install the requirements and execute `run_client.py`
|
103 |
```bash
|
104 |
cd WhisperBot
|