File size: 3,168 Bytes
c2b26cc 9ca46b6 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 |
---
base_model:
- Qwen/Qwen2-VL-2B
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- rknn
- rkllm
- chat
- vision
- rk3588
- multimodal
---
## 3ib0n's RKLLM Guide
These models and binaries require an RK3588 board running rknpu driver version 0.9.7 or above
## Steps to reproduce conversion
```shell
# Download and setup miniforge3
curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
bash Miniforge3-$(uname)-$(uname -m).sh
# activate the base environment
source ~/miniforge3/bin/activate
# create and activate a python 3.8 environment
conda create -n rknn-llm-1.1.4 python=3.8
conda activate rknn-llm-1.1.4
# clone the lastest rknn-llm toolkit
git clone https://github.com/airockchip/rknn-llm.git
# update the following 4 files to your desired models and output locations
cd rknn-llm/examples/rkllm_multimodal_demo
nano export/export_vision.py # update model path and output path
nano export/export_vision_rknn.py # update model path
nano export/export_rkllm.py # update input and output paths
nano data/make_input_embeds_for_quantize.py # update model path
# intstall necessary dependencies for the above
pip install transformers accelerate torchvision rknn-toolkit2==2.2.1
pip install --upgrade torch pillow # necessary to use vision models with opset_version=18
# export vision models and create input embeddings
cd export/
python export_vision.py
python export_vision_rknn.py
cd ..
python data/make_input_embeds_for_quantize.py
# install rkllm and export the language model
pip install ../../rkllm-toolkit/packages/rkllm_toolkit-1.1.4-cp38-cp38-linux_x86_64.whl
python export/export_rkllm.py
```
## Steps to build and run demo
```shell
# Dwonload the correct toolchain for working with rkllm
# Documentation here: https://github.com/airockchip/rknn-llm/blob/main/doc/Rockchip_RKLLM_SDK_EN_1.1.0.pdf
wget https://developer.arm.com/-/media/Files/downloads/gnu-a/10.2-2020.11/binrel/gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz
tar -xz gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz
# ensure that the gcc compiler path is set to the location where the toolchain dowloaded earlier is unpacked
nano deploy/build-linux.sh # update the gcc compiler path
# compile the demo app
cd delpoy/
./build-linux.sh
```
## Steps to run the app
More information and original guide: https://github.com/airockchip/rknn-llm/tree/main/examples/rkllm_multimodel_demo
```shell
# push install dir to device
adb push ./install/demo_Linux_aarch64 /data
# push model file to device
adb push qwen2_vl_2b_vision_rk3588.rknn /data/models
adb push Qwen2-VL-2B-Instruct.rkllm /data/models
# push demo image to device
adb push ../data/demo.jpg /data/demo_Linux_aarch64
adb shell
cd /data/demo_Linux_aarch64
# export lib path
export LD_LIBRARY_PATH=./lib
# soft link models dir
ln -s /data/models .
# run imgenc
./imgenc models/qwen2_vl_2b_vision_rk3588.rknn demo.jpg
# run llm(Pure Text Example)
./llm models/Qwen2-VL-2B-Instruct.rkllm 128 512
# run demo(Multimodal Example)
./demo demo.jpg models/qwen2_vl_2b_vision_rk3588.rknn models/Qwen2-VL-2B-Instruct.rkllm 128 512
``` |