README.md

---
license: agpl-3.0
language:
- en
base_model: microsoft/Florence-2-base-ft
tags:
- rknn
---

# Florence-2-base-ft-ONNX-RKNN2

### (English README see below)

ONNX/RKNN2部署Florence-2视觉多模态大模型! 

- 推理速度(RKNN2)：RK3588推理一张768x768图片, 使用`<MORE_DETAILED_CAPTION>`指令, 总时间需要~4.5秒
- 内存占用(RKNN2)：约2GB

## 使用方法

1. 克隆项目到本地

2. 安装依赖

```bash
pip install transformers onnxruntime pillow numpy<2
```

如果需要使用rknn推理, 还需要安装rknn-toolkit2-lite2. 

3. 修改项目路径
分词器和预处理配置仍然需要使用原项目中的文件. 将onnx/onnxrun.py或onnx/rknnrun.py中的对应路径修改为项目所在路径. 
```python
AutoProcessor.from_pretrained(
    "path/to/Florence-2-base-ft-ONNX-RKNN2",
    trust_remote_code=True
)
```

4. 运行
```bash
python onnx/onnxrun.py # 或 python onnx/rknnrun.py
```

## RKNN模型转换

你需要提前安装rknn-toolkit2 v2.1.0或更高版本. 

```bash
cd onnx
python convert.py all
```

注意: RKNN模型不支持运行时任意长度的输入, 所以需要提前确定好输入的shape. 之后可以修改convert.py中的`vision_size`, `vision_tokens`, `prompt_tokens`来修改输入的shape. 

## 存在的问题(rknn)
- ~~已知转换vision encoder时输入分辨率>=640x640时会报错`Buffer overflow!`然后转换失败, 所以目前是使用512x512的输入, 推理质量会下降~~ (已解决)
- ~~在同分辨率下推理精度相比onnxruntime有显著下降, 已确定问题出在vision encoder部分. 如果对精度要求高, 这部分可以换成onnxruntime~~ (已解决, 不过现在貌似仍然会丢失一点精度)
- ~~vision encoder推理需要1.5秒, 占比最大, 但其中大半时间都在做Transpose, 也许还可以优化~~ (已优化)
- decode阶段因为kvcache的长度不断变化, 貌似无法简单的使用NPU推理. 不过用onnxruntime应该也足够了

## 参考
- [microsoft/Florence-2-base-ft](https://huggingface.co/microsoft/Florence-2-base-ft)
- [onnx-community/Florence-2-base-ft](https://huggingface.co/onnx-community/Florence-2-base-ft)
- [florence2-webgpu](https://huggingface.co/spaces/Xenova/florence2-webgpu)


# English README

# Florence-2-base-ft-ONNX-RKNN2

ONNX/RKNN2 deployment for Florence-2 visual-language multimodal large model!

- Inference speed (RKNN2): RK3588 inference with a 768x768 image, using the `<MORE_DETAILED_CAPTION>` instruction, takes ~4.5 seconds in total.
- Memory usage (RKNN2): Approximately 2GB

## Usage

1. Clone the project locally

2. Install dependencies

```bash
pip install transformers onnxruntime pillow numpy<2
```

If you need to use rknn for inference, you also need to install rknn-toolkit2-lite2.

3. Modify project paths
The tokenizer and preprocessing configurations still need to use files from the original project. Modify the corresponding paths in onnx/onnxrun.py or onnx/rknnrun.py to the project's location.
```python
AutoProcessor.from_pretrained(
    "path/to/Florence-2-base-ft-ONNX-RKNN2",
    trust_remote_code=True
)
```

4. Run
```bash
python onnx/onnxrun.py # or python onnx/rknnrun.py
```

## RKNN Model Conversion

You need to install rknn-toolkit2 v2.1.0 or higher in advance.

```bash
cd onnx
python convert.py all
```

Note: The RKNN model does not support arbitrary input lengths at runtime, so you need to determine the input shape in advance. You can modify `vision_size`, `vision_tokens`, and `prompt_tokens` in convert.py to change the input shape.

## Existing Issues (rknn)

- ~~It is known that converting the vision encoder with an input resolution >= 640x640 will result in a `Buffer overflow!` error and conversion failure. Therefore, the current input is 512x512, which will reduce inference quality.~~ (Solved.)
- ~~Inference accuracy is significantly lower compared to onnxruntime at the same resolution. The problem has been identified in the vision encoder part. If high accuracy is required, this part can be replaced with onnxruntime.~~ (Solved, although there is still some precision loss present.)
- ~~Vision encoder inference takes 1.5 seconds, accounting for the largest proportion, but most of this time is spent on Transpose op, which may be further optimized.~~ (Solved.)
- The decode phase seems unable to simply use NPU inference because the length of kvcache keeps changing. However, using onnxruntime should be sufficient.

## References

- [microsoft/Florence-2-base-ft](https://huggingface.co/microsoft/Florence-2-base-ft)
- [onnx-community/Florence-2-base-ft](https://huggingface.co/onnx-community/Florence-2-base-ft)
- [florence2-webgpu](https://huggingface.co/spaces/Xenova/florence2-webgpu)