Converted BAAI/bge-m3 model (dense retriever only) in onnx fp16/int8 format for use with Vespa Embedding.

  • BAAI-bge-m3_fp16.onnx (fp16)
  • BAAI-bge-m3_quantized.onnx (int8 quantized)

The model was quantized using the optimum toolkit.

Example of vespa services.xml:

Notice: FP16 works well with Vespa versions 8.325.46 and above.

<component id="bge_m3" type="hugging-face-embedder">
    <transformer-model
        url="https://huggingface.co/hotchpotch/vespa-onnx-BAAI-bge-m3-only-dense/resolve/main/BAAI-bge-m3_fp16.onnx" />
    <!-- or int8 quantization model
    <transformer-model
    url="https://huggingface.co/hotchpotch/vespa-onnx-BAAI-bge-m3-only-dense/resolve/main/BAAI-bge-m3_quantized.onnx"
    />
    -->
    <tokenizer-model
        url="https://huggingface.co/hotchpotch/vespa-onnx-BAAI-bge-m3-only-dense/resolve/main/tokenizer.json" />
    <normalize>true</normalize>
    <pooling-strategy>cls</pooling-strategy>
</component>

deploy

# FP16 model has a larger file size, which can result in longer deployment times.
vespa deploy --wait 1800 .

Tips: conver to int8 quantized

# https://github.com/vespa-engine/sample-apps/blob/master/simple-semantic-search/export_hf_model_from_hf.py
./export_hf_model_from_hf.py --hf_model BAAI/bge-m3 --output_dir bge-m3
optimum-cli onnxruntime quantize --onnx_model ./bge-m3  -o bge-m3-large_quantized --avx512_vnni

Tips: convert to fp16

# https://github.com/vespa-engine/sample-apps/blob/master/simple-semantic-search/export_hf_model_from_hf.py
./export_hf_model_from_hf.py --hf_model BAAI/bge-m3 --output_dir bge-m3
# https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/float16.py

import onnx
from onnxruntime.transformers.float16 import convert_float_to_float16

onnx_model = onnx.load("bge-m3/BAAI-bge-m3.onnx")
model_fp16 = convert_float_to_float16(onnx_model, disable_shape_infer=True)
onnx.save(model_fp16, "bge-m3/BAAI-bge-m3_fp16.onnx")

License

The license for this model is based on the original license (found in the LICENSE file in the project's root directory), which is the MIT License.

Attribution

All credits for this model go to the authors of BAAI/bge-m3 and the associated researchers and organizations. When using this model, please be sure to attribute the original authors.

Downloads last month
18
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.