--- license: apache-2.0 pipeline_tag: feature-extraction tags: - clip - vision datasets: - Ziyang/yfcc15m - conceptual_captions ---

UForm

Multi-Modal Inference Library
For Semantic Search Applications

--- UForm is a Multi-Modal Modal Inference package, designed to encode Multi-Lingual Texts, Images, and, soon, Audio, Video, and Documents, into a shared vector space! This is model card of the __English only model__ with: * 4 layers BERT (2 layers for unimodal encoding and rest layers for multimodal encoding) * ViT-S/16 (image resolution is 224x224) If you need Multilingual model, check [this](https://huggingface.co/unum-cloud/uform-vl-multilingual). ## Evaluation The following metrics were obtained with multimodal re-ranking (text-to-image retrieval): | Dataset |Recall@1 | Recall@5 | Recall@10 | | :------ | ------: | --------: | --------: | | Zero-Shot Flickr | 0.565 | 0.790 | 0.860 | | Zero-Shot MS-COCO | 0.281 | 0.525 | 0.645 | ImageNet-Top1: 0.361 \ ImageNet-Top5: 0.608 ## Installation ```bash pip install uform[onnx-gpu] ``` ## Usage To load the model: ```python import uform model, processor = uform.get_model_onnx('unum-cloud/uform-vl-english-small', device='gpu', dtype='fp16') ``` To encode data: ```python from PIL import Image text = 'a small red panda in a zoo' image = Image.open('red_panda.jpg') image_data = processor.preprocess_image(image) text_data = processor.preprocess_text(text) image_features, image_embedding = model.encode_image(image_data, return_features=True) text_features, text_embedding = model.encode_text(text_data, return_features=True) score, joint_embedding = model.encode_multimodal( image_features=image_features, text_features=text_features, attention_mask=text_data['attention_mask'], return_scores=True ) ``` There are two options to calculate semantic compatibility between an image and a text: cosine similarity and [Matching Score](#matching-score). ### Cosine Similarity The `similarity` will belong to the `[-1, 1]` range, `1` meaning the absolute match. __Pros__: - Computationally cheap. - Only unimodal embeddings are required, unimodal encoding is faster than joint encoding. - Suitable for retrieval in large collections. __Cons__: - Takes into account only coarse-grained features. ### Matching Score Unlike cosine similarity, unimodal embedding are not enough. Joint embedding will be needed and the resulting `score` will belong to the `[0, 1]` range, `1` meaning the absolute match. __Pros__: - Joint embedding captures fine-grained features. - Suitable for re-ranking – sorting retrieval result. __Cons__: - Resource-intensive. - Not suitable for retrieval in large collections.