File size: 2,730 Bytes
ab28109
 
3d571f2
 
 
 
 
 
 
ab28109
3d571f2
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
c5ba562
3d571f2
 
 
 
 
 
0a543d3
3d571f2
 
 
0a543d3
 
3d571f2
 
 
0a543d3
3d571f2
 
0a543d3
 
3d571f2
 
 
b09aca2
3d571f2
0359bc0
 
3d571f2
 
 
 
 
 
 
 
 
 
 
0359bc0
3d571f2
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
---
license: apache-2.0
pipeline_tag: feature-extraction
tags:
- clip
- vision
datasets:
- Ziyang/yfcc15m
- conceptual_captions
---
<h1 align="center">UForm</h1>
<h3 align="center">
Multi-Modal Inference Library<br/>
For Semantic Search Applications<br/>
</h3>

---

UForm is a Multi-Modal Modal Inference package, designed to encode Multi-Lingual Texts, Images, and, soon, Audio, Video, and Documents, into a shared vector space!

This is model card of the __English only model__ with:

* 12 layers BERT (6 layers for unimodal encoding and rest layers for multimodal encoding)
* ViT-L/14 (image resolution is 224x224)
* Multiple embedding sizes: 64, 256, 512, 768


If you need Multilingual model, check [this](https://huggingface.co/unum-cloud/uform-vl-multilingual).

## Evaluation

The following metrics were obtained with multimodal re-ranking (text-to-image retrieval):

| Dataset   |Recall@1 |  Recall@5 | Recall@10 |
| :------   | ------: | --------: | --------: |
| Zero-Shot Flickr    | 0.693 | 0.875 | 0.923 |
| Zero-Shot MS-COCO   | 0.382 | 0.617 | 0.728 |

ImageNet-Top1: 0.518 \
ImageNet-Top5: 0.756

## Installation

```bash
pip install uform[onnx-gpu]
```

## Usage

To load the model:

```python
import uform
model, processor = uform.get_model_onnx('unum-cloud/uform-vl-english-large', device='gpu', dtype='fp32')
```

To encode data:

```python
from PIL import Image

text = 'a small red panda in a zoo'
image = Image.open('red_panda.jpg')

image_data = processor.preprocess_image(image)
text_data = processor.preprocess_text(text)

image_features, image_embedding = model.encode_image(image_data, return_features=True)
text_features, text_embedding = model.encode_text(text_data, return_features=True)
score, joint_embedding = model.encode_multimodal(
    image_features=image_features,
    text_features=text_features,
    attention_mask=text_data['attention_mask'],
    return_scores=True
)
```

There are two options to calculate semantic compatibility between an image and a text: cosine similarity and [Matching Score](#matching-score).

### Cosine Similarity

__Pros__:

- Computationally cheap.
- Only unimodal embeddings are required, unimodal encoding is faster than joint encoding.
- Suitable for retrieval in large collections.

__Cons__:

- Takes into account only coarse-grained features.


### Matching Score

Unlike cosine similarity, unimodal embedding are not enough.
Joint embedding will be needed and the resulting `score` will belong to the `[0, 1]` range, `1` meaning the absolute match.

__Pros__:

- Joint embedding captures fine-grained features.
- Suitable for re-ranking – sorting retrieval result.

__Cons__:

- Resource-intensive.
- Not suitable for retrieval in large collections.