kimihailv commited on
Commit
637710b
1 Parent(s): f3767e2

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +134 -0
README.md CHANGED
@@ -1,3 +1,137 @@
1
  ---
2
  license: apache-2.0
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ pipeline_tag: feature-extraction
4
+ tags:
5
+ - clip
6
+ - vision
7
+ datasets:
8
+ - Ziyang/yfcc15m
9
+ - conceptual_captions
10
  ---
11
+ <h1 align="center">UForm</h1>
12
+ <h3 align="center">
13
+ Multi-Modal Inference Library<br/>
14
+ For Semantic Search Applications<br/>
15
+ </h3>
16
+
17
+ ---
18
+
19
+ UForm is a Multi-Modal Modal Inference package, designed to encode Multi-Lingual Texts, Images, and, soon, Audio, Video, and Documents, into a shared vector space!
20
+
21
+ This is model card of the __English only model__ with:
22
+
23
+ * 12 layers BERT (6 layers for unimodal encoding and rest layers for multimodal encoding)
24
+ * ViT-L/14 (image resolution is 224x224)
25
+ * Multiple embedding sizes: 64, 256, 512, 768
26
+
27
+
28
+ If you need Multilingual model, check [this](https://huggingface.co/unum-cloud/uform-vl-multilingual).
29
+
30
+ ## Evaluation
31
+
32
+ The following metrics were obtained with multimodal re-ranking (text-to-image retrieval):
33
+
34
+ | Dataset |Recall@1 | Recall@5 | Recall@10 |
35
+ | :------ | ------: | --------: | --------: |
36
+ | Zero-Shot Flickr | 0.693 | 0.875 | 0.923 |
37
+ | Zero-Shot MS-COCO | 0.382 | 0.617 | 0.728 |
38
+
39
+ ImageNet-Top1: 0.518 \
40
+ ImageNet-Top5: 0.756
41
+
42
+ ## Installation
43
+
44
+ ```bash
45
+ pip install uform[onnx] # for cpu
46
+ pip install uform[onnx-gpu] # for gpu
47
+ ```
48
+
49
+ ## Usage
50
+
51
+ To load the model:
52
+
53
+ ```python
54
+ import uform
55
+
56
+ model = uform.get_model_onnx('unum-cloud/uform-vl-english-large', device='cpu', dtype='fp32')
57
+ ```
58
+
59
+ To encode data:
60
+
61
+ ```python
62
+ from PIL import Image
63
+
64
+ text = 'a small red panda in a zoo'
65
+ image = Image.open('red_panda.jpg')
66
+
67
+ image_data = model.preprocess_image(image)
68
+ text_data = model.preprocess_text(text)
69
+
70
+ image_embedding = model.encode_image(image_data)
71
+ text_embedding = model.encode_text(text_data)
72
+ score, joint_embedding = model.encode_multimodal(
73
+ image_features=image_features,
74
+ text_features=text_features,
75
+ attention_mask=text_data['attention_mask'],
76
+ return_scores=True
77
+ )
78
+ ```
79
+
80
+ To get features:
81
+
82
+ ```python
83
+ image_features, image_embedding = model.encode_image(image_data, return_features=True)
84
+ text_features, text_embedding = model.encode_text(text_data, return_features=True)
85
+ ```
86
+
87
+ These features can later be used to produce joint multimodal encodings faster, as the first layers of the transformer can be skipped:
88
+
89
+ ```python
90
+ joint_embedding = model.encode_multimodal(
91
+ image_features=image_features,
92
+ text_features=text_features,
93
+ attention_mask=text_data['attention_mask']
94
+ )
95
+ ```
96
+
97
+ There are two options to calculate semantic compatibility between an image and a text: [Cosine Similarity](#cosine-similarity) and [Matching Score](#matching-score).
98
+
99
+ ### Cosine Similarity
100
+
101
+ ```python
102
+ import torch.nn.functional as F
103
+
104
+ similarity = F.cosine_similarity(image_embedding, text_embedding)
105
+ ```
106
+
107
+ The `similarity` will belong to the `[-1, 1]` range, `1` meaning the absolute match.
108
+
109
+ __Pros__:
110
+
111
+ - Computationally cheap.
112
+ - Only unimodal embeddings are required, unimodal encoding is faster than joint encoding.
113
+ - Suitable for retrieval in large collections.
114
+
115
+ __Cons__:
116
+
117
+ - Takes into account only coarse-grained features.
118
+
119
+
120
+ ### Matching Score
121
+
122
+ Unlike cosine similarity, unimodal embedding are not enough.
123
+ Joint embedding will be needed and the resulting `score` will belong to the `[0, 1]` range, `1` meaning the absolute match.
124
+
125
+ ```python
126
+ score = model.get_matching_scores(joint_embedding)
127
+ ```
128
+
129
+ __Pros__:
130
+
131
+ - Joint embedding captures fine-grained features.
132
+ - Suitable for re-ranking – sorting retrieval result.
133
+
134
+ __Cons__:
135
+
136
+ - Resource-intensive.
137
+ - Not suitable for retrieval in large collections.