Hugging Face's logo

jinaai
/
clip-models

ONNX
🇪🇺 Region: EU
Model card Files Community
selmisskilig commited on
Commit
7b28699
•
1 Parent(s): 46d5201

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +112 -36
README.md CHANGED
@@ -1,39 +1,115 @@
1
  ---
2
  license: apache-2.0
3
  ---
4
- # Clip Models for clip-as-service
5
-
6
-
7
- | Model | PyTorch | ONNX | TensorRT | Output Dimension |
8
- | ------------------------------------- | ------- | ---- | -------- | ---------------- |
9
- | RN50::openai | ✅ | ✅ | ✅ | 1024 |
10
- | RN50::yfcc15m | ✅ | ✅ | ✅ | 1024 |
11
- | RN50::cc12m | ✅ | ✅ | ✅ | 1024 |
12
- | RN101::openai | ✅ | ✅ | ✅ | 512 |
13
- | RN101::yfcc15m | ✅ | ✅ | ✅ | 512 |
14
- | RN50x4::openai | ✅ | ✅ | ✅ | 640 |
15
- | RN50x16::openai | ✅ | ✅ | ❌ | 768 |
16
- | RN50x64::openai | ✅ | ✅ | ❌ | 1024 |
17
- | ViT-B-32::openai | ✅ | ✅ | ✅ | 512 |
18
- | ViT-B-32::laion2b_e16 | ✅ | ✅ | ✅ | 512 |
19
- | ViT-B-32::laion400m_e31 | ✅ | ✅ | ✅ | 512 |
20
- | ViT-B-32::laion400m_e32 | ✅ | ✅ | ✅ | 512 |
21
- | ViT-B-32::laion2b-s34b-b79k | ✅ | ✅ | ❌ | 512 |
22
- | ViT-B-16::openai | ✅ | ✅ | ✅ | 512 |
23
- | ViT-B-16::laion400m_e31 | ✅ | ✅ | ✅ | 512 |
24
- | ViT-B-16::laion400m_e32 | ✅ | ✅ | ✅ | 512 |
25
- | ViT-B-16-plus-240::laion400m_e31 | ✅ | ✅ | 🚧 | 640 |
26
- | ViT-B-16-plus-240::laion400m_e32 | ✅ | ✅ | 🚧 | 640 |
27
- | ViT-L-14::openai | ✅ | ✅ | ❌ | 768 |
28
- | ViT-L-14::laion400m_e31 | ✅ | ✅ | ❌ | 768 |
29
- | ViT-L-14::laion400m_e32 | ✅ | ✅ | ❌ | 768 |
30
- | ViT-L-14::laion2b-s32b-b82k | ✅ | ✅ | ❌ | 768 |
31
- | ViT-L-14-336::openai | ✅ | ✅ | ❌ | 768 |
32
- | ViT-H-14::laion2b-s32b-b79k | ✅ | ✅ | ❌ | 1024 |
33
- | ViT-g-14::laion2b-s12b-b42k | ✅ | ✅ | ❌ | 1024 |
34
- | M-CLIP/LABSE-Vit-L-14 | ✅ | ✅ | ❌ | 768 |
35
- | M-CLIP/XLM-Roberta-Large-Vit-B-32 | ✅ | ✅ | 🚧 | 512 |
36
- | M-CLIP/XLM-Roberta-Large-Vit-B-16Plus | ✅ | ✅ | 🚧 | 640 |
37
- | M-CLIP/XLM-Roberta-Large-Vit-L-14 | ✅ | ✅ | ❌ | 768 |
38
-
39
- ✅ = Supported — 🚧 = Working in progress — ❌ = Not supported
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
  ---
4
+ # Welcome to CLIP-as-service!
5
+
6
+ CLIP-as-service is a low-latency high-scalability service for embedding images and text. It can be easily integrated as a microservice into neural search solutions.
7
+
8
+ ⚡ Fast: Serve CLIP models with TensorRT, ONNX runtime and PyTorch w/o JIT with 800QPS[*]. Non-blocking duplex streaming on requests and responses, designed for large data and long-running tasks.
9
+
10
+ 🫐 Elastic: Horizontally scale up and down multiple CLIP models on single GPU, with automatic load balancing.
11
+
12
+ 🐥 Easy-to-use: No learning curve, minimalist design on client and server. Intuitive and consistent API for image and sentence embedding.
13
+
14
+ 👒 Modern: Async client support. Easily switch between gRPC, HTTP, WebSocket protocols with TLS and compression.
15
+
16
+ 🍱 Integration: Smooth integration with neural search ecosystem including Jina and DocArray. Build cross-modal and multi-modal solutions in no time.
17
+
18
+ [*] with default config (single replica, PyTorch no JIT) on GeForce RTX 3090.
19
+
20
+ ## Try it!
21
+
22
+ ## Install
23
+
24
+ [PyPI](https://img.shields.io/pypi/v/clip_client?color=%23ffffff&label=%20) is the latest version.
25
+
26
+ Make sure you are using Python 3.7+. You can install the client and server independently. It is **not required** to install both: e.g. you can install `clip_server` on a GPU machine and `clip_client` on a local laptop.
27
+
28
+ Client
29
+
30
+ ```bash
31
+ pip install clip-client
32
+ ```
33
+
34
+ Server (PyTorch)
35
+
36
+ ```
37
+ pip install clip-server
38
+ ```
39
+ Server (ONNX)
40
+
41
+ ```
42
+ pip install "clip_server[onnx]"
43
+ ```
44
+ Server (TensorRT)
45
+
46
+ ```
47
+ pip install nvidia-pyindex
48
+ pip install "clip_server[tensorrt]"
49
+ ```
50
+ Server on [Google Colab](https://colab.research.google.com/github/jina-ai/clip-as-service/blob/main/docs/hosting/cas-on-colab.ipynb)
51
+
52
+
53
+
54
+ ## Quick check
55
+
56
+ After installing, you can run the following commands for a quick connectivity check.
57
+
58
+ ### Start the server
59
+
60
+ Start PyTorch Server
61
+ ```bash
62
+ python -m clip_server
63
+ ```
64
+ Start ONNX Server
65
+ ```bash
66
+ python -m clip_server onnx-flow.yml
67
+ ```
68
+ Start TensorRT Server
69
+ ```bash
70
+ python -m clip_server tensorrt-flow.yml
71
+ ```
72
+
73
+ At the first time starting the server, it will download the default pretrained model, which may take a while depending on your network speed. Then you will get the address information similar to the following:
74
+
75
+ ```text
76
+ ╭────────────── 🔗 Endpoint ───────────────╮
77
+ │ 🔗 Protocol GRPC │
78
+ │ 🏠 Local 0.0.0.0:51000 │
79
+ │ 🔒 Private 192.168.31.62:51000 │
80
+ | 🌍 Public 87.105.159.191:51000 |
81
+ ╰──────────────────────────────────────────╯
82
+ ```
83
+
84
+ This means the server is ready to serve. Note down the three addresses shown above, you will need them later.
85
+
86
+ ### Connect from client
87
+
88
+ ```{tip}
89
+ Depending on the location of the client and server. You may use different IP addresses:
90
+ - Client and server are on the same machine: use local address, e.g. `0.0.0.0`
91
+ - Client and server are connected to the same router: use private network address, e.g. `192.168.3.62`
92
+ - Server is in public network: use public network address, e.g. `87.105.159.191`
93
+ ```
94
+
95
+ Run the following Python script:
96
+
97
+ ```python
98
+ from clip_client import Client
99
+
100
+ c = Client('grpc://0.0.0.0:51000')
101
+ c.profile()
102
+ ```
103
+
104
+ will give you:
105
+
106
+ ```text
107
+ Roundtrip 16ms 100%
108
+ ├── Client-server network 8ms 49%
109
+ └── Server 8ms 51%
110
+ ├── Gateway-CLIP network 2ms 25%
111
+ └── CLIP model 6ms 75%
112
+ {'Roundtrip': 15.684750003856607, 'Client-server network': 7.684750003856607, 'Server': 8, 'Gateway-CLIP network': 2, 'CLIP model': 6}
113
+ ```
114
+
115
+ It means the client and the server are now connected. Well done!