---
license: mit
library_name: open_clip
pipeline_tag: zero-shot-image-classification
---

CoreML versions of [laion/CLIP-ViT-H-14-laion2B-s32B-b79K](/laion/CLIP-ViT-H-14-laion2B-s32B-b79K). 

On my baseline M1 they run about 4x faster than the equivalent pytorch models run on the `mps` device (~6 image embeddings per second vs 1.5 images/sec for torch+mps), and according to `asitop` profiling, using about 3/4 of the energy to do so (6W average vs 8W for torch+mps).

There are separate models for the image and text encoders. Sorry, I don't know how to put them both into one file. 

Conversion code is in [clip-to-coreml.ipynb](clip-to-coreml.ipynb).

# Usage

You'll need to use the original CLIP preprocessor (or write your own preprocessing). eg:

```
from transformers import CLIPProcessor
import coremltools as ct
from PIL import Image

preprocessor = CLIPProcessor.from_pretrained("laion/CLIP-ViT-H-14-laion2B-s32B-b79K")

model_coreml_image = ct.models.MLModel('CLIP-ViT-H-14-laion2B-s32B-b79K.image-encoder.mlprogram')
model_coreml_text = ct.models.MLModel('CLIP-ViT-H-14-laion2B-s32B-b79K.text-encoder.mlprogram')

image = Image.open("example.jpg")
preprocessed_image = preprocessor(text=None, images=image, return_tensors="pt", padding=True)
image_embedding = model_coreml.predict({'input_image_preprocessed': preprocessed_image.pixel_values})['output_embedding']

text = 'example text'
preprocessed_text = preprocessor(text=text, images=None, return_tensors="pt", padding=True)
text_embedding = model_coreml_text.predict({'input_text_token_ids': preprocessed_text.input_ids})['output_embedding'])
```

Please credit me if you use this.

---
license: mit
---