---
license: apache-2.0
---
# Marqo E-commerce Embedding Models
In this work, we introduce two state-of-the-art embedding models for e-commerce: 
Marqo-Ecommerce-B and Marqo-Ecommerce-L. 
They are over 30% better compared to Amazon Titan Embedding services for e-commerce retrieval tasks.

**Released Content**: 
1) Marqo-Ecommerce-B and Marqo-Ecommerce-L embedding models
2) GoogleShopping-1m and AmazonProducts-3m for evaluation
3) Evaluation Code

<img src="performance.png" alt="multi split visual" width="500"/>

## Models

| **Embedding Model** | **#Params (m)** | **Dimension** | **HuggingFace**                    | **Download .pt**                                                                                            |
|---------------------| --- |---------------|------------------------------------|-------------------------------------------------------------------------------------------------------------|
| Marqo-Ecommerce-B   | 203 | 768           | Marqo/marqo-ecommerce-embeddings-B | [link](https://marqo-gcl-public.s3.us-west-2.amazonaws.com/marqo-general-ecomm/marqo-ecomm-embeddings-b.pt) |
| Marqo-Ecommerce-L   | 652 | 1024          | Marqo/marqo-ecommerce-embeddings-L | [link](https://marqo-gcl-public.s3.us-west-2.amazonaws.com/marqo-general-ecomm/marqo-ecomm-embeddings-l.pt)                                       |

### HuggingFace with OpenCLIP
```
pip install open_clip_torch
```
```python
from PIL import Image
import open_clip
import requests
import torch

# Specify model from Hugging Face Hub
model_name = 'hf-hub:Marqo/marqo-ecommerce-embeddings-B'
model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms(model_name)
tokenizer = open_clip.get_tokenizer(model_name)

# Preprocess the image and tokenize text inputs
# Load an example image from a URL
img = Image.open(requests.get('https://raw.githubusercontent.com/marqo-ai/marqo-FashionCLIP/main/docs/fashion-hippo.png', stream=True).raw)
image = preprocess_val(img).unsqueeze(0)
text = tokenizer(["a hat", "a t-shirt", "shoes"])

# Perform inference
with torch.no_grad(), torch.cuda.amp.autocast():
    image_features = model.encode_image(image, normalize=True)
    text_features = model.encode_text(text, normalize=True)

    # Calculate similarity probabilities
    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

# Display the label probabilities
print("Label probs:", text_probs)
# [9.9955e-01, 4.4712e-04, 4.4010e-06]]
```
### HuggingFace with transformers
```python
from transformers import AutoModel, AutoProcessor
import torch
from PIL import Image
import requests
# model_name= 'Marqo/marqo-ecommerce-embeddings-L'
model_name = 'Marqo/marqo-ecommerce-embeddings-B'

model_1 = AutoModel.from_pretrained(model_name, trust_remote_code=True)
processor_1 = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)

img = Image.open(requests.get('https://raw.githubusercontent.com/marqo-ai/marqo-FashionCLIP/main/docs/fashion-hippo.png', stream=True).raw).convert("RGB")
image_1 = [img]
text_1 = ["a hat", "a t-shirt", "shoes"]
processed_1 = processor_1(text=text_1, images=image_1, padding='max_length', return_tensors="pt")
processor_1.image_processor.do_rescale = False
with torch.no_grad():
    image_features_1 = model_1.get_image_features(processed_1['pixel_values'], normalize=True)
    text_features_1 = model_1.get_text_features(processed_1['input_ids'], normalize=True)

    text_probs_1 = (100 * image_features_1 @ text_features_1.T).softmax(dim=-1)
    
print(text_probs_1)
# [9.9955e-01, 4.4712e-04, 4.4010e-06]]
```

### Evaluation with GCL
```
git clone https://github.com/marqo-ai/GCL
```
Install the packages required by GCL. 
```
cd ./GCL
MODEL=hf-hub:Marqo/marqo-ecommerce-B
outdir=/MarqoModels/GE/marqo-ecommerce-B/gs-title2image2
hfdataset=Marqo/google-shopping-general-eval
python  evals/eval_hf_datasets_v1.py \
      --model_name $MODEL \
      --hf-dataset $hfdataset \
      --output-dir $outdir \
      --batch-size 1024 \
      --num_workers 8 \
      --left-key "['title']" \
      --right-key "['image']" \
      --img-or-txt "[['txt'], ['img']]" \
      --left-weight "[1]" \
      --right-weight "[1]" \
      --run-queries-cpu \
      --top-q 4000 \
      --doc-id-key item_ID \
      --context-length "[[64], [0]]"
```


## Detailed Performance
**GoogleShopping-Text2Image Retrieval.**

| **Embedding Model** | **mAP** | **P@10** | **R@10** | **MRR** |
| --- | --- | --- | --- | --- |
| Marqo-Ecommerce-L | **0.682** | **0.089** | **0.878** | **0.683** |
| Marqo-Ecommerce-B | 0.623 | 0.084 | 0.832 | 0.624 |
| Amazon-Titan-MultiModal | 0.475 | 0.065 | 0.648 | 0.475 |
| ViT-B-16-SigLip | 0.476 | 0.067 | 0.660 | 0.477 |
| ViT-L-16-SigLip | 0.540 | 0.073 | 0.722 | 0.540 |


**GoogleShopping-Category2Image Retrieval.**  

| **Embedding Model** | **mAP** | **P@10** | **MRR** | **nDCG@10** |
| --- | --- | --- | --- | --- |
| Marqo-Ecommerce-L | **0.463** | **0.652** | **0.822** | **0.666** |
| Marqo-Ecommerce-B | 0.423 | 0.629 | 0.810 | 0.644 |
| Amazon-Titan-MultiModal | 0.246 | 0.429 | 0.642 | 0.446 |
| ViT-B-16-SigLip | 0.277 | 0.458 | 0.660 | 0.473 |
| ViT-L-16-SigLip | 0.324 | 0.497 | 0.687 | 0.509 |

**AmazonProducts-Text2Image Retrieval.** 

| **Embedding Model** | **mAP** | **P@10** | **R@10** | **MRR** |
| --- | --- | --- | --- | --- |
| Marqo-Ecommerce-L | **0.658** | **0.096** | **0.854** | **0.663** |
| Marqo-Ecommerce-B | 0.592 | 0.089 | 0.795 | 0.597 |
| Amazon-Titan-MultiModal | 0.456 | 0.064 | 0.627 | 0.457 |
| ViT-B-16-SigLip | 0.480 | 0.070 | 0.650 | 0.484 |
| ViT-L-16-SigLip | 0.544 | 0.077 | 0.715 | 0.548 |