Conditional ViT - B/16 - Text
Introduced in LRVSF-Fashion: Extending Visual Search with Referring Instructions, Lepage et al. 2023
Data | Code | Models | Spaces |
---|---|---|---|
Full Dataset | Training Code | Categorical Model | LRVS-F Leaderboard |
Test set | Benchmark Code | Textual Model | Demo |
General Infos
Model finetuned from CLIP ViT-B/16 on LRVSF at 224x224. The conditioning text is preprocessed by a frozen Sentence T5-XL.
Research use only.
How to Use
from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel
import torch
model = AutoModel.from_pretrained("Slep/CondViT-B16-txt")
processor = AutoProcessor.from_pretrained("Slep/CondViT-B16-txt")
url = "https://huggingface.co/datasets/Slep/LAION-RVS-Fashion/resolve/main/assets/108856.0.jpg"
img = Image.open(requests.get(url, stream=True).raw)
txt = "a brown bag"
inputs = processor(images=[img], texts=[txt])
raw_embedding = model(**inputs)
normalized_embedding = torch.nn.functional.normalize(raw_embedding, dim=-1)
- Downloads last month
- 38
Inference API (serverless) does not yet support model repos that contain custom code.
Dataset used to train Slep/CondViT-B16-txt
Evaluation results
- R@1 +10K Dist. on LAION - Referred Visual Search - Fashiontest set LRVSF Leaderboard94.18 ± 0.86
- R@5 +10K Dist. on LAION - Referred Visual Search - Fashiontest set LRVSF Leaderboard98.78 ± 0.32
- R@10 +10K Dist. on LAION - Referred Visual Search - Fashiontest set LRVSF Leaderboard99.25 ± 0.30
- R@20 +10K Dist. on LAION - Referred Visual Search - Fashiontest set LRVSF Leaderboard99.71 ± 0.17
- R@50 +10K Dist. on LAION - Referred Visual Search - Fashiontest set LRVSF Leaderboard99.79 ± 0.13
- R@1 +100K Dist. on LAION - Referred Visual Search - Fashiontest set LRVSF Leaderboard87.07 ± 1.30
- R@5 +100K Dist. on LAION - Referred Visual Search - Fashiontest set LRVSF Leaderboard95.28 ± 0.61
- R@10 +100K Dist. on LAION - Referred Visual Search - Fashiontest set LRVSF Leaderboard96.99 ± 0.44
- R@20 +100K Dist. on LAION - Referred Visual Search - Fashiontest set LRVSF Leaderboard98.04 ± 0.36
- R@50 +100K Dist. on LAION - Referred Visual Search - Fashiontest set LRVSF Leaderboard98.98 ± 0.26