Hugging Face's logo

yuvalkirstain
/
PickScore_v1

Zero-Shot Image Classification
Transformers
PyTorch
Safetensors
clip
Inference Endpoints
Model card Files Community
2
PickScore_v1
File size: 3,223 Bytes 
6a9262a
5d7a24c
 
a05aeb3
1e86ddc
5d7a24c
 
 
 
 
 
a05aeb3
5d7a24c
 
 
 
 
 
1e86ddc
5d7a24c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1e86ddc
 
 
 
 
 
 
 
 
 
 
 
5d7a24c
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
 
# Model Card for PickScore v1

This model is a scoring function for images generated from text. It takes as input a prompt and a generated image and outputs a score. 
It can be used as a general scoring function, and for tasks such as human preference prediction, model evaluation, image ranking, and more. 
See our paper [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://arxiv.org/abs/2305.01569) for more details.


## Model Details

### Model Description

This model was finetuned from CLIP-H using the [Pick-a-Pic dataset](https://huggingface.co/datasets/yuvalkirstain/pickapic_v1).

### Model Sources [optional]

<!-- Provide the basic links for the model. -->

- **Repository:** [See the PickScore repo](https://github.com/yuvalkirstain/PickScore)
- **Paper:** [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://arxiv.org/abs/2305.01569).
- **Demo [optional]:** TODO

## How to Get Started with the Model

Use the code below to get started with the model.

```python
# import
from transformers import AutoProcessor, AutoModel

# load model
device = "cuda"
processor_name_or_path = "laion/CLIP-ViT-H-14-laion2B-s32B-b79K"
model_pretrained_name_or_path = "yuvalkirstain/PickScore_v1"

processor = AutoProcessor.from_pretrained(processor_name_or_path)
model = AutoModel.from_pretrained(model_pretrained_name_or_path).eval().to(device)

def calc_probs(prompt, images):
    
    # preprocess
    image_inputs = processor(
        images=images,
        padding=True,
        truncation=True,
        max_length=77,
        return_tensors="pt",
    ).to(device)
    
    text_inputs = processor(
        text=prompt,
        padding=True,
        truncation=True,
        max_length=77,
        return_tensors="pt",
    ).to(device)


    with torch.no_grad():
        # embed
        image_embs = model.get_image_features(**image_inputs)
        image_embs = image_embs / torch.norm(image_embs, dim=-1, keepdim=True)
    
        text_embs = model.get_text_features(**text_inputs)
        text_embs = text_embs / torch.norm(text_embs, dim=-1, keepdim=True)
    
        # score
        scores = model.logit_scale.exp() * (text_embs @ image_embs.T)[0]
        
        # get probabilities if you have multiple images to choose from
        probs = torch.softmax(scores, dim=-1)
    
    return probs.cpu().tolist()

pil_images = [Image.open("my_amazing_images/1.jpg"), Image.open("my_amazing_images/2.jpg")]
prompt = "fantastic, increadible prompt"
print(calc_probs(prompt, pil_images))
```
## Training Details

### Training Data

This model was trained on the [Pick-a-Pic dataset](https://huggingface.co/datasets/yuvalkirstain/pickapic_v1).


### Training Procedure 

TODO - add paper.


## Citation [optional]

If you find this work useful, please cite:

```bibtex
@misc{kirstain2023pickapic,
      title={Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation}, 
      author={Yuval Kirstain and Adam Polyak and Uriel Singer and Shahbuland Matiana and Joe Penna and Omer Levy},
      year={2023},
      eprint={2305.01569},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
```

**APA:**

[More Information Needed]