Zero-Shot Image Classification
TiC-CLIP
vision
fartashf commited on
Commit
afe3519
1 Parent(s): e8a1bb8

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +27 -0
README.md CHANGED
@@ -55,6 +55,8 @@ The models can also be used to resume a training or as initialization for new tr
55
  Please follow instructions in our [GitHub repo](https://github.com/apple/ml-tic-clip) to create the evaluation sets or follow [DataComp](https://github.com/mlfoundations/datacomp) for the standard evaluations on 38 datasets.
56
 
57
  The following snippet assumes the TiC-DataComp data has been prepared and following the instructions in the GitHub repo.
 
 
58
  ```bash
59
  YEAR=2016 # There are no models before 2016 since data from 2014-2016 were compined into one year
60
  REPO="apple/TiC-CLIP-bestpool-oracle"
@@ -74,7 +76,10 @@ torchrun --nproc_per_node 8 --nnodes 1 \
74
  --save_frequency 1 \
75
  --resume
76
  popd
 
77
 
 
 
78
  ## Evaluate Model
79
  # Evaluate a ViT-B/16 model on TiC/Retrieval/Yearly/$YEAR and
80
  # TiC/DataCompNet/Yearly/$YEAR
@@ -83,6 +88,28 @@ python ../dataset_creation/tic-datacomp/generate_tasklist.py --yaml-path tasklis
83
  python evaluate.py --data_dir data/ --train_output_dir ./results --use_model "ViT-B-16 $YEAR.pt" --skip_hf --skip_db --skip_notification
84
  ```
85
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
86
  ## Training Details
87
 
88
  ### Training Data
 
55
  Please follow instructions in our [GitHub repo](https://github.com/apple/ml-tic-clip) to create the evaluation sets or follow [DataComp](https://github.com/mlfoundations/datacomp) for the standard evaluations on 38 datasets.
56
 
57
  The following snippet assumes the TiC-DataComp data has been prepared and following the instructions in the GitHub repo.
58
+
59
+ ### Training
60
  ```bash
61
  YEAR=2016 # There are no models before 2016 since data from 2014-2016 were compined into one year
62
  REPO="apple/TiC-CLIP-bestpool-oracle"
 
76
  --save_frequency 1 \
77
  --resume
78
  popd
79
+ ```
80
 
81
+ ### Evaluation
82
+ ```bash
83
  ## Evaluate Model
84
  # Evaluate a ViT-B/16 model on TiC/Retrieval/Yearly/$YEAR and
85
  # TiC/DataCompNet/Yearly/$YEAR
 
88
  python evaluate.py --data_dir data/ --train_output_dir ./results --use_model "ViT-B-16 $YEAR.pt" --skip_hf --skip_db --skip_notification
89
  ```
90
 
91
+ ### OpenCLIP Load and Inference Example
92
+ ```python
93
+ import open_clip
94
+ from huggingface_hub import hf_hub_download
95
+ filename = hf_hub_download(repo_id="apple/TiC-CLIP-bestpool-cumulative", filename="checkpoints/2016.pt")
96
+ model, _, preprocess = open_clip.create_model_and_transforms('ViT-B-16', filename)
97
+ tokenizer = open_clip.get_tokenizer('ViT-B-16')
98
+
99
+ image = preprocess(Image.open("image.png").convert('RGB')).unsqueeze(0)
100
+ text = tokenizer(["a diagram", "a dog", "a cat"])
101
+
102
+ with torch.no_grad(), torch.cuda.amp.autocast():
103
+ image_features = model.encode_image(image)
104
+ text_features = model.encode_text(text)
105
+ image_features /= image_features.norm(dim=-1, keepdim=True)
106
+ text_features /= text_features.norm(dim=-1, keepdim=True)
107
+
108
+ text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
109
+
110
+ print("Label probs:", text_probs)
111
+ ```
112
+
113
  ## Training Details
114
 
115
  ### Training Data