updated files to be like bioclip

README.md

@@ -0,0 +1,300 @@
+---
+license:
+- mit
+language:
+- en
+library_name: open_clip
+tags:
+- zero-shot-image-classification
+- clip
+- biology
+- CV
+- images
+- animals
+- species
+- taxonomy
+- rare species
+- endangered species
+- evolutionary biology
+- multimodal
+- knowledge-guided
+datasets:
+- imageomics/TreeOfLife-10M
+- iNat21
+- BIOSCAN-1M
+- EOL
+---
+
+
+# Model Card for BioCLIP
+
+<!-- 
+This modelcard has been generated using [this raw template](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/modelcard_template.md?plain=1). And further altered to suit Imageomics Institute needs -->
+
+BioCLIP is a foundation model for the tree of life, built using CLIP architecture as a vision model for general organismal biology. 
+It is trained on [TreeOfLife-10M](https://huggingface.co/datasets/imageomics/TreeOfLife-10M), our specially-created dataset covering over 450K taxa--the most biologically diverse ML-ready dataset available to date. 
+Through rigorous benchmarking on a diverse set of fine-grained biological classification tasks, BioCLIP consistently outperformed existing baselines by 16% to 17% absolute. 
+Through intrinsic evaluation, we found that BioCLIP learned a hierarchical representation aligned to the tree of life, which demonstrates its potential for robust generalizability.
+
+**See the `examples/` directory for examples of how to use BioCLIP in zero-shot and few-shot settings.**
+
+## Model Details
+
+### Model Description
+
+BioCLIP is based on OpenAI's [CLIP](https://openai.com/research/clip). 
+We trained the model on [TreeOfLife-10M](https://huggingface.co/datasets/imageomics/TreeOfLife-10M) from OpenAI's ViT-B/16 checkpoint, using [OpenCLIP's](https://github.com/mlfoundations/open_clip) code.
+BioCLIP is trained with the standard CLIP objective to imbue the model with an understanding, not just of different species, but of the hierarchical structure that relates species across the tree of life. 
+In this way, BioCLIP offers potential to aid biologists in discovery of new and related creatures, since it does not see the 454K different taxa as distinct classes, but as part of an interconnected hierarchy. 
+
+
+- **Developed by:** Samuel Stevens, Jiaman Wu, Matthew J. Thompson, Elizabeth G. Campolongo, Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M. Dahdul, Charles Stewart, Tanya Berger-Wolf, Wei-Lun Chao, and Yu Su
+- **Model type:** Vision Transformer (ViT-B/16)
+- **License:** MIT
+- **Fine-tuned from model:** OpenAI CLIP, ViT-B/16
+
+This model was developed for the benefit of the community as an open-source product, thus we request that any derivative products are also open-source.
+
+### Model Sources
+
+- **Repository:** [BioCLIP](https://github.com/Imageomics/BioCLIP)
+- **Paper:** BioCLIP: A Vision Foundation Model for the Tree of Life ([arXiv](https://doi.org/10.48550/arXiv.2311.18803))
+- **Demo:** [BioCLIP Demo](https://huggingface.co/spaces/imageomics/bioclip-demo)
+
+## Uses
+
+BioCLIP has been extensively evaluated on species classification tasks across many different subtrees of the tree of life.
+The ViT-B/16 vision encoder is recommended as a base model for any computer vision task for biology; we expect it to outperform general domain models with the same architecture on biology-specific tasks.
+
+
+### Direct Use
+
+See the demo [here](https://huggingface.co/spaces/imageomics/bioclip-demo) for examples of zero-shot classification.
+It can also be used in a few-shot setting with a KNN; please see [our paper](https://doi.org/10.48550/arXiv.2311.18803) for details for both few-shot and zero-shot settings without fine-tuning.
+
+
+## Bias, Risks, and Limitations
+
+This model was developed from the original CLIP model, thus many of the concerns discussed in ([Radford et al. 2021](https://proceedings.mlr.press/v139/radford21a/radford21a.pdf)) apply. 
+We encourage the concerned/curious user to read their extensive ethics statement, while we focus our attention on the biological perspective which is unique to BioCLIP. 
+ - No specific geographic information (eg., GPS coordinates) are included in training, so the species classification does not pose a direct threat to animals through aiding poachers, as it cannot inform them of their location.
+ - BioCLIP is designed to aid in scientific discovery through an association of images to the hierarchical taxonomy structure. As with many--if not all--models currently in production, it is important to retain the context that it is meant to assist biologists in their work, not replace them. As such, we caution against over-reliance on model predictions.
+
+### Recommendations
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. 
+More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+BioCLIP can be used with the `open_clip` library:
+
+```py
+import open_clip
+
+model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms('hf-hub:imageomics/bioclip')
+tokenizer = open_clip.get_tokenizer('hf-hub:imageomics/bioclip')
+```
+
+## Training Details
+
+### Compute Infrastructure
+
+Training was performed on 8 NVIDIA A100-80GB GPUs distributed over 2 nodes on [OSC's](https://www.osc.edu/) Ascend HPC Cluster with global batch size 32,768 for 4 days.
+
+Based on [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://doi.org/10.48550/arXiv.1910.09700), that's 132.71 kg of CO<sub>2</sub> eq., or 536km driven by an average ICE car.
+
+### Training Data
+
+This model was trained on [TreeOfLife-10M](https://huggingface.co/datasets/imageomics/TreeOfLife-10M), which is a compilation of images matched to [Linnaean taxonomic rank](https://www.britannica.com/science/taxonomy/The-objectives-of-biological-classification) from kingdom through species. They are also matched with common (vernacular) name of the subject of the image where available. For more information, please see our dataset, [TreeOfLife-10M](https://huggingface.co/datasets/imageomics/TreeOfLife-10M).
+
+### Training Hyperparameters
+
+- **Training regime:** fp16 mixed precision.
+
+We resize images to 224 x 224 pixels.
+We use a maximum learning rate of 1e4 with 1000 linear warm-up steps, then use cosine decay to 0 over 100 epochs.
+We also use a weight decay of 0.2 and a batch size of 32K.
+
+## Evaluation
+
+### Testing Data
+
+We tested BioCLIP on the following collection of 10 biologically-relevant tasks.
+ - [Meta-Album](https://paperswithcode.com/dataset/meta-album): Specifically, we used the Plankton, Insects, Insects 2, PlantNet, Fungi, PlantVillage, Medicinal Leaf, and PlantDoc datasets from Set-0 through Set-2 (Set-3 was still not released as of our publication/evaluation (Nov. 2023)).
+ - [Birds 525](https://www.kaggle.com/datasets/gpiosenka/100-bird-species): We evaluated on the 2,625 test images provided with the dataset.
+ - [Rare Species](https://huggingface.co/datasets/imageomics/rare-species): A new dataset we curated for the purpose of testing this model and to contribute to the ML for Conservation community. It consists of 400 species labeled Near Threatened through Extinct in the Wild by the [IUCN Red List](https://www.iucnredlist.org/), with 30 images per species. For more information, see our dataset, [Rare Species](https://huggingface.co/datasets/imageomics/rare-species).
+
+For more information about the contents of these datasets, see Table 2 and associated sections of [our paper](https://doi.org/10.48550/arXiv.2311.18803).
+
+### Metrics
+
+We use top-1 and top-5 accuracy to evaluate models, and validation loss to choose the best performing checkpoints from training.
+
+### Results
+
+We compare BioCLIP to OpenAI's CLIP and OpenCLIP's LAION-2B checkpoint.
+Here are the zero-shot classification results on our benchmark tasks. 
+Please see [our paper](https://doi.org/10.48550/arXiv.2311.18803) for few-shot results.
+
+<table cellpadding="0" cellspacing="0">
+  <thead>
+    <tr>
+      <th rowspan="2">Model</th>
+      <th colspan="4">Animals</th>
+      <th colspan="5">Plants & Fungi</th>
+      <th rowspan="2">Rare Species</th>
+      <th rowspan="2">Mean</th>
+    </tr>
+    <tr>
+      <th>Birds 525</th>
+      <th>Plankton</th>
+      <th>Insects</th>
+      <th>Insects 2</th>
+      <th>PlantNet</th>
+      <th>Fungi</th>
+      <th>PlantVillage</th>
+      <th>Med. Leaf</th>
+      <th>PlantDoc</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>CLIP</td>
+      <td>49.9</td>
+      <td>3.2</td>
+      <td>9.1</td>
+      <td>9.8</td>
+      <td>58.5</td>
+      <td>10.2</td>
+      <td>5.4</td>
+      <td>15.9</td>
+      <td>26.1</td>
+      <td>31.8</td>
+      <td>21.9</td>
+    </tr>
+    <tr>
+      <td>OpenCLIP</td>
+      <td>54.7</td>
+      <td>2.2</td>
+      <td>6.5</td>
+      <td>9.6</td>
+      <td>50.2</td>
+      <td>5.7</td>
+      <td>8.0</td>
+      <td>12.4</td>
+      <td>25.8</td>
+      <td>29.8</td>
+      <td>20.4</td>
+    </tr>
+    <tr>
+      <td>BioCLIP</td>
+      <td><b>72.1</b></td>
+      <td><b>6.1</b></td>
+      <td><b>34.8</b></td>
+      <td><b>20.4</b></td>
+      <td><b>91.4</b></td>
+      <td>40.7</td>
+      <td><b>24.4</b></td>
+      <td><b>38.6</b></td>
+      <td><b>28.4</b></td>
+      <td><b>38.0</b></td>
+      <td><b>39.4</b></td>
+    </tr>
+    <tr>
+      <td>iNat21 Only</td>
+      <td>56.1</td>
+      <td>2.6</td>
+      <td>30.7</td>
+      <td>11.5</td>
+      <td>88.2</td>
+      <td><b>43.0</b></td>
+      <td>18.4</td>
+      <td>25.6</td>
+      <td>20.5</td>
+      <td>21.3</td>
+      <td>31.7</td>
+    </tr>
+  </tbody>
+</table>
+
+
+### Summary
+
+BioCLIP outperforms general-domain baselines by 17% on average for zero-shot.
+
+### Model Examination
+
+We encourage readers to see Section 4.6 of [our paper](https://doi.org/10.48550/arXiv.2311.18803).
+In short, BioCLIP forms representations that more closely align to the taxonomic hierarchy compared to general-domain baselines like CLIP or OpenCLIP.
+
+
+## Citation 
+
+**BibTeX:**
+
+```
+@software{bioclip2023,
+  author = {Samuel Stevens and Jiaman Wu and Matthew J. Thompson and Elizabeth G. Campolongo and Chan Hee Song and David Edward Carlyn and Li Dong and Wasila M. Dahdul and Charles Stewart and Tanya Berger-Wolf and Wei-Lun Chao and Yu Su},
+  doi = {10.57967/hf/1511},
+  month = nov,
+  title = {BioCLIP},
+  version = {v0.1},
+  year = {2023}
+}
+```
+
+Please also cite our paper:
+
+```
+@inproceedings{stevens2024bioclip,
+  title = {{B}io{CLIP}: A Vision Foundation Model for the Tree of Life}, 
+  author = {Samuel Stevens and Jiaman Wu and Matthew J Thompson and Elizabeth G Campolongo and Chan Hee Song and David Edward Carlyn and Li Dong and Wasila M Dahdul and Charles Stewart and Tanya Berger-Wolf and Wei-Lun Chao and Yu Su},
+  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
+  year = {2024}
+}
+
+```
+
+
+Please also consider citing OpenCLIP, iNat21 and BIOSCAN-1M:
+```
+@software{ilharco_gabriel_2021_5143773,
+  author={Ilharco, Gabriel and Wortsman, Mitchell and Wightman, Ross and Gordon, Cade and Carlini, Nicholas and Taori, Rohan and Dave, Achal and Shankar, Vaishaal and Namkoong, Hongseok and Miller, John and Hajishirzi, Hannaneh and Farhadi, Ali and Schmidt, Ludwig},
+  title={OpenCLIP},
+  year={2021},
+  doi={10.5281/zenodo.5143773},
+}
+```
+```
+@misc{inat2021,
+  author={Van Horn, Grant and Mac Aodha, Oisin},
+  title={iNat Challenge 2021 - FGVC8},
+  publisher={Kaggle},
+  year={2021},
+  url={https://kaggle.com/competitions/inaturalist-2021}
+}
+```
+```
+@inproceedings{gharaee2023step,
+  author={Gharaee, Z. and Gong, Z. and Pellegrino, N. and Zarubiieva, I. and Haurum, J. B. and Lowe, S. C. and McKeown, J. T. A. and Ho, C. Y. and McLeod, J. and Wei, Y. C. and Agda, J. and Ratnasingham, S. and Steinke, D. and Chang, A. X. and Taylor, G. W. and Fieguth, P.},
+  title={A Step Towards Worldwide Biodiversity Assessment: The {BIOSCAN-1M} Insect Dataset},
+  booktitle={Advances in Neural Information Processing Systems ({NeurIPS}) Datasets \& Benchmarks Track},
+  year={2023},
+}
+```
+
+## Acknowledgements
+
+The authors would like to thank Josef Uyeda, Jim Balhoff, Dan Rubenstein, Hank Bart, Hilmar Lapp, Sara Beery, and colleagues from the Imageomics Institute and the OSU NLP group for their valuable feedback. We also thank the BIOSCAN-1M team and the iNaturalist team for making their data available and easy to use, and Jennifer Hammack at EOL for her invaluable help in accessing EOL’s images.
+
+The [Imageomics Institute](https://imageomics.org) is funded by the US National Science Foundation's Harnessing the Data Revolution (HDR) program under [Award #2118240](https://www.nsf.gov/awardsearch/showAward?AWD_ID=2118240) (Imageomics: A New Frontier of Biological Information Powered by Knowledge-Guided Machine Learning). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
+
+
+## Model Card Authors
+
+Elizabeth G. Campolongo, Samuel Stevens, and Jiaman Wu
+
+## Model Card Contact
+
+[stevens.994@osu.edu](mailto:stevens.994@osu.edu)

examples/README.md

@@ -0,0 +1,18 @@
+# Examples
+
+## Zero-Shot Classification
+
+```sh
+pip install torch  # whatever version you want
+pip install open_clip_torch numpy tqdm torchvision
+```
+
+Suppose you want to evaluate BioCLIP on zero-shot classification on two tasks, `<DATASET-NAME>` and `<DATASET2-NAME>`.
+You can use `examples/zero_shot.py` to get top1 and top5 accuracy assuming your tasks are arranged as `torchvision`'s [`ImageFolder`](https://pytorch.org/vision/stable/generated/torchvision.datasets.ImageFolder.html) wants.
+
+```sh
+python examples/zero_shot.py \
+  --datasets <DATASET-NAME>=<DATASET-FOLDER> <DATASET2-NAME>=<DATASET2-FOLDER>
+```
+
+This will write to `logs/bioclip-zero-shot/results.json` with your results.

examples/zero_shot.py

@@ -0,0 +1,298 @@
+"""
+Do zero-shot image classification.
+
+Writes the output to a plaintext and JSON format in the logs directory.
+"""
+import argparse
+import ast
+import contextlib
+import json
+import logging
+import os
+import random
+import sys
+
+import numpy as np
+import open_clip
+import torch
+import torch.nn.functional as F
+from torchvision import datasets
+from tqdm import tqdm
+
+log_format = "[%(asctime)s] [%(levelname)s] [%(name)s] %(message)s"
+logging.basicConfig(level=logging.INFO, format=log_format)
+logger = logging.getLogger("main")
+
+openai_templates = [
+    lambda c: f"a bad photo of a {c}.",
+    lambda c: f"a photo of many {c}.",
+    lambda c: f"a sculpture of a {c}.",
+    lambda c: f"a photo of the hard to see {c}.",
+    lambda c: f"a low resolution photo of the {c}.",
+    lambda c: f"a rendering of a {c}.",
+    lambda c: f"graffiti of a {c}.",
+    lambda c: f"a bad photo of the {c}.",
+    lambda c: f"a cropped photo of the {c}.",
+    lambda c: f"a tattoo of a {c}.",
+    lambda c: f"the embroidered {c}.",
+    lambda c: f"a photo of a hard to see {c}.",
+    lambda c: f"a bright photo of a {c}.",
+    lambda c: f"a photo of a clean {c}.",
+    lambda c: f"a photo of a dirty {c}.",
+    lambda c: f"a dark photo of the {c}.",
+    lambda c: f"a drawing of a {c}.",
+    lambda c: f"a photo of my {c}.",
+    lambda c: f"the plastic {c}.",
+    lambda c: f"a photo of the cool {c}.",
+    lambda c: f"a close-up photo of a {c}.",
+    lambda c: f"a black and white photo of the {c}.",
+    lambda c: f"a painting of the {c}.",
+    lambda c: f"a painting of a {c}.",
+    lambda c: f"a pixelated photo of the {c}.",
+    lambda c: f"a sculpture of the {c}.",
+    lambda c: f"a bright photo of the {c}.",
+    lambda c: f"a cropped photo of a {c}.",
+    lambda c: f"a plastic {c}.",
+    lambda c: f"a photo of the dirty {c}.",
+    lambda c: f"a jpeg corrupted photo of a {c}.",
+    lambda c: f"a blurry photo of the {c}.",
+    lambda c: f"a photo of the {c}.",
+    lambda c: f"a good photo of the {c}.",
+    lambda c: f"a rendering of the {c}.",
+    lambda c: f"a {c} in a video game.",
+    lambda c: f"a photo of one {c}.",
+    lambda c: f"a doodle of a {c}.",
+    lambda c: f"a close-up photo of the {c}.",
+    lambda c: f"a photo of a {c}.",
+    lambda c: f"the origami {c}.",
+    lambda c: f"the {c} in a video game.",
+    lambda c: f"a sketch of a {c}.",
+    lambda c: f"a doodle of the {c}.",
+    lambda c: f"a origami {c}.",
+    lambda c: f"a low resolution photo of a {c}.",
+    lambda c: f"the toy {c}.",
+    lambda c: f"a rendition of the {c}.",
+    lambda c: f"a photo of the clean {c}.",
+    lambda c: f"a photo of a large {c}.",
+    lambda c: f"a rendition of a {c}.",
+    lambda c: f"a photo of a nice {c}.",
+    lambda c: f"a photo of a weird {c}.",
+    lambda c: f"a blurry photo of a {c}.",
+    lambda c: f"a cartoon {c}.",
+    lambda c: f"art of a {c}.",
+    lambda c: f"a sketch of the {c}.",
+    lambda c: f"a embroidered {c}.",
+    lambda c: f"a pixelated photo of a {c}.",
+    lambda c: f"itap of the {c}.",
+    lambda c: f"a jpeg corrupted photo of the {c}.",
+    lambda c: f"a good photo of a {c}.",
+    lambda c: f"a plushie {c}.",
+    lambda c: f"a photo of the nice {c}.",
+    lambda c: f"a photo of the small {c}.",
+    lambda c: f"a photo of the weird {c}.",
+    lambda c: f"the cartoon {c}.",
+    lambda c: f"art of the {c}.",
+    lambda c: f"a drawing of the {c}.",
+    lambda c: f"a photo of the large {c}.",
+    lambda c: f"a black and white photo of a {c}.",
+    lambda c: f"the plushie {c}.",
+    lambda c: f"a dark photo of a {c}.",
+    lambda c: f"itap of a {c}.",
+    lambda c: f"graffiti of the {c}.",
+    lambda c: f"a toy {c}.",
+    lambda c: f"itap of my {c}.",
+    lambda c: f"a photo of a cool {c}.",
+    lambda c: f"a photo of a small {c}.",
+    lambda c: f"a tattoo of the {c}.",
+]
+
+
+def parse_args(args):
+    class ParseKwargs(argparse.Action):
+        def __call__(self, parser, namespace, values, option_string=None):
+            kw = {}
+            for value in values:
+                key, value = value.split("=")
+                try:
+                    kw[key] = ast.literal_eval(value)
+                except (ValueError, SyntaxError):
+                    # fallback to string (avoid need to escape on command line)
+                    kw[key] = str(value)
+            setattr(namespace, self.dest, kw)
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--datasets",
+        type=str,
+        default=None,
+        nargs="+",
+        help="Path to dirs(s) with validation data. In the format NAME=PATH.",
+        action=ParseKwargs,
+    )
+    parser.add_argument(
+        "--logs", type=str, default="./logs", help="Where to write logs"
+    )
+    parser.add_argument(
+        "--exp", type=str, default="bioclip-zero-shot", help="Experiment name."
+    )
+    parser.add_argument(
+        "--workers", type=int, default=8, help="Number of dataloader workers per GPU."
+    )
+    parser.add_argument(
+        "--batch-size", type=int, default=64, help="Batch size per GPU."
+    )
+    parser.add_argument(
+        "--precision",
+        choices=["amp", "amp_bf16", "amp_bfloat16", "bf16", "fp32"],
+        default="amp",
+        help="Floating point precision.",
+    )
+    parser.add_argument("--seed", type=int, default=0, help="Default random seed.")
+    args = parser.parse_args(args)
+    os.makedirs(os.path.join(args.logs, args.exp), exist_ok=True)
+
+    return args
+
+
+def make_txt_features(model, classnames, templates, args):
+    tokenizer = open_clip.get_tokenizer("hf-hub:imageomics/bioclip")
+    with torch.no_grad():
+        txt_features = []
+        for classname in tqdm(classnames):
+            classname = " ".join(word for word in classname.split("_") if word)
+            texts = [template(classname) for template in templates]  # format with class
+            texts = tokenizer(texts).to(args.device)  # tokenize
+            class_embeddings = model.encode_text(texts)
+            class_embedding = F.normalize(class_embeddings, dim=-1).mean(dim=0)
+            class_embedding /= class_embedding.norm()
+            txt_features.append(class_embedding)
+        txt_features = torch.stack(txt_features, dim=1).to(args.device)
+    return txt_features
+
+
+def accuracy(output, target, topk=(1,)):
+    pred = output.topk(max(topk), 1, True, True)[1].t()
+    correct = pred.eq(target.view(1, -1).expand_as(pred))
+    return [correct[:k].reshape(-1).float().sum(0, keepdim=True).item() for k in topk]
+
+
+def get_autocast(precision):
+    if precision == "amp":
+        return torch.cuda.amp.autocast
+    elif precision == "amp_bfloat16" or precision == "amp_bf16":
+        # amp_bfloat16 is more stable than amp float16 for clip training
+        return lambda: torch.cuda.amp.autocast(dtype=torch.bfloat16)
+    else:
+        return contextlib.suppress
+
+
+def run(model, txt_features, dataloader, args):
+    autocast = get_autocast(args.precision)
+    cast_dtype = open_clip.get_cast_dtype(args.precision)
+
+    top1, top5, n = 0.0, 0.0, 0.0
+
+    with torch.no_grad():
+        for images, targets in tqdm(dataloader, unit_scale=args.batch_size):
+            images = images.to(args.device)
+            if cast_dtype is not None:
+                images = images.to(dtype=cast_dtype)
+            targets = targets.to(args.device)
+
+            with autocast():
+                image_features = model.encode_image(images)
+                image_features = F.normalize(image_features, dim=-1)
+                logits = model.logit_scale.exp() * image_features @ txt_features
+
+            # Measure accuracy
+            acc1, acc5 = accuracy(logits, targets, topk=(1, 5))
+            top1 += acc1
+            top5 += acc5
+            n += images.size(0)
+
+    top1 = top1 / n
+    top5 = top5 / n
+    return top1, top5
+
+
+def evaluate(model, data, args):
+    results = {}
+
+    logger.info("Starting zero-shot classification.")
+
+    for split in data:
+        logger.info("Building zero-shot %s classifier.", split)
+
+        classnames = data[split].dataset.classes
+        classnames = [name.replace("_", " ") for name in classnames]
+
+        txt_features = make_txt_features(model, classnames, openai_templates, args)
+
+        logger.info("Got text features.")
+        top1, top5 = run(model, txt_features, data[split], args)
+
+        logger.info("%s-top1: %.3f", split, top1 * 100)
+        logger.info("%s-top5: %.3f", split, top5 * 100)
+
+        results[f"{split}-top1"] = top1 * 100
+        results[f"{split}-top5"] = top5 * 100
+
+        logger.info("Finished zero-shot %s.", split)
+
+    logger.info("Finished zero-shot classification.")
+
+    return results
+
+
+if __name__ == "__main__":
+    args = parse_args(sys.argv[1:])
+
+    if torch.cuda.is_available():
+        # This enables tf32 on Ampere GPUs which is only 8% slower than
+        # float16 and almost as accurate as float32
+        # This was a default in pytorch until 1.12
+        torch.backends.cuda.matmul.allow_tf32 = True
+        torch.backends.cudnn.benchmark = True
+        torch.backends.cudnn.deterministic = False
+
+    # Init torch device
+    if torch.cuda.is_available():
+        device = "cuda:0"
+        torch.cuda.set_device(device)
+    else:
+        device = "cpu"
+    args.device = device
+
+    # Random seeding
+    torch.manual_seed(args.seed)
+    np.random.seed(args.seed)
+    random.seed(args.seed)
+
+    # Load model.
+    model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms(
+        "hf-hub:imageomics/bioclip"
+    )
+
+    # Write datasets
+    params_file = os.path.join(args.logs, args.exp, "params.json")
+    with open(params_file, "w") as fd:
+        params = {name: getattr(args, name) for name in vars(args)}
+        json.dump(params, fd, sort_keys=True, indent=4)
+
+    # Initialize datasets.
+    data = {}
+    for split, path in args.datasets.items():
+        data[split] = torch.utils.data.DataLoader(
+            datasets.ImageFolder(path, transform=preprocess_val),
+            batch_size=args.batch_size,
+            num_workers=args.workers,
+            sampler=None,
+            shuffle=False,
+        )
+
+    model.eval()
+    results = evaluate(model, data, args)
+
+    results_file = os.path.join(args.logs, args.exp, "results.json")
+    with open(results_file, "w") as fd:
+        json.dump(results, fd, indent=4, sort_keys=True)

open_clip_config.json

@@ -0,0 +1,30 @@
+{
+  "model_cfg": {
+    "embed_dim": 512,
+    "vision_cfg": {
+      "image_size": 224,
+      "layers": 12,
+      "width": 768,
+      "patch_size": 16
+    },
+    "text_cfg": {
+      "context_length": 77,
+      "vocab_size": 49408,
+      "width": 512,
+      "heads": 8,
+      "layers": 12
+    }
+  },
+  "preprocess_cfg": {
+    "mean": [
+      0.48145466,
+      0.4578275,
+      0.40821073
+    ],
+    "std": [
+      0.26862954,
+      0.26130258,
+      0.27577711
+    ]
+  }
+}

pytorch_model.bin

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:82bb1b89f807169102c57e1e950f29e93a92180d8bb98d54ecc173790105ce84
-size 1795824654

special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<|startoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,33 @@
+{
+  "add_prefix_space": false,
+  "bos_token": {
+    "__type": "AddedToken",
+    "content": "<|startoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "do_lower_case": true,
+  "eos_token": {
+    "__type": "AddedToken",
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "errors": "replace",
+  "model_max_length": 77,
+  "pad_token": "<|endoftext|>",
+  "special_tokens_map_file": "./special_tokens_map.json",
+  "tokenizer_class": "CLIPTokenizer",
+  "unk_token": {
+    "__type": "AddedToken",
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}