kakaobrain/vit-l16-coyo-labeled-300m-i1k512

Vision Transformer (large-sized model)

Vision Transformer (ViT) model pre-trained on COYO-Labeled-300M (300 million images, 21,841 classes) at resolution 224x224. It was introduced in the paper An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale by Dosovitskiy et al. However, since the JFT-300M is a private dataset, we tried to reproduce it using the publicly available COYO-Labeled-300M dataset.

Thanks to Hugging Face team for converting weights of ViT trained in Tensorflow to be used on Pytorch, JAX/Flax and Tensorflow in Hugging Face.

Model description

The Vision Transformer (ViT) is a transformer model pretrained on a large collection of images in a supervised fashion, namely COYO-Labeled-300M, at a resolution of 224x224 pixels.

Images are presented to the model as a sequence of fixed-size patches (resolution 16x16), which are linearly embedded. One also adds a [CLS] token to the beginning of a sequence to use it for classification tasks. One also adds absolute position embeddings before feeding the sequence to the layers of the Transformer.

This ViT model is finetuned with Imagenet-1k at resolution 384x384. Please see details here

Intended uses & limitations

You can use weights from ViT models for image classification, downstream. Codes for reproduction are also provided. Please see this github repository for pretraining and finetuning code.

How to use

Here is how to use this model in PyTorch:

WIP

Here is how to use this model in JAX/Flax:

WIP

Here is how to use this model in Tensorflow:

WIP

Training data

The ViT model was pretrained on COYO-Labeled-300M, a dataset consisting of 300 million images and 21k classes.

Training procedure

Preprocessing

The exact details of preprocessing of images during training/validation can be found here.

Images are inception-cropped to the same resolution (224x224) and normalized across the RGB channels with mean (0.5, 0.5, 0.5) and standard deviation (0.5, 0.5, 0.5).

Pretraining

The model was trained on TPUv3 hardware. All model variants are trained with a batch size of 4096 and learning rate warmup of 10k steps. Pre-training resolution is 224. More detail, please see here

Evaluation results

Model	Upstream Dataset	Resolution	ImageNet (downstream)	ImageNet-ReaL (dwonstream)	Public
ViT-L/16	JFT-300M	512	87.76	90.54	X
ViT-L/16	COYO-Labeled-300M	512	87.24 (-0.52)	90.03 (-0.51)	O
ViT-L/16	JFT-300M	384	87.12	89.99	X
ViT-L/16	COYO-Labeled-300M	384	86.72 (-0.4)	89.84 (-0.15)	O

Citation

@misc{kakaobrain2022coyo-vit,
  title         = {COYO-ViT},
  author        = {Lee, Sungjun and Park, Beomhee},
  year          = {2022},
  howpublished  = {\url{https://github.com/kakaobrain/coyo-vit}},
}

@misc{kakaobrain2022coyo-700m,
  title         = {COYO-700M: Image-Text Pair Dataset},
  author        = {Byeon, Minwoo and Park, Beomhee and Kim, Haecheon and Lee, Sungjun and Baek, Woonhyuk and Kim, Saehoon},
  year          = {2022},
  howpublished  = {\url{https://github.com/kakaobrain/coyo-dataset}},
}

@misc{dosovitskiy2020image,
    title   = {An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
    author  = {Alexey Dosovitskiy and Lucas Beyer and Alexander Kolesnikov and Dirk Weissenborn and Xiaohua Zhai and Thomas Unterthiner and Mostafa Dehghani and Matthias Minderer and Georg Heigold and Sylvain Gelly and Jakob Uszkoreit and Neil Houlsby},
    year    = {2020},
    eprint  = {2010.11929},
    archivePrefix = {arXiv},
    primaryClass = {cs.CV}
}

License

The source codes are licensed under Apache 2.0 License.