copying from MCG-NJU/videomae-base-short

README.md

@@ -0,0 +1,84 @@
+---
+license: "cc-by-nc-4.0"
+tags:
+- vision
+- video-classification
+---
+
+# VideoMAE (base-sized model, pre-trained only) 
+
+VideoMAE model pre-trained on Kinetics-400 for 800 epochs in a self-supervised way. It was introduced in the paper [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://arxiv.org/abs/2203.12602) by Tong et al. and first released in [this repository](https://github.com/MCG-NJU/VideoMAE). 
+
+Disclaimer: The team releasing VideoMAE did not write a model card for this model so this model card has been written by the Hugging Face team.
+
+## Model description
+
+VideoMAE is an extension of [Masked Autoencoders (MAE)](https://arxiv.org/abs/2111.06377) to video. The architecture of the model is very similar to that of a standard Vision Transformer (ViT), with a decoder on top for predicting pixel values for masked patches.
+
+Videos are presented to the model as a sequence of fixed-size patches (resolution 16x16), which are linearly embedded. One also adds a [CLS] token to the beginning of a sequence to use it for classification tasks. One also adds fixed sinus/cosinus position embeddings before feeding the sequence to the layers of the Transformer encoder.
+
+By pre-training the model, it learns an inner representation of videos that can then be used to extract features useful for downstream tasks: if you have a dataset of labeled videos for instance, you can train a standard classifier by placing a linear layer on top of the pre-trained encoder. One typically places a linear layer on top of the [CLS] token, as the last hidden state of this token can be seen as a representation of an entire video.
+
+## Intended uses & limitations
+
+You can use the raw model for predicting pixel values for masked patches of a video, but it's mostly intended to be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=videomae) to look for fine-tuned versions on a task that interests you.
+
+### How to use
+
+Here is how to use this model to predict pixel values for randomly masked patches:
+
+```python
+from transformers import VideoMAEImageProcessor, VideoMAEForPreTraining
+import numpy as np
+import torch
+
+num_frames = 16
+video = list(np.random.randn(16, 3, 224, 224))
+
+processor = VideoMAEImageProcessor.from_pretrained("MCG-NJU/videomae-base-short")
+model = VideoMAEForPreTraining.from_pretrained("MCG-NJU/videomae-base-short")
+
+pixel_values = processor(video, return_tensors="pt").pixel_values
+
+num_patches_per_frame = (model.config.image_size // model.config.patch_size) ** 2
+seq_length = (num_frames // model.config.tubelet_size) * num_patches_per_frame
+bool_masked_pos = torch.randint(0, 2, (1, seq_length)).bool()
+
+outputs = model(pixel_values, bool_masked_pos=bool_masked_pos)
+loss = outputs.loss
+```
+
+For more code examples, we refer to the [documentation](https://huggingface.co/transformers/main/model_doc/videomae.html#).
+
+## Training data
+
+(to do, feel free to open a PR)
+
+## Training procedure
+
+### Preprocessing
+
+(to do, feel free to open a PR)
+
+### Pretraining
+
+(to do, feel free to open a PR)
+
+## Evaluation results
+
+(to do, feel free to open a PR)
+
+### BibTeX entry and citation info
+
+```bibtex
+misc{https://doi.org/10.48550/arxiv.2203.12602,
+  doi = {10.48550/ARXIV.2203.12602},
+  url = {https://arxiv.org/abs/2203.12602},
+  author = {Tong, Zhan and Song, Yibing and Wang, Jue and Wang, Limin},
+  keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},
+  title = {VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training},
+  publisher = {arXiv},
+  year = {2022},
+  copyright = {Creative Commons Attribution 4.0 International}
+}
+```

config.json

@@ -0,0 +1,29 @@
+{
+  "architectures": [
+    "VideoMAEForPreTraining"
+  ],
+  "attention_probs_dropout_prob": 0.0,
+  "decoder_hidden_size": 384,
+  "decoder_intermediate_size": 1536,
+  "decoder_num_attention_heads": 6,
+  "decoder_num_hidden_layers": 4,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.0,
+  "hidden_size": 768,
+  "image_size": 224,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "model_type": "videomae",
+  "norm_pix_loss": true,
+  "num_attention_heads": 12,
+  "num_channels": 3,
+  "num_frames": 16,
+  "num_hidden_layers": 12,
+  "patch_size": 16,
+  "qkv_bias": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.21.0.dev0",
+  "tubelet_size": 2,
+  "use_mean_pooling": false
+}

preprocessor_config.json

@@ -0,0 +1,18 @@
+{
+  "do_center_crop": true,
+  "do_normalize": true,
+  "do_resize": true,
+  "feature_extractor_type": "VideoMAEFeatureExtractor",
+  "image_mean": [
+    0.485,
+    0.456,
+    0.406
+  ],
+  "image_std": [
+    0.229,
+    0.224,
+    0.225
+  ],
+  "resample": 2,
+  "size": 224
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3e1d7c6ed86b5285ad1608cc996fe8333794646c72ead9b9ad401f38921384ae
+size 376924301