Adding initial files

README.md

@@ -0,0 +1,55 @@
+---
+language: en
+license: apache-2.0
+datasets:
+- bookcorpus
+- wikipedia
+- cc_news
+---
+
+# BigBird base model
+
+BigBird, is a sparse-attention based transformer which extends Transformer based models, such as BERT to much longer sequences. Moreover, BigBird comes along with a theoretical understanding of the capabilities of a complete transformer that the sparse model can handle.
+
+It is a pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in this [paper](https://arxiv.org/abs/2007.14062) and first released in this [repository](https://github.com/google-research/bigbird).
+
+Disclaimer: The team releasing BigBird did not write a model card for this model so this model card has been written by the Hugging Face team.
+
+## Model description
+
+BigBird relies on **block sparse attention** instead of normal attention (i.e. BERT's attention) and can handle sequences up to a length of 4096 at a much lower compute cost compared to BERT. It has achieved SOTA on various tasks involving very long sequences such as long documents summarization, question-answering with long contexts.
+
+## How to use `TODO: Update`
+
+Here is how to use this model to get the features of a given text in Flax:
+
+```python
+from transformers import BigBirdTokenizer, FlaxBigBirdModel
+
+model_id = "flax-community/bigband"
+
+# by default its in `block_sparse` mode with num_random_blocks=3, block_size=64
+model = FlaxBigBirdModel.from_pretrained(model_id)
+
+# you can change `attention_type` to full attention like this:
+model = FlaxBigBirdModel.from_pretrained(model_id, attention_type="original_full")
+
+# you can change `block_size` & `num_random_blocks` like this:
+model = FlaxBigBirdModel.from_pretrained(model_id, block_size=16, num_random_blocks=2)
+
+tokenizer = BigBirdTokenizer.from_pretrained(model_id)
+
+text = "Replace me by any text you'd like."
+inputs = tokenizer(text, return_tensors="jax")
+output = model(**inputs)
+```
+
+## Training Data `TODO: Update`
+
+This model is pre-trained on four publicly available datasets: **Books**, **CC-News**, **Stories** and **Wikipedia**. It used same sentencepiece vocabulary as RoBERTa (which is in turn borrowed from GPT2).
+
+## Training Procedure `TODO: Update`
+
+Document longer than 4096 were split into multiple documents and documents that were much smaller than 4096 were joined. Following the original BERT training, 15% of tokens were masked and model is trained to predict the mask.
+
+Model is warm started from RoBERTa’s checkpoint.

config.json

@@ -0,0 +1,30 @@
+{
+  "architectures": [
+    "BigBirdForPreTraining"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "attention_type": "block_sparse",
+  "block_size": 64,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu_new",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 4096,
+  "model_type": "big_bird",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_random_blocks": 3,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "rescale_embeddings": false,
+  "transformers_version": "4.4.0.dev0",
+  "type_vocab_size": 2,
+  "use_bias": true,
+  "use_cache": true,
+  "vocab_size": 50358
+}

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "eos_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "unk_token": {"content": "<unk>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "sep_token": {"content": "[SEP]", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "pad_token": {"content": "<pad>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "cls_token": {"content": "[CLS]", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "mask_token": {"content": "[MASK]", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true}}

spiece.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fdc81e1fc9d42e0c08b86d5b280d05d7c5e9747c4231c648f2b56b8e1d893c82
+size 845731

tokenizer_config.json

@@ -0,0 +1 @@
+{"bos_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "eos_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "unk_token": {"content": "<unk>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "pad_token": {"content": "<pad>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "sep_token": {"content": "[SEP]", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "mask_token": {"content": "[MASK]", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "cls_token": {"content": "[CLS]", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "model_max_length": 4096, "name_or_path": "google/bigbird-roberta-large"}