initial commit

README.md

@@ -0,0 +1,54 @@
+---
+language: "mn"
+tags:
+- bert
+- mongolian
+- uncased
+---
+
+# BERT-BASE-MONGOLIAN-UNCASED
+[Link to Official Mongolian-BERT repo](https://github.com/tugstugi/mongolian-bert)
+
+## Model description
+This repository contains pre-trained Mongolian [BERT](https://arxiv.org/abs/1810.04805) models trained by [tugstugi](https://github.com/tugstugi), [enod](https://github.com/enod) and [sharavsambuu](https://github.com/sharavsambuu).
+Special thanks to [nabar](https://github.com/nabar) who provided 5x TPUs.
+
+This repository is based on the following open source projects: [google-research/bert](https://github.com/google-research/bert/),
+[huggingface/pytorch-pretrained-BERT](https://github.com/huggingface/pytorch-pretrained-BERT) and [yoheikikuta/bert-japanese](https://github.com/yoheikikuta/bert-japanese).
+
+#### How to use
+
+```python
+from transformers import pipeline, AutoTokenizer, BertForMaskedLM
+
+tokenizer = AutoTokenizer.from_pretrained('tugstugi/bert-base-mongolian-uncased')
+model = BertForMaskedLM.from_pretrained('tugstugi/bert-base-mongolian-uncased')
+
+## declare task ##
+pipe = pipeline(task="fill-mask", model=model, tokenizer=tokenizer)
+
+## example ##
+input_  = 'Миний [MASK] хоол идэх нь тун чухал.'
+
+output_ = pipe(input_)
+for i in range(len(output_)):
+    print(output_[i])
+
+```
+
+
+## Training data
+Mongolian Wikipedia and the 700 million word Mongolian news data set  [[Pretraining Procedure](https://github.com/tugstugi/mongolian-bert#pre-training)]
+
+### BibTeX entry and citation info
+
+```bibtex
+@misc{mongolian-bert,
+  author = {Tuguldur, Erdene-Ochir and Gunchinish, Sharavsambuu and Bataa, Enkhbold},
+  title = {BERT Pretrained Models on Mongolian Datasets},
+  year = {2019},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  howpublished = {\url{https://github.com/tugstugi/mongolian-bert/}}
+}
+```

added_tokens.json

@@ -0,0 +1 @@
+{"<pad>": 32000}

config.json

@@ -0,0 +1,21 @@
+{
+  "architectures": [
+    "BertForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "type_vocab_size": 2,
+  "vocab_size": 32000,
+  "tokenizer_class": "AlbertTokenizer"
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:866380e7756622c93c94b7ffc28cf036b3e621f5e2f596b2d7886c55577d771b
+size 445054385

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": "[CLS]", "eos_token": "[SEP]", "unk_token": "<unk>", "sep_token": "[SEP]", "pad_token": "<pad>", "cls_token": "[CLS]", "mask_token": "[MASK]"}

spiece.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19914d955b6d028592b0a331a43b7235b860e5e4964e46e8c004a21eee0b2b27
+size 1007226

tf_model.h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d07d2e1fdb92e85536c0330f5f5c7b46dd0304b3e6722868bd6b2df47a65c7e4
+size 545136208

tokenizer_config.json

@@ -0,0 +1 @@
+{"keep_accents": true, "do_lower_case": true}