Update README.md

README.md

@@ -0,0 +1,66 @@
+# Releasing Hindi ELECTRA model
+
+This is a first attempt at a Hindi language model trained with Google Research's [ELECTRA](https://github.com/google-research/electra).  **I don't modify ELECTRA until we get into finetuning**
+
+Tokenization and training CoLab: https://colab.research.google.com/drive/1R8TciRSM7BONJRBc9CBZbzOmz39FTLl_
+
+Blog post: https://medium.com/@mapmeld/teaching-hindi-to-electra-b11084baab81
+
+I was greatly influenced by: https://huggingface.co/blog/how-to-train
+
+## Corpus
+
+Download: https://drive.google.com/drive/u/1/folders/1WikYHHMI72hjZoCQkLPr45LDV8zm9P7p
+
+The corpus is two files:
+- Hindi CommonCrawl deduped by OSCAR https://traces1.inria.fr/oscar/
+- latest Hindi Wikipedia ( https://dumps.wikimedia.org/hiwiki/20200420/ ) + WikiExtractor to txt 
+
+Bonus notes:
+- Adding English wiki text or parallel corpus could help with cross-lingual tasks and training
+
+## Vocabulary
+
+https://drive.google.com/file/d/1-02Um-8ogD4vjn4t-wD2EwCE-GtBjnzh/view?usp=sharing
+
+Bonus notes:
+- Created with HuggingFace Tokenizers; could be longer or shorter, review ELECTRA vocab_size param
+
+## Pretrain TF Records
+
+[build_pretraining_dataset.py](https://github.com/google-research/electra/blob/master/build_pretraining_dataset.py) splits the corpus into training documents
+
+Set the ELECTRA model size and whether to split the corpus by newlines.  This process can take hours on its own.
+
+https://drive.google.com/drive/u/1/folders/1--wBjSH59HSFOVkYi4X-z5bigLnD32R5
+
+Bonus notes:
+- I am not sure of the meaning of the corpus newline split (what is the alternative?) and given this corpus, which creates the better training docs
+
+## Training
+
+Structure your files, with data-dir named "trainer" here
+
+```
+trainer
+- vocab.txt
+- pretrain_tfrecords
+-- (all .tfrecord... files)
+- models
+-- modelname
+--- checkpoint
+--- graph.pbtxt
+--- model.*
+```
+
+CoLab notebook gives examples of GPU vs. TPU setup
+
+[configure_pretraining.py](https://github.com/google-research/electra/blob/master/configure_pretraining.py)
+
+Model https://drive.google.com/drive/folders/1cwQlWryLE4nlke4OixXA7NK8hzlmUR0c?usp=sharing
+
+## Using this model with Transformers
+
+Sample movie reviews classifier: https://colab.research.google.com/drive/1mSeeSfVSOT7e-dVhPlmSsQRvpn6xC05w
+
+Slightly outperforms Multilingual BERT on these Hindi Movie Reviews from https://github.com/sid573/Hindi_Sentiment_Analysis