Update README.md

README.md

@@ -3,33 +3,75 @@ language: "nl"
 thumbnail: "https://github.com/iPieter/RobBERT/raw/master/res/robbert_logo.png"
 tags:
 - Dutch
+- Flemish
 - RoBERTa
 - RobBERT
 license: mit
 datasets:
 - oscar
-- Shuffled Dutch section of the OSCAR corpus (https://oscar-corpus.com/)
+- oscar (NL)
+- dbrd
+- lassy-ud
+- europarl-mono
+- conll2002
+widget:
+- text: "Hallo, ik ben RobBERT, een <mask> taalmodel van de KU Leuven"
 ---
 
-# RobBERT
+<p align="center"> 
+    <img src="https://github.com/iPieter/RobBERT/raw/master/res/robbert_logo_with_name.png" alt="RobBERT: A Dutch RoBERTa-based Language Model" width="75%">
+ </p>
+
+# RobBERT: Dutch RoBERTa-based Language Model.
+
+[RobBERT](https://github.com/iPieter/RobBERT) is the state-of-the-art Dutch BERT model. It is a large pre-trained general Dutch language model that can be fine-tuned on a given dataset to perform any text classification, regression or token-tagging task. As such, it has been successfully used by many [researchers](https://scholar.google.com/scholar?oi=bibs&hl=en&cites=7180110604335112086) and [practitioners](https://huggingface.co/models?search=robbert) for achieving state-of-the-art performance for a wide range of Dutch natural language processing tasks, including:
+
+- [Emotion detection](https://www.aclweb.org/anthology/2021.wassa-1.27/)
+- Sentiment analysis ([book reviews](https://arxiv.org/pdf/2001.06286.pdf), [news articles](https://biblio.ugent.be/publication/8704637/file/8704638.pdf)*)
+- [Coreference resolution](https://arxiv.org/pdf/2001.06286.pdf)
+- Named entity recognition ([CoNLL](https://arxiv.org/pdf/2001.06286.pdf), [job titles](https://arxiv.org/pdf/2004.02814.pdf)*, [SoNaR](https://github.com/proycon/deepfrog))
+- Part-of-speech tagging ([Small UD Lassy](https://arxiv.org/pdf/2001.06286.pdf), [CGN](https://github.com/proycon/deepfrog))
+- [Zero-shot word prediction](https://arxiv.org/pdf/2001.06286.pdf)
+- [Humor detection](https://arxiv.org/pdf/2010.13652.pdf)
+- [Cyberbulling detection](https://www.cambridge.org/core/journals/natural-language-engineering/article/abs/automatic-classification-of-participant-roles-in-cyberbullying-can-we-detect-victims-bullies-and-bystanders-in-social-media-text/A2079C2C738C29428E666810B8903342)
+- [Correcting dt-spelling mistakes](https://gitlab.com/spelfouten/dutch-simpletransformers/)*
+
+and also achieved outstanding, near-sota results for:
 
-## Model description
+- [Natural language inference](https://arxiv.org/pdf/2101.05716.pdf)*
+- [Review classification](https://medium.com/broadhorizon-cmotions/nlp-with-r-part-5-state-of-the-art-in-nlp-transformers-bert-3449e3cd7494)*
 
-[RobBERT v2](https://github.com/iPieter/RobBERT) is a Dutch state-of-the-art [RoBERTa](https://arxiv.org/abs/1907.11692)-based language model.
+\* *Note that several evaluations use RobBERT-v1, and that the second and improved RobBERT-v2 outperforms this first model on everything we tested*
+
+*(Also note that this list is not exhaustive. If you used RobBERT for your application, we are happy to know about it! Send us a mail, or add it yourself to this list by sending a pull request with the edit!)*
+
+More in-depth information about RobBERT can be found in our [blog post](https://people.cs.kuleuven.be/~pieter.delobelle/robbert/), [our paper](https://arxiv.org/abs/2001.06286) and [the RobBERT Github repository](https://github.com/iPieter/RobBERT)
 
-More detailled information can be found in the [RobBERT paper](https://arxiv.org/abs/2001.06286).
 
 ## How to use
 
+RobBERT uses the [RoBERTa](https://arxiv.org/abs/1907.11692) architecture and pre-training but with a Dutch tokenizer and training data. RoBERTa is the robustly optimized English BERT model, making it even more powerful than the original BERT model. Given this same architecture, RobBERT can easily be finetuned and inferenced using [code to finetune RoBERTa](https://huggingface.co/transformers/model_doc/roberta.html) models and most code used for BERT models, e.g. as provided by [HuggingFace Transformers](https://huggingface.co/transformers/) library.
+
+By default, RobBERT has the masked language model head used in training. This can be used as a zero-shot way to fill masks in sentences. It can be tested out for free on [RobBERT's Hosted infererence API of Huggingface](https://huggingface.co/pdelobelle/robbert-v2-dutch-base?text=De+hoofdstad+van+Belgi%C3%AB+is+%3Cmask%3E.). You can also create a new prediction head for your own task by using any of HuggingFace's [RoBERTa-runners](https://huggingface.co/transformers/v2.7.0/examples.html#language-model-training), [their fine-tuning notebooks](https://huggingface.co/transformers/v4.1.1/notebooks.html) by changing the model name to `pdelobelle/robbert-v2-dutch-base`, or use the original fairseq [RoBERTa](https://github.com/pytorch/fairseq/tree/master/examples/roberta) training regimes.
+
+Use the following code to download the base model and finetune it yourself, or use one of our finetuned models (documented on  [our project site](https://people.cs.kuleuven.be/~pieter.delobelle/robbert/)).
+
 ```python
 from transformers import RobertaTokenizer, RobertaForSequenceClassification
 tokenizer = RobertaTokenizer.from_pretrained("pdelobelle/robbert-v2-dutch-base")
 model = RobertaForSequenceClassification.from_pretrained("pdelobelle/robbert-v2-dutch-base")
 ```
 
-## Performance Evaluation Results
+Starting with `transformers v2.4.0` (or installing from source), you can use AutoTokenizer and AutoModel.
+You can then use most of [HuggingFace's BERT-based notebooks](https://huggingface.co/transformers/v4.1.1/notebooks.html) for finetuning RobBERT on your type of Dutch language dataset.
 
-All experiments are described in more detail in our [paper](https://arxiv.org/abs/2001.06286).
+
+## Technical Details From The Paper
+
+
+### Our Performance Evaluation Results
+
+All experiments are described in more detail in our [paper](https://arxiv.org/abs/2001.06286), with the code in [our GitHub repository](https://github.com/iPieter/RobBERT).
 
 ### Sentiment analysis
 Predicting whether a review is positive or negative using the [Dutch Book Reviews Dataset](https://github.com/benjaminvdb/110kDBRD).
@@ -109,7 +151,7 @@ Using the [CoNLL 2002 evaluation script](https://www.clips.uantwerpen.be/conll20
 | RobBERT v2        | 89.08                    |
 
 
-## Training procedure
+## Pre-Training Procedure Details
 
 We pre-trained RobBERT using the RoBERTa training regime.
 We pre-trained our model on the Dutch section of the [OSCAR corpus](https://oscar-corpus.com/), a large multilingual corpus which was obtained by language classification in the Common Crawl corpus.
@@ -132,7 +174,7 @@ Using the [Fairseq library](https://github.com/pytorch/fairseq/tree/master/examp
 In between training jobs on the computing cluster, 2 Nvidia 1080 Ti's also covered some parameter updates for RobBERT v2.
 
 
-## Limitations and bias
+## Investigating Limitations and Bias
 
 In the [RobBERT paper](https://arxiv.org/abs/2001.06286), we also investigated potential sources of bias in RobBERT.
 
@@ -150,15 +192,35 @@ By augmenting the DBRB Dutch Book sentiment analysis dataset with the stated gen
 
 
 
-## BibTeX entry and citation info
+## How to Replicate Our Paper Experiments
+Replicating our paper experiments is [described in detail on teh RobBERT repository README](https://github.com/iPieter/RobBERT#how-to-replicate-our-paper-experiments).
+
+## Name Origin of RobBERT
+
+Most BERT-like models have the word *BERT* in their name (e.g. [RoBERTa](https://huggingface.co/transformers/model_doc/roberta.html), [ALBERT](https://arxiv.org/abs/1909.11942), [CamemBERT](https://camembert-model.fr/), and [many, many others](https://huggingface.co/models?search=bert)).
+As such, we queried our newly trained model using its masked language model to name itself *\<mask\>bert* using [all](https://huggingface.co/pdelobelle/robbert-v2-dutch-base?text=Mijn+naam+is+%3Cmask%3Ebert.) [kinds](https://huggingface.co/pdelobelle/robbert-v2-dutch-base?text=Hallo%2C+ik+ben+%3Cmask%3Ebert.) [of](https://huggingface.co/pdelobelle/robbert-v2-dutch-base?text=Leuk+je+te+ontmoeten%2C+ik+heet+%3Cmask%3Ebert.) [prompts](https://huggingface.co/pdelobelle/robbert-v2-dutch-base?text=Niemand+weet%2C+niemand+weet%2C+dat+ik+%3Cmask%3Ebert+heet.), and it consistently called itself RobBERT.
+We thought it was really quite fitting, given that RobBERT is a [*very* Dutch name](https://en.wikipedia.org/wiki/Robbert) *(and thus clearly a Dutch language model)*, and additionally has a high similarity to its root architecture, namely [RoBERTa](https://huggingface.co/transformers/model_doc/roberta.html).
+
+Since *"rob"* is a Dutch words to denote a seal, we decided to draw a seal and dress it up like [Bert from Sesame Street](https://muppet.fandom.com/wiki/Bert) for the [RobBERT logo](https://github.com/iPieter/RobBERT/blob/master/res/robbert_logo.png).
+
+## Credits and citation
+
+This project is created by [Pieter Delobelle](https://people.cs.kuleuven.be/~pieter.delobelle), [Thomas Winters](https://thomaswinters.be) and [Bettina Berendt](https://people.cs.kuleuven.be/~bettina.berendt/).
+If you would like to cite our paper or model, you can use the following BibTeX:
 
-```bibtex
-@misc{delobelle2020robbert,
-    title={RobBERT: a Dutch RoBERTa-based Language Model},
-    author={Pieter Delobelle and Thomas Winters and Bettina Berendt},
-    year={2020},
-    eprint={2001.06286},
-    archivePrefix={arXiv},
-    primaryClass={cs.CL}
-}
 ```
+@inproceedings{delobelle2020robbert,
+    title = "{R}ob{BERT}: a {D}utch {R}o{BERT}a-based {L}anguage {M}odel",
+    author = "Delobelle, Pieter  and
+      Winters, Thomas  and
+      Berendt, Bettina",
+    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2020",
+    month = nov,
+    year = "2020",
+    address = "Online",
+    publisher = "Association for Computational Linguistics",
+    url = "https://www.aclweb.org/anthology/2020.findings-emnlp.292",
+    doi = "10.18653/v1/2020.findings-emnlp.292",
+    pages = "3255--3265"
+}
+```