mrm8488
/

electricidad-base-discriminator

@@ -1,13 +1,18 @@
 ---
 language: es
 thumbnail: https://i.imgur.com/uxAvBfh.png
 ---
 ## ELECTRICIDAD: The Spanish Electra [Imgur](https://imgur.com/uxAvBfh)
-**Electricidad-base-discriminator** (uncased) is a ```base``` Electra like model (discriminator in this case) trained on a + 20 GB of  the [OSCAR](https://oscar-corpus.com/) Spanish corpus.
 As mentioned in the original [paper](https://openreview.net/pdf?id=r1xMH1BtvB):
 **ELECTRA** is a new method for self-supervised language representation learning. It can be used to pre-train transformer networks using relatively little compute. ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a [GAN](https://arxiv.org/pdf/1406.2661.pdf). At small scale, ELECTRA achieves strong results even when trained on a single GPU. At large scale, ELECTRA achieves state-of-the-art results on the [SQuAD 2.0](https://rajpurkar.github.io/SQuAD-explorer/) dataset.
@@ -19,8 +24,8 @@ For a detailed description and experimental results, please refer the paper [ELE
 |Name| # Value|
 |-----|--------|
-|Layers|	12   |
-|Hidden |768 	|
 |Params| 110M|
 ## Evaluation metrics (for discriminator) 🧾
@@ -70,7 +75,18 @@ As you can see there are **1s** in the places where the model detected a fake to
 [NER](https://huggingface.co/mrm8488/electricidad-base-finetuned-ner)
-[Paraphrase Identification](https://huggingface.co/mrm8488/RuPERTa-base-finetuned-pawsx-es)

 ---
 language: es
 thumbnail: https://i.imgur.com/uxAvBfh.png
+tags:
+ - Spanish
+ - Electra
+datasets:
+ -large_spanish_corpus
 ---
 ## ELECTRICIDAD: The Spanish Electra [Imgur](https://imgur.com/uxAvBfh)
+**Electricidad-base-discriminator** (uncased) is a ```base``` Electra like model (discriminator in this case) trained on a [Large Spanish Corpus](https://github.com/josecannete/spanish-corpora) (aka BETO's corpus)
 As mentioned in the original [paper](https://openreview.net/pdf?id=r1xMH1BtvB):
 **ELECTRA** is a new method for self-supervised language representation learning. It can be used to pre-train transformer networks using relatively little compute. ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a [GAN](https://arxiv.org/pdf/1406.2661.pdf). At small scale, ELECTRA achieves strong results even when trained on a single GPU. At large scale, ELECTRA achieves state-of-the-art results on the [SQuAD 2.0](https://rajpurkar.github.io/SQuAD-explorer/) dataset.
 |Name| # Value|
 |-----|--------|
+|Layers|\t12   |
+|Hidden |768 \t|
 |Params| 110M|
 ## Evaluation metrics (for discriminator) 🧾
 [NER](https://huggingface.co/mrm8488/electricidad-base-finetuned-ner)
+### Spanish LM model comparison 📊
+| Dataset     | Metric   | RoBERTa-b | RoBERTa-l | BETO   | mBERT  | BERTIN | Electricidad-b |
+|-------------|----------|-----------|-----------|--------|--------|--------|---------|
+| UD-POS      | F1       | 0.9907    | 0.9901    | 0.9900 | 0.9886 | 0.9904 | 0.9818  |
+| Conll-NER   | F1       | 0.8851    | 0.8772    | 0.8759 | 0.8691 | 0.8627 | 0.7954  |
+| Capitel-POS | F1       | 0.9846    | 0.9851    | 0.9836 | 0.9839 | 0.9826 | 0.9816  |
+| Capitel-NER | F1       | 0.8959    | 0.8998    | 0.8771 | 0.8810 | 0.8741 | 0.8035  |
+| STS         | Combined | 0.8423    | 0.8420    | 0.8216 | 0.8249 | 0.7822 | 0.8065  |
+| MLDoc       | Accuracy | 0.9595    | 0.9600    | 0.9650 | 0.9560 | 0.9673 | 0.9490  |
+| PAWS-X      | F1       | 0.9035    | 0.9000    | 0.8915 | 0.9020 | 0.8820 | **0.9045**  |
+| XNLI        | Accuracy | 0.8016    | 0.7958    | 0.8130 | 0.7876 | 0.7864 | 0.7878  |