Update README.md

README.md

@@ -5,6 +5,7 @@ tags:
 - albertina-pt*
 - albertina-ptpt
 - albertina-ptbr
+- albertina-ptbr-nobrwac
 - fill-mask
 - bert
 - deberta
@@ -13,7 +14,6 @@ tags:
 - foundation model
 license: other
 datasets:
-- brwac
 - PORTULAN/glue-ptpt
 - assin2
 - dlb/plue
@@ -24,14 +24,8 @@ widget:
 ---
 
 
-# Albertina PT-BR
+# Albertina PT-BR No-brWaC
 
----
-<img align="left" width="40" height="40" src="https://github.githubassets.com/images/icons/emoji/unicode/1f917.png">
-<p style="text-align: center;">&nbsp;&nbsp;&nbsp;&nbsp;<b>We just released</b> the base models and <b>Albertina PT-BR nobrwac</b>, 
-  trained on a data set with most permissive license.</p>
-
----
 
 **Albertina PT-*** is a foundation, large language model for the **Portuguese language**.
 
@@ -73,10 +67,11 @@ Please use the above cannonical reference when using or citing this model.
 
 # Model Description
 
-**This model card is for Albertina-PT-BR-V2**, with 900M parameters, 24 layers and a hidden size of 1536.
+**This model card is for Albertina-PT-BR No-brWaC**, with 900M parameters, 24 layers and a hidden size of 1536.
 
-TODO:Licence
+Albertina-PT-BR No-brWaC is distributed under an [MIT license](https://huggingface.co/PORTULAN/albertina-ptpt/blob/main/LICENSE).
 
+DeBERTa is distributed under an [MIT license](https://github.com/microsoft/DeBERTa/blob/master/LICENSE).
 
 
 <br>
@@ -84,24 +79,15 @@ TODO:Licence
 # Training Data
 
 
-**Albertina PT-BR** was trained over the 2.7 billion token [BrWac](https://huggingface.co/datasets/brwac) data set.
-
-TODO:
-**Albertina PT-BR-V2** was trained with [OSCAR](https://huggingface.co/datasets/oscar-corpus/OSCAR-2301): the OSCAR data set includes documents in more than one hundred languages, including Portuguese, and it is widely used in the literature. It is the result of a selection performed over the [Common Crawl](https://commoncrawl.org/) data set, crawled from the Web, that retains only pages whose metadata indicates permission to be crawled, that performs deduplication, and that removes some boilerplate, among other filters. Given that it does not discriminate between the Portuguese variants, we performed extra filtering by retaining only documents whose meta-data indicate the Internet country code top-level domain of Brazil. We used the January 2023 version of OSCAR, which is based on the November/December 2022 version of Common Crawl.
-
-
-[**Albertina PT-PT**](https://huggingface.co/PORTULAN/albertina-ptpt), in turn, was trained over a 2.2 billion token data set that resulted from gathering some openly available corpora of European Portuguese from the following sources:
-
-- [OSCAR](https://huggingface.co/datasets/oscar-corpus/OSCAR-2301): the OSCAR data set includes documents in more than one hundred languages, including Portuguese, and it is widely used in the literature. It is the result of a selection performed over the [Common Crawl](https://commoncrawl.org/) data set, crawled from the Web, that retains only pages whose metadata indicates permission to be crawled, that performs deduplication, and that removes some boilerplate, among other filters. Given that it does not discriminate between the Portuguese variants, we performed extra filtering by retaining only documents whose meta-data indicate the Internet country code top-level domain of Portugal. We used the January 2023 version of OSCAR, which is based on the November/December 2022 version of Common Crawl.
-- [DCEP](https://joint-research-centre.ec.europa.eu/language-technology-resources/dcep-digital-corpus-european-parliament_en): the Digital Corpus of the European Parliament is a multilingual corpus including documents in all official EU languages published on the European Parliament&#39;s official website. We retained its European Portuguese portion.
-- [Europarl](https://www.statmt.org/europarl/): the European Parliament Proceedings Parallel Corpus is extracted from the proceedings of the European Parliament from 1996 to 2011. We retained its European Portuguese portion.
-- [ParlamentoPT](https://huggingface.co/datasets/PORTULAN/parlamento-pt): the ParlamentoPT is a data set we obtained by gathering the publicly available documents with the transcription of the debates in the Portuguese Parliament.
-
-
+**Albertina PT-BR No-brWac** was trained over a 3.7 billion token curated selection of documents from the [OSCAR](https://huggingface.co/datasets/oscar-corpus/OSCAR-2301) data set.
+The OSCAR data set includes documents in more than one hundred languages, including Portuguese, and it is widely used in the literature. 
+It is the result of a selection performed over the [Common Crawl](https://commoncrawl.org/) data set, crawled from the Web, that retains only pages whose metadata indicates permission to be crawled, that performs deduplication, and that removes some boilerplate, among other filters. 
+Given that it does not discriminate between the Portuguese variants, we performed extra filtering by retaining only documents whose meta-data indicate the Internet country code top-level domain of Brazil. 
+We used the January 2023 version of OSCAR, which is based on the November/December 2022 version of Common Crawl.
 
 ## Preprocessing
 
-We filtered the PT-PT corpora using the [BLOOM pre-processing](https://github.com/bigscience-workshop/data-preparation) pipeline, resulting in a data set of 8 million documents, containing around 2.2 billion tokens.
+We filtered the PT-BR corpora using the [BLOOM pre-processing](https://github.com/bigscience-workshop/data-preparation) pipeline.
 We skipped the default filtering of stopwords since it would disrupt the syntactic structure, and also the filtering for language identification given the corpus was pre-selected as Portuguese.
 
 
@@ -109,27 +95,12 @@ We skipped the default filtering of stopwords since it would disrupt the syntact
 
 As codebase, we resorted to the [DeBERTa V2 XLarge](https://huggingface.co/microsoft/deberta-v2-xlarge), for English.
 
-To train **Albertina-PT-BR** the BrWac data set was tokenized with the original DeBERTA tokenizer with a 128 token sequence truncation and dynamic padding.
-The model was trained using the maximum available memory capacity resulting in a batch size of 896 samples (56 samples per GPU without gradient accumulation steps).
-We chose a learning rate of 1e-5 with linear decay and 10k warm-up steps based on the results of exploratory experiments. 
-In total, around 200k training steps were taken across 50 epochs.
-The model was trained for 1 day and 11 hours on a2-megagpu-16gb Google Cloud A2 VMs with 16 GPUs, 96 vCPUs and 1.360 GB of RAM.
-
-TODO:
-To train **Albertina-PT-BR-nobrwac** the OSCAR data set was tokenized with the original DeBERTA tokenizer with a 128 token sequence truncation and dynamic padding.
-The model was trained using the maximum available memory capacity resulting in a batch size of 896 samples (56 samples per GPU without gradient accumulation steps).
-We chose a learning rate of 1e-5 with linear decay and 10k warm-up steps based on the results of exploratory experiments. 
+To train [**Albertina PT-PT No-brWac**](https://huggingface.co/PORTULAN/albertina-ptpt-nobrwac), the data set was tokenized with the original DeBERTa tokenizer with a 128 token sequence truncation and dynamic padding.
+The model was trained using the maximum available memory capacity resulting in a batch size of 896 samples (56 samples per GPU).
+We chose a learning rate of 1e-5 with linear decay and 10k warm-up steps. 
 In total, around 200k training steps were taken across 50 epochs.
 The model was trained for 1 day and 13 hours on a2-megagpu-16gb Google Cloud A2 VMs with 16 GPUs, 96 vCPUs and 1.360 GB of RAM.
 
-
-To train [**Albertina PT-PT**](https://huggingface.co/PORTULAN/albertina-ptpt), the data set was tokenized with the original DeBERTa tokenizer with a 128 token sequence truncation and dynamic padding.
-The model was trained using the maximum available memory capacity resulting in a batch size of 832 samples (52 samples per GPU and applying gradient accumulation in order to approximate the batch size of the PT-BR model).
-Similarly to the PT-BR variant above, we opted for a learning rate of 1e-5 with linear decay and 10k warm-up steps.
-However, since the number of training examples is approximately twice of that in the PT-BR variant, we reduced the number of training epochs to half and completed only 25 epochs, which resulted in approximately 245k steps.
-The model was trained for 3 days on a2-highgpu-8gb Google Cloud A2 VMs with 8 GPUs, 96 vCPUs and 680 GB of RAM.
-
-
 <br>
 
 # Evaluation
@@ -137,7 +108,7 @@ The model was trained for 3 days on a2-highgpu-8gb Google Cloud A2 VMs with 8 GP
 The two model versions were evaluated on downstream tasks organized into two groups.
 
 In one group, we have the two data sets from the [ASSIN 2 benchmark](https://huggingface.co/datasets/assin2), namely STS and RTE, that were used to evaluate the previous state-of-the-art model [BERTimbau Large](https://huggingface.co/neuralmind/bert-large-portuguese-cased).
-In the other group of data sets, we have the translations into PT-BR and PT-PT of the English data sets used for a few of the tasks in the widely-used [GLUE benchmark](https://huggingface.co/datasets/glue), which allowed us to test both Albertina-PT-* variants on a wider variety of downstream tasks.
+In the other group of data sets, we have the translations into PT-BR of the English data sets used for a few of the tasks in the widely-used [GLUE benchmark](https://huggingface.co/datasets/glue), which allowed us to test both Albertina-PT-* variants on a wider variety of downstream tasks.
 
 
 ## ASSIN 2
@@ -145,10 +116,11 @@ In the other group of data sets, we have the translations into PT-BR and PT-PT o
 [ASSIN 2](https://huggingface.co/datasets/assin2) is a **PT-BR data** set of approximately 10.000 sentence pairs, split into 6.500 for training, 500 for validation, and 2.448 for testing, annotated with semantic relatedness scores (range 1 to 5) and with binary entailment judgments.
 This data set supports the task of semantic textual similarity (STS), which consists of assigning a score of how semantically related two sentences are; and the task of recognizing textual entailment (RTE), which given a pair of sentences, consists of determining whether the first entails the second.
 
-| Model               | RTE (Accuracy) | STS (Pearson)|
-|---------------------|----------------|--------------|
-| **Albertina-PT-BR** | **0.9130**     | **0.8676**   |
-| BERTimbau-large     | 0.8913         | 0.8531       |
+| Model                        | RTE (Accuracy) | STS (Pearson)|
+|------------------------------|----------------|--------------|
+| **Albertina-PT-BR**          | **0.9130**     | **0.8676**   |
+| **Albertina-PT-BR No-brWaC** | 0.8950         | 0.8547       |
+| BERTimbau-large              | 0.8913         | 0.8531       |
 
 
 ## GLUE tasks translated
@@ -159,22 +131,15 @@ We address four tasks from those in PLUE, namely:
 - and two inference tasks: RTE, for recognizing textual entailment and WNLI, for coreference and natural language inference.
   
 
-| Model               | RTE (Accuracy) | WNLI (Accuracy)| MRPC (F1) | STS-B (Pearson) |
-|---------------------|----------------|----------------|-----------|-----------------|
-| **Albertina-PT-BR** | 0.7545         | 0.4601         | 0.9071    | **0.8910**      | 
-| BERTimbau-large     | 0.6546         | **0.5634**     | 0.887     | 0.8842          |
+| Model                        | RTE (Accuracy) | WNLI (Accuracy)| MRPC (F1) | STS-B (Pearson) |
+|------------------------------|----------------|----------------|-----------|-----------------|
+| **Albertina-PT-BR No-brWaC** | 0.7798         | 0.5070         | **0.9167**| 0.8743 
+| **Albertina-PT-BR**          | 0.7545         | 0.4601         | 0.9071    | **0.8910**      | 
+| BERTimbau-large              | 0.6546         | **0.5634**     | 0.8870    | 0.8842          |
 | |   |      |  |           |
-| **Albertina-PT-PT** | **0.7960**     | 0.4507         | **0.9151**| 0.8799          |
+| **Albertina-PT-PT**          | **0.7960**     | 0.4507         | 0.9151    | 0.8799          |
 
 
-We resorted to [GLUE-PT](https://huggingface.co/datasets/PORTULAN/glue-ptpt), a **PT-PT version of the GLUE** benchmark. 
-We automatically translated the same four tasks from GLUE using [DeepL Translate](https://www.deepl.com/), which specifically provides translation from English to PT-PT as an option.
-
-| Model               | RTE (Accuracy) | WNLI (Accuracy)| MRPC (F1) | STS-B (Pearson) |
-|---------------------|----------------|----------------|-----------|-----------------|
-| **Albertina-PT-PT** | **0.8339**     | **0.4225**     | **0.9171**| 0.8801          | 
-| |   |      |  |           |
-| **Albertina-PT-BR** | 0.7942         | 0.4085         | 0.9048    | **0.8847**      |
 
 <br>
 
@@ -184,14 +149,14 @@ You can use this model directly with a pipeline for masked language modeling:
 
 ```python
 >>> from transformers import pipeline
->>> unmasker = pipeline('fill-mask', model='PORTULAN/albertina-ptbr')
+>>> unmasker = pipeline('fill-mask', model='PORTULAN/albertina-ptbr-nobrwac')
 >>> unmasker("A culinária brasileira é rica em sabores e [MASK], tornando-se um dos maiores patrimônios do país.")
 
-[{'score': 0.6145166158676147, 'token': 23395, 'token_str': 'aromas', 'sequence': 'A culinária brasileira é rica em sabores e aromas, tornando-se um dos maiores patrimônios do país.'},
-{'score': 0.1720353364944458, 'token': 21925, 'token_str': 'cores', 'sequence': 'A culinária brasileira é rica em sabores e cores, tornando-se um dos maiores patrimônios do país.'},
-{'score': 0.1438736468553543, 'token': 10392, 'token_str': 'costumes', 'sequence': 'A culinária brasileira é rica em sabores e costumes, tornando-se um dos maiores patrimônios do país.'},
-{'score': 0.02997930906713009, 'token': 117371, 'token_str': 'cultura', 'sequence': 'A culinária brasileira é rica em sabores e cultura, tornando-se um dos maiores patrimônios do país.'},
-{'score': 0.015540072694420815, 'token': 22647, 'token_str': 'nuances', 'sequence': 'A culinária brasileira é rica em sabores e nuances, tornando-se um dos maiores patrimônios do país.'}]
+[{'score': 0.3866911828517914, 'token': 23395, 'token_str': 'aromas', 'sequence': 'A culinária brasileira é rica em sabores e aromas, tornando-se um dos maiores patrimônios do país.'},
+{'score': 0.2926434874534607, 'token': 10392, 'token_str': 'costumes', 'sequence': 'A culinária brasileira é rica em sabores e costumes, tornando-se um dos maiores patrimônios do país.'},
+{'score': 0.1913347691297531, 'token': 21925, 'token_str': 'cores', 'sequence': 'A culinária brasileira é rica em sabores e cores, tornando-se um dos maiores patrimônios do país.'},
+{'score': 0.06453365087509155, 'token': 117371, 'token_str': 'cultura', 'sequence': 'A culinária brasileira é rica em sabores e cultura, tornando-se um dos maiores patrimônios do país.'},
+{'score': 0.019388679414987564, 'token': 22647, 'token_str': 'nuances', 'sequence': 'A culinária brasileira é rica em sabores e nuances, tornando-se um dos maiores patrimônios do país.'}]
 
 
 ```
@@ -202,7 +167,7 @@ The model can be used by fine-tuning it for a specific task:
 >>> from transformers import AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainer
 >>> from datasets import load_dataset
 
->>> model = AutoModelForSequenceClassification.from_pretrained("PORTULAN/albertina-ptbr", num_labels=2)
+>>> model = AutoModelForSequenceClassification.from_pretrained("PORTULAN/albertina-ptbr-nobrwac", num_labels=2)
 >>> tokenizer = AutoTokenizer.from_pretrained("PORTULAN/albertina-ptbr")
 >>> dataset = load_dataset("PORTULAN/glue-ptpt", "rte")