jarodrigues commited on
Commit
1005f1e
1 Parent(s): 7f89d8c

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -76,7 +76,7 @@ DeBERTa is distributed under an [MIT license](https://github.com/microsoft/DeBER
76
 
77
  # Training Data
78
 
79
- **Albertina PT-PT Base** was trained over a 2.2 billion token data set that resulted from gathering some openly available corpora of European Portuguese from the following sources:
80
 
81
  - [OSCAR](https://huggingface.co/datasets/oscar-corpus/OSCAR-2301): the OSCAR data set includes documents in more than one hundred languages, including Portuguese, and it is widely used in the literature. It is the result of a selection performed over the [Common Crawl](https://commoncrawl.org/) data set, crawled from the Web, that retains only pages whose metadata indicates permission to be crawled, that performs deduplication, and that removes some boilerplate, among other filters. Given that it does not discriminate between the Portuguese variants, we performed extra filtering by retaining only documents whose meta-data indicate the Internet country code top-level domain of Portugal. We used the January 2023 version of OSCAR, which is based on the November/December 2022 version of Common Crawl.
82
  - [DCEP](https://joint-research-centre.ec.europa.eu/language-technology-resources/dcep-digital-corpus-european-parliament_en): the Digital Corpus of the European Parliament is a multilingual corpus including documents in all official EU languages published on the European Parliament's official website. We retained its European Portuguese portion.
@@ -97,8 +97,8 @@ We skipped the default filtering of stopwords since it would disrupt the syntact
97
 
98
  As codebase, we resorted to the [DeBERTa V1 Base](https://huggingface.co/microsoft/deberta-base), for English.
99
 
100
- To train **Albertina-PT-PT Base**, the data set was tokenized with the original DeBERTa tokenizer with a 128 token sequence truncation and dynamic padding.
101
- The model was trained using the maximum available memory capacity resulting in a batch size of 3072 samples (192 samples per GPU and applying gradient accumulation in order to approximate the batch size of the PT-BR model).
102
  We opted for a learning rate of 1e-5 with linear decay and 10k warm-up steps.
103
  A total of 200 training epochs were performed resulting in approximately 180k steps.
104
  The model was trained for one day on a2-megagpu-16gb Google Cloud A2 VMs with 16 GPUs, 96 vCPUs and 1.360 GB of RAM.
 
76
 
77
  # Training Data
78
 
79
+ [**Albertina-PT-PT Base**](https://huggingface.co/PORTULAN/albertina-ptpt-base) was trained over a 2.2 billion token data set that resulted from gathering some openly available corpora of European Portuguese from the following sources:
80
 
81
  - [OSCAR](https://huggingface.co/datasets/oscar-corpus/OSCAR-2301): the OSCAR data set includes documents in more than one hundred languages, including Portuguese, and it is widely used in the literature. It is the result of a selection performed over the [Common Crawl](https://commoncrawl.org/) data set, crawled from the Web, that retains only pages whose metadata indicates permission to be crawled, that performs deduplication, and that removes some boilerplate, among other filters. Given that it does not discriminate between the Portuguese variants, we performed extra filtering by retaining only documents whose meta-data indicate the Internet country code top-level domain of Portugal. We used the January 2023 version of OSCAR, which is based on the November/December 2022 version of Common Crawl.
82
  - [DCEP](https://joint-research-centre.ec.europa.eu/language-technology-resources/dcep-digital-corpus-european-parliament_en): the Digital Corpus of the European Parliament is a multilingual corpus including documents in all official EU languages published on the European Parliament's official website. We retained its European Portuguese portion.
 
97
 
98
  As codebase, we resorted to the [DeBERTa V1 Base](https://huggingface.co/microsoft/deberta-base), for English.
99
 
100
+ To train [**Albertina-PT-PT Base**](https://huggingface.co/PORTULAN/albertina-ptpt-base), the data set was tokenized with the original DeBERTa tokenizer with a 128 token sequence truncation and dynamic padding.
101
+ The model was trained using the maximum available memory capacity resulting in a batch size of 3072 samples (192 samples per GPU).
102
  We opted for a learning rate of 1e-5 with linear decay and 10k warm-up steps.
103
  A total of 200 training epochs were performed resulting in approximately 180k steps.
104
  The model was trained for one day on a2-megagpu-16gb Google Cloud A2 VMs with 16 GPUs, 96 vCPUs and 1.360 GB of RAM.