README.md

---
language: ca
license: apache-2.0
tags:
- "catalan"
- "masked-lm"
- "distilroberta"
widget:
- text: "El Català és una llengua molt <mask>."
- text: "Salvador Dalí va viure a <mask>."
- text: "La Costa Brava té les millors <mask> d'Espanya."
- text: "El cacaolat és un batut de <mask>."
- text: "<mask> és la capital de la Garrotxa."
- text: "Vaig al <mask> a buscar bolets."
- text: "Antoni Gaudí vas ser un <mask> molt important per la ciutat."
- text: "Catalunya és una referència en <mask> a nivell europeu."
---

# DistilRoBERTa-base-ca

## Overview
- **Architecture:** DistilRoBERTa-base
- **Language:** Catalan
- **Task:** Fill-Mask
- **Data:** Crawling

## Model description

This model is a distilled version of [projecte-aina/roberta-base-ca-v2](https://huggingface.co/projecte-aina/roberta-base-ca-v2). 
It follows the same training procedure as [DistilBERT](https://arxiv.org/abs/1910.01108), using the implementation of Knowledge Distillation 
from the paper's [official repository](https://github.com/huggingface/transformers/tree/main/examples/research_projects/distillation).

The resulting architecture consists of 6 layers, 768 dimensional embeddings and 12 attention heads. 
This adds up to a total of 82M parameters, which is considerably less than the 125M of standard RoBERTa-base models. 
This makes the model lighter and faster than the original, at the cost of a slightly lower performance.

## Training

### Training procedure

This model has been trained using a technique known as Knowledge Distillation, which is used to shrink networks to a reasonable size while minimizing the loss in performance.

It basically consists in distilling a large language model (the teacher) into a more lightweight, energy-efficient, and production-friendly model (the student).

So, in a “teacher-student learning” setup, a relatively small student model is trained to mimic the behavior of a larger teacher model. As a result, the student has lower inference time and the ability to run in commodity hardware.

### Training data

The training corpus consists of several corpora gathered from web crawling and public corpora, as shown in the table below:

| Corpus                   | Size (GB)  |
|--------------------------|------------|
| Catalan Crawling         | 13.00      |
| RacoCatalá               | 8.10       |
| Catalan Oscar            | 4.00       |
| CaWaC                    | 3.60       |
| Cat. General Crawling    | 2.50       |
| Wikipedia                | 1.10       |
| DOGC                     | 0.78       |
| Padicat                  | 0.63       |
| ACN                      | 0.42       |
| Nació Digital            | 0.42       |
| Cat. Government Crawling | 0.24       |
| Vilaweb                  | 0.06       |
| Catalan Open Subtitles   | 0.02       |
| Tweets                   | 0.02       |

## Evaluation

### Evaluation benchmark

This model has been fine-tuned on the downstream tasks of the [Catalan Language Understanding Evaluation benchmark (CLUB)](https://club.aina.bsc.es/), which includes the following datasets:

| Dataset   | Task| Total   | Train  | Dev   | Test  |
|:----------|:----|:--------|:-------|:------|:------|
| AnCora    | NER | 13,581  | 10,628 | 1,427 | 1,526 |
| AnCora    | POS | 16,678  | 13,123 | 1,709 | 1,846 |
| STS-ca    | STS | 3,073   | 2,073  | 500   | 500   |
| TeCla     | TC  | 137,775 | 110,203| 13,786| 13,786|
| TE-ca     | RTE | 21,163  | 16,930 | 2,116 | 2,117 |
| CatalanQA | QA  | 21,427  | 17,135 | 2,157 | 2,135 |
| XQuAD-ca  | QA  |   -     |   -    |   -   | 1,189 |

### Evaluation results

This is how it compares to its teacher when fine-tuned on the aforementioned downstream tasks:

|      Model  \  Task     |NER (F1)|POS (F1)|STS-ca (Comb.)|TeCla (Acc.)|TEca (Acc.)|CatalanQA (F1/EM)| XQuAD-ca <sup>1</sup> (F1/EM) | 
| ------------------------|:-------|:-------|:-------------|:-----------|:----------|:----------------|:------------------------------|
| RoBERTa-base-ca-v2      | 89.29  | 98.96  | 79.07        | 74.26      | 83.14     | 89.50/76.63     | 73.64/55.42                   |
| DistilRoBERTa-base-ca   | 87.88  | 98.83  | 77.26        | 73.20      | 76.00     | 84.07/70.77     | 62.93/45.08                   |

<sup>1</sup> : Trained on CatalanQA, tested on XQuAD-ca.