eduagarcia
commited on
Update README.md
Browse files
README.md
CHANGED
@@ -116,30 +116,34 @@ We adopted the standard [RoBERTa hyperparameters](https://arxiv.org/abs/1907.116
|
|
116 |
|
117 |
## Evaluation
|
118 |
|
119 |
-
<!-- This section describes the evaluation protocols and provides the results. -->
|
120 |
-
|
121 |
-
### Testing Data, Factors & Metrics
|
122 |
-
|
123 |
-
#### Testing Data
|
124 |
-
|
125 |
The model was evaluated on ["PortuLex" benchmark](eduagarcia/portuguese_benchmark), a four-task benchmark designed to evaluate the quality and performance of language models in the Portuguese legal domain.
|
126 |
|
127 |
-
|
128 |
-
|
129 |
-
|
130 |
-
|
131 |
-
|
132 |
-
|
133 |
-
|
134 |
-
|
135 |
-
[
|
136 |
-
|
137 |
-
|
138 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
139 |
|
140 |
## Citation
|
141 |
|
142 |
-
|
143 |
[More Information Needed]
|
144 |
|
145 |
## Acknowledgment
|
|
|
|
|
|
116 |
|
117 |
## Evaluation
|
118 |
|
|
|
|
|
|
|
|
|
|
|
|
|
119 |
The model was evaluated on ["PortuLex" benchmark](eduagarcia/portuguese_benchmark), a four-task benchmark designed to evaluate the quality and performance of language models in the Portuguese legal domain.
|
120 |
|
121 |
+
Macro F1-Score (\%) for multiple models evaluated on PortuLex benchmark test splits:
|
122 |
+
|
123 |
+
| **Model** | **LeNER** | **UlyNER-PL** | **FGV-STF** | **RRIP** | **Average (%)** |
|
124 |
+
|----------------------------------------------------------------------------|-----------|-----------------|-------------|:---------:|-----------------|
|
125 |
+
| | | Coarse/Fine | Coarse | | |
|
126 |
+
| [BERTimbau-base](https://dl.acm.org/doi/abs/10.1007/978-3-030-61377-8_28) | 88.34 | 86.39/83.83 | 79.34 | 82.34 | 83.78 |
|
127 |
+
| [BERTimbau-large](https://dl.acm.org/doi/abs/10.1007/978-3-030-61377-8_28) | 88.64 | 87.77/84.74 | 79.71 | **83.79** | 84.60 |
|
128 |
+
| [Albertina-PT-BR-base](https://arxiv.org/abs/2305.06721) | 89.26 | 86.35/84.63 | 79.30 | 81.16 | 83.80 |
|
129 |
+
| [Albertina-PT-BR-xlarge](https://arxiv.org/abs/2305.06721) | 90.09 | 88.36/**86.62** | 79.94 | 82.79 | 85.08 |
|
130 |
+
| [BERTikal-base](https://arxiv.org/abs/2110.15709) | 83.68 | 79.21/75.70 | 77.73 | 81.11 | 79.99 |
|
131 |
+
| [JurisBERT-base](https://repositorio.ufms.br/handle/123456789/5119) | 81.74 | 81.67/77.97 | 76.04 | 80.85 | 79.61 |
|
132 |
+
| [BERTimbauLAW-base](https://repositorio.ufms.br/handle/123456789/5119) | 84.90 | 87.11/84.42 | 79.78 | 82.35 | 83.20 |
|
133 |
+
| [Legal-XLM-R-base](https://arxiv.org/abs/2306.02069) | 87.48 | 83.49/83.16 | 79.79 | 82.35 | 83.24 |
|
134 |
+
| [Legal-XLM-R-large](https://arxiv.org/abs/2306.02069) | 88.39 | 84.65/84.55 | 79.36 | 81.66 | 83.50 |
|
135 |
+
| [Legal-RoBERTa-PT-large](https://arxiv.org/abs/2306.02069) | 87.96 | 88.32/84.83 | 79.57 | 81.98 | 84.02 |
|
136 |
+
| RoBERTaTimbau-base | 89.68 | 87.53/85.74 | 78.82 | 82.03 | 84.29 |
|
137 |
+
| RoBERTaLegalPT-base | 90.59 | 85.45/84.40 | 79.92 | 82.84 | 84.57 |
|
138 |
+
| RoBERTaLexPT-base | **90.73** | **88.56**/86.03 | **80.40** | 83.22 | **85.41** |
|
139 |
+
|
140 |
+
In summary, RoBERTaLexPT consistently achieves top legal NLP effectiveness despite its base size.
|
141 |
+
With sufficient pre-training data, it can surpass overparameterized models. The results highlight the importance of domain-diverse training data over sheer model scale.
|
142 |
|
143 |
## Citation
|
144 |
|
|
|
145 |
[More Information Needed]
|
146 |
|
147 |
## Acknowledgment
|
148 |
+
|
149 |
+
This work has been supported by the AI Center of Excellence (Centro de Excelência em Inteligência Artificial – CEIA) of the Institute of Informatics at the Federal University of Goiás (INF-UFG).
|