|
--- |
|
language: |
|
- ar |
|
- en |
|
- multilingual |
|
license: mit |
|
tags: |
|
- bert |
|
- roberta |
|
- exbert |
|
datasets: |
|
- arabic_billion_words |
|
- cc100 |
|
- gigaword |
|
- oscar |
|
- wikipedia |
|
--- |
|
|
|
# An English-Arabic Bilingual Encoder |
|
|
|
``` |
|
from transformers import AutoModelForMaskedLM, AutoTokenizer |
|
tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/roberta-large-eng-ara-128k") |
|
model = AutoModelForMaskedLM.from_pretrained("jhu-clsp/roberta-large-eng-ara-128k") |
|
``` |
|
|
|
`roberta-large-eng-ara-128k` is an English�Arabic bilingual encoders of 24-layer Transformers (d\_model= 1024), the same size as XLM-R large. We use the same Common Crawl corpus as XLM-R for pretraining. Additionally, we also use English and Arabic Wikipedia, Arabic Gigaword (Parker et al., 2011), Arabic OSCAR (Ortiz Su�rez et al., 2020), Arabic News Corpus (El-Khair, 2016), and Arabic OSIAN (Zeroual et al.,2019). In total, we train with 9.2B words of Arabic text and 26.8B words of English text, more than either XLM-R (2.9B words/23.6B words) or GigaBERT v4 (Lan et al., 2020) (4.3B words/6.1B words). We build an English�Arabic joint vocabulary using SentencePiece (Kudo and Richardson, 2018) with size of 128K. We additionally enforce coverage of all Arabic characters after normalization. |
|
|
|
## Pretraining Detail |
|
|
|
We pretrain each encoder with a batch size of 2048 sequences and 512 sequence length for 250K steps from scratch roughly 1/24 the amount of pretraining compute of XLM-R. Training takes 8 RTX6000 GPUs roughly three weeks. We follow the pretraining recipe of RoBERTa (Liu et al., 2019) and XLM-R. We omit the next sentence prediction task and use a learning rate of 2e-4, Adam optimizer, and linear warmup of 10K steps then decay linearly to 0, multilingual sampling alpha of 0.3, and the fairseq (Ott et al., 2019) implementation. |
|
|
|
## Citation |
|
|
|
Please cite this paper for reference: |
|
|
|
```bibtex |
|
@inproceedings{yarmohammadi-etal-2021-everything, |
|
title = "Everything Is All It Takes: A Multipronged Strategy for Zero-Shot Cross-Lingual Information Extraction", |
|
author = "Yarmohammadi, Mahsa and |
|
Wu, Shijie and |
|
Marone, Marc and |
|
Xu, Haoran and |
|
Ebner, Seth and |
|
Qin, Guanghui and |
|
Chen, Yunmo and |
|
Guo, Jialiang and |
|
Harman, Craig and |
|
Murray, Kenton and |
|
White, Aaron Steven and |
|
Dredze, Mark and |
|
Van Durme, Benjamin", |
|
booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing", |
|
year = "2021", |
|
} |
|
``` |
|
|
