AUB MIND LAB commited on
Commit
4d67dc9
1 Parent(s): 3b5808c

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +104 -101
README.md CHANGED
@@ -1,132 +1,133 @@
1
  ---
2
  language: ar
 
 
 
 
 
 
3
  ---
4
 
5
- # AraBERT : Pre-training BERT for Arabic Language Understanding
6
- <img src="https://github.com/aub-mind/arabert/blob/master/arabert_logo.png" width="100" align="left"/>
7
 
8
- **AraBERT** is an Arabic pretrained lanaguage model based on [Google's BERT architechture](https://github.com/google-research/bert). AraBERT uses the same BERT-Base config. More details are available in the [AraBERT PAPER](https://arxiv.org/abs/2003.00104v2) and in the [AraBERT Meetup](https://github.com/WissamAntoun/pydata_khobar_meetup)
9
 
10
- There are two version off the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the [Farasa Segmenter](http://alt.qcri.org/farasa/segmenter.html).
11
 
12
- The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words. The training corpora are a collection of publically available large scale raw arabic text ([Arabic Wikidumps](https://archive.org/details/arwiki-20190201), [The 1.5B words Arabic Corpus](https://www.semanticscholar.org/paper/1.5-billion-words-Arabic-Corpus-El-Khair/f3eeef4afb81223df96575adadf808fe7fe440b4), [The OSIAN Corpus](https://www.aclweb.org/anthology/W19-4619), Assafir news articles, and 4 other manually crawled news websites (Al-Akhbar, Annahar, AL-Ahram, AL-Wafd) from [the Wayback Machine](http://web.archive.org/))
13
 
14
- We evalaute both AraBERT models on different downstream tasks and compare it to [mBERT]((https://github.com/google-research/bert/blob/master/multilingual.md)), and other state of the art models (*To the extent of our knowledge*). The Tasks were Sentiment Analysis on 6 different datasets ([HARD](https://github.com/elnagara/HARD-Arabic-Dataset), [ASTD-Balanced](https://www.aclweb.org/anthology/D15-1299), [ArsenTD-Lev](https://staff.aub.edu.lb/~we07/Publications/ArSentD-LEV_Sentiment_Corpus.pdf), [LABR](https://github.com/mohamedadaly/LABR), [ArSaS](http://lrec-conf.org/workshops/lrec2018/W30/pdf/22_W30.pdf)), Named Entity Recognition with the [ANERcorp](http://curtis.ml.cmu.edu/w/courses/index.php/ANERcorp), and Arabic Question Answering on [Arabic-SQuAD and ARCD](https://github.com/husseinmozannar/SOQAL)
15
 
16
- **Update 2 (21/5/2020) :**
17
- Added support for the farasapy segmenter https://github.com/MagedSaeed/farasapy in the ``preprocess_arabert.py`` which is ~6x faster than the ``py4j.java_gateway``, consider setting ``use_farasapy=True`` when calling preprocess and pass it an instance of ``FarasaSegmenter(interactive=True)`` with interactive set to ``True`` for faster segmentation.
18
 
19
- **Update 1 (21/4/2020) :**
20
- Fixed an issue with ARCD fine-tuning which drastically improved performance. Initially we didn't account for the change of the ```answer_start``` during preprocessing.
21
- ## Results (Acc.)
22
- Task | prev. SOTA | mBERT | AraBERTv0.1 | AraBERTv1
23
- ---|:---:|:---:|:---:|:---:
24
- HARD |95.7 [ElJundi et.al.](https://www.aclweb.org/anthology/W19-4608/)|95.7|**96.2**|96.1
25
- ASTD |86.5 [ElJundi et.al.](https://www.aclweb.org/anthology/W19-4608/)| 80.1|92.2|**92.6**
26
- ArsenTD-Lev|52.4 [ElJundi et.al.](https://www.aclweb.org/anthology/W19-4608/)|51|58.9|**59.4**
27
- AJGT|93 [Dahou et.al.](https://dl.acm.org/doi/fullHtml/10.1145/3314941)| 83.6|93.1|**93.8**
28
- LABR|**87.5** [Dahou et.al.](https://dl.acm.org/doi/fullHtml/10.1145/3314941)|83|85.9|86.7
29
- ANERcorp|81.7 (BiLSTM-CRF)|78.4|**84.2**|81.9
30
- ARCD|mBERT|EM:34.2 F1: 61.3|EM:51.14 F1:82.13|**EM:54.84 F1: 82.15**
31
 
32
- *If you tested AraBERT on a public dataset and you want to add your results to the table above, open a pull request or contact us. Also make sure to have your code available online so we can add it as a reference*
33
 
34
- ## How to use
35
 
36
- You can easily use AraBERT since it is almost fully compatible with existing codebases (Use this repo instead of the official BERT one, the only difference is in the ```tokenization.py``` file where we modify the _is_punctuation function to make it compatible with the "+" symbol and the "[" and "]" characters)
37
 
38
- To use HuggingFace's Transformer repository you only need to provide a list of token that forces the model to not split them, also make sure that the text is pre-segmented:
39
- **Not all libraries built on top of transformers support the `never_split` argument**
40
- ```python
41
- from transformers import AutoTokenizer, AutoModel
42
- from arabert.preprocess_arabert import never_split_tokens, preprocess
43
- from farasa.segmenter import FarasaSegmenter
44
-
45
- arabert_tokenizer = AutoTokenizer.from_pretrained(
46
- "aubmindlab/bert-base-arabert",
47
- do_lower_case=False,
48
- do_basic_tokenize=True,
49
- never_split=never_split_tokens)
50
- arabert_model = AutoModel.from_pretrained("aubmindlab/bert-base-arabert")
51
-
52
- #Preprocess the text to make it compatible with AraBERT using farasapy
53
- farasa_segmenter = FarasaSegmenter(interactive=True)
54
-
55
- #or you can use a py4j JavaGateway to the farasa Segmneter .jar but it's slower
56
- #(see update 2)
57
- #from py4j.java_gateway import JavaGateway
58
- #gateway = JavaGateway.launch_gateway(classpath='./PATH_TO_FARASA/FarasaSegmenterJar.jar')
59
- #farasa = gateway.jvm.com.qcri.farasa.segmenter.Farasa()
60
 
61
- text = "ولن نبالغ إذا قلنا إن هاتف أو كمبيوتر المكتب في زمننا هذا ضروري"
62
- text_preprocessed = preprocess( text,
63
- do_farasa_tokenization = True,
64
- farasa = farasa_segmenter,
65
- use_farasapy = True)
66
 
67
- >>>text_preprocessed: "و+ لن نبالغ إذا قل +نا إن هاتف أو كمبيوتر ال+ مكتب في زمن +نا هذا ضروري"
 
 
 
 
 
 
 
68
 
69
- arabert_tokenizer.tokenize(text_preprocessed)
70
 
71
- >>> ['و+', 'لن', 'نبال', '##غ', 'إذا', 'قل', '+نا', 'إن', 'هاتف', 'أو', 'كمبيوتر', 'ال+', 'مكتب', 'في', 'زمن', '+نا', 'هذا', 'ضروري']
72
- ```
73
 
74
- **AraBERTv0.1 is compatible with all existing libraries, since it needs no pre-segmentation.**
75
- ```python
76
- from transformers import AutoTokenizer, AutoModel
77
 
78
- arabert_tokenizer = AutoTokenizer.from_pretrained("aubmindlab/bert-base-arabertv01",do_lower_case=False)
79
- arabert_model = AutoModel.from_pretrained("aubmindlab/bert-base-arabertv01")
80
 
81
- text = "ولن نبالغ إذا قلنا إن هاتف أو كمبيوتر المكتب في زمننا هذا ضروري"
82
- arabert_tokenizer.tokenize(text)
83
 
84
- >>> ['ولن', 'ن', '##بالغ', 'إذا', 'قلنا', 'إن', 'هاتف', 'أو', 'كمبيوتر', 'المكتب', 'في', 'زمن', '##ن', '##ا', 'هذا', 'ضروري']
85
- ```
86
 
 
 
87
 
88
- The ```araBERT_(Updated_Demo_TF).ipynb``` Notebook is a small demo using the AJGT dataset using TensorFlow (GPU and TPU compatible).
 
 
 
 
 
 
89
 
90
- **Coming Soon :** Fine-tunning demo using HuggingFace's Trainer API
91
 
92
- **AraBERT on ARCD**
93
- During the preprocessing step the ```answer_start``` character position needs to be recalculated. You can use the file ```arcd_preprocessing.py``` as shown below to clean, preprocess the ARCD dataset before running ```run_squad.py```. More detailed Colab notebook is available in the [SOQAL repo](https://github.com/husseinmozannar/SOQAL).
94
- ```bash
95
- python arcd_preprocessing.py \
96
- --input_file="/PATH_TO/arcd-test.json" \
97
- --output_file="arcd-test-pre.json" \
98
- --do_farasa_tokenization=True \
99
- --use_farasapy=True \
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
100
  ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
101
  ```bash
102
- python SOQAL/bert/run_squad.py \
103
- --vocab_file="/PATH_TO_PRETRAINED_TF_CKPT/vocab.txt" \
104
- --bert_config_file="/PATH_TO_PRETRAINED_TF_CKPT/config.json" \
105
- --init_checkpoint="/PATH_TO_PRETRAINED_TF_CKPT/" \
106
- --do_train=True \
107
- --train_file=turk_combined_all_pre.json \
108
- --do_predict=True \
109
- --predict_file=arcd-test-pre.json \
110
- --train_batch_size=32 \
111
- --predict_batch_size=24 \
112
- --learning_rate=3e-5 \
113
- --num_train_epochs=4 \
114
- --max_seq_length=384 \
115
- --doc_stride=128 \
116
- --do_lower_case=False\
117
- --output_dir="/PATH_TO/OUTPUT_PATH"/ \
118
- --use_tpu=True \
119
- --tpu_name=$TPU_ADDRESS \
120
  ```
121
- ## Model Weights and Vocab Download
122
- Models | AraBERTv0.1 | AraBERTv1
123
- ---|:---:|:---:
124
- TensorFlow|[Drive Link](https://drive.google.com/open?id=1-kVmTUZZ4DP2rzeHNjTPkY8OjnQCpomO) | [Drive Link](https://drive.google.com/open?id=1-d7-9ljKgDJP5mx73uBtio-TuUZCqZnt)
125
- PyTorch| [Drive_Link](https://drive.google.com/open?id=1-_3te42mQCPD8SxwZ3l-VBL7yaJH-IOv)| [Drive_Link](https://drive.google.com/open?id=1-69s6Pxqbi63HOQ1M9wTcr-Ovc6PWLLo)
126
 
127
- **You can find the PyTorch models in HuggingFace's Transformer Library under the ```aubmindlab``` username**
 
 
 
128
 
129
- ## If you used this model please cite us as:
 
 
130
  ```
131
  @inproceedings{antoun2020arabert,
132
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
@@ -135,10 +136,12 @@ PyTorch| [Drive_Link](https://drive.google.com/open?id=1-_3te42mQCPD8SxwZ3l-VBL7
135
  pages={9}
136
  }
137
  ```
138
- ## Acknowledgments
139
  Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn't have done it without this program, and to the [AUB MIND Lab](https://sites.aub.edu.lb/mindlab/) Members for the continous support. Also thanks to [Yakshof](https://www.yakshof.com/#/) and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.
140
 
141
- ## Contacts
142
- **Wissam Antoun**: [Linkedin](https://www.linkedin.com/in/giulio-ravasio-3a81a9110/) | [Twitter](https://twitter.com/wissam_antoun) | [Github](https://github.com/WissamAntoun) | <wfa07@mail.aub.edu> | <wissam.antoun@gmail.com>
143
 
144
  **Fady Baly**: [Linkedin](https://www.linkedin.com/in/fadybaly/) | [Twitter](https://twitter.com/fadybaly) | [Github](https://github.com/fadybaly) | <fgb06@mail.aub.edu> | <baly.fady@gmail.com>
 
 
 
1
  ---
2
  language: ar
3
+ datasets:
4
+ - wikipedia
5
+ - OSIAN
6
+ - 1.5B Arabic Corpus
7
+ widget:
8
+ - text: " عاصمة لبنان هي [MASK] ."
9
  ---
10
 
11
+ # !!! A newer version of this model is available !!! [AraBERTv2](https://huggingface.co/aubmindlab/bert-base-arabertv02)
 
12
 
 
13
 
14
+ # AraBERT v1 & v2 : Pre-training BERT for Arabic Language Understanding
15
 
16
+ <img src="https://raw.githubusercontent.com/aub-mind/arabert/master/arabert_logo.png" width="100" align="left"/>
17
 
18
+ **AraBERT** is an Arabic pretrained lanaguage model based on [Google's BERT architechture](https://github.com/google-research/bert). AraBERT uses the same BERT-Base config. More details are available in the [AraBERT Paper](https://arxiv.org/abs/2003.00104) and in the [AraBERT Meetup](https://github.com/WissamAntoun/pydata_khobar_meetup)
19
 
20
+ There are two versions of the model, AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were splitted using the [Farasa Segmenter](http://alt.qcri.org/farasa/segmenter.html).
 
21
 
 
 
 
 
 
 
 
 
 
 
 
 
22
 
23
+ We evalaute AraBERT models on different downstream tasks and compare them to [mBERT]((https://github.com/google-research/bert/blob/master/multilingual.md)), and other state of the art models (*To the extent of our knowledge*). The Tasks were Sentiment Analysis on 6 different datasets ([HARD](https://github.com/elnagara/HARD-Arabic-Dataset), [ASTD-Balanced](https://www.aclweb.org/anthology/D15-1299), [ArsenTD-Lev](https://staff.aub.edu.lb/~we07/Publications/ArSentD-LEV_Sentiment_Corpus.pdf), [LABR](https://github.com/mohamedadaly/LABR)), Named Entity Recognition with the [ANERcorp](http://curtis.ml.cmu.edu/w/courses/index.php/ANERcorp), and Arabic Question Answering on [Arabic-SQuAD and ARCD](https://github.com/husseinmozannar/SOQAL)
24
 
25
+ # AraBERTv2
26
 
27
+ ## What's New!
28
 
29
+ AraBERT now comes in 4 new variants to replace the old v1 versions:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
30
 
31
+ More Detail in the AraBERT folder and in the [README](https://github.com/aub-mind/arabert/blob/master/AraBERT/README.md) and in the [AraBERT Paper](https://arxiv.org/abs/2003.00104v2)
 
 
 
 
32
 
33
+ Model | HuggingFace Model Name | Size (MB/Params)| Pre-Segmentation | DataSet (Sentences/Size/nWords) |
34
+ ---|:---:|:---:|:---:|:---:
35
+ AraBERTv0.2-base | [bert-base-arabertv02](https://huggingface.co/aubmindlab/bert-base-arabertv02) | 543MB / 136M | No | 200M / 77GB / 8.6B |
36
+ AraBERTv0.2-large| [bert-large-arabertv02](https://huggingface.co/aubmindlab/bert-large-arabertv02) | 1.38G 371M | No | 200M / 77GB / 8.6B |
37
+ AraBERTv2-base| [bert-base-arabertv2](https://huggingface.co/aubmindlab/bert-base-arabertv2) | 543MB 136M | Yes | 200M / 77GB / 8.6B |
38
+ AraBERTv2-large| [bert-large-arabertv2](https://huggingface.co/aubmindlab/bert-large-arabertv2) | 1.38G 371M | Yes | 200M / 77GB / 8.6B |
39
+ AraBERTv0.1-base| [bert-base-arabertv01](https://huggingface.co/aubmindlab/bert-base-arabertv01) | 543MB 136M | No | 77M / 23GB / 2.7B |
40
+ AraBERTv1-base| [bert-base-arabert](https://huggingface.co/aubmindlab/bert-base-arabert) | 543MB 136M | Yes | 77M / 23GB / 2.7B |
41
 
42
+ All models are available in the `HuggingFace` model page under the [aubmindlab](https://huggingface.co/aubmindlab/) name. Checkpoints are available in PyTorch, TF2 and TF1 formats.
43
 
44
+ ## Better Pre-Processing and New Vocab
 
45
 
46
+ We identified an issue with AraBERTv1's wordpiece vocabulary. The issue came from punctuations and numbers that were still attached to words when learned the wordpiece vocab. We now insert a space between numbers and characters and around punctuation characters.
 
 
47
 
48
+ The new vocabulary was learnt using the `BertWordpieceTokenizer` from the `tokenizers` library, and should now support the Fast tokenizer implementation from the `transformers` library.
 
49
 
50
+ **P.S.**: All the old BERT codes should work with the new BERT, just change the model name and check the new preprocessing dunction
51
+ **Please read the section on how to use the [preprocessing function](#Preprocessing)**
52
 
53
+ ## Bigger Dataset and More Compute
 
54
 
55
+ We used ~3.5 times more data, and trained for longer.
56
+ For Dataset Sources see the [Dataset Section](#Dataset)
57
 
58
+ Model | Hardware | num of examples with seq len (128 / 512) |128 (Batch Size/ Num of Steps) | 512 (Batch Size/ Num of Steps) | Total Steps | Total Time (in Days) |
59
+ ---|:---:|:---:|:---:|:---:|:---:|:---:
60
+ AraBERTv0.2-base | TPUv3-8 | 420M / 207M |2560 / 1M | 384/ 2M | 3M | -
61
+ AraBERTv0.2-large | TPUv3-128 | 420M / 207M | 13440 / 250K | 2056 / 300K | 550K | -
62
+ AraBERTv2-base | TPUv3-8 | 520M / 245M |13440 / 250K | 2056 / 300K | 550K | -
63
+ AraBERTv2-large | TPUv3-128 | 520M / 245M | 13440 / 250K | 2056 / 300K | 550K | -
64
+ AraBERT-base (v1/v0.1) | TPUv2-8 | - |512 / 900K | 128 / 300K| 1.2M | 4 days
65
 
66
+ # Dataset
67
 
68
+ The pretraining data used for the new AraBERT model is also used for Arabic **GPT2 and ELECTRA**.
69
+
70
+ The dataset consists of 77GB or 200,095,961 lines or 8,655,948,860 words or 82,232,988,358 chars (before applying Farasa Segmentation)
71
+
72
+ For the new dataset we added the unshuffled OSCAR corpus, after we thoroughly filter it, to the previous dataset used in AraBERTv1 but with out the websites that we previously crawled:
73
+ - OSCAR unshuffled and filtered.
74
+ - [Arabic Wikipedia dump](https://archive.org/details/arwiki-20190201) from 2020/09/01
75
+ - [The 1.5B words Arabic Corpus](https://www.semanticscholar.org/paper/1.5-billion-words-Arabic-Corpus-El-Khair/f3eeef4afb81223df96575adadf808fe7fe440b4)
76
+ - [The OSIAN Corpus](https://www.aclweb.org/anthology/W19-4619)
77
+ - Assafir news articles. Huge thank you for Assafir for giving us the data
78
+
79
+ # Preprocessing
80
+
81
+ It is recommended to apply our preprocessing function before training/testing on any dataset.
82
+ **Install farasapy to segment text for AraBERT v1 & v2 `pip install farasapy`**
83
+
84
+ ```python
85
+ from arabert.preprocess import ArabertPreprocessor
86
+
87
+ model_name="bert-base-arabertv01"
88
+ arabert_prep = ArabertPreprocessor(model_name=model_name, keep_emojis=True)
89
+
90
+ text = "ولن نبالغ إذا قلنا إن هاتف أو كمبيوتر المكتب في زمننا هذا ضروري"
91
+ arabert_prep.preprocess(text)
92
+ ```
93
+
94
+ ## Accepted_models
95
  ```
96
+ bert-base-arabertv01
97
+ bert-base-arabert
98
+ bert-base-arabertv02
99
+ bert-base-arabertv2
100
+ bert-large-arabertv02
101
+ bert-large-arabertv2
102
+ araelectra-base
103
+ aragpt2-base
104
+ aragpt2-medium
105
+ aragpt2-large
106
+ aragpt2-mega
107
+ ```
108
+
109
+ # TensorFlow 1.x models
110
+
111
+ The TF1.x model are available in the HuggingFace models repo.
112
+ You can download them as follows:
113
+ - via git-lfs: clone all the models in a repo
114
  ```bash
115
+ curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash
116
+ sudo apt-get install git-lfs
117
+ git lfs install
118
+ git clone https://huggingface.co/aubmindlab/MODEL_NAME
119
+ tar -C ./MODEL_NAME -zxvf /content/MODEL_NAME/tf1_model.tar.gz
 
 
 
 
 
 
 
 
 
 
 
 
 
120
  ```
121
+ where `MODEL_NAME` is any model under the `aubmindlab` name
 
 
 
 
122
 
123
+ - via `wget`:
124
+ - Go to the tf1_model.tar.gz file on huggingface.co/models/aubmindlab/MODEL_NAME.
125
+ - copy the `oid sha256`
126
+ - then run `wget https://cdn-lfs.huggingface.co/aubmindlab/aragpt2-base/INSERT_THE_SHA_HERE` (ex: for `aragpt2-base`: `wget https://cdn-lfs.huggingface.co/aubmindlab/aragpt2-base/3766fc03d7c2593ff2fb991d275e96b81b0ecb2098b71ff315611d052ce65248`)
127
 
128
+
129
+ # If you used this model please cite us as :
130
+ Google Scholar has our Bibtex wrong (missing name), use this instead
131
  ```
132
  @inproceedings{antoun2020arabert,
133
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
 
136
  pages={9}
137
  }
138
  ```
139
+ # Acknowledgments
140
  Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn't have done it without this program, and to the [AUB MIND Lab](https://sites.aub.edu.lb/mindlab/) Members for the continous support. Also thanks to [Yakshof](https://www.yakshof.com/#/) and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.
141
 
142
+ # Contacts
143
+ **Wissam Antoun**: [Linkedin](https://www.linkedin.com/in/wissam-antoun-622142b4/) | [Twitter](https://twitter.com/wissam_antoun) | [Github](https://github.com/WissamAntoun) | <wfa07@mail.aub.edu> | <wissam.antoun@gmail.com>
144
 
145
  **Fady Baly**: [Linkedin](https://www.linkedin.com/in/fadybaly/) | [Twitter](https://twitter.com/fadybaly) | [Github](https://github.com/fadybaly) | <fgb06@mail.aub.edu> | <baly.fady@gmail.com>
146
+
147
+