|
--- |
|
|
|
title: README |
|
|
|
emoji: 👁 |
|
|
|
colorFrom: orange |
|
|
|
colorTo: indigo |
|
|
|
sdk: static |
|
|
|
pinned: false |
|
|
|
--- |
|
|
|
|
|
<div> |
|
<img src="https://raw.githubusercontent.com/NCAI-Research/CALM/main/assets/logo.png" width="380" alt="CALM Logo" /> |
|
<p class="mb-2" style="font-size:30px;font-weight:bold"> |
|
CALM: Collaborative Arabic Language Model |
|
</p> |
|
<p class="mb-2"> |
|
The CALM project is joint effort lead by <u><a target="_blank" href="https://sdaia.gov.sa/ncai/?Lang=en">NCAI</a></u> in collaboration with |
|
<u><a target="_blank" href="https://yandex.com/">Yandex</a></u>, <u><a href="https://huggingface.co/">HuggingFace</a></u> and <u><a href="http://www.washington.edu/">UW</a></u> to train an Arabic language model with |
|
volunteers from around the globe. The project is an adaptation of the framework proposed at the NeurIPS 2021 demonstration: |
|
<u><a target="_blank" href="https://huggingface.co/training-transformers-together">Training Transformers Together</a></u>. |
|
</p> |
|
<p class="mb-2"> |
|
One of the main obstacles facing many researchers in the Arabic NLP community is the lack of computing resources that are needed for training large models. Models with |
|
leading performane on Arabic NLP tasks, such as <u><a target="_blank" href="https://github.com/aub-mind/arabert">AraBERT</a></u>, |
|
<u><a href="https://github.com/CAMeL-Lab/CAMeLBERT" target="_blank" >CamelBERT</a></u>, |
|
<u><a href="https://huggingface.co/aubmindlab/araelectra-base-generator" target="_blank" >AraELECTRA</a></u>, and |
|
<u><a href="https://huggingface.co/qarib">QARiB</a></u>, |
|
took days to train on TPUs. In the spirit of democratization of AI and community enabling, a core value at NCAI, CALM aims to demonstrate the effectiveness |
|
of collaborative training and form a community of volunteers for ANLP researchers with basic level cloud GPUs who wish to train their own models collaboratively. |
|
</p> |
|
<p class="mb-2"> |
|
CALM trains a single BERT model on a dataset that combines MSA, Oscar and Arabic Wikipedia, and dialectal data for the gulf region from existing open source datasets. |
|
Each volunteer GPU trains the model locally at its own pace on a portion of the dataset while another portion is being streamed in the background to reduces local |
|
memory consumption. Computing the gradients and aggregating them is performed in a distributed manner, based on the computing abilities of each participating |
|
volunteer. Details of the distributed training process are further described in the paper |
|
<u><a target="_blank" href="https://papers.nips.cc/paper/2021/hash/41a60377ba920919939d83326ebee5a1-Abstract.html">Deep Learning in Open Collaborations</a></u>. |
|
</p> |
|
|
|
<p class="mb-2" style="font-size:20px;font-weight:bold"> |
|
How to participate in training? |
|
</p> |
|
<p class="mb-2"> |
|
To join the collaborative training, all you have to do is to keep a notebook running for at <b>least 15 minutes</b>, you're free to close it after that and join again |
|
in another time. There are few steps before running the notebook: |
|
</p> |
|
|
|
<ul class="mb-2"> |
|
<li>👉 Create an account on <u><a target="_blank" href="https://huggingface.co">Huggingface</a></u>.</li> |
|
<li>👉 Join the <u><a target="_blank" href="https://huggingface.co/CALM">NCAI-CALM Organization</a></u> on Huggingface through the invitation link shared with you by email.</li> |
|
<li>👉 Get your Access Token, it's later required in the notebook. |
|
</li> |
|
</ul> |
|
|
|
<p class="h2 mb-2" style="font-size:18px;font-weight:bold">How to get my Huggingface Access Token</p> |
|
<ul class="mb-2"> |
|
<li>👉 Go to your <u><a target="_blank" href="https://huggingface.co">HF account</a></u>.</li> |
|
<li>👉 Go to Settings ⇒ Access Tokens.</li> |
|
<li>👉 Generate a new Access Token and enter any name for "what's this token for".</li> |
|
<li>👉 Select <code>read</code> role.</li> |
|
<li>👉 Copy your access token.</li> |
|
<li>👉 In cell 4, it will ask you for an Access Token, paste it there.</li> |
|
</ul> |
|
|
|
<p class="mb-2" style="font-size:20px;font-weight:bold"> |
|
Start training |
|
</p> |
|
<p class="mb-2">Pick one of the following methods to run the training code. |
|
<br /><em>NOTE: Kaggle gives you around 40 hrs per week of GPU time, so it's preferred over Colab, unless you have Colab Pro or Colab Pro+.</em></p> |
|
<ul class="mb-2"> |
|
<li>👉 <span><a href="https://www.kaggle.com/prmais/volunteer-gpu-notebook"> |
|
<img style="display:inline;margin:0px" src="https://img.shields.io/badge/kaggle-Open%20in%20Kaggle-blue.svg"/> |
|
</a></span> <b> (recommended)</b> <br /> |
|
</li> |
|
<li>👉 <span><a href="https://colab.research.google.com/github/NCAI-Research/CALM/blob/main/notebooks/volunteer-gpu-notebook.ipynb"> |
|
<img style="display:inline;margin:0px" src="https://colab.research.google.com/assets/colab-badge.svg"/> |
|
</a></span> |
|
</li> |
|
<li>👉 Running locally: If you have additional local computing GPUs, please visit our discord channel for instructions to set it. |
|
</li> |
|
</ul> |
|
|
|
<p class="mb-2" style="font-size:20px;font-weight:bold"> |
|
Issues or questions? |
|
</p> |
|
|
|
<p class="mb-2"> |
|
Feel free to reach us on <u><a target="_blank" href="https://discord.gg/peU5Nx77">Discord</a></u> if you have any questions 🙂 |
|
</p> |
|
</div> |
|
|
