|
--- |
|
license: mit |
|
language: |
|
- fr |
|
pipeline_tag: fill-mask |
|
tags: |
|
- medical |
|
- clinical |
|
datasets: |
|
- rntc/biomed-fr |
|
--- |
|
|
|
<a href=https://camembert-model.fr> |
|
<img width="300px" src="https://camembert-model.fr/authors/admin/avatar_huac8a9374dbd7d6a2cb77224540858ab4_463389_270x270_fill_lanczos_center_3.png"> |
|
</a> |
|
|
|
# CamemBERT-bio : a Tasty French Language Model Better for your Health |
|
|
|
CamemBERT-bio is a state-of-the-art french biomedical language model built using continual-pretraining from [camembert-base](https://huggingface.co/camembert-base). |
|
It was trained on a french public biomedical corpus of 413M words containing scientific documments, drug leaflets and clinical cases extrated from theses and articles. |
|
It shows 2.54 points of F1 score improvement on average on 5 different biomedical named entity recognition tasks compared to [camembert-base](https://huggingface.co/camembert-base). |
|
|
|
## Absract |
|
|
|
Clinical data in hospitals are increasingly accessible for research through clinical data warehouses, however these documents are unstructured. It is therefore necessary to extract information from medical |
|
reports to conduct clinical studies. Transfer learning with BERT-like models such as CamemBERT |
|
has allowed major advances, especially for named entity recognition. However, these models are |
|
trained for plain language and are less efficient on biomedical data. This is why we propose a new |
|
french public biomedical dataset on which we have continued the pre-training of CamemBERT. Thus, |
|
we introduce a first version of CamemBERT-bio, a specialized public model for the french biomedical |
|
domain that shows 2.54 points of F1 score improvement on average on different biomedical named |
|
entity recognition tasks. |
|
|
|
- **Developed by:** Rian Touchent, Eric Villemonte de La Clergerie |
|
- **Logo by:** Alix Chagué |
|
- **License:** MIT |
|
|
|
<!-- ### Model Sources [optional] --> |
|
|
|
<!-- Provide the basic links for the model. --> |
|
|
|
<!-- - **Website:** camembert-bio-model.fr --> |
|
<!-- - **Paper [optional]:** [More Information Needed] --> |
|
<!-- - **Demo [optional]:** [More Information Needed] --> |
|
|
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. --> |
|
|
|
| **Corpus** | **Details** | **Size** | |
|
|------------|--------------------------------------------------------------------|------------| |
|
| ISTEX | diverse scientific literature indexed on ISTEX | 276 M | |
|
| CLEAR | drug leaflets | 73 M | |
|
| E3C | various documents from journals, drug leaflets, and clinical cases | 64 M | |
|
| Total | | 413 M | |
|
|
|
|
|
### Training Procedure |
|
|
|
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. --> |
|
|
|
We used continual-pretraining from [camembert-base](https://huggingface.co/camembert-base). |
|
We trained the model using the Masked Language Modeling (MLM) objective with Whole Word Masking for 50k steps during 39 hours |
|
with 2 Tesla V100. |
|
|
|
## Evaluation |
|
|
|
<!-- This section describes the evaluation protocols and provides the results. --> |
|
|
|
### Fine-tuning |
|
|
|
For fine-tuning, we utilized Optuna to select the hyperparameters. |
|
The learning rate was set to 5e-5, with a warmup ratio of 0.224 and a batch size of 16. |
|
The fine-tuning process was carried out for 2000 steps. |
|
For prediction, a simple linear layer was added on top of the model. |
|
Notably, none of the CamemBERT layers were frozen during the fine-tuning process. |
|
|
|
### Scoring |
|
|
|
To evaluate the performance of the model, we used the seqeval tool in strict mode with the IOB2 scheme. |
|
For each evaluation, the best fine-tuned model on the validation set was selected to calculate the final score on the test set. |
|
To ensure reliability, we averaged over 10 evaluations with different seeds. |
|
|
|
### Results |
|
|
|
| Style | Dataset | Score | CamemBERT | CamemBERT | |
|
| :----------- | :------ | :---- | :---------------: | :-------------------: | |
|
| Clinical | CAS1 | F1 | 70\.50 ~~±~~ 1.75 | **73\.03 ~~±~~ 1.29** | |
|
| | | P | 70\.12 ~~±~~ 1.93 | 71\.71 ~~±~~ 1.61 | |
|
| | | R | 70\.89 ~~±~~ 1.78 | **74\.42 ~~±~~ 1.49** | |
|
| | CAS2 | F1 | 79\.02 ~~±~~ 0.92 | **81\.66 ~~±~~ 0.59** | |
|
| | | P | 77\.3 ~~±~~ 1.36 | **80\.96 ~~±~~ 0.91** | |
|
| | | R | 80\.83 ~~±~~ 0.96 | **82\.37 ~~±~~ 0.69** | |
|
| | E3C | F1 | 67\.63 ~~±~~ 1.45 | **69\.85 ~~±~~ 1.58** | |
|
| | | P | 78\.19 ~~±~~ 0.72 | **79\.11 ~~±~~ 0.42** | |
|
| | | R | 59\.61 ~~±~~ 2.25 | **62\.56 ~~±~~ 2.50** | |
|
| Drug leaflets | EMEA | F1 | 74\.14 ~~±~~ 1.95 | **76\.71 ~~±~~ 1.50** | |
|
| | | P | 74\.62 ~~±~~ 1.97 | **76\.92 ~~±~~ 1.96** | |
|
| | | R | 73\.68 ~~±~~ 2.22 | **76\.52 ~~±~~ 1.62** | |
|
| Scientific | MEDLINE | F1 | 65\.73 ~~±~~ 0.40 | **68\.47 ~~±~~ 0.54** | |
|
| | | P | 64\.94 ~~±~~ 0.82 | **67\.77 ~~±~~ 0.88** | |
|
| | | R | 66\.56 ~~±~~ 0.56 | **69\.21 ~~±~~ 1.32** | |
|
|
|
|
|
## Environmental Impact estimation |
|
|
|
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly --> |
|
|
|
- **Hardware Type:** 2 x Tesla V100 |
|
- **Hours used:** 39 hours |
|
- **Provider:** INRIA clusters |
|
- **Compute Region:** Paris, France |
|
- **Carbon Emitted:** 0.84 kg CO2 eq. |
|
|
|
<!-- ## Citation [optional] --> |
|
|
|
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. --> |
|
|
|
<!-- **BibTeX:** --> |
|
|
|
<!-- [More Information Needed] --> |
|
|
|
<!-- **APA:** --> |
|
|
|
<!-- [More Information Needed] --> |
