|
--- |
|
language: |
|
- fr |
|
tags: |
|
- music |
|
- rap |
|
- lyrics |
|
- bert |
|
library_name: transformers |
|
--- |
|
# WatiBERT: Fine-Tuned BERT Model for French Rap Lyrics |
|
|
|
## Overview |
|
|
|
__WatiBERT__ is a __BERT__ model fine-tuned on __french rap lyrics__ sourced from __Genius__. Used dataset size was __323MB__, corresponding to __85M tokens__ after tokenization. |
|
|
|
This model is designed to understand and analyze the __semantic relationships__ within the context of __French rap__, providing a valuable tool for research in __French slang__, and __music writing__. |
|
|
|
## Model Details |
|
|
|
The model is based on the __FlauBERT Large Cased__ architecture and has been fine-tuned with the following hyperparameters: |
|
|
|
| Parameter | Value | |
|
|----------------------|--------------| |
|
| Epochs | 5 | |
|
| Train Batch Size | 16 | |
|
| Learning Rate | 2e-5 | |
|
| Weight Decay | 0.01 | |
|
| Warmup Ratio | 0.1 | |
|
| Dropout | 0.1 | |
|
| Mask Token | <special1> | |
|
|
|
## Versions |
|
|
|
The model was trained using __AWS SageMaker__ on a single __ml.p3.2xlarge__ instance with the following software versions: |
|
|
|
| Requirement | Version | |
|
|----------------------|--------------| |
|
| Transformers Library | 4.6 | |
|
| PyTorch | 1.7 | |
|
| Python | 3.6 | |
|
|
|
## Installation |
|
|
|
1. **Install Required Python Libraries**: |
|
|
|
```bash |
|
pip install transformers |
|
``` |
|
|
|
## Loading the Model |
|
|
|
To load the WatiBERT model, use the following Python code: |
|
|
|
```python |
|
from transformers import AutoModel, AutoTokenizer |
|
|
|
# Load the tokenizer and model |
|
tokenizer = AutoTokenizer.from_pretrained("rapminerz/WatiBERT-large-cased") |
|
model = AutoModel.from_pretrained("rapminerz/WatiBERT-large-cased") |
|
``` |
|
|
|
## Using the Model |
|
|
|
BERT Models being masked-models, you can fill missing words to check it out |
|
|
|
```python |
|
def fill_mask(sentence, topk): |
|
inputs = tokenizer(sentence, return_tensors="pt") |
|
mask_token_index = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)[1] |
|
outputs = model(**inputs) |
|
logits = outputs.logits |
|
top_tokens_ids = logits[0, mask_token_index, :].topk(topk, dim=1).indices[0] |
|
top_tokens = [tokenizer.decode(token_id) for token_id in top_tokens_ids] |
|
return top_tokens |
|
|
|
sentence = "La <special1> est morte hier, ils faisaient pas le poids (gang)" |
|
fill_mask(sentence, 1) |
|
['concurrence'] |
|
|
|
sentence = "On s'en souviendra comme le coup de tête de <special1>..." |
|
fill_mask(sentence, 1) |
|
['Zidane'] |
|
|
|
sentence = "Et quand je serai en haut j'achêterai une <special1> à ma daronne !" |
|
fill_mask(sentence, 1) |
|
['villa'] |
|
|
|
sentence = "Tout ce qui m'importe c'est faire du <special1> !" |
|
fill_mask(sentence, 5) |
|
['chiffre', 'cash', 'fric', 'sale', 'blé'] |
|
``` |
|
|
|
## Usages |
|
|
|
This model can be then fined tune to serveral tasks such as : text classification, named entity recognition, question answering, text summerization, text generation, text completion, paraphrasing, language translation, sentiment analysis... |
|
|
|
## Purpose and Disclaimer |
|
|
|
This model is designed for academic and research purposes only. It is not intended for commercial use. The creators of this model do not endorse or promote any specific views or opinions that may be represented in the dataset. |
|
|
|
## Contact |
|
|
|
For any questions or issues, please contact the repository owner, __RapMinerz__, at rapminerz.contact@gmail.com. |
