language:
- fr
tags:
- music
- rap
- lyrics
- bert
library_name: transformers
WatiBERT: Fine-Tuned BERT Model for French Rap Lyrics
Overview
WatiBERT is a BERT model fine-tuned on french rap lyrics sourced from Genius. Used dataset size was 323MB, corresponding to 85M tokens after tokenization.
This model is designed to understand and analyze the semantic relationships within the context of French rap, providing a valuable tool for research in French slang, and music lyrics analysis.
Model Details
The model is based on the FlauBERT Large Cased architecture and has been fine-tuned with the following hyperparameters:
| Parameter | Value |
|---|---|
| Epochs | 5 |
| Train Batch Size | 16 |
| Learning Rate | 2e-5 |
| Weight Decay | 0.01 |
| Warmup Ratio | 0.1 |
| Dropout | 0.1 |
Versions
The model was trained using AWS SageMaker on a single ml.p3.2xlarge instance with the following software versions:
| Requirement | Version |
|---|---|
| Transformers Library | 4.6 |
| PyTorch | 1.7 |
| Python | 3.6 |
Installation
Install Required Python Libraries:
pip install transformers
Loading the Model
To load the WatiBERT model, use the following Python code:
from transformers import FlaubertTokenizer, FlaubertWithLMHeadModel
# Load the tokenizer and model
tokenizer = FlaubertTokenizer.from_pretrained("rapminerz/WatiBERT-large-cased")
model = FlaubertWithLMHeadModel.from_pretrained("rapminerz/WatiBERT-large-cased")
Using the Model
BERT Models being masked-models, you can fill missing words to check it out
def fill_mask(sentence, topk):
inputs = tokenizer(sentence, return_tensors="pt")
mask_token_index = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)[1]
outputs = model(**inputs)
logits = outputs.logits
top_tokens_ids = logits[0, mask_token_index, :].topk(topk, dim=1).indices[0]
top_tokens = [tokenizer.decode(token_id) for token_id in top_tokens_ids]
return top_tokens
sentence = "La <special1> est morte hier, ils faisaient pas le poids (gang)"
fill_mask(sentence, 1)
['concurrence']
sentence = "On s'en souviendra comme le coup de tête de <special1>..."
fill_mask(sentence, 1)
['Zidane']
sentence = "Et quand je serai en haut j'achêterai une <special1> à ma daronne !"
fill_mask(sentence, 1)
['villa']
sentence = "Tout ce qui m'importe c'est faire du <special1> !"
fill_mask(sentence, 5)
['chiffre', 'cash', 'fric', 'sale', 'blé']
Usages
This model can be then fined tune to serveral tasks such as : text classification, named entity recognition, question answering, text summerization, text generation, text completion, paraphrasing, language translation, sentiment analysis...
Purpose and Disclaimer
This model is designed for academic and research purposes only. It is not intended for commercial use. The creators of this model do not endorse or promote any specific views or opinions that may be represented in the dataset.
Please mention @RapMinerz if you use our models
Contact
For any questions or issues, please contact the repository owner, RapMinerz, at rapminerz.contact@gmail.com.