mT0-XL-detox-orpo
Resources:
Model Information
This is a multilingual 3.7B text detoxification model for 9 languages built on TextDetox 2024 shared task based on mT0-XL. The model was trained in a two-step setup: the first step is full fine-tuning on different parallel text detoxification datasets, and the second step is ORPO alignment on a self-annotated preference dataset collected using toxicity and similarity classifiers. See the paper for more details.
In terms of human evaluation, the model is a second-best approach on the TextDetox 2024 shared task. More precisely, the model shows state-of-the-art performance for the Ukrainian language, top-2 scores for Arabic, and near state-of-the-art performance for other languages.
Example usage
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
model = AutoModelForSeq2SeqLM.from_pretrained('s-nlp/mt0-xl-detox-orpo', device_map="auto")
tokenizer = AutoTokenizer.from_pretrained('s-nlp/mt0-xl-detox-orpo')
LANG_PROMPTS = {
'zh': '排毒:',
'es': 'Desintoxicar: ',
'ru': 'Детоксифицируй: ',
'ar': 'إزالة السموم: ',
'hi': 'विषहरण: ',
'uk': 'Детоксифікуй: ',
'de': 'Entgiften: ',
'am': 'መርዝ መርዝ: ',
'en': 'Detoxify: ',
}
def detoxify(text, lang, model, tokenizer):
encodings = tokenizer(LANG_PROMPTS[lang] + text, return_tensors='pt').to(model.device)
outputs = model.generate(**encodings.to(model.device),
max_length=128,
num_beams=10,
no_repeat_ngram_size=3,
repetition_penalty=1.2,
num_beam_groups=5,
diversity_penalty=2.5,
num_return_sequences=5,
early_stopping=True,
)
return tokenizer.batch_decode(outputs, skip_special_tokens=True)
Citation
@misc{rykov2024smurfcatpan2024textdetox,
title={SmurfCat at PAN 2024 TextDetox: Alignment of Multilingual Transformers for Text Detoxification},
author={Elisei Rykov and Konstantin Zaytsev and Ivan Anisimov and Alexandr Voronin},
year={2024},
eprint={2407.05449},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2407.05449},
}
- Downloads last month
- 31
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.