bart-base-styletransfer-subjective-to-neutral

Model description

This facebook/bart-base model has been fine-tuned on the Wiki Neutrality Corpus (WNC) - a parallel corpus of 180,000 biased and neutralized sentence pairs along with contextual sentences and metadata. The model can be used to transfer style in text from subjectively biased to neutrally toned.

The development and modeling efforts that produced this model are documented in detail through this blog series.

Intended uses & limitations

The model is intended purely as a research output for NLP and data science communities. We imagine this model will be used by researchers to better understand the limitations, robustness, and generalization of text style transfer models. Ultimately, we hope this model will inspire future work on text style transfer and serve as a benchmarking tool for the style attribute of subjectivity bias, specifically.

Any production use of this model - whether commercial or not - is currently not intended. This is because, as the team at OpenAI points out, large langauge models like BART reflect biases inherent to the systems they were trained on, so we do not recommend that they be deployed into systems that interact with humans, unless the deployers first carry out a study of biases relevant to the intended use-case. Neither the model nor the WNC dataset has been sufficiently evaluated for performance and bias. Our efforts quantified model performance using two custom evaluation metrics, neither of which have been correlated to human evaluation for the task.

As we discuss in the blog series, since the WNC is a parallel dataset and we formulate the learning task as a supervised problem, the model indirectly adopts Wikipedia's NPOV policy as the definition for "neutrality" and "subjectivity". The NPOV policy may not fully reflect an end users assumed/intended meaning of subjectivity because the notion of subjectivity itself can be...well, subjective.

We discovered through our exploratory work that the WNC does contain data quality issues that will contribute to unintended bias in the model. For example, some NPOV revisions introduce factual information outside the context of the prompt as a means to correct bias. We believe these factual based edits are out of scope for a subjective-to-neutral style transfer modeling task, but exist here nonetheless.

How to use

This model can be used directly with a HuggingFace pipeline for text2text-generation.

>>> from transformers import pipeline

>>> styletransfer = pipeline(
    task="text2text-generation",
    model="cffl/bart-base-styletransfer-subjective-to-neutral",
    max_length=200,
)
>>> input_text = "chemical abstracts service (cas), a prominent division of the american chemical society, is the world's leading source of chemical information."
>>> styletransfer(input_text)

[{'generated_text': 'chemical abstracts service (cas), a division of the american chemical society, is a source of chemical information.'}]

Training procedure

For modeling, we made extensive use of the Huggingface transformers library by initializing the BartForConditionalGeneration model with facebook/bart-base pretrained weights and adapting the summarization fine-tuning script for our TST-specific needs. We fine-tune the model for 15 epochs on an NVIDIA Tesla V100 GPU with a batch size of 32. (Note that when fine-tuning the model with the parallel examples, the noising function is turned off so an uncorrupted document is passed to BART's encoder and decoder.)

Please refer to our blog series for a discussion of evaluation metrics and results.