|
--- |
|
license: afl-3.0 |
|
--- |
|
|
|
# Generating Declarative Statements from QA Pairs |
|
|
|
There are already some rule-based models that can accomplish this task, but I haven't seen any transformer-based models that can do so. Therefore, I trained this model based on `Bart-base` to transform QA pairs into declarative statements. |
|
|
|
I compared the this model with other rule base models, including |
|
|
|
> [paper1](https://aclanthology.org/D19-5401.pdf) (2019), which proposes **2 Encoder Pointer-Gen model** |
|
|
|
and |
|
|
|
> [paper2](https://arxiv.org/pdf/2112.03849.pdf) (2021), which proposes **RBV2 model** |
|
|
|
**Here are results compared to 2 Encoder Pointer-Gen model (on testset released by paper1)** |
|
|
|
Test on testset |
|
|
|
| Model | 2 Encoder Pointer-Gen(2019) | BART-base | |
|
| ------- | --------------------------- | ---------- | |
|
| BLEU | 74.05 | **78.878** | |
|
| ROUGE-1 | 91.24 | **91.937** | |
|
| ROUGE-2 | 81.91 | **82.177** | |
|
| ROUGE-L | 86.25 | **87.172** | |
|
|
|
Test on NewsQA testset |
|
|
|
| Model | 2 Encoder Pointer-Gen | BART | |
|
| ------- | --------------------- | ---------- | |
|
| BLEU | 73.29 | **74.966** | |
|
| ROUGE-1 | **95.38** | 89.328 | |
|
| ROUGE-2 | **87.18** | 78.538 | |
|
| ROUGE-L | **93.65** | 87.583 | |
|
|
|
Test on free_base testset |
|
|
|
| Model | 2 Encoder Pointer-Gen | BART | |
|
| ------- | --------------------- | ---------- | |
|
| BLEU | 75.41 | **76.082** | |
|
| ROUGE-1 | **93.46** | 92.693 | |
|
| ROUGE-2 | **82.29** | 81.216 | |
|
| ROUGE-L | **87.5** | 86.834 | |
|
|
|
|
|
|
|
**As paper2 doesn't release its own dataset, it's hard to make a fair comparison. But according to results in paper2, the Bleu and ROUGE score of their model is lower than that of MPG, which is exactly the 2 Encoder Pointer-Gen model.** |
|
|
|
| Model | BLEU | ROUGE-1 | ROUGE-2 | ROUGE-L | |
|
| ------------ | ---- | ------- | ------- | ------- | |
|
| RBV2 | 74.8 | 95.3 | 83.1 | 90.3 | |
|
| RBV2+BERT | 71.5 | 93.9 | 82.4 | 89.5 | |
|
| RBV2+RoBERTa | 72.1 | 94 | 83.1 | 89.8 | |
|
| RBV2+XLNET | 71.2 | 93.6 | 82.3 | 89.4 | |
|
| MPG | 75.8 | 94.4 | 87.4 | 91.6 | |
|
|
|
There are reasons to believe that this model performs better than RBV2. |
|
|
|
To sum up, this model performs nearly as well as the SOTA rule-based model evaluated with BLEU and ROUGE score. However the sentence pattern is lack of diversity. |
|
|
|
(It's worth mentioning that even though I tried my best to conduct objective tests, the testsets I could find were more or less different from what they introduced in the paper.) |
|
|
|
## How to use |
|
|
|
|
|
```python |
|
from transformers import BartTokenizer, BartForConditionalGeneration |
|
tokenizer = BartTokenizer.from_pretrained("MarkS/bart-base-qa2d") |
|
model = BartForConditionalGeneration.from_pretrained("MarkS/bart-base-qa2d") |
|
|
|
input_text = "question: what day is it today? answer: Tuesday" |
|
input = tokenizer(input_text, return_tensors='pt') |
|
output = model.generate(input.input_ids) |
|
result = tokenizer.batch_decode(output, skip_special_tokens=True) |
|
``` |
|
|
|
|
