metadata

language: en
tags:
  - summarization
model-index:
  - name: google/pegasus-multi_news
    results:
      - task:
          type: summarization
          name: Summarization
        dataset:
          name: multi_news
          type: multi_news
          config: default
          split: test
        metrics:
          - type: rouge
            value: 47.5861
            name: ROUGE-1
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiODk5MjA3OWU0OTQ5OWM2MjdlZGFiNjAwNzQ4ODQ3ODUxMDg1MmVmMmZlM2I5MmZjNGFlYjIwZmZhNjE5ZDliZiIsInZlcnNpb24iOjF9.2JV0rRhAnYf7IZ5YTH9VWL1ZpKhsIowmfLvIy33vs7V3tML9TVEhIUwui6zwoyc7VnoemEp5-Y9990F5_jaIBw
          - type: rouge
            value: 18.7885
            name: ROUGE-2
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNGY1MDEyMjM3N2FlZWRmMDI5ZmJkOWIwZmRiNzA5M2IxZDU4Zjk2NzI5MGI3YTUxMDkxMThmZjliOTcwYWI4MyIsInZlcnNpb24iOjF9.4EQqViSHQ9i0dGLb1-6VixCQ4PmuQZ6d7gKiJr70xNZtBt124Y4hkVz-yNTKaiJ3fOxD7XP372eH1743UfdbAw
          - type: rouge
            value: 24.9734
            name: ROUGE-L
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiN2UyMGFlODNhNDM4NTVjZTM5ZWQ3YzVmYTYzODBmYjk3N2ExNTZiZGQ0YWJjZTA1NGQ3NDIyZTllY2U1OTFlNCIsInZlcnNpb24iOjF9.Ue926PfDVV2HyQ-H09fXgCFuey-mA1iKvnxEoHlvQZTvfqxq0_gWCL4bNS5DfawQGO2owMLAo_yguVhYwnoXAA
          - type: rouge
            value: 43.2093
            name: ROUGE-LSUM
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYzNiODk1ZDc5NDA0NDY0NjdiNzM3YTc5N2ZhM2FjZDY4NWQ5ZDUzMTNlMzA3YWVhMGIzMDExMmI4MTRkMjFiNiIsInZlcnNpb24iOjF9.rb1KNGXrJLBiV7CPMuc3YkYP6UyoZ9m7oxiHPFqZZ7hKGMW_6KFO0mpXeGAbX6OI_oVo6oNh1RwseQrOum4yAA
          - type: loss
            value: 2.3634445667266846
            name: loss
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNDk0OGRiMzk2ZDhlYjYzYjUyNTI3MzcwMzZiZWVjYzFlN2QwYTI2ZjM3NzM3NzY2ZDUxYTVlMjZhMTFlZDk1ZCIsInZlcnNpb24iOjF9.avci5v52cAP5kppL3mb2TTV59hhPgYo3TJtCyyKbDVTKmDHxZoNzTGUoswHzew8oTicEvHegsbcn5tyeP73cBA
          - type: gen_len
            value: 223.0068
            name: gen_len
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZDk5M2VmZGJkYjc2MmRjNDFhNzg3NTU0ZTFjNzM2NDQzNGI2NjliMzdjMTBhMTNmMTA3NWY1NDQyZTI2MmQxMyIsInZlcnNpb24iOjF9.Cj_cJRmr-tj_-BWl0ij0qGfRTUtqMhlmFydW3C76rJfN1IQAFUsXB_LW7YYHbZwC8pAPuWfXjiYo6N4LFFH0Aw

Pegasus Models

See Docs: here

Original TF 1 code here

Authors: Jingqing Zhang, Yao Zhao, Mohammad Saleh and Peter J. Liu on Dec 18, 2019

Maintained by: @sshleifer

Task: Summarization

The following is copied from the authors' README.

Mixed & Stochastic Checkpoints

We train a pegasus model with sampled gap sentence ratios on both C4 and HugeNews, and stochastically sample important sentences. The updated the results are reported in this table.

dataset	C4	HugeNews	Mixed & Stochastic
xsum	45.20/22.06/36.99	47.21/24.56/39.25	47.60/24.83/39.64
cnn_dailymail	43.90/21.20/40.76	44.17/21.47/41.11	44.16/21.56/41.30
newsroom	45.07/33.39/41.28	45.15/33.51/41.33	45.98/34.20/42.18
multi_news	46.74/17.95/24.26	47.52/18.72/24.91	47.65/18.75/24.95
gigaword	38.75/19.96/36.14	39.12/19.86/36.24	39.65/20.47/36.76
wikihow	43.07/19.70/34.79	41.35/18.51/33.42	46.39/22.12/38.41 *
reddit_tifu	26.54/8.94/21.64	26.63/9.01/21.60	27.99/9.81/22.94
big_patent	53.63/33.16/42.25	53.41/32.89/42.07	52.29/33.08/41.66 *
arxiv	44.70/17.27/25.80	44.67/17.18/25.73	44.21/16.95/25.67
pubmed	45.49/19.90/27.69	45.09/19.56/27.42	45.97/20.15/28.25
aeslc	37.69/21.85/36.84	37.40/21.22/36.45	37.68/21.25/36.51
billsum	57.20/39.56/45.80	57.31/40.19/45.82	59.67/41.58/47.59

The "Mixed & Stochastic" model has the following changes:

trained on both C4 and HugeNews (dataset mixture is weighted by their number of examples).
trained for 1.5M instead of 500k (we observe slower convergence on pretraining perplexity).
the model uniformly sample a gap sentence ratio between 15% and 45%.
importance sentences are sampled using a 20% uniform noise to importance scores.
the sentencepiece tokenizer is updated to be able to encode newline character.

(*) the numbers of wikihow and big_patent datasets are not comparable because of change in tokenization and data:

wikihow dataset contains newline characters which is useful for paragraph segmentation, the C4 and HugeNews model's sentencepiece tokenizer doesn't encode newline and loose this information.
we update the BigPatent dataset to preserve casing, some format cleanings are also changed, please refer to change in TFDS.

The "Mixed & Stochastic" model has the following changes (from pegasus-large in the paper):

trained on both C4 and HugeNews (dataset mixture is weighted by their number of examples). trained for 1.5M instead of 500k (we observe slower convergence on pretraining perplexity). the model uniformly sample a gap sentence ratio between 15% and 45%. importance sentences are sampled using a 20% uniform noise to importance scores. the sentencepiece tokenizer is updated to be able to encode newline character.

Citation



@misc{zhang2019pegasus,
    title={PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization},
    author={Jingqing Zhang and Yao Zhao and Mohammad Saleh and Peter J. Liu},
    year={2019},
    eprint={1912.08777},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}