---
language: en
license: apache-2.0
tags:
- azureml
- t5
- summarization
- deepspeed
datasets:
- samsum
widget:
- text: 'Kevin: Hey man, are you excited to watch Finding Nemo tonight?

    Henry: Yea, I can''t wait to watch that same movie for the 89th time. Is Nate
    coming over to watch it with us tonight?

    Kevin: Yep, he said he''ll be arriving a bit later at around 7 since he gets off
    of work at 6. Have you taken out the garbage yet? It''s starting to make the kitchen
    really smell.

    Henry: Oh I forgot. I''ll do that once I''m finished with my assignment for my
    math class. I didn''t get to start on it until an hour ago, and it''s due in 30
    minutes.

    Kevin: Okay dude, you should take it out as soon as possible. By the way, Nate
    is bringing his girlfriend and their cat too.

    Henry: Nice, I''m really looking forward to seeing them again.'
model-index:
- name: henryu-lin/t5-large-samsum-deepspeed
  results:
  - task:
      type: summarization
      name: Summarization
    dataset:
      name: samsum
      type: samsum
      config: samsum
      split: train
    metrics:
    - type: rouge
      value: 40.8694
      name: ROUGE-1
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZjgxNDg4YjM4YjY0MDVhZmY5ZjQ1YTgyN2RhOGIwY2M5YjljMjAwYjI0ZWEzMzMxMzBlYmE5MjY3ODM1MjI4YiIsInZlcnNpb24iOjF9.NkOSwlWC_r8ewewRk1X9KJxaTEWZ0lDz0SuABLeUf1tESeTBowSJJBXgwiYb7gjpHnipfcK2HczlNRl-KzdDAA
    - type: rouge
      value: 19.223
      name: ROUGE-2
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNjc3ZTY3ZWU0OWE5Zjc1ZjdiZWE4NDQ0YzI3MTMxNTM3ZDRjY2Y1YWM1OWQyOWMwMTZlMmRlZTI5ZGNkMmI5OSIsInZlcnNpb24iOjF9.4jHtzkDGNLPHSC7RN9Hi5jeiLy9F3JwBpDKdCjkiDmZY_cgHHCTr5v6QTr7VISZNQdCNg27iO0d8ohSIxVVXBg
    - type: rouge
      value: 31.0688
      name: ROUGE-L
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNmQzZjhhMjJjZWFkYWViODlhMDQ4MTY0YWI4NzRjZWMyMGZlMDQ1MjA0Yzk0MTczOWU0ODMyYjQ3NGEwOTZhNiIsInZlcnNpb24iOjF9.nUPHLaP5n_7YYbtV6ms0-fOGtPvEx826Ivsv-MfKiUVKyxTJ-9G_xbECK2cS1XQxuO05tlWhO89zz03vsNkuAQ
    - type: rouge
      value: 38.3786
      name: ROUGE-LSUM
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNTJlMGUwMTE4NWRkMWZkYzU0NzA3ODQzOWE2MWY3MWU2YmFjNzQ0ZTU2MDZiYTY0ZmY2N2U4NmUyOTY5NDRkMiIsInZlcnNpb24iOjF9.-T68JCuA99EVzu4fIOJN-Vyu-d__RYvfnKPaLu4pJ2cOmRVKh2Qc6pHnjXDP2powPu2R6pD6KcANZhEE4AEVAw
    - type: loss
      value: 2.184831380844116
      name: loss
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNTJkZjk3Y2YwM2E4ZDQxYzZkOGNkYzY3N2I4NzkyZmU4ZjYyYjk1Y2FiZjRkN2I1MTEzZmI4Y2FjNjBiYWNjZiIsInZlcnNpb24iOjF9.7OgmxB2mQ7CYH9p9p56bf7cAjkA6YflzB75zd0-O1WYgrsEibX-Zb2H6-0SMqxD-drWrRrEpma1Tu1fWSkBhDQ
    - type: gen_len
      value: 42.2081
      name: gen_len
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMmEyZmYyOGVmN2JlYzM2MzI4YzU3ZDhjYmYwZDlhZTliNzk0ODVjNjU5MGNlMmRjOGZiOTk0MGU1YWM0NDcxMCIsInZlcnNpb24iOjF9.b9-F5AzXERN-pVcH61r23kaqdKO4iX79mQPRnoZ_riZ91o6UihsNftdGa50vgleloGDwkKT4aR6PNMZujCRZDQ
---

## `t5-large-samsum-deepspeed`
This model was trained using Microsoft's `AzureML` and `DeepSpeed`'s ZeRO 2 optimization. It was fine-tuned on the `SAMSum` corpus from `t5-large` checkpoint.

More information on the fine-tuning process (includes samples and benchmarks):  
*(currently still WIP, major updates coming soon: 7/6/21~7/9/21)*

## Resource Usage
These results are retrieved from AzureML Studio's resource monitoring module. All experiments were ran on AzureML's low priority clusters.

| key | value |
| --- | ----- |
| AzureML SKU | ND40rs_v2 (8 X V100 32GB) |
| Region | US West 2 |
| Run Duration | 12m 47.13s |
| Compute Cost (LowPriority/Dedicated) | $0.94/$4.69 (USD) |
| Average CPU Utilization | 51.2% |
| Average GPU Utilization | 42.0% |
| GPU Memory Usage (Avg/Peak) | 24.85/28.79 (GB) |
| Total GPU Energy Usage | 670.38 (kJ) |

*Compute cost is calculated from run duration and SKU's price per hour. Updated SKU pricing could be found here: https://azure.microsoft.com/en-us/pricing/details/machine-learning/  
*Peak memory usage is calculated from average peak across all utilized GPUs.  

### Carbon Emissions
These results are obtained using `codecarbon`. The carbon emission is estimated from training runtime only (excluding setup and evaluation runtime).  
CodeCarbon: https://github.com/mlco2/codecarbon  

| key | value |
| --- | ----- |
| timestamp | 2021-07-08T06:29:27 |
| duration | 515.5018835067749 |
| emissions | 0.043562840982919106 |
| energy_consumed | 0.14638051405550773 |
| country_name | USA |
| region | Washington |
| cloud_provider | azure |
| cloud_region | westus2 |

## Hyperparameters
```yaml
fp16: True
per device batch size: 8
effective batch size: 64
epoch: 3.0
learning rate: 1e-4
weight decay: 0.1
seed: 1
```
*Same `per device batch size` for evaluations

### DeepSpeed
Optimizer = `AdamW`, Scheduler = `WarmupDecayLR`, Offload = `none`
```json
  "zero_optimization": {
    "stage": 2,
    "allgather_partitions": true,
    "allgather_bucket_size": 1300000000,
    "overlap_comm": true,
    "reduce_scatter": true,
    "reduce_bucket_size": 1300000000,
    "contiguous_gradients": true
  }
```

## Usage
```python
from transformers import pipeline
summarizer = pipeline("summarization", model="henryu-lin/t5-large-samsum-deepspeed")

conversation = '''Kevin: Hey man, are you excited to watch Finding Nemo tonight?
    Henry: Yea, I can't wait to watch that same movie for the 89th time. Is Nate coming over to watch it with us tonight?
    Kevin: Yep, he said he'll be arriving a bit later at around 7 since he gets off of work at 6. Have you taken out the garbage yet? It's starting to make the kitchen really smell.
    Henry: Oh I forgot. I'll do that once I'm finished with my assignment for my math class. I didn't get to start on it until an hour ago, and it's due in 30 minutes.
    Kevin: Okay dude, you should take it out as soon as possible. By the way, Nate is bringing his girlfriend and their cat too.
    Henry: Nice, I'm really looking forward to seeing them again.
'''
summarizer(conversation)
```

## Results
| ROUGE | Score |
| ----- | ----- |
| eval_rouge1 | 53.0823 |
| eval_rouge2 | 28.7097 |
| eval_rougeL | 43.939 |
| eval_rougeLsum | 49.067 |
| predict_rouge1 | 51.6716 |
| predict_rouge2 | 26.5372 |
| predict_rougeL | 42.9681 |
| predict_rougeLsum | 47.4084 |

| Metric | Value |
| ------ | ----- |
| eval_gen_len | 26.4071 |
| predict_gen_len | 25.9451 |
| train_loss | 1.3212629926497115 |
| eval_loss | 1.23828125 |
| predict_loss | 1.2333984375 |
| train_runtime | 515.2198 |
| train_samples | 14732 |
| train_samples_per_second | 85.781 |
| train_steps_per_second | 1.345 |
| eval_runtime | 61.275 |
| eval_samples | 818 |
| eval_samples_per_second | 13.35 |
| eval_steps_per_second | 0.212 |
| predict_runtime | 63.3732 |
| predict_samples | 819 |
| predict_samples_per_second | 12.923 |
| predict_steps_per_second | 0.205 |
| total_steps | 693 |
| total_flos | 7.20140924616704e+16 |