|
--- |
|
datasets: |
|
- samsum |
|
language: |
|
- en |
|
metrics: |
|
- rouge |
|
- bleu |
|
library_name: transformers |
|
pipeline_tag: summarization |
|
--- |
|
|
|
# t5-small-finetuned |
|
|
|
## Model Description |
|
- **Purpose and Use**: This model is designed for abstractive text summarization with a focus on the SAMSum Dialogue Dataset. |
|
- **Model Architecture**: The architecture is based on a fine-tuned T5-small model, which consists of 60 million parameters. |
|
- **Training Data**: Trained on the SAMSum Dialogue Dataset, which comprises approximately 15,000 dialogue-summary pairs. |
|
|
|
## Training Procedure |
|
- **Preprocessing**: Data preprocessing involved the removal of irrelevant tags and tokenization to ensure data consistency. |
|
- **Training Details**: The model was fine-tuned over 4 epochs with a learning rate of 2e-5 and a batch size of 2, utilizing gradient accumulation for optimization. |
|
- **Infrastructure**: Training was conducted using GPU acceleration and the Hugging Face Trainer API, with progress monitored via TensorBoard. |
|
|
|
## Evaluation Results |
|
- **Metrics Used**: Evaluation metrics included ROUGE-1, ROUGE-2, ROUGE-L, BLEU, and Cosine Similarity. |
|
- **Performance**: The fine-tuned T5-small model demonstrated superior efficiency and effectiveness in summarization tasks, outperforming its larger counterparts. |
|
|
|
## Validation and Test Set Performance |
|
|
|
| Metric | Validation Set | Test Set | |
|
|----------|--------------------|--------------| |
|
| ROUGE-1 | 0.5667 | 0.5536 | |
|
| ROUGE-2 | 0.2923 | 0.2718 | |
|
| ROUGE-L | 0.5306 | 0.5210 | |
|
|
|
The table above presents the performance of the model on both the validation and test sets, indicating the quality of content overlap and structural fluency in the summaries generated. |
|
|
|
## Performance Metrics Comparison Across Models |
|
|
|
| Model | ROUGE-1 | ROUGE-2 | ROUGE-L | BLEU | Cosine Similarity | |
|
|----------|---------|---------|---------|------|-------------------| |
|
| My Model | 0.3767 | 0.1596 | 0.2896 | 9.52 | 0.7698 | |
|
| T5 Large | 0.3045 | 0.0960 | 0.2315 | 4.82 | 0.6745 | |
|
| Bart | 0.3189 | 0.0989 | 0.2352 | 6.28 | 0.6961 | |
|
| Pegasus | 0.2702 | 0.0703 | 0.2093 | 3.88 | 0.6432 | |
|
|
|
In the table above shows results on 50 samples for the test set that is being compared across various models. |
|
|
