Average scores by model
 |
The checkpoint-41448 (T5-GenQ-TDC-v1) model outperforms query-gen-msmarco-t5-base-v1 across all ROUGE metrics. The largest performance gap is in ROUGE2, where checkpoint-41448 achieves 56.11, whereas query-gen-msmarco-t5-base-v1 scores 15.29. ROUGE1, ROUGEL, and ROUGELSUM scores are very similar in both trends, with checkpoint-41448 consistently scoring above 72, while query-gen-msmarco-t5-base-v1 stays below 41. |
Density comparison
 |
```T5-GenQ-TDC-v1``` - Higher concentration of high ROUGE scores, especially near 100%, indicating strong text overlap with references. ```query-gen-msmarco-t5-base-v1``` โ more spread-out distribution, with multiple peaks at 10-40%, suggesting greater variability but lower precision. ROUGE-1 & ROUGE-L: ```T5-GenQ-TDC-v1``` peaks at 100%, while ```query-gen-msmarco-t5-base-v1``` has lower, broader peaks. ROUGE-2: ```query-gen-msmarco-t5-base-v1``` has a high density at 0%, indicating many low-overlap outputs. |
Histogram comparison
 |
```T5-GenQ-TDC-v1``` โ higher concentration of high ROUGE scores, especially near 100%, indicating strong text overlap with references. ```query-gen-msmarco-t5-base-v1``` โ more spread-out distribution, with peaks in the 10-40% range, suggesting greater variability but lower precision. ROUGE-1 & ROUGE-L: ```T5-GenQ-TDC-v1``` shows a rising trend towards higher scores, while ```query-gen-msmarco-t5-base-v1``` has multiple peaks at lower scores. ROUGE-2: ```query-gen-msmarco-t5-base-v1``` has a high concentration of low-score outputs, whereas ```T5-GenQ-TDC-v1``` generates more accurate text. |
Scores by generated query length
 |
This visualization analyzes average ROUGE scores and score differences across different word sizes. Consistently High ROUGE Scores (Sizes 2-8): ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-LSUM scores remain stable and above 60% across most word sizes. Significant Drop at Size 9: A sharp decline in scores at size 9, followed by a spike at size 10, indicates a possible edge case in the dataset. Stable Score Differences (Sizes 3-9): After the initial spike at size 2, score differences stay close to zero, indicating consistent performance across phrase lengths. |
Semantic similarity distribution
 |
This histogram visualizes the distribution of cosine similarity scores, which measure the semantic similarity between paired texts. The majority of similarity scores cluster near 1.0, indicating that most text pairs are highly similar. A gradual increase in frequency is observed as similarity scores rise, with a sharp peak at 1.0. Lower similarity scores (0.0โ0.4) are rare, suggesting fewer instances of dissimilar text pairs. |
Semantic similarity score against ROUGE scores
 |
This scatter plot matrix compares semantic similarity (cosine similarity) with ROUGE scores, showing their correlation. Higher similarity โ Higher ROUGE scores, indicating strong n-gram overlap in semantically similar texts. ROUGE-1 & ROUGE-L show the strongest correlation, while ROUGE-2 has more variability. Some low-similarity outliers still achieve moderate ROUGE scores, suggesting surface-level overlap without deep semantic alignment. These plots help assess how semantic similarity aligns with n-gram overlap metrics in text evaluation. |
Performance on Another Dataset
To assess the performance of our fine-tuned query generation model, we conducted evaluations on a dataset containing real user queries, which was not part of the fine-tuning data. The goal was to verify the modelโs generalizability and effectiveness in generating high-quality queries for e-commerce products.
Average scores by model
|
The figure presents a comparison of ROUGE scores between GenQ-TDC-v1 (checkpoint-41448) and the base model (query-gen-msmarco-t5-base-v1).
ROUGE scores (ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-LSUM) measure the similarity between generated queries and ground-truth user queries.
The fine-tuned model outperforms the base model across all ROUGE metrics, with significant improvements in ROUGE-1, ROUGE-2, and ROUGE-LSUM. These improvements indicate that the fine-tuned model generates queries that are more aligned with real user queries, making it a better fit for practical e-commerce applications. |
Semantic similarity distribution
|
The figure shows the distribution of semantic similarity scores (cosine similarity between the generated queries and real user queries.
The distribution shows a peak around 0.1 - 0.2, with a secondary peak near 0.6 - 0.7. The bimodal nature of the distribution suggests that while a portion of generated queries are highly aligned with real user queries, some still exhibit lower similarity, likely due to domain-specific variations. A significant number of generated queries achieve a similarity score of 0.6 or higher, confirming that the fine-tuned model effectively captures user intent better than a generic, pre-trained model. |
Better Generalization: Even though the fine-tuned model was not trained on this dataset, it generalizes better than the original model on real user queries.
Improved Query Quality: The fine-tuned model produces more relevant, structured, and user-aligned queries which is critical for enhancing search and recommendation performance in e-commerce.
Robust Semantic Alignment: Higher semantic similarity scores indicate that queries generated by the fine-tuned model better match user intent, leading to improved search and retrieval performance.