Make a Strong Teacher with Label Assistance: A Novel Knowledge Distillation Approach for Semantic Segmentation

In this paper, we introduce a novel <PRE_TAG>knowledge distillation</POST_TAG> approach for the <PRE_TAG>semantic segmentation</POST_TAG> task. Unlike previous methods that rely on power-trained teachers or other modalities to provide additional knowledge, our approach does not require complex <PRE_TAG>teacher model</POST_TAG>s or information from extra sensors. Specifically, for the <PRE_TAG>teacher model</POST_TAG> training, we propose to <PRE_TAG>noise</POST_TAG> the label and then incorporate it into input to effectively boost the lightweight teacher performance. To ensure the robustness of the <PRE_TAG>teacher model</POST_TAG> against the introduced <PRE_TAG>noise</POST_TAG>, we propose a <PRE_TAG>dual-path consistency</POST_TAG> training strategy featuring a distance loss between the outputs of two paths. For the student model training, we keep it consistent with the standard distillation for simplicity. Our approach not only boosts the efficacy of <PRE_TAG>knowledge distillation</POST_TAG> but also increases the flexibility in selecting teacher and <PRE_TAG>student model</POST_TAG>s. To demonstrate the advantages of our Label Assisted Distillation (LAD) method, we conduct extensive experiments on five challenging datasets including <PRE_TAG>Cityscapes</POST_TAG>, <PRE_TAG>ADE20K</POST_TAG>, PASCAL-VOC, COCO-Stuff 10K, and COCO-Stuff 164K, five popular models: FCN, PSPNet, DeepLabV3, STDC, and OCRNet, and results show the effectiveness and generalization of our approach. We posit that incorporating labels into the input, as demonstrated in our work, will provide valuable insights into related fields. Code is available at https://github.com/skyshoumeng/Label_Assisted_Distillation.