Exploring Self-Supervised Learning with U-Net Masked Autoencoders and EfficientNet B7 for Improved Classification

We present a self-supervised U-Net-based masked autoencoder and noise removal model designed to reconstruct original images. Once adequately trained, this model extracts high-level features, which are then combined with features from the EfficientNet B7 model. These integrated features are subsequently fed into dense layers for classification. Among the approaches of masked input and Gaussian noise removal, we selected the best U-Net reconstruction model. Additionally, we explored various configurations, including EfficientNet with attention, <PRE_TAG>attention fusion</POST_TAG> of the autoencoder, and classification utilizing <PRE_TAG>U-Net encoder features</POST_TAG>. The best performance was achieved with EfficientNet B7 combined with <PRE_TAG>U-Net encoder features</POST_TAG>. We employed the Adam optimizer with a learning rate of 0.0001, achieving a top accuracy of 0.94 on the validation set.