Effective and Efficient Self-supervised Masked Model Based on Mixed Feature Training

Qingjiu Kang¹Feng Liu²Chunliu Cai^3*

• ¹Guangzhou University, Guangzhou, China
• ²Scientific Research Department, Guangzhou Preschool Teachers College, Guangzhou, China
• ³Library, Guangdong University of Foreign Studies, Guangzhou, China

Under the influence of Masked Language Modeling (MLM), Masked Image Modeling (MIM) employs an attention mechanism to perform masked training on images. However, processing a single image requires numerous iterations and substantial computational resources to reconstruct the masked regions, resulting in high computational complexity and significant time costs. To address this issue, we propose an Effective and Efficient self-supervised Masked model based on Mixed feature training (EESMM). First, we stack two images for encoding and input the fused features into the network, which not only reduces computational complexity but also enables the learning of more features. Second, during decoding, we obtain the decoding features corresponding to the original images based on the decoding features of the two input original images and the mixed images, and then construct a corresponding loss function to enhance feature representation. EESMM significantly reduces pre-training time without sacrificing accuracy, achieving 83% accuracy on ImageNet in just 363 hours using four V100 GPUs—only one-tenth of the training time required by SimMIM. This validates that the method can substantially accelerate the pre-training process without noticeable performance degradation.