| Summary: | The unique giant murals of Yongle Palace, as valuable cultural heritage, have suffered damage and urgently require restoration. Deep learning inpainting algorithm offers promising solutions for image restoration. However, the restoration of unique giant murals presents two challenges: 1) The unique and scarce data of mural poses significant training difficulties, and 2) the giant size leads to a wider range of defect types and sizes, increasing the complexity of restoration. To address these challenges, a 3M-Hybrid model is proposed. Firstly, on the data level, based on the frequency characteristics of mural data, multi-frequency complementary learning is employed to enhance the model’s restoration capability. Secondly, on the model structure level, a pre-trained Visual Transformer (VIT) is integrated into the Convolutional Neural Networks (CNN) module to alleviate data scarcity and reduce domain bias. Finally, seam and structural distortion problems arising from repairing oversized defects are mitigated by multi-scale and multi-perspective strategies, including data segmentation and fusion. Experimental results demonstrate the efficacy of our proposed model. In regular-sized mural restoration, it improves Structural Similarity Image Measurement (SSIM) and Peak Signal-to-Noise Ratio (PSNR) by 14.61% and 4.73%, respectively, compared to the best model among four representative CNN models. Additionally, it achieves favorable results in the final restoration of giant murals.
|
|---|