MULTIMODAL FUSIONS FOR DEFECT DETECTION OF PHOTOVOLTAIC PANELS BY MASK R-CNN AND HAWKFISH OPTIMIZATION ALGORITHM

Nazar Almukhtar^*Sefer Kurnaz

• Altinbas Universitesi Mahmutbey Teknoloji Yerleskesi, Istanbul, Türkiye

Accurate detection of photovoltaic (PV) module defects requires integrating information from multiple sensing modalities, as single-image approaches often fail under real-world variability. This paper proposes a multimodal segmentation framework that fuses RGB, infrared (IR), and electroluminescence (EL) imagery within a modified Mask R-CNN architecture. RGB provides structural and surface context, EL captures subsurface micro-cracks, and IR contributes complementary thermal signatures associated with hotspots, cell mismatch, and localized heating fault characteristics that cannot be observed in RGB or EL alone. A dedicated alignment pipeline based on homography and ECC refinement ensures geometric consistency across modalities, while a feature-level Fusion Attention Block adaptively combines modality-specific features. Hyperparameters and fusion weights are automatically tuned using the HawkFish Optimization Algorithm (HFOA), improving stability and segmentation accuracy. Experiments on statistically paired RGB–EL–IR datasets demonstrate that incorporating IR imagery significantly enhances the detection of thermally driven defects and reduces false negatives in low-contrast regions. The proposed framework achieves state-of-the-art performance, confirming the value of thermal information in multimodal PV inspection.