Targetless LiDAR–Camera Extrinsic Calibration via Semantic Distribution Alignment

Xi Chen^1,2Bingyu Sun^1*

• ¹Hefei Institutes of Physical Science, Chinese Academy of Sciences (CAS), Hefei, China
• ²University of Science and Technology of China, Hefei, China

LiDAR–camera fusion systems are widely used in robotic localization and perception, where 3 accurate extrinsic calibration is crucial for multi-sensor fusion in field and autonomous robotics. 4 During long-term operation, however, the extrinsic parameters can drift because of vibration 5 and other disturbances, degrading localization accuracy and perception stability. Traditional 6 target-based calibration requires dedicated boards and controlled environments, making on-7 site recalibration inconvenient, while targetless approaches often suffer from highly non-convex 8 objectives and limited robustness in challenging outdoor scenes. To address these issues, 9 we propose a targetless LiDAR–camera extrinsic calibration method that casts calibration as 10 minimizing a semantic distribution consistency risk on SE(3). We align semantic probability 11 distributions from the two sensing modalities in the image domain and freeze the pixel sampling 12 measure at an anchor pose, so that pixel weighting no longer depends on the current extrinsic 13 estimate and the objective landscape remains stable during optimization. On top of this anchor-14 fixed measure, we introduce a direction-aware weighting strategy that emphasizes pixels sensitive 15 to yaw perturbations, improving the conditioning of rotation estimation. We further use a globally 16 balanced Jensen–Shannon divergence to mitigate semantic class imbalance and enhance 17 robustness. Experiments on the KITTI Odometry dataset show that the proposed method reliably 18 converges from substantial initial perturbations and yields stable extrinsic estimates, indicating its 19 promise for maintaining long-term LiDAR–camera calibration in real-world robotic systems.