Currently, the UAVs mainly used for crack detection include DJ Mavic 2 Pro, DJ M210-RTK, DJ Mini 2, and DJ Mavic Air 2 (Xu et al., 2023a). Due to the large span of bridges inspected by UAVs, the main parameters to consider are hovering time and hovering accuracy. Additionally, since it involves storing appearance images of large-span bridges, there is also a high requirement for memory. Through an analysis of the parameters of the four UAVs, DJ Mavic Air 2 is the best option. Its hover time is 33 min, hover accuracy is ± 0.1, and memory size is 8 GB. To ensure consistent experimental conditions, the distance between the UAV and the target surface is fixed at 5.0m, and the normal direction of the lens is perpendicular to the target surface. The images will be taken in clear weather conditions and natural lighting. To ensure sufficient image data, the UAV will cover the entire bridge deck and tower surface, with a 50% overlap in the images taken.

In the process of picture-taking and crack detection of concrete bridges, a geometric model of camera imaging is required to determine the interrelationship between the three-dimensional geometric position of a point on the surface of the bridge structure and its corresponding point in the image. In the UAVIAT, the calibration of camera parameters is the key link, and its calculation process and calibration results directly affect the authenticity of the crack detection results. The conversion model of the ground coordinate system to the pixel coordinate system is introduced below.

1) Transformation from ground coordinate system to camera coordinate system.

In order to accurately describe the motion trajectory of the UAV and obtain its position information, a transformation model from the ground coordinate system X G , Y G , Z G to the camera coordinate system X C , Y C , Z C has been established. They represent the angle of camera rotation around the X_G, Y_G, and Z_G axes, respectively (Figure 2). Then, the rotation matrix R from the ground coordinate system to the image coordinate system can be obtained, as shown in Eq. 1. R = 1 0 0 0 cos ⁡ σ sin ⁡ σ 0 − ⁡ sin ⁡ σ cos ⁡ σ ⋅ cos ⁡ μ 0 − ⁡ sin ⁡ μ 0 1 0 sin ⁡ μ 0 cos ⁡ μ ⋅ cos ⁡ ε sin ⁡ ε 0 − ⁡ sin ⁡ ε cos ⁡ ε 0 0 0 1 (1)

In addition to the rotation transformation, there is also a translation transformation between the UAV and the ground. Therefore, the coordinate transformation matrix from the ground coordinate system to the camera coordinate system is given by Eq. 2. M 1 = R T O 1 (2) where T = T x T y T z T is the translation transformation matrix. T x , T y and T z represent the displacement of the UAV in the X_G, Y_G, and Z_G directions, respectively.

2) Transformation from camera coordinate system to image coordinate system.

As shown in Figure 3A, the transformation from the camera coordinate system to the image coordinate system X , Y , Z follows the pinhole imaging principle. The Z_C axis of the camera coordinate system is on the same line as the Z-axis of the image coordinate system. P is a point in space, P₁ is its projection point in the plane X_CO_CY_C, P₂ is the imaging point of P in the XOY plane of the image coordinate system, f is the focal length, and z is the distance from point P to point P₁. The conversion matrix F from camera coordinates to image coordinates is given by Eq. 3. F = f / z 0 0 0 0 f / z 0 0 0 0 1 0 (3)

3) Transformation from image coordinate system to pixel coordinate system.

The pixel coordinate system reflects the arrangement of pixels in the CMOS chip of the camera, as shown in Figure 3B. The image coordinate system and the pixel coordinate system u , v are in a translation relationship, and the transformation matrix D between them is shown in Eq. 4. D = 1 / d X 0 u 0 0 1 / d Y v 0 0 0 1 (4) where u 0 , v 0 is the coordinate of the image coordinate system origin in the pixel coordinate system; d X and d Y represent the physical dimensions of the pixel in the X and Y directions, respectively.

In summary, the formula for transforming the ground coordinate system to the pixel coordinate system is shown in Eq. 5. Eq. 6 represents the calculation of the intrinsic parameter matrix of the camera. u v 1 = D g F g M 1 g X G Y G Z G 1 = M 2 g M 1 g X G Y G Z G 1 (5) M 2 = D g F = f / z d X 0 u 0 0 0 f / z d Y v 0 0 0 0 1 0 (6) where u , v , 1 T is the homogeneous coordinate in the pixel coordinate system, and X G , Y G , Z G , 1 T is the homogeneous coordinate in the ground coordinate system. M₁ is the external parameter matrix of the camera, and M₂ is the internal parameter matrix of the camera.

In this article, entropy and sharpness are utilized as two image quality parameters to assess the quality of the images captured from the UAV.10 (Duque et al., 2018). The formula for judging the image quality is given by Eq. 7 and Eq. 8. The criteria for judging the image quality are:

1) The image sharpness is greater than the average value of sharpness.

The image satisfying any of these points is considered a qualified image. S k ≥ 1 N ∑ k = 1 N 1 P s u m ∑ j = 1 P s u m ∑ i = 1 8 d G i / d x i (7) S k ≤ 1 N ∑ k = 1 N 1 P s u m ∑ j = 1 P s u m ∑ i = 1 8 d G i d x i , E k ≤ 1 N ∑ k = 1 N − ∑ i = 0 255 p i ⁡ log 2 p i (8) where N is the total number of captured bridge images, S k is the sharpness of the kth image, P s u m is the total number of pixels in the image, d G i is the amplitude of the grayscale variation, and d x i is the distance increment between pixels. d G i / d x i is the gradient vector between pixels, which is calculated based on the pixel values of the eight neighboring pixels for each pixel. p i is the occurrence probability of the gray value i in the image, which is obtained from the grayscale histogram.

The bridge surface images that meet the quality requirements are input into the crack detection model YOLOv7. In the beginning, the input image enters the main feature extraction network Backbone, which performs crack feature extraction by convolutional normalization and activation function. In the feature extraction process, the image will be compressed in height and width first, then the channel expansion will be performed, and finally, the three effective crack feature layers will be formed. Then, the three effective crack feature layers obtained at Backbone will be fused by the FPN framework, which aims to combine the crack feature information at different scales for enhanced feature extraction of the image. FPN performs up-sampling and down-sampling on the crack features to achieve the fusion of crack features. After passing through the Backbone and FPN, three enhanced effective crack feature layers will be obtained. Each crack feature layer has a width, height, and number of channels. At this point, the crack feature map can be viewed as a collection of feature points, with three prior boxes at each feature point, each of which has the same number of crack features as the number of channels. Eventually, the RepConv structure is introduced in the Head part to equivalent the complex residual structure to a normal 3 × 3 convolution. This can reduce the complexity of the network while ensuring the same prediction performance.

The task of locating and identifying cracks focuses on details such as background color and crack position. Therefore, in order to enhance the model’s perception of crack features and improve the detection effect on small targets, three SE channel attention modules were added to the YOLOv7 model. The SE attention module has three steps:

1) Using adaptive global average pooling to compress the length and width of the crack feature layer, leaving only the information of the channel dimension C.

Neural network model training is the process of optimizing the parameters in the network and reducing the losses using a backpropagation algorithm. Loss is the penalty for inaccurate predictions during the training process and describes the difference between the model’s predicted results and the actual results. Eq. 9 is the formula for calculating the loss function of YOLOv7 during the training process. L = L C I o U + L o b j + L c l s (9) where L is the total loss of the model, L C I o U is the bounding box regression loss used to measure the deviation between the predicted and ground truth crack boxes, L o b j is the confidence loss used to measure the accuracy of crack localization, and L c l s is the classification loss used to measure the accuracy of predicting the presence of cracks. The calculation formulas for L o b j and L c l s are shown in Eq. 10 and Eq. 11. L o b j = − ∑ i = 0 S 2 ∑ j = 0 B I i , j o b C i j ⁡ log C i j ^ + 1 − C i j log 1 − C i j ^ − λ n o ∑ i = 0 S 2 ∑ j = 0 B I i , j n o C i j ⁡ log C i j ^ + 1 − C i j log 1 − C i j ^ (10) L c l s = − ∑ i = 0 S 2 I i , j o b ∑ P i j ⁡ log P i j ^ + 1 − P i j log 1 − P i j ^ (11) where B and S 2 are the priori box and the feature map scale. I i , j o b and I i , j n o denote crack and no crack in the jth prior box of the ith grid. C i j and C i j ^ denote the confidence of the predicted and labeled box. λ n o is the weight coefficient. P i j and P i j ^ denote the classification probability of the predicted and labeled box.

When calculating the regression loss, the EIoU was used instead of complete intersection over union (CIoU) to consider the effects of overlapping area, centroid distance, and aspect ratio of the target and prediction boxes simultaneously. The EIoU takes into account the width and height loss, which minimizes the difference between the width and height of the target and prediction boxes, thereby accelerating convergence and improving regression accuracy. Since cracks are small and occupy a small proportion of the background, the focal loss function is introduced to balance the proportion of foreground and background data samples (Lin et al., 2020; Wang et al., 2023a). Finally, the Focal-EIoU loss function is obtained, as shown in Eq. 12 and Eq. 13. L E I o U = 1 − I o U + ρ 2 b , b g t c w 2 + c h 2 + w − w g t 2 c w 2 + h − h g t 2 c h 2 (12) L F o c a l − E I o U = I o U γ L E I o U (13) where I o U is the ratio of the intersection of the area of the target box and the predicted box to the concatenated set. b , w and h are the coordinates of the center point, width, and height of the predicted box, respectively. b g t , w g t and h g t are the coordinates of the center point, width, and height of the target box, respectively. ρ 2 b , b g t represents the distance between two center coordinates. c w and c h are the minimum width and height of the outlier boxes of the target and prediction boxes. λ is a parameter controlling the degree of outlier suppression.

The YOLOv7 model, which integrates the SE attention module and the Focal-EIoU Loss, is named YOLOv7-CD, as shown in Figure 4. The red part indicates the added SE attention module, and the red arrow represents the changed computation path.

The larger the batch size, the higher the GPU performance requirements and the batch size is usually a power of 2 (Dong et al., 2021). Therefore, in this experiment, the batch size was set to 2, 4, and 8 based on the actual hardware configuration. And the performance of two learning rate optimization algorithms, SGD and Adam, was compared simultaneously. To ensure the reliability of batch size optimization, the initial learning rate was set to 0.01, and 50 epochs were trained in the freezing phase and 250 epochs in the thawing phase on the QL_CRACK dataset. Figure 7 shows the relationship between Precision, Recall, F1, and AP with different batch sizes for different learning rate optimization algorithms. The blue color represents the SGD optimization algorithm, and the grey color represents the Adam optimization algorithm. The results are also summarized in Table 2.

Combined with Figure 7 and Table 2, it can be seen that Precision is the largest for Number 3 and Number 5, with 95.51% and 94.99%, respectively; Recall is the largest for Number 5 and Number 6, with 96.56% and 96.80%, respectively; F1 is the largest for Number 3, Number 5 and Number 6, all with 0.96. When the batch size is 4 and 8, the precision, recall, and F1 are all higher than when the batch size is 2.

When the batch size is increased from 2 to 4, the AP for SGD and Adam optimization algorithms improved by 1.08% and 0.76%, respectively, while when the batch size is increased from 4 to 8, the AP only improved by 0.03% and 0.1%, respectively. This indicates that the AP obtained by different optimization algorithms (SGD and Adam) have similar trends with the batch size, both of which have a large change in the batch size from 2 to 4 and a small change in the batch size from 4 to 8. The AP for the SGD and Adam optimization algorithms are the largest at a batch size of 8, but the difference with a batch size of 4 is small. Considering both the saving of video memory and the speed of training, the batch size of the freezing phase was set to 8, and the training batch size of the thawing phase was set to 4 during the experiment.

The learning rate affects how fast the algorithm converges to the regionally minimal value. A suitable learning rate allows the algorithm to descend in the direction of the maximum gradient in appropriate steps, and the learning rate can be effectively optimized by the decreasing gradient of the loss function. Since the YOLOv7 model uses the learning rate optimization algorithm, only the initial learning rate and the optimization algorithm can be considered in the optimization (Wang et al., 2023b).

In this manuscript, the range of the initial learning rate was set from 10^−5.0 to 10^−1.0, with a step size of 10 set in sequence (Mayr et al., 2018; Xu et al., 2023b; Yao et al., 2023b). The minimum learning rate was set to 0.01 times the initial learning rate. The batch size follows the optimal solution mentioned above (batch size = 4), and the Epoch is set to 100. Table 3 shows the training loss under different working conditions. Figures 8A, B show the convergence of the loss function values with the initial learning rate for the SGD and Adam optimization algorithms, respectively. From Figure 8A, it can be seen that the loss function converges fastest when the initial learning rate is set to 10⁻³ when the SGD optimization algorithm is used, and the loss function fluctuates more in the early stage when the initial learning rate is 10⁻¹, indicating that the learning rate is set too large at this time and the model does not converge well. From Figure 8B, it can be seen that the convergence curves of the loss function are smooth when the Adam optimization algorithm is used, but in the convergence process, the loss function values of each working condition have the following relationships: Number 14>Number 5, 11and 12>No. 14. When the initial learning rate is 10⁻³, the model convergence effect is obviously better than other working conditions.

Figure 8C shows the variation of the loss function with the initial learning rate when the SGD and Adam optimization algorithms were used. Combining Figure 8C and Table 3, it can be seen that the minimum loss is 0.0175 for an initial learning rate of 10⁻³, and the maximum is 0.0224 for an initial learning rate of 10⁻¹ when the SGD optimization algorithm was used. And the minimum loss is 0.0100 for an initial learning rate of 10⁻³, and the maximum is 0.0175 for an initial learning rate of 10⁻¹ when the Adam optimization algorithm was used. The trend of the loss curve with the initial learning rate is similar for the SGD and Adam optimization algorithms, both decreasing first and then increasing, and there is an optimal initial learning rate. In addition, the loss functions of the Adam optimization algorithm are lower than those of the SGD optimization algorithm. The optimal result is that the learning rate optimization algorithm is set to Adam, and the initial learning rate is set to 0.001.