Distributed fusion architecture is a classical and typical structure in multi-sensor fusion due to its high speed and reliability (Sun et al., 2017). In distributed fusion, the measurement results of each sensor are processed to obtain local estimates and error covariance. Then the processing results are sent to the fusion node to conflate them into global state estimation and the estimated error covariance (Wu et al., 2021). Figure 1 shows a distributed model for the fusion in radar and infrared sensors (Yang et al., 2016). For single target tracking, radar and infrared sensors track the target respectively and generate dependent target trajectories in their local information processing center, then send the local trajectory information to the fusion center for data fusion.

In some tasks, deeper neural networks can extract higher-level features and perform excellently. However, too deep networks may cause the notorious problem of vanishing or exploding gradients and degrade the accuracy. To solve these problems, He et al. proposed a residual network composed of a series of residual blocks (He et al., 2016a). Figure 2A shows the original residual module, which can be expressed as (He et al., 2016b).

where x_l and x_l+1 are the input and output in the l-th layer, and ℱ is the residual function. f is a ReLU (Nair and Hinton, 2010) function. The residual block contains two parts: identity mapping and residual mapping. The left part of Figure 2A is the identity mapping, and the right part of Figure 2A is the residual part expressed as ℱ(x_l , W_l ), which usually contains 2 or 3 convolutional layers. In many cases, the dimensions of input x_l and output x_l+1 are discrepant, so it is necessary to employ a 1×1 convolution operation to maintain the dimension in input and output consistent, whose schematic diagram is shown in Figure 2B , which can be expressed as (He et al., 2016b).

where h(x_l ) is the identity skip connection and W l ' is the 1×1 convolution kernel.

The residual network can be formulated as (He et al., 2016b)

for any deeper block L and any shallower block l. The formula 3 indicates the feature x_L in any deeper residual block L which can be represented as the feature x_l in any shallower block l add the residual function, which leads to nice backward propagation properties that the gradient of layers will not vanish even when the weights are arbitrarily small (He et al., 2016b). Moreover, experiments with the various usages of activation function were carried out in (He et al., 2016b). The order of the activation function in the network will affect the performance of the residual network. The structure of the improved residual unit shown in Figure 2C has the best performance. In this structure, the batch normalization (BN) and ReLU activation function are placed before the convolution layer, and the activation function after addition is moved to the residual part.

Attention mechanism can be traced to the last century, which was mostly applied to machine translation tasks. It has become an essential concept in artificial intelligence because it conforms to some laws of human cognition and can improve the interpretability of neural networks. Therefore, the attention mechanism is widely applied, such as natural language processing, speech recognition and computer vision (Mnih et al., 2014; Vaswani et al., 2017; Bhatti et al., 2022c). In the computer vision domain, many researchers have studied attention mechanism and proposed corresponding methods to acquire nice performance. A residual attention network built by stacking attention modules is proposed in (Wang et al., 2017) which are designed to generate attention-aware features, achieving outstanding recognition performance. A novel architecture unit termed the “Squeeze-and-Excitatio”(SE) block that adaptively recalibrates the channel feature strength by explicitly modelling the interdependence between channels is introduced in (Hu et al., 2020). The structure of SE block is shown in Figure 3 , where U is a feature map with the size of W×H×C, ⨂ and refers to channel-wise multiplication, so X and U have the same size. Moreover, edge-guided attention mechanisms which can produce visually appealing images also attract the attention of many researchers (Bhatti et al., 2021). Zhao et al. (Zhao et al., 2019a) propose an edge guidance network (EGNet) which solves the problems of rough boundary in object detection through the complementarity of the object and salient edge information.

Enlighted by the advantages of distributed structure and the residual module, we propose a novel distributed fusion architecture for infrared and visible images based on the residual module and attention of edge and multiscale channel, RADFNet. The RADFNet is an end-to-end fusion network, the overall structure of which is shown in Figure 4 . It contains four parts: the feature extraction for the visible image, the feature extraction for the infrared image, the fusion for features, and the compensation for edge information. The infrared and visible image fusion process is formulated as follows.

The visible image features extraction branch can be formulated as

where V ₀, the visible image input in the architecture, is the V in the Figure 4 . V_i is the representation of V ₀ after the residual module, multiscale channel attention and compensation of edge information. VR_i means the residual module acting on the V_{i -1} and ATT_i is the multiscale channel attention module designed to obtain coarse-to-fine features from the outcome of VR_{i -1}. Vi represents the features in different levels of V₀ with different scales, wherein V_i has a higher level than V_{i- 1} . E_Vi is the edge information feature map obtained by EAT_i with input E _{V i−1} configured to compensate for the edge information of the feature map achieved by residual module and multiscale channel attention module. ⓜ refers to the maximum value in the homologous channel and position in the feature map. The features in the visible image with separate scales are extracted through the above steps. Then, they are fed into the fusion channel to fuse at each layer, which can fully utilize the multi-scale information from perceptible images. In this method, more texture information with high spatial resolution retains, which can enhance the quality of the fused image.

The infrared image feature extraction branch can be formulated as

where I ₀, the infrared image input in the architecture, is the I in Figure 4 . I_i is the representation of I ₀ after the residual module, multiscale channel attention and compensation of edge information. IR_i means the residual module acting on the I_i-1 and ATT_i is the multiscale channel attention module designed to obtain coarse-to-fine features from the outcome of IR_{i -1} . I_i represents the features in different levels of I ₀ with different scales, wherein I _i has ahigher level than I_{i -1}. E _{I i} is the edge information feature map obtained by EAT_i with input E _{I i−1} configured to compensate for the edge information of the feature map achieved by residual module and multiscale channel attention module. ⓜ refers to the maximum value in the homologous channel and position in the feature map. The features in the infrared image with distinct scales are extracted through the above steps. Then they are constituted into the fusion channel to fuse at each layer, which can fully utilize the multi-scale information from infrared images. As a result, rich target information is used for highlighting the target in the fused image.

The channel of feature fusion can be defined as

where V ₀ and I ₀, which are visible image and infrared image inputs in the fusion architecture, are the V and I in Figure 4 respectively. FUI ₁ , FUI ₂ , FUI ₃, and FUI ₄ are the fusion results with different level features using corresponding rules. F_i refers to the fusion rule of the relevant layer features. FUI_i is the fusion result of the i-th extracted features V_i , I _i and the different scales from previous fusion result FUI_{i -1}. It realizes the layer-by-layer fusion so it can make the best use of the information from multisource images and then improve the quality of the fused image.

The infrared and visible image fusion model RADFNet set out in the present paper is constituted of three channels. The RADFNet structure is exhibited in Figure 5 . RADFNet contains four parts: the features extraction branch of the visible image and infrared image, the features fusion branch, and the edge attention module compensating edge information for the extracted features. The left and right branches in Figure 5 are intended to extract the features in visible and infrared images respectively. The middle branch fuses the features extracted by the two branches with the results from the previous step layer by layer, and the last layer generates the fused image. For a convolutional layer, ‘k×k,(in,out)’ means the kernel size is k×k, the input channel is in and the output channel is out. In the network, BN indicates batch normalization that is utilized to speed up the training and make the training more stable, and ReLU denotes the linear rectification function.

The RADFNet adopts four-layers network structure. The VR₁₋₄ and IR₁₋₄ are the residual networks which extract image features. Because the residual network has the advantages of mitigating gradient disappearance or gradient explosion and protecting the information integrity, the networks we designed can extract meaningful features and ensure the information integrity simultaneously. Besides, ATTi processes the features extracted by residual block VRi or IRi to obtain coarse-to-fine features. EATi acquires the edge information and then compensates edge information for the extracted feature map. The ⓜ refers to the operation for achieving the maximum value in the homologous channel and the homologous position in the feature map. The FUPi generates FUIi by fusing features extracted by the other two branches with the FUI_i−1 generated by FUP_i−1 when i is not 1. When i is 1, the concatenated infrared and visible image is fed into the FUP1 to generate FUI₁. The ⊕ is the concatenation operation in channel-wise, and the 1×1 244 convolution layer in the last fusion layer constructs fusion images.

In the process of infrared and visible image fusion, image feature extraction is exceptionally significant. However, in practical situations, numerous detailed information loses in the process of feature extraction. Inspired by SENet (Hu et al., 2020), the multiscale channel attention network is proposed to process the features extracted by the residual network to obtain the coarse-to-fine features, which can retain more detailed information in the feature map. As shown in Figure 6 , the structure enclosed by the dotted line is the multiscale channel attention module. The features which lose a lot of details extracted by VRi or IRi are used as input in ATTi. Then, the 1×1, 2×2, and 4×4 average pooling operations are performed to generate multiscale features which contain more necessary spatial information. Moreover, the channel attention mechanism is utilized to enhance channel correlation information between features. The multiscale channel attention network is trained to learn the weight W t i k for the k-th feature f t i k of the t-th pooling scale in the ATTi which can be formulated as

where G(z) denotes the global average pooling operation. ∑ x , y f t i k ( x , y ) means the sum of the k-th feature with the t-th pooling scale in ATTi. (x, y) refers to the position in feature map, and H,W means the height and width of the feature map. δ refers to the ReLU function, w ₁∈ℝ ^k×k and w ₂∈ℝ ^k×k , σ denotes the sigmoid function. Then the channel-wise multiplication is implemented between W t i k and the up-sampled features which can be expressed as U P ( f t i k ) , ensuring the multiscale features have the same size as the input. Based on this, the reweighted features are obtained and then the attention map can be achieved as follows:

where π denotes the instance normalization (Ulyanov et al., 2016) and ⓜ refers to the operation for acquiring the maximum value in the homologous channel and position in the feature map. Through the above method, the coarse-to-fine attention map Fi is obtained. The attention map not only emphasizes more critical features and neglects secondary ones but also reserves more necessarily detailed information.

Generally, the edge information of an image refers to the sudden change in local grayscale value, color component and texture structure. The edge information from images which is helpful to distinguish objects, can effectively attract attention of people due to human visual characteristics. Enlightened by previous work, we utilize an edge feature map extraction model from the shallower to deeper to obtain the enhanced edge maps, which are designed to compensate for textural information for the fused image.

For the sake of acquiring the edge information used to compensate fused images, we obtain the gradient map from the source images. The process of obtaining the gradient maps ∇g by inputting a gray-scale image f with the size h × w is defined as

where f(x, y) means the pixel at position (x, y). Moreover, we perform the enhanced operation to obtain the more obvious gradient information:

where H={1,…,h–1} and W={1,…,w–1}. The (x, y) represent the position at the gradient map. Through the above steps, we get the gradient image G with the abundant enhanced edge information.

Subsequently, we feed the gradient images from infrared and visible images into the edge attention module to generate edge attention feature maps with enhanced edge information. Then, the feature maps will be entered into the extraction branch to compensate edge information for the extracted features by IRi or VRi. The structure diagram of the edge attention module is shown in Figure 7 . The edge attention module generates E _Vi and E _Ii layer by layer, which is then used to compensate edge information for the feature maps V_i and I_i extracted by VR _i and IR _i respectively. Therefore, compensated feature maps fused to generate the fused images retain more edge information.

For infrared and visible image fusion, it is difficult to provide the ground truth of fused images for networks to train a model. However, the requirement to retain salient target information in the infrared image and the texture information in the visible image is determined. Inspired by this requirement, the loss function we employ is as follows:

where the L_pixel constrains the fused image to contain more target information from the image pair facilitating target tracking and the L_texture forces the fused images to contain more texture details which can effectively improve the identification of objects in images.

Specifically, the exact definition of L_pixel is expressed as follows:

where m is the batch size that is the number of training samples used in each iteration. The I_f means the fused image with the input image pair {I_ir , I_vis }, and the max (·) denotes the element-wise maximum selection. Through the maximum selection strategy, the fused images have the prominent target information.

Moreover, we hope the fused images contain significant target information and simultaneously preserve great textural details from source images. However, the L_pixel has very limited constraints on textural details. Therefore, the L_texture is introduced to force the fused image to retain more textural information and the L_texture is defined as:

where the m is the batch size, the I_f means the fused image with the input image pair {I_ir , I_vis }, and the max (·) denotes the element-wise maximum selection. The ∇ indicates the Sobel gradient operator and the |·| means the absolute operation. The element-wise maximum selection strategy can make the fused images obtain the most significant edge textural information.