Insulator Contamination Perception Based on Feature Fusion of Infrared Image and Meteorological Parameters

Polluted insulators seriously threaten the safe and stable operation of power grids, which attaches great significance to insulator contamination perception. Among the present methods, the non-contact approaches based on infrared images have gradually been widely used, as they are much more safe and are of low cost. However, the thermal effect of insulators is largely affected by meteorological conditions, which makes the infrared image-based methods less accurate. To solve the above problem, we take infrared image and meteorological parameters including humidity and temperature as input, and propose a feature fusion model to perceive insulator contamination in different weather conditions. Firstly, different feature extraction networks are used to perform feature extraction on the two types of data; secondly, the two features are concatenated to fuse together; thirdly, further feature extraction is performed and contamination is classified according to the pollution severity. Case studies show that the proposed method can better explore the relationship between humidity, temperature and pollution level of the insulators, thus can better separate the contamination grades and outperform the conventional infrared image based methods.


INTRODUCTION
Functioning as electrical insulation and mechanical support, outdoor insulators play an important role in power systems (He et al., 2006). However, with the increasing pollution caused by gas emissions from factories and cars, bad weather like fog-haze, rain and snow conditions, the insulators are contaminated more severe and are more easily and frequently to flashover (Fofana et al., 2020;3, 2020;Dong et al., 2014;Liu et al., 2020), which can seriously endanger the safe and stable operation of power grids, and even cause huge economic losses. Therefore, it is necessary to perceive the pollution level of insulators and take measures in time.
There are mainly two kinds of contamination perception methods for insulators: contact methods and non-contact methods (Jin et al., 2017). The contact ones include leakage current (Banik et al., 2016;Wang et al., 2019;Salem et al., 2020), surface conductivity Zhong et al., 2018), ESDD (Equivalent Salt Deposit Density)  and so on. Generally speaking, the above approaches have clear physical meaning and are relatively accurate. However, they often require online monitoring equipments with large quantity and high prices, what's more, some of the equipments need to be implemented in power off situation. Based on the acoustic, discharge and heat phenomenon of polluted insulators in some cases, non-contact methods including infrared images , ultraviolet images  and (Liu et al., 2021) photothermal radiometry are increasingly applied to insulator contamination perception, because of their advantages of safety and immunity to electromagnetic interference.
Among those non-contact approaches, the infrared imagebased ones are mostly used because of their low costs. From the perspective of the elements they consider, these methods can be divided into two kinds: 1) those based on infrared images only; 2) those based on infrared images and meteorological parameters.
For contaminated insulators, the leakage current on their surfaces will increase under wet conditions, and the more severe the contamination, the greater the leakage current, which can be detected through infrared images. As a result, there are many methods perceiving insulator pollution based on infrared images. In , the authors analyze insulator contamination through the combination of infrared images shot in different angles-to deal with the limitations caused by angle diversification. In addition, to cope with the limitation of manually-designed features, they use Deep Convolutional Neural Networks (DCNNs) to automatically extract features. Jiang and Xia (2010) study the influence of contamination on deteriorated insulators, and conclude that the infrared image based methods are capable of detecting polluted deteriorated insulators in high-voltage. However, these methods lack the consideration of meteorological parameters, which can influence the temperature distribution of the contaminated insulators, to be specific, the same polluted insulator may behave differently in infrared images under different humidity.
The temperature distribution of infrared images depends on both contamination level and meteorological parameters, thus the methods consider both these two factors can be more generalized and accurate. Based on the color features extracted from infrared images and relative humidity, He et al. (2019) accomplish contamination level detection using Radial Basis Function Neural Network (RBFNN); For a contaminated insulator, its infrared images can give contamination information when it in wet conditions, while the visible images are valid to detect contamination level when it in dry conditions. Therefore, Jin et al. (2018) fuse the infrared and visible images to cope with the impact of humidity on noncontact methods. However, the above two kinds are both based on manually designed features, whose perception accuracy are highly dependent on expert experience.
In this study, taking both infrared image and meteorological conditions into consideration, we propose an end-to-end feature fusion method to detect pollution level of insulators in an automatic and accurate way, the contributions are as follows: 1. Considering the differences of infrared images under different weather conditions, the insulator pollution perception problem is mapped to a probability problem of pollution degree, infrared image and meteorological conditions. And the way to solve this problem is to calculate the conditional probability of each pollution severity under the corresponding inputs.
2. The insulator contamination perception methods based on infrared images largely depend on the meteorological conditions, to cope with this problem, we take both infrared image and meteorological parameters as input, and propose a feature fusion method to fuse these two types of data, from which the relationship between contamination grades, temperature and humidity is explored, and the contamination perception in different environmental conditions improves a lot.
3. Instead of manually designing features for inputs, the feature fusion method proposed can automatically extract features for the two kinds of data, which is more accurate and can be trained and tested in an end-to-end way.
The rest of this paper is organized as follows: Section II gives the basic principles of infrared image based contamination perception methods, based on this, the influences of meteorological parameters on infrared images are analyzed; Section III proposes the feature fusion method, which includes the feature extraction part, the feature fusion part and the contamination classification part; Section IV describes the data preparation, the model training and model testing process; Section V verifies the effectiveness of the proposed method from evaluation results and model comparison; and Section VI concludes this paper.

BASIC THEORY
Infrared images manifest infrared energy radiated from objects with temperature higher than absolute zero. Because of the abnormal temperature distribution caused by anomaly or faults of the objects, events can be detected through infrared image analysis, which has been widely used in biomedicine (Vainer, 2019), transportation control (Wang et al., 2020a), quality inspection (Tumas and Serackis, 2017), power systems (Wang et al., 2020b) and other fields.
For contaminated insulators in power systems, their pollution degrees can be detected through infrared images. To be specific, the infrared effect of a contaminated insulator is associated with the heat produced by leakage current on its surface, which is affected by the pollution on its surface and meteorological parameters, such as temperature and humidity. For example, even the same contaminated insulator can show different temperature distribution under different meteorological circumstances, which indicates the necessity to consider both thermal effect on infrared image and meteorological circumstance when analyzing the contamination degree of insulators. The factors affecting infrared effect are analyzed as follows:

Contamination Grade
In a humid atmosphere, the contamination grades of insulators are positively correlated with the heat they produce: The pollution on a polluted insulator absorbs moisture in the atmosphere, which causes a decrease in the surface resistance, and an increase in its conductivity. Therefore, the leakage current through the surface is significantly increased and thus causes thermal effect that can be detected by infrared images. In addition, when under a certain humidity condition, the higher the contamination grade, the larger the decrease of the surface resistance, and the larger the leakage current. While for a clean insulator without contamination, the leakage current is always Frontiers in Energy Research | www.frontiersin.org September 2021 | Volume 9 | Article 746378 small and there are no obvious heating phenomenon even with very high humidity.

Humidity
A certain degree of humidity is necessary for contaminated insulators to produce leakage current: When the humidity is relatively small, both contaminated insulators and clean insulators produce small or even no leakage current, thus there are no clear differences between their corresponding infrared images; When subjected to a wet condition with relatively high humidity, however, the surfaces of the polluted insulators will become conductive, and the thermal effect caused by the leakage current will be reflected on the infrared images, while for the clean insulators, the increase in humidity does not affect their thermal effect. What's more, for the same contaminated insulator, it can present differently when in different wet condition, to be more specific, the higher the humidity value, the more heat it produces and the higher the temperature in infrared image.

Temperature
The environmental temperature affects both the leakage current and the infrared image: 1) The leakage current can be influenced in two ways: on the one hand, the change of the environmental temperature often yields change of the environmental humidity, which will indirectly affect the leakage current produced; on the other hand, it affects the evaporation rate of the contamination layer, and may affect the pollution grade of the insulators in the end.
2) The temperature distribution of infrared images can be affected by environmental temperature (Zou et al., 2014), which in turn can influence the contamination perception accuracy. In short, there is a complicated nonlinear relationship between leakage current and pollution level, temperature and humidity. When using infrared images to perceive contamination grade, environmental humidity and temperature are necessary to be considered.

CONTAMINATION GRADE PERCEPTION BASED ON FEATURE FUSION METHOD
We model the pollution degree perception problem of insulators as a conditional probability function based on temperature, humidity and infrared image characteristics. In this paper, we divide the contamination degree into five classes according to the pollution severity, then under the interaction of humidity, temperature and the contamination on the insulator surface, the probability of the contaminated insulator belonging to each category can be characterized as where c represents the contamination class, h, t represents humidity and temperature, respectively, and in is infrared image shoot under this environmental circumstance. Based on Eq. 1, we convert the contamination perception problem into a conditional probability problem, and propose a feature fusion method to tackle this. In the following parts of this chapter, we firstly give the overall framework of the proposed method, and then illustrate the details of feature extraction, feature fusion and contamination perception. Figure 1 depicts the overall feature fusion framework consisting of four parts: 1) the input part, which includes two kinds of parameters-infrared image and meteorological parameters including humidity and temperature. What's more, the input data aligns in time, which means that they are collected at the same time; 2) the feature extraction part, where the two kinds of data are fed into different feature extraction networks-the meteorological parameters are extracted by an artificial neural network (ANN), and the infrared image is extracted by a convolutional neural network (CNN)-because of the differences in data format and physical meaning; 3) the data fusion part, where the two kinds of features are concatenated and fed into two fully connected layers to extract features further; 4) the output part, where and the contamination grade is outputted.

Meteorological Parameters Feature Extraction Network
Artificial Neural Network (ANN) is a highly nonlinear dynamic system with a directed graph as its topological structure. It learns features and processes information by adjusting the interconnection relationship between internal neurons, so it is of strong self-learning, self-adaptation and high fault tolerance (Kamesh and Rani, 2017;Hu et al., 2018;Bhatt and Gandhi, 2019;Shen et al., 2021).
For a polluted insulator, humidity and temperature can influence its infrared effect on infrared image, at the same time, the two meteorological parameters also interact and influence each other. Therefore, we take humidity and temperature parameters as a whole, and feed them into an ANN to extract the interaction between the two parameters, and learn the features with regard to the contamination grade. The reason we choose ANN to extract features of meteorological parameters is that the input is relatively simple-with only two parameters, in this way, we suppose that a rather simple network is enough to extract the information needed. Besides, ANN has been widely used in many scenes with meteorological parameters as parts of the inputs (Matsumoto et al., 1993;Bhatt and Gandhi, 2019;Madhiarasan, 2020), and the satisfactory results has provided intuitions to our problem.
The proposed feature extraction network is as shown in Figure 2, which consists of 2 1 × 10 fully connected layers and 1 1 × 5 fully connected layer (FC); and Relu is used to activate the fully connected layers. After the feature extraction, we get a 1 × 5 sized feature. What needs to be pointed out is that, we test the analysis results under different number of hidden layers and the neurons in each layer, and the structure of the proposed one can get relatively better performance when it used to fuse the two kinds of data.
Before feeding into the ANN, we use Eq. 2 to normalize the two parameters, to solve the problem caused by their different scales. x in where x i represents the i − th parameter, i 1, 2; x in is the n − th parameter of x i , which corresponds to the infrared image at the n − th sampling moment; min(x i ) and max(x i ) are the minim and maxim of the two parameters.

Infrared Image Feature Extraction Network
One of the most popular image classification approaches today involves the convolutional neural network (CNN), which has been widely used in classification (Simonyan and Zisserman, 2014;He et al., 2016), object detection (Ren et al., 2017), segmentation (Long et al., 2015) and so on. Unlike traditional hand-crafted features, CNN unifies the feature learning and classification parts, and automatically and jointly solves the two problems together, which in turn greatly improves the accuracy of image classification (Hatami et al., 2017). As is shown in Figure 3, in this paper, we take part of VGG-16 as feature extraction network of infrared images, for it is one of the classical classification networks with good results, and meets the contamination classification goal of this paper. In Figure 3, conv (64 × 3 × 3) × 2 represents the same 2 convolutional layers, and each of them has 64 filters sized 3 × 3, and the same to conv (128 × 3 × 3) × 2, conv (256 × 3 × 3) × 3 and so on. maxpooling (2 × 2) is the max pooling layer sized 2 × 2. It can be seen that the convolutional layers are all sized 3 × 3, and the pooling layers are all sized 2 × 2. In addition, all the convolutional layers are activated by Relu.
We normalize the infrared images to 224 × 224 × 3 before feeding them into the feature extraction network. During the feature extraction process, the width and height of a image are reduced to half of the original each time it passes through a pooling layer, and the channel size doubles every time it goes  through the convolutional layer blocks (the network between two neighboring maxpooling represents a block). After the final maxpooling layer, we get a 7 × 7 × 512 sized feature, then it is fed into two fully connected layers activated by Relu function, to further feature extracted. Finally, a feature sized 1 × 5 is outputted.

Feature Fusion and Classification Network
As is shown in Figure 4, we fuse the two kinds of features by concatenating them. And after that, we get a 1 × 10 sized feature.
To make the fusion effectively, it requires at least one nonlinear stage to successfully capture feature from the fused feature (Srivastava and Salakhutdinov, 2014;Ramachandram and Taylor, 2017). Therefore, we use two fully connected layers activated by Relu to further analyze the fused feature. Here, to avoid overfitting of the network, we set the dropout ratio of the FC layer followed by the concatenation layer to be 0.5. To classify the contamination, we divide the polluted insulators into five grades, and use softmax to calculate the probability that the input belonging to each class, as is shown in Eq. 3.
is the input, which is determined by the network before the softmax; p(x) represents the output of softmax, which is the conditional probability of the input belonging to each category; y is the label of the input; m represents the class number, which is 5 in this paper (we category the contamination into 5 classes). Finally, the grade with the biggest possibility is the class outputted.

EXPERIMENTAL SETUPS
As is shown in Figure 5, the implementation of the proposed method includes three parts: data acquisition and labeling, training phase and testing phase.

Data Acquisition and Labeling
To collect polluted insulator image samples in different weather conditions, we use a mixture of NaCl and kaolin to contaminate the XP-70 insulators, and then, in a fog chamber, we shot infrared images in different humidity and temperature, where the relative humidity ranges from 60 to 100% in step of 5% each; and the temperature ranges from 0°C to 40°C in step of 5°C each. As is illustrated in 2.2, a ceratin value of humidity is a must for the polluted insulators to generate leakage current and radiate to show thermal effect. Therefore, we don't collect infrared images with the humidity less than 60%, as in that circumstance, the insufficient humidity will cause almost no leakage current generated by the contaminated insulators, and the contamination degree can't be perceived by the infrared images. We use the equivalent salt deposit density (ESDD) (mg/cm 2 ) to discriminate the insulator contamination severity. According to Chinese national standard 'GB/T 5582-1993', the contamination can be classified into 5 grades according to the pollution content, as is shown in Table 1. In addition, we divide the data into a training and testing set according to 5 : 1, which are used to train the network and evaluate the performance of the proposed method, respectively.

Training Phase
During the training phase, we employ transfer learning to make the network converge faster and solve the problem of relatively small samples, which consists of two stages: In the first stage, we pretrain VGG-16 network based on the ILSVRC-2012 dataset, which contains more than 1 million training samples, and then the parameters of the network are saved. In the second stage, based on the shared parameters of the pretrained VGG-16, we retrain the feature fusion network taking both infrared and meteorological parameters pairs as input. The idea here is that, there are some similarities for the image classification problems, for example, they all have to extract some profile information based on the point and line characteristics, thus they can share some of the parameters for the same network regarding to different classification goals. Besides, the network has been trained to some extent and the loss function has arrived to a relatively small value, which leads to a relatively small modification compared with training from a completely new network. During the training process, we take the same training strategies as that in VGG-16 (Simonyan and Zisserman, 2014), and aim to make the loss function small enough, as is shown in Eq. 4.
where L represents the loss; p i is the probability from softmax, p p i represents the label of the input, which is the real class of the input data; n is the number of samples, which equals to 1,000 in the VGG-16 and 5 in the feature fusion network proposed.

Testing Phase
After training process, the structure and parameters of the proposed network is frozen in the test stage. In this phase, we take testing set as input, and perceive the contamination grades of the polluted insulators, the evaluation indicators are as follows: As is shown in Eq. 5-8four indicators-including p (precision), R (recall), AP (average precision), and AR (average recall)-are used to evaluate the proposed approach.  m represents the class number, which equals to 5 in this paper.

CASE STUDIES
To validate the effectiveness of the proposed method, two insulator contamination perception methods are compared: 1) VGG-16; 2) feature fusion method proposed in this paper. For the two networks, we take the same training set to train them, and use the same test set to evaluate them. The only difference is that only infrared images are fed into VGG-16, while the data pairs (infrared image and meteorological parameters) are taken as input for the feature fusion network.

Evaluation Results
Table 2 tells the average results of the two methods, Figure 6 and Figure 7 illustrate the performance between five classes, from which we can see that, the fusion method proposed in this paper effectively improves the precision and recall (increases by 7.97 and 12.46% respectively), in addition, it has better results between different contamination classes.
To further analyze the necessity of considering meteorological parameters in insulator pollution perception, we divide humidity into four classes-60 ∼ 70%, 70 ∼ 80%, 80 ∼ 90%, 90 ∼ 100%-and compare the AP of the two methods, as is shown in Figure 8. And we also divide the temperature into four ranges-0 ∼ 10°C, 11 ∼ 20°C, 21 ∼ 30°C, 31 ∼ 40°C-and compare the AP of the two methods, as is shown in Figure 9.
From Figure 8, we get two conclusions: 1) both two methods perform better in high humidity. However, the data pairs input model is more balanced in each humidity range, while the infrared image input model is more dependent on humidity-the variance of AP of the former is 0.47%, while is 0.63% of the latter; 2) the method proposed in this paper outperforms the single-input method in all humidity levels.
From Figure 9, both two methods have relatively balanced performances in each temperature range, while the method proposed in this paper outperforms the single-input method in all the temperature classes.
The above phenomenons can be explained as follows: 1) Both humidity and temperature affect the leakage current of    the contaminated insulators, thus further influence the infrared image and the classification perception. When the infrared image alone is used as input, the model cannot explore the relationship between humidity, temperature and the pollution level of the insulator, so it has relatively low precision; While for the fusion method proposed in this paper, it takes humidity, temperature and infrared image as input, which can better explore the relationship between weather conditions and thermal radiation, so it can better judge the contamination level.
2) The temperature does not affect the performance between different contamination classes, as the two methods perform similar in the four temperature ranges within their own (the precision change for the two methods are within 5%). Still, it is still one of the key factors influencing the classification performance, as our method outperforms the single-input method in every temperature level, which may be explained by the influence of the temperature to the humidity.

Class Separability Analysis
To reveal the ability of the proposed model to distinguish different levels of pollution, latent discriminant analysis (LDA) (Hussein Mouzannar and Awad, 2018) is performed on testing set to examine the separability of the two methods (the feature fusion model and VGG-16). We extract the features before softmax, and plot the distribution of the data after performing LDA, which reduces the dimensionality of the data to 2. As is shown in Figure 10, compared with VGG-16, the feature fusion method is more capable of distinguishing different levels of pollution, which is the reason of better classification ability.

Model Complexity Comparison
Compared with VGG-16, the feature fusion method is less complicated: Although it has more parameters in the meteorological parameters feature extraction part, the fused feature extraction part of the proposed network has much less parameters than VGG-16 -As the former has two 1 × 10 FC layers, while the latter has two 1 × 4096 FC layers.

CONCLUSION
The accuracy of infrared image based insulator contamination methods largely depend on weather conditions. In this paper, we take the constructed meteorological parameters and the unstructured infrared image as input, and propose a feature fusion model to classify insulator contamination in different meteorological conditions. The two types of data are different in data form and physical meaning, which makes it difficult to fuse them. In this way, we extract the features of the two types of data, and represent them with the same data form, then two features are fused and further analyzed to classify the contamination. To be specific, the ANN and VGG-16 are used to extract features of meteorological parameters and infrared image, respectively. Then two features are fused by concatenation and further feature extracted by two nonlinear layers (Relu activated fully connected layers). In the end, the contamination level perception is performed by softmax, from which we get the probability of the input belonging to each contamination grade.
Compared with the infrared image input model, the method proposed in this paper can better separate the contamination classification, and perform better in different meteorological conditions. Besides, it has less parameters and is less complicated.
The model in this paper only test samples under the condition of humidity greater than 60%, while for those in the case of humidity less than 60%, there may be no obvious differences between the infrared images of each pollution level. In the future, we will focus on finding more features to make the method more applicable to different meteorological conditions.

DATA AVAILABILITY STATEMENT
The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.

AUTHOR CONTRIBUTIONS
HW: methodology, data collection, simulation, data analysis, writing original draft, recording image; BW: methodology, September 2021 | Volume 9 | Article 746378 8 recording image data, draft review and editing; ML: methodology, draft review and editing; PL: data collection, simulation; HM: writing review and editing; FM: writing review and editing.

FUNDING
This paper is supported by the National Natural Science Foundation of China (Grant No.51777142,51907096).