Instead of directly estimating posterior probability p(y|x), the generative methods pay attention to learning the class distributions P(x|y) or the joint distribution p(x, y) = p(x|y)p(y), to compute the posterior probability using Bayes' formula. In this framework, SSL can be modeled as a missing data problem while the unlabeled data can be used to optimize the marginal distribution p(x)=∑y∈Yp(x|y)p(y). The joint log-likelihood on both labeled data set D_L and unlabeled data set D_U is naturally considered an objective function (Chapelle et al., 2006) as follows: (1)∑(xi,yi)∈DLlogπyip(xi|yi,θ)+∑xi∈DUlog∑y=1Mπyp(xi|y,θ), where π_y = p(y|θ) is class prior. Previous studies under this framework use auto-encoder (Rasmus et al., 2015) or variational auto-encoder (Kingma and Welling, 2013) to model p(x, y). Unfortunately, since the neural network has excessive representational power, optimizing marginal distribution cannot guarantee to achieve the right joint distribution, making those approaches perform less well in large datasets.

To better estimate p(x|y), generative adversarial network (GAN) (Goodfellow et al., 2014) has been introduced to the SSL framework. GAN is well-known for high-quality realistic image generation, Radford et al. (2015) employed fake examples from a conditional GAN as additional training data. Salimans et al. (2016) strengthened the discriminator to classify input data as well as distinguish fake examples from real data. Besides, BadGAN (Dai et al., 2017) argued that generating low-quality examples which lie in low-density areas between different classes can better guide the classifier position its the decision boundary. However, those approaches are based on practice and lack theoretical analysis, showing weak performance compared with newly emerged approaches.

Graph-based methods operate on a weighted graph G = (V, E) with adjacency matrix A, where the vertex set V is composed of all data samples denoted as D = D_L ∪ D_U, and the elements of adjacency matrix A_ij are based on similarities between vertices x_i, x_j ∈ V. The smoothness assumption states that close data points should have similar predictions. Label Propagation (LP) iteratively propagated the class posterior of each node to its neighbors, faster through a short connection between data nodes, until a global equilibrium is reached. Zhou et al. (2003) showed that the same solution is arrived at by enforcing the smoothness term or equivalently minimizing the energy on the graph as follows: (2)R(f)=12∑i,jAij(fi-fj)2=fTΔf Here, f_i is the pseudo label of the i-th data sample, Δ = D − A is the traditional graph Laplacian where D is a diagonal matrix with Dii=∑j=1nAij. If f is replaced with f(X) which is the output of a parameterized function on all data samples, the R(f) reaches a graph Laplacian regularizer which forces the function f to be harmonic. Some modifications to Equation (2) are made to mitigate the effect of outliers in the graph (Gong et al., 2015; Tu et al., 2015). An inevitable drawback of these approaches is that their performance largely relies on the quality of the input graph.

Our study is inspired by a series of recent methods which utilize the LP with a dynamically constructed graph in the optimization process. Acting like EM algorithms, those methods alternate between the two steps, the first step is to use the embeddings obtained by the deep neural network to construct the nearest neighbor graph, and then LP is performed on this graph to infer pseudo-label for the unlabeled images. After that, the network is trained using both labeled and pseudo-labeled data. In addition to the time cost by LP, this method just uses standard back-propagation methods to train the deep neural network, making it fast and efficient. Luo et al. (2018) used the graph constructed from embeddings obtained by the teacher model which is acting better than the student model. Kamnitsas et al. (2018) utilize LP to infer clusters in network middle representation, then encourage points in the same cluster to be closer. Iscen et al. (2019) introduce entropy as an uncertainty measure of pseudo-labels, then reduce the cost of uncertain examples. All of those approaches construct the graph actively in the optimization process to improve the quality of pseudo-labels.

We assume that the input space is X⊆ℝN. We have a collection of l labeled samples X_L: = {x₁, ..., x_l} with xi∈X, their labels are given by Y_L = {y₁, ...., y_l} with y_i ∈ C, where C = {1, ..., c} is the set of discrete labels for c classes. In addition, u extra samples X_U = {x_l+1, ..., x_l+u} are given without any label information. The whole set of samples is denoted as D = X_L ∪ X_U. The transductive goal in SSL is to find the possible label set Ŷ for all unlabeled samples, while the inductive goal is to find a classifier f:X↦ℝc which can generalize well on unseen samples by utilizing all samples D and label Y_L. In this study, we focus on the inductive settings and use the convolutional neural network (CNN) as the classifier in our experiments.

Given a randomly initialized neural network f parameterized by θ, we introduce a new optimization process for SSL that can be summarized as follows. First, we perform pre-training of the network using only labeled data to warm it up, where we introduce mixup data argumentation. Then, we start the iterative SSL training process, perform label propagation to infer pseudo-labels, and optimize the network with both labeled and pseudo-labeled data. We point out two critical improvements compared with previous approaches: mixup regularization with pseudo-labeled data and deformed graph Laplacian-based label propagation. In addition, we incorporate the pseudo-label certainty and class-balancing strategy from the study by Iscen et al. (2019) into our approach. A graphical view of the proposed approach is shown in Figure 1.

In the early stage of the training process, the neural network f is composed of randomly initialized weight parameters. The output of the neural network is chaotic and has little semantic information about the input images. In previous studies, the network is pre-trained using the labeled samples by minimizing supervised cost only. Standard optimization techniques are employed in this procedure. The optimization target is the expectation of loss function ℓ over the labeled data distribution P_D as follows: (3)Rsu(fθ)=𝔼(x,y)~DL[ℓ(fθ(x),y)], where DL(x,y)=1l∑i=1lδ(x=xi,y=yi) is a point distribution that is employed to estimate the true data distribution P_D when labeled data set X_L and correspondent label Y_L are given. With fewer labeled data in the SSL setting, the point distribution is hardly able to estimate the true data distribution P_D, causing the model overfitting and degeneration. Previous studies take measures to mitigate this problem, using a larger learning rate and fewer pre-training epochs (Iscen et al., 2019). Other regularization techniques such as dropout and weight decay are adopted by Luo et al. (2018) and Miyato et al. (2019).

In this study, we introduce mixup (Zhang et al., 2018) in the pre-training procedure. Instead of simple point distribution, mixup proposes a vicinal distribution to estimate P_D, whose generative process is summarized as follows: (4)(xi,yi),(xj,yj)~DL,          λ~Beta(α,α),          x: =λxi+(1-λ)xj,          y: =λyi+(1-λ)yj, where α is a hyperparameter α ∈ (0, ∞), which controls the strength of interpolation between data pairs. Notably, D_M will degrade to D_L as α → 0. We denote the mixup distribution as D_M(D_L, α). By replacing the D_L with mixup distribution in Equation (3), we have as follows: (5)Rsu(f)=𝔼(x,y)~DM(DL,αsu)[ℓ(fθ(x),y)]. In a nutshell, we just randomly sample two data points each time and perform standard supervised training using the mixed data point. Notably, the interpolation between y_i and y_j is an interpolation between two one-hot encoded probability vectors. This modification does not influence the optimization process, which means any optimizer and network architecture can be applied with this regularization.

We summarize the benefits of introducing mixup mainly in two-folds as follows:

Overfitting problem is greatly alleviated. While the output posterior probability of the classifier is forced to transit linearly from class to class, the decision boundaries between classes are pushed into the intermediate area, reducing the number of undesirable results when predicting outside the train examples. The experiments will demonstrate the test error of the pre-trained network is reduced by introducing mixup regularization.

Given a classification network f with pre-trained parameters θ, we clarify how to infer the possible label for unlabeled examples, which is employed to guide the SSL optimization process in the following subsection. First, we construct a graph based on network internal representations of all examples, and then, we apply label propagation on the graph to infer pseudo labels.

In most cases, the deep neural network can be seen as a sequence of non-linear layers or transformations, with each transformation giving an internal representation of the input. While the low-level representation captures more details of the input image, the high-level representation contains more semantic information about the input image, and the last layer maps the input feature into class probabilities. In a nutshell, network f can be decomposed as f = g ◦ h where h:X↦ℝd is a feature extractor and g : ℝ^d ↦ ℝ^c is the output layer usually consisting of a fully-connected layer with softmax. We take h as a low-dimensional feature extractor and denote the feature vector for the i-th example as v_i: = h(x_i). We extract all features set of D as V = {v₁, .., v_l, v_l+1, ..., v_n} for similarity computation.

The next is to construct a graph on N data nodes. While computing the full N × N affinity matrix A may be intractable, we approximate it by constructing a nearest neighbor graph and only count the similarity between nodes and their k nearest neighbors. Thus, we create a graph with a sparse affinity matrix A ∈ ℝ^{n × n} and its elements as follows: (6)aij: ={s(vi,vj),ifi≠j∧vi∈NNk(vj)0,otherwise where NN_k(v_i) denotes the set of k nearest neighbors of v_i in D, and s is the similarity function. The choice of s is quite flexible. Since we need to approximate the semantic similarity of two instances, we adopt the Gaussian similarity function s(vi,vj)=exp(∥vi-vj∥2/σ2) with hyperparameter σ. Notably, approximate nearest neighbor (ANN) algorithms can be applied to accelerate the graph construction for large N.

Hereafter, let W: = A + A^T be the symmetric affinity matrix and W=D-1/2WD-1/2 be its normalized counterpart, D is the diagonal degree matrix, in which the element is defined by Dii: =∑j=1nWij. Further more, the volume of the graph is formulated as v=∑i=1ndii.

After defining those parameters, we are going to describe our LP algorithm by defining input and output as two n × c matrix Y, Z. Y is the matrix of the given label with rows {y₁, ..., y_l, y_l+1, ...y_n}, where the first l rows are corresponding one-hot encoded labels of each labeled example, and the rest are all zero vectors. Z is the desired class posterior probabilities which are solved by minimizing the following cost function: (7)minZQ(Z)=12[β∑i,j=1nwij∥zidii-zjdjj∥2          +γ∑i=1n(1-dii/v)∥zi∥2+∑i=1n∥zi-yi∥2]          =12[βtr(ZTΔZ)+γtr(ZT(I-D/v)Z)          +∥Z-Y∥F2]. Here, z_i is the i-th row of matrix Z, Δ=I-W is the normalized graph Laplacian and ∥·∥_F is the Frobenius norm. The first term encourages smoothness where similar examples tend to induce the same predictions, and the last term attempts to maintain predictions for labeled examples (Zhou et al., 2003). In addition, the outlier which is indicated by a lower degree d_ii is forced to have a weak label in the second term. Thus, the degree of smoothness and weakness of outliers is controlled by weight parameters β and γ individually. To find the optimal solution of Z, we set the derivative of Equation (7) with respect to Z to 0 and obtain as follows: (8)βΔZ*+γ(I-D/v)Z*+Z*-Y=0. Thus, the optimal Z is defined as follows: (9)Z*=(I+βΔ+γ(I-D/v))-1Y. Let k1=β1+β+γ,k2=γ(1+β+γ)v and ignore the constant part in Equation (9). We have as follows: (10)Z*=(I-k1W-k2D)-1Y. Directly computing Z^* by Equation (10) is often intractable for large n because the inverse matrix (I-k1W-k2D)-1 is not sparse, Instead, we use the conjugate gradient (CG) method to solve the linear system as follows: (11)(I-k1W-k2D)-1Z=Y. This method is applicable because (I-k1W-k2D)-1 is a positive-definite matrix. The CG method has been adopted in many LP applications (Zhou et al., 2003; Gong et al., 2015; Tu et al., 2015; Iscen et al., 2019). Finally, the pseudo-label for an unlabeled example is given as follows: (12)yi^=argmaxjzij. Equation (11) is a hard assignment by evaluating the most confident class of each example; however, the contrast between classes can reflect the certainty of each example. Following Iscen et al. (2019), we associate a measure of confidence to each unlabeled example of calculating the entropy of Z: (13)wi: =1-H(ẑi)log(c), where zi^ is the normalized counterpart of z_i, in other words, ẑij=zij/∑kzik, function H:ℝ^c ↦ ℝ is the entropy function.

Given the pseudo-label and confidence measure of all unlabeled data, we associate them with each example and denote the point distribution of unlabeled data as DU(x,y,w)=1u∑i=ll+uδ(x=xi,y=ŷi,w=wi). To take the confidence coefficient into account, we propose the mixup distribution of unlabeled data as D_MU whose generative process is summarized as follows: (14)(xi,yi^,wi),(xj,yj^,wj)~DU,          λ~Beta(α,α),          x: =λxi+(1-λ)xj,          ŷ: =λyi^+(1-λ)yj^,          w: =λwi+(1-λ)wj, In this generative process, the input data are interpolated randomly between two examples, while the pseudo-label and confidence score are interpolated with the same proportion as well.

The abundant unlabeled data are used in the training process by minimizing the following cost along with labeled data. (15)Runsu(fθ)=𝔼(x,ŷ,w)~DMU(DU,αunsu)[wℓ(fθ(x),ŷ)]. While the interpolation of labeled data can help form better network representation, in the unsupervised part, introducing interpolation of unlabeled data can lead to two benefits as follows: (1) The decision boundaries are pushed far away from unlabeled data, which are a desired property of low-density separation assumption. The model is forced to make neutral predictions in the middle zone of different samples or namely different clusters. (2) The clusters in hidden space are encouraged to have only one class of pseudo label, respectively. Considering the clusters in hidden space, if data points in one cluster are pseudo-labeled by two different classes, the mixup loss term will tear this cluster apart. The middle of this cluster is both encouraged to have neutral predictions as the interpolation of edge points or encouraged to have clear predictions as the middle points of the cluster.

Moreover, we employ differential privacy by directly adding noise to the latent representation of the deep neural network in the training progress: (16)fθ*(x)=g(h(x)+ϵ), where ϵ is randomly sampled from N(0,σ2). This procedure is inspired by the Ladder network (Rasmus et al., 2015). By adding noise to its latent representation, the neural network will have more resistance to the dataset bias and form a more coherent latent space.

Finally, we finetune the network by minimizing the following objective function using both labeled and unlabeled data: (17)Rall(fθ,DL,DU)=Rsu(fθ,DL)+λRunsu(fθ*,DU), where λ is a coefficient that controls the effects of the unsupervised term.

We summarize our approach with the above definitions. Given a convolution neural network f with randomly initialized weights θ, we begin by training the network with mixup regularization for T epochs using the supervised loss term (Equation 5), and then, we start the following iterative process. First, we extract feature vector set V on the entire training set X and construct a nearest neighbor graph by computing the adjacency matrix via Equation (6). Second, we perform label propagation by solving the linear system (Equation 11) and assign pseudo-label and confidence score by Equations (12) and (13). Finally, we train the network for one epoch by minimizing the cost (Equation 17) on both labeled and unlabeled data set. This iterative process is repeated for T′ epochs.

The whole training process is summarized in Algorithm 1, where procedure Optimize() refers to the mini-batch optimization of the given loss term for one epoch. In our experiment, we randomly sample a mini-batch of data and perform mixup interpolation within this mini-batch, and this strategy is used to reduce I/O consumption and report no harm to the result in the study by Zhang et al. (2018). The procedure NearestNeighborGraph() refers to the nearest neighbor graph construction based on the feature vector set V and the computing of edge value in the graph.

We conduct experiments on three datasets, such as Cifar10, Cifar100, and Mini-imagenet. Cifar10 is widely used in related study, and Mini-imagenet is adopted by Iscen et al. (2019), to evaluate the proposed method on a large-scale dataset. Those datasets are commonly used in SSL setting by randomly taking a certain amount of labels and all image data to train the network and evaluate on the test set for fair a comparison with fully supervised methods, while the use of other labels in the training process is forbidden.

In this section, we present a comparison with the state-of-the-art methods. We choose representative methods from three categories, such as generative SSL methods (BadGAN; Dai et al., 2017), consistency-based SSL methods [VAT; (Miyato et al., 2019), MT (Tarvainen and Valpola, 2017), and ICT (Verma et al., 2019)], and graph-based SSL methods (LP; Iscen et al., 2019). The performance of various methods on three datasets is represented in Tables 1–3, respectively.

The proposed method outperforms other methods with the same network architecture. On the Cifar10 dataset, our method achieves a significant error rate reduction (~20%) compared with our precedent method (Iscen et al., 2019), showing that our method exactly amends its weakness and successfully mitigates the performance gap between the graph-based SSL framework and other SSL methods. Compared with the best consistency-based method (Verma et al., 2019) to our knowledge, our method performs slightly weaker with 4,000 labels in total but outperforms it with fewer labeled images. This shows the advantage of traditional graph-based SSL learning that it can make more effective utilization of available labels, which are still applicable when it comes to modern deep learning architecture. We also try to use even fewer labels to evaluate the robustness of our method. Holding 500 labels (~1% of all), our method still achieves 18.56% error rate on the Cifar10 dataset.

We conduct ablation studies to investigate the impact of mixup regularization on the pseudo-labels. To access the quality of pseudo-labels, accuracy is an important indicator. Despite this, we utilize the confidence score to calculate the weighted accuracy of pseudo-labels: Accweighted=1u∑i=1uwiδ(yi^=yi). During the iterative optimization process, if the models are not capable of correcting wrong pseudo-labels, the confidence of those mistakes will increase and eventually get close to 1, leading to Acc_weighted ≈ Acc. The weighted accuracy indicator can reflect if the model really learns something useful from unlabeled images or if it just remembers the pseudo-labels. Figure 2 shows the progress of accuracy and the weighted accuracy of pseudo-labels throughout the training process. The experiments are conducted on Mini-imagenet with 100 labels per class. The results show that if no mixup regularization is applied or only applying mixup regularization on labeled data, the accuracy of pseudo-labels only increases in the beginning and tends to be stable in the following epochs, while the weighted accuracy curve keeps declining until getting close to the accuracy curve. These results imply that without regularization, the deep neural network just remembers the pseudo-labels due to its excessive representational ability. In contrast, our regularization method successfully alleviates such undesired phenomenon, and the accuracy of pseudo-labels is keep increasing during the training process.

In Table 4, we compare the performance of Mini-imagenet. The results show that our method greatly reduces the error rate compared with the baseline method even on a high-resolution image dataset. To investigate the effectiveness of differential privacy in the proposed method, we vary the noise scale from 0 to 1.0 and report the performance on different datasets in Table 5. The results clearly show that the added noise reduces the error rate of the final model.