Given a code instance, we wish to identify and correct potential flaw in it, which might lead to a failure in execution after successful compilation. Each bad code instance contains exactly one flaw.

Formaly speaking, given an input code instance x, we wish to map it to an output code instance y and we seek to model P ( y | x ) . A code is “repaired” if the flaw that x contains is fixed in the output y. The “repair rate” is defined as the fraction between the number of code instances fixed and the total number of code instances that the model was applied on. We use repair rate as the evaluation metric in our experiments.

For this purpose, we applied two major families of seq2seq models: GRU and Transformer. We use learnable embedding layers, which allows the model to recognize the relationship between different words in the vocabulary. For the encoder, we applied pyramid encoder, where a pyramid module is added in between layers of regular multilayer encoders. For the purpose of testing generality of pyramid encoder, we combined it with different attention mechanisms.

In language correction, character-level reasoning is a more commonly applied method, Xie et al., (2016). However, in code correction, we apply word-level models. A “word” here is defined as a code syntax (e.g., “void”, “{“, space, “ = “, “int”, newline, etc.) or a custom variable name. The reason is that the basic building blocks of a code instance are related to the syntax. In the field of programming language processing, out-of-vocabulary (OOV) is less a problem than in natural language due to a fixed syntax pool.

In order to prevent the model suffering from vast variation of variable names, we performed a certain degree of variable renaming. We focused on renaming function names in our dataset while keeping other variables unchanged. This method reduced vocabulary size to ∼1,000 and was proven to be effective in improving the performance.

We include our preprocessing method to a code instance in the Appendix.

Given a multilayer seq2seq encoder, its input at i th layer at step t is x t ( i ) and the output is h t ( i ) : h t ( i ) = Layer ( i ) ( x t ( i ) ) (1) In standard seq2seq models, the output of the i th layer h ( i ) is directly used as input of the i + 1 th layer, x ( i + 1 ) : x t ( i + 1 ) = h t ( i ) (2) and the time step t = 1,2 , … , T , the layer number i = 1,2 , … , N . Note that x t ( 0 ) is the embedded representation of the input instance.

For pyramid encoder, we introduce a pyramid module in between h ( i ) and x ( i + 1 ) as Eq. 3 follows: x t ′ ( i + 1 ) = tanh ( W pyr ( h 2 t ( i ) , h 2 t + 1 ( i ) ) + b p y r ) (3) This module reduced the length of the input x ( i ) by half each time it is applied. The length of final output of the encoder is T / 2 N − 1 . One could also take a bigger window such as 3, 4, 5… depending on their needs. The hope is that pyramid structure will extract the important information and reduce the redundant information of each of the neighboring hidden state, therefore reducing the training cost while keeping the accuracy of the correction. This is conceptually similar to a convolution, but without using filters.

For our GRU models, we used multilayer bidirectional GRU and we implemented pyramid encoder as described first in Xie et al., (2016): f t ( i ) = GRU ( f t − 1 ( i ) , x t ( i ) ) (4) b t ( i ) = GRU ( b t + 1 ( i ) , x t ( i ) ) (5) h t ( i ) = f t ( i ) + b t ( i ) (6) x t ′ ( i + 1 ) = tanh ( W pyr ( h 2 t ( i ) , h 2 t + 1 ( i ) ) + b p y r ) (7) where x t ′ ( i + 1 ) denotes the input to next layer, f t ( i ) and b t ( i ) denote output from a forward and a backward GRU, respectively. GRU (Gated Recurrent Unit) is a RNN (Recurrent Neural Network) type model that includes a gating mechanism in the following equations (Cho et al., 2014): r t = σ ( W i r x t + b i r + W h r h ˜ t − 1 + b h r (8) z t = σ ( W i z x t + b i z + W h z h ˜ t − 1 + b h z (9) n t = tanh ( W i n x t + b i n + r t * W h n h ˜ t − 1 + b h n ) (10) h ˜ t = ( 1 − z t ) * n t + z t * h ˜ t − 1 (11) where h ˜ t is the hidden state at step t, which is denoted by f t in Eq. 4 and b t in Eq. 5. r t , z t , and n t are the reset, update, and new gates, respectively. σ is the sigmoid function.

Transformer is a novel family of seq2seq model that works very differently than RNN type models. In the original Transformer (see Figure 3), a Feed Forward layer directly takes in the output from the Multihead attention layer c att , accompanied by a residual connection, shown in c att ( i ) = MultiHeadAtt ( x ( i ) ) + x ( i ) (12) x ( i + 1 ) = c att ( i ) + FeedForward ( c att ( i ) ) (13) In our model, we concatenated the neighboring elements in c att before we feed it into the Feed Forward. As a result, the dimension of the first Linear layer in the Feed Forward layer has to change from [ d model × d ff ] to [ 2 d model × d ff ] . Here we use the same notation as in Vaswani et al., (2017), where d model is the size of input, output, and attention vectors and d ff is the number of neurons in the Feed Forward layer. The residual connection also has to be changed accordingly; we tried two different approaches, simply averaging the neighboring element (Eq. 14) or concatenating the neighboring element and passing it through another affine transformation to recover its dimensions (Eq. 15). For simplicity, we denote the former method with subscript “ave” and the latter with subscript “aff”. x t ′ , ave ( i + 1 ) = ( c att , 2 t ( i ) + c att , 2 t + 1 ( i ) ) 2 + FeedForward [ ( c att , 2 t ( i ) , c att , 2 t + 1 ( i ) ) ] (14) x t ′ , aff ( i + 1 ) = tanh [ W aff ( c att , 2 t ( i ) , c att , 2 t + 1 ( i ) ) + b aff ] + FeedForward [ ( c att , 2 t ( i ) , c att , 2 t + 1 ( i ) ) ] (15) In our experiments, both methods show close performance. Therefore when showing the results, unless otherwise specified, we use the results of “ave” version.

For our GRU models, we compared a regular multilayer unidirectional GRU: h ¯ t ( i ) = GRU ( h ¯ t − 1 ( i ) , h ¯ t ( i − 1 ) ) (16) In our experiment, we did a comparison study on Bahdanau attention (Eq. 17) and different Luong attentions. Bahdanau attention is described in following set of equations. u t k = ( W 1 h ¯ t ( M ) + b 1 ) ⊤ ( W 2 h k ( N ) + b 2 ) (17) α t k = u t k ∑ j u t j (18) a t = ∑ j α t j h j ( N ) (19) Here, u is the alignment score, h and h ¯ denote the hidden state in encoder and decoder, respectively. M, N are the number of layers in decoder and encoder, respectively. a t is the context vector, which will be concatenated with the decoder hidden state of last layer for predicting the next word y t ^ .

Luong’s global attentions are generalizations to Bahdanau attention, but using different alignment score calculation methods. For simplicity, we omit the superscript ( M ) and ( N ) . u t k = { h ¯ t ⊤ h k dot h ¯ t ⊤ W a h k general v a ⊤ tanh ( W a [ h ¯ t , h k ] ) concat (20) We also tried one example of Luong’s local attention, which is done by imposing a Gaussian on Eq. 19 at a desired attention center p t : a t = ∑ j ( α t j h j ) exp ( − ( j − p t ) 2 2 σ 2 ) (21) p t = S ⋅ sigmoid ( W p h ¯ t ) (22) where S denotes the total length of the hidden state from the last encoding layer and σ is a parameter chosen manually.

We use beam search in test and validation where text generation is involved. For each time step, we rank candidates based on their total negative logarithmic probability to current decoding time step t dec : score = − ∑ t t dec log ( P ( y ^ ) ) (23) The search stops when there are five completed candidates.

In all our experiments, we used a learnable embedding layer which embeds each “word” into a vector of length 400.

In our GRU models, we used a 3-layer bidirectional encoder; the size of the hidden states are 400 in all three layers. We used a 3-layer unidirectional decoder; the size of the hidden states are also 400.

In our Transformer models, following the original study, we used d model = 512 and d f f = 2048 . We used 3-layer encoder and 3-layer decoder.

We did a coarse parameter space search to find these parameters chosen to be roughly optimal. But we did not fine-tune these parameters, because (1) we show that the overall performance of seq2seq model on PLC problem is satisfying and (2) we are more concerned about comparison between different attention mechanisms and between pyramid encoder and regular encoder.

We train our models on a GeForce GTX 1080 Ti graphic card. The metric we use for evaluation is the repair rate, which is the fraction of instances that are repaired after the model’s edit. Since we performed beam search with beam width 5, each time a correction is being performed, we generate five correction candidates. Here we have two metrics in measuring the performance: one-candidate repair rate and five-candidate repair rate. The former corresponds to the scenario of code autocorrection, where there is no human judgment involved. The latter corresponds to correction suggesting, where the machine will identify an error and provide suggestions for the programmer for further judgment. The comparisons of the repair rates for considered models and their counterparts with pyramid encoder are listed in Table 1 and Table 2. For comparison, we have attempted to test other machine-learning-based PLC tools that have been made. Gupta et al., (2018) take error messages while compiling as input, but our dataset focuses on logic flaws in programs that do not have syntax errors; therefore, this tool is not applicable. Pu et al., (2016) do not provide an open source repository, nor any documentations of their code. We have successfully trained Gupta et al., (2017) on our C/C++ dataset and included it in our work for comparison. Unfortunately, a tokenizer is required for preprocessing the data into a certain format, and they only provided that for C/C++, but not java.

From these results we see that pyramid encoder has close performance to regular encoder in most of the models we applied to, except for Luong’s local attention. The reason is that the encoder output in pyramid encoder is very “coarse-grained”; each output position now represents information from 2 ( N − 1 ) words. This results in two drawbacks specifically to local attention: one, a much more “blurry” attention center and two, a much broader attention window. As a result, the attention is much less targeted, which damages the performance. Therefore, in the rest of the article, we will exclude this attention mechanism from our discussion.

Since pyramid encoder reduces the sequence lengths in higher layers, one can expect a smaller training cost per batch in both GRU and Transformer models. To quantify this effect, for each of the regular encoder-pyramid encoder model pairs in Table 1, we set the same batch size and compare the average training speed in words per second, as shown in Table 3. Here the batch size is chosen so that it optimizes the training speed on the given GPU for each model. In the model, we also included number of epochs for the model to converge.

Apparently it takes similar number of epochs to converge for the same type of model with pyramid encoder and regular encoder. However, pyramid encoders largely increase the training speed, between 50 and 130%. Therefore it could easily shorten the training time by two to four folds while the same performance is achieved. As an example, Figure 4 shows the learning curve for GRU model with general Luong’s attention, comparing the regular encoder and pyramid encoder.

The last thing we compared is the memory cost of the pyramid encoder and the regular encoder. This measure is crucial in some scenarios, where your input instances are very long; therefore, the memory of GPU is only capable of holding a very small batch. In code correction, this is often the case.

The metric we use for comparison is memory cost per instance, k, which is defined as k = ΔMemory usage ΔBatch size (24) Figure 5 shows the calculation process of k. Define ℰ = 1 / k as memory efficiency. We calculated the k and ℰ value for each of the models we applied, shown in Table 4. We also included the number of parameters in each model, from which we see that each pair of models has roughly the same model size.

The pyramid encoder could increase the memory efficiency by 20%–600% depending on the attention mechanisms used, while only increase the memory occupied by the model itself by around 10%. One should note that the memory efficiency directly affects the maximum batch size one is able to use on a single GPU, and therefore affects the utility of the GPU. For example, for regular GRU with Bahdanau Attention, the memory of a GeForce GTX 1080 Ti graphic card can only support a batch size of 8, which does not fully utilize the GPU. With pyramid encoder, it can support up to 60 instances each batch. In practice, this will drastically reduce the training time by increasing the GPU utility, together with the smaller computational cost of pyramid encoder as addressed in previous section.

Figure 6 shows the repair rate of the models with respect to the input length. We omitted the result of Transformer, Bahdanau’s attention, and Luong’s general attention, because they are qualitatively similar to the result of Luong’s dot attention. Despite the different attention mechanisms, these seq2seq models (with pyramid encoder or regular encoder) are relatively robust to longer input lengths. The performance drops at around 250 words and above 500 words are likely resulting from the shortage of samples, which one can easily observe from Figure 7, the length histogram of source instances and target instances. The histogram also shows that the majority of code instances contains several hundred words, while natural language sentences are typically not longer than 50 words. This feature of code instances calls for a much higher computational resource requirement for PLC problems than NLC problems, which makes pyramid structure especially useful.

In this section we give several examples of successful corrections from our Pyramid GRU model on Juliet C/C++ Test Suite for closer examination of model and datasets. The red striked out texts denote the original faulted instance, and blue buffed texts are the reparation done by the model.

Example 1: Memory allocation match

The flawed code creates a char variable whose size does not match its concatenating destination. The model is able to correct it so that their size matches each other.

Example 2: Redundant Code

This is an example that the model deletes repeated code where a variable is freed twice.

Example 3: Possible Overflow

Here we show a slightly questionable example of correction provided by the dataset. In order to prevent potential string overflow emerging from environment variable, the repair suggestion given by the Juliet Test Suite is to abort the entire part of concatenating the environment string and replace the variable with an arbitrary string “*.*”. This “correction” is easy for the model to learn; however, it has changed the original purpose of the program.

Example 4: Correction Across Functions

In this example, the models demonstrate the ability of making connections across the whole instance, between different functions. Here it prevents potential overflow in the sink function caused by a variable that was passed from the main function by adding an “if condition”.

In the spirit of comparative study, we attempted to compare our method to Deepfix (Gupta et al., 2017), the only machine-learning-based PLC method that made their code and dataset open to the public, to the best of our knowledge. Unfortunately, the attempt of applying Deepfix onto Juliet Test Suite has failed, because Deepfix is aimed only to correct syntax errors and a compiler is used as the evaluator, marking any programs that could pass the compiling stage as “correct”. This apparently contradicts the spirit of “identify logic errors from syntactically correct programs”.

The difficulty that we are facing here comes from a more general problem in the field of Machine Learning PLC; the field is still disorganized and works in the field are uncorrelated. Each group might be using their own dataset and design their systems to match the specific purpose of that dataset. Comparative work is difficult to conduct not only because the datasets are hard to obtain due to private policies, but also because issues raised in PLC are versatile; each model is designed and optimized to best address the problem occurring in their particular dataset.

For the above reasons, we had to back off to a weaker comparative study, using seq2seq models on the dataset from Deepfix. Deepfix uses a generated dataset, originated from students’ submission to an introductory C course in a web-based tutoring system (Das et al., 2016). For each student submission, they generate up to five syntax errors in the code instance, including replacing “}; ” with “; }”, deleting a semicolon, add an extra “}”, replacing a semicolon with a period, and replacing a comma with a semicolon. If all of the syntax errors were fixed, then consider such a program as successfully repaired.

Table 5 shows the comparison of repair rate of our seq2seq models compared to the method applied by Deepfix. We observe that pyramid encoder performs worse than regular encoder on this particular dataset. This is expected from how the dataset was generated. The generated syntax errors are extremely local in Deepfix’s dataset. The fix usually only involves changing one token or two neighboring tokens, leaving the rest of the entire code piece unchanged. Therefore, while a pyramid encoder summarizes the information from neighboring tokens, it also blurs the local information.

We also observed that, in Luong’s attention, dot has the best performance in this dataset and Bahdanau’s attention performs the worst. After observing the dataset carefully, we came up with the following hypothesis: in this dataset, the network is only required to simply copy the original token most part of the instance and locally fix one or two tokens. This means that in the majority of times, for each decoder hidden state h ¯ t , the normalized attention score score ( h ¯ t , h t ' ) needs to be close to one where t = t ' and close to 0 everywhere else. In Luong’s attention, a dot, which simply do an inner product of hidden states, could do the job easier, because latent vectors are mostly orthogonal to each other in the latent space due to high dimensionality. On the other hand, Bahdanau attention, which does an affine transformation to every hidden state h t , may overcomplicate the problem and fail to capture the correct attention.

One main difficulty that researchers often come across when attempting to apply machine learning methods to PLC problems is the availability of suitable datasets. Although there are many datasets and shared tasks available on Software Assurance Reference Dataset (2006), most of them include less than 1,000 examples. This makes neural-network-based methods nearly impossible. To tackle this problem, we take the idea of transfer learning from Pan and Yang (2009).

Our idea is to take the encoder part of the model that was trained on Juliet Test Suite and attach it to a untrained decoder, which was designed for the specific problem. We aim to take the advantage that codes written in the same coding language share the same syntax library and same construction rules.

Since many datasets available only provide the faulted code and their corresponding fault categories, here we give an example of fault classification using transfer learning, applying the model pretrained on Juliet Test Suite for C/C++ on ITC benchmark (Charles (2015)).