The advent of computers to psychological and educational assessment has made it possible to analyze behavioral processes and sequences of actions through information captured in computer-generated log-files (records of all actions taken while working on a computerized assessment; Bunderson et al., 1989). Researchers are no longer limited to measuring the final outcome of an assessment (e.g., solved vs. not solved) but can also investigate the steps and actions resulting in the specific outcome through analyzes of test-taking behaviors. In other words, analyzing log-files allows researchers to make inferences about the latent cognitive processes involved in solving tasks from overt behavior (Greiff et al., 2015b). Log-files may, for example, inform researchers of specific mistakes made while working on a problem that may be indicative of a misunderstanding of the problem at hand (Ifenthaler et al., 2012). Identifying specific test-taking behaviors that lead to successful and unsuccessful performance has proven to be a treasure chest for the improvement of interventions and teaching enabling the differentiation of instructions and scaffolding and providing students with avenues for learning individually tailored to their needs.

A field that has made much use of log-file analysis in the last years is the field of complex problem-solving (CPS; e.g., Goldhammer et al., 2014; Greiff et al., 2016). Analyzing students’ behavior through log-files, it was shown that the application of the vary-one-thing-at-a-time strategy (VOTAT; Tschirgi, 1980), also referred to as “control of variables strategy” (Chen and Klahr, 1999), could explain a great deal of students’ performance in solving complex problems (Greiff et al., 2015b). Others noted, however, that simply identifying those students that applied the VOTAT strategy is not sufficient to fully explain why some students successfully solve a task whereas others do not (Kuhn and Dean, 2005). There must be other differences in metastrategic behavior that distinguish students that apply the VOTAT strategy and successfully solve a problem and those students that apply the strategy but fail. The aim of this paper is to use data mining techniques to analyze CPS log-files to find differences in behavior that indicate successful and unsuccessful behavior beyond the already established strategies.

Traces of behavior have been gathered in psychology studies since the 1930s (Skinner, 1938). Today, modern computer-based applications of psychological assessment make it very easy to capture a variety of interaction behaviors and save them to log files for later analysis. These interaction data have been referred to virtually synonymously as “log-file data” (Arroyo and Woolf, 2005), “discrete action protocols” (Fu, 2001), or “process data” (Zoanetti, 2010), only listing the most common names. Behavioral log-files are indicators of human behavior as observed by automatic sensors that capture and record actions displayed while interacting with the assessment. They may include behavior as diverse as rich audio and video recordings or low-level keystrokes.

Complex tasks, allowing for multiple behaviors that lead to a correct solution, produce valuable log-files with sufficient variation among participants for a meaningful interpretation. The study of how individuals engage with such complex tasks is therefore synonymous with problem-solving (Vista et al., 2016). Exploration of the processes employed in problem-solving or in engaging with complex tasks can provide information about the cognitive skills that underlie successful resolution of the problems or tasks (O’Neil et al., 2003; Griffin and Care, 2015). Indicators of these cognitive skills can be deduced from behaviors, which are captured in the form of attempted or completed processes in problem-solving tasks.

Problem-solving tasks that are particularly rich in log-file data are CPS tasks. Throughout this paper, CPS is understood as “(…) the successful interaction with task environments that are dynamic (i.e., change as a function of the user’s interventions and/or as a function of time) and in which some, if not all, of the environment’s regularities, can only be revealed by successful exploration and integration of the information gained in that process” (Buchner in Frensch and Funke, 1995, p. 14). CPS tasks thus differ from static problem-solving tasks in that they require active interaction between the problem solver and the problem resulting in very meaningful log-file data (Greiff et al., 2015a).

The aim of this study is to use exploratory educational data mining techniques in explaining CPS behavior. We go beyond the already established VOTAT strategy, exploring differences in behavior between students that applied the VOTAT strategy to a complex problem and successfully solved it and those that applied the strategy but failed to solve the complex problem. To analyze students’ behavior, we chose the n-gram approach (Damashek, 1995) introduced above to classify students that applied the VOTAT strategy into successful and unsuccessful problem-solvers based on their behavior. Applying the n-gram approach, we summarize the participating students’ behavior while solving the complex problem into a set of short sequences that can be used to find behaviors that are indicative of whether a student that applied the VOTAT strategy will also solve the complex problem. Next to presenting the empirical example, we will illustrate the methodological steps necessary to apply the n-gram approach to log-file data of CPS behavior.

As can be seen from Table 4, the task was relatively difficult with only 544 (38.9%) of the students solving the task correctly. Moreover, 666 (47.6%) of the students applied the VOTAT strategy. Applying the VOTAT strategy, generally, lead to a substantially higher likelihood of solving the problem (χ² = 401.10; df = 1; p < 0.001). However, 143 (21.5%) of the students that applied the VOTAT strategy did not solve the problem.

In the exploratory analysis, we attempt to understand this observation by finding behavioral differences among the students that applied the VOTAT strategy by using the n-gram approach. There was no significant difference between the absolute number of behaviors observed for either group of students [t(664) = 0.52; p = 0.601; d = 0.05] nor the time spent working on the task [t(664) = 0.27; p = 0.790; d = 0.03]. Table 5 displays the results of the chi-square feature selection model analyzing differences in likelihoods of specific n-grams for students that applied the VOTAT strategy and solved the problem and those that did not. Note that the possible behaviors were reduced to working on the scenario (S) and changing the model (M). N-grams with higher chi-square values are more discriminative between the two groups. Moreover, Table 5 indicates whether the n-grams were more typical of students that solved the problem or of those that did not.

As can be seen from Table 5, the informational value of the n-grams increases with their length, while the general pattern does not change. The most discriminative sequence of behavior was consistently the one indicating working maximally long in the scenario (SS, SSS, and SSSS), which was always more typical of the students that did not solve the task. This was followed by the sequence of behavior indicating working maximally long in the model (only statistically significant for MMM and MMMM), which was always more typical of students solving the task. Generally, the sequences indicating repeated changes in the scenario were associated with failing to solve the problem (statistically significant for MSS, MSSS, and SSSM), whereas the sequences indicating repeated changes in the model were associated with solving the problem (statistically significant for SMMM). The discriminative value (high chi-square values) was highest for the sequences with the longest uninterrupted sequences of one specific behavior (M or S) and least for those that indicated frequent changes between working on the scenario and working on the model (e.g., SMSM).

The aim of this study was to use exploratory educational data mining techniques in explaining problem-solving behavior. We chose one of the most established types of CPS tasks (based on the MicroDYN approach; Greiff et al., 2015a), for which the optimal strategy is well known (i.e., the VOTAT strategy; Tschirgi, 1980). However, not all students applying the VOTAT strategy also solved the tasks correctly implying that simply observing whether or not the strategy was applied is not sufficient to understand why some students succeed in solving CPS tasks while others do not (Kuhn and Dean, 2005).

Describing the whole string of behaviors observed for each individual student as a set of n-grams of different length (Damashek, 1995) allowed us to exploratively search for differences in the behavior observed within those students that applied the VOTAT strategy and successfully solved the task and those that applied the strategy but still failed to solve the task. The empirical example illustrates that given enough complexity, there are substantial differences in the frequencies of observed n-grams between the two groups. Interpreting those differences, however, requires some understanding of the task and what it takes to solve it (Banovic et al., 2016).

Correctly applied, the VOTAT strategy requires problem-solvers to make only minimal changes in the scenario, register the effects and then immediately plot the findings in the model (Wüstenberg et al., 2012). Any deviations from this algorithm will increase the cognitive load (Sweller, 2011) on the problem-solver as important information (i.e., either changes made in the scenario or findings resulting from these changes) need to be stored in working memory (Sweller, 1988). Inspecting the differences in behaviors between students that applied the VOTAT strategy and successfully solved the task and those that applied the strategy but still failed, the general pattern seemed to be that the students that solved the task spent fewer rounds continuously working on the scenario (e.g., SS, SSS, or SSSS) but more rounds working on the model (e.g., MMM and MMMM). Students that did not solve the task, thus, did not immediately plot their findings, thereby increasing their cognitive load and, in turn, the task’s difficulty (Kirschner, 2002). Our findings thus highlight the importance of metastrategic competencies that enable a person to not only apply the correct strategy to solve a problem but to make use of the information gained in the process. Metastrategic competencies encompass awareness, understanding, monitoring, and management of one’s strategic performance of many kinds of cognitive tasks (Kuhn and Pearsall, 1998). As becomes obvious from our analyses, students that did not solve the problems correctly either lacked understanding of the VOTAT strategy or were not able to manage their use of the strategy. Due to the exploratory nature of our analyses, our interpretations are post hoc though and should be corroborated by experimental studies.

There are other limitations to be considered. In focusing only on the students applying the VOTAT strategy we reduced our sample to N = 666, excluding almost half of the initial sample from our analyses. However, since the aim of our paper was to find behavior differences between students that applied the VOTAT strategy and successfully solved the problem and those students that applied the VOTAT strategy but failed, students that did not apply the VOTAT strategy at all were irrelevant to our analyses. Future studies should extend our analyses to explore differences in behavior across all students not selected by a priori defined strategies.

Moreover, coding of the log-files into changes in the scenario and changes in the model does not allow differentiating between different changes applied to the input variables within one round of changes to the scenario (e.g., manipulations of only one variable vs. manipulations of multiple variables). However, this simplification allows for a relatively straight-forward interpretation of the resulting n-grams. A more detailed coding of changes to the scenario, on the other hand, would lead to an exponentially higher number of potential behavior sequences most of which would most likely have very little information value due to their specificity. The potential variance in changes in the input variables between successful and unsuccessful students is further reduced by the fact that all participants included in our analyses applied the VOTAT strategy (i.e., manipulated all input variables at least once individually while keeping the others constant). Since manipulating all input variables individually once is sufficient to solve the task, all further manipulations, regardless of whether single or multiple variables, will result in unnecessary additional information increasing cognitive load. Testing these assumptions will, however, require additional information to be logged (for more on the completeness of log data see Kroehne and Goldhammer, 2018).

Finally, the n-gram approach showcased in this paper is not the only explorative educational data mining approach applicable to CPS log-files, of course. Other studies have applied analyses of the interaction of behavior displayed while solving tasks such as Network Analysis (Wooldridge et al., 2018), or included the temporal order of behaviors in their analyses by displaying them as complex directed networks (Vista et al., 2016). All of these approaches share the aim of understanding problem-solving behavior on a very detailed level and the difficulties that come with that aim. Most importantly, any increase in task specificity (e.g., longer n-grams) comes, necessarily, with a decrease in generalizability. In that, perfect understanding of students’ behavior in one task may be meaningless to understand performance in another task unless the structural similarities between these tasks are well understood and theoretically described. Future studies should, therefore, investigate the generalizability of behavior across different problem-solving tasks.

The findings show the potential benefit of applying explorative educational data mining approaches such as the n-gram approach in addition to searching for a priori defined strategies. Knowledge about how and why students that actually apply the correct strategy to solve a problem fail to actually solve it has implications for the instruction or training of CPS tasks. So far, interventions aimed at increasing CPS performance have relied on repeatedly confronting problem-solvers with problems of a similar nature (e.g., Kretzschmar and Süß, 2015). Training lead to an increase in performance and, in fact, also to an increase in strategic prowess (Lotz et al., 2017). However, no dedicated strategy training has been published to the best of our knowledge. Based on our findings, such a strategy training should consider to not only teach the VOTAT strategy but also metastrategic knowledge such as the handling of information gained through the application of VOTAT (Zohar and Peled, 2008).

In summary, our paper showcased the n-gram approach on a CPS task. The detailed description of the data provided some indication toward behavioral differences within students that apply the correct strategy toward a problem and solve it as opposed to those that apply the correct strategy and fail. We hope that the paper will help other scholars in finding ways to analyze and interpret log-file data themselves. After all, the exploitation of this rich resource through dedicated analyses is still in its infancy and we believe that it is a treasure trove worth hunting for.

Ethics approval was granted for the data collection (see Vainikainen, 2014).

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

SG is one of two authors of the commercially available COMPRO test that is based on the multiple complex systems approach and that employs the MicroDYN approach. However, for any research and educational purposes, a free version of MicroDYN tasks is available and he receives royalties for COMPRO. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.