Applying Learning Analytics in Online Environments: Measuring Learners’ Engagement Unobtrusively

Prior to the emergence of Big Data and technologies such as Learning Analytics (LA), classroom research focused mainly on measuring learning outcomes of a small sample through tests. Research on online environments shows that learners’ engagement is a critical precondition for successful learning and lack of engagement is associated with failure and dropout. LA helps instructors to track, measure and visualize students’ online behavior and use such digital traces to improve instruction and provide individualized support, i.e., feedback. This paper examines 1) metrics or indicators of learners’ engagement as extracted and displayed by LA, 2) their relationship with academic achievement and performance, and 3) some freely available LA tools for instructors and their usability. The paper concludes with making recommendations for practice and further research by considering challenges associated with using LA in classrooms.


INTRODUCTION
Data-driven Decision-Making (DDDM) requires the instructors to actively adapt their instruction to evidence gathered in multiple-format from multiple sources. Digital environments such as LMS and MOOCs generate a large amount of data-Big Data-about students' learning, performance and engagement activities called 'digital traces'. Since students' engagement is reported to be positively associated with learning and productivity, Learning Analytics (LA) can be used to collect and analyze these digital traces to track, measure and visualize students' engagement patterns with materials/ content, instructors, and peers in real time (during the learning process). Teachers can use data from LA to guide their pedagogical decisions, improve instruction and provide customized support, i.e., individualized feedback. This paper aims at exploring the complex, multidimensional nature of engagement as well as the use of LA in quantifying and measuring the relationship between engagement and learning outcomes. Furthermore, some freely available LA tools, their usability and challenges will be presented. After a review of engagement conceptualizations, the remainder of the paper aims to delineate the following questions: identification, 3) literature search, 4) data evaluation, 5) data analysis and presentation of instructional principle suggested. As the first step, articles on "students engagement," "learning analytics," "conventional and online courses," and 'higher education' were considered as eligible candidates to be included. The main identified problem was "inadequacy of traditional evaluation methods in measuring students engagement." Literature search was first performed on three journals related to education and educational technology: Educational Technology & Society, Journal of Educational Research, and British Journal of Educational Technology. Later, a so-called cross-referencing was conducted: the reference sections of relevant articles were consulted to find further articles. About 84 articles were selected and put into a quick review (abstracts only). Eventually, a total of 37 articles were scanned thoroughly.

STUDENTS' ENGAGEMENT: A MULTIDIMENSIONAL CONSTRUCT
Over the past 30 years, educational research has shown a growing interest in understanding "engagement." Engagement, as a form of involvement and commitment, is defined as the "effort, time and energy students invest in academic experience and educational activities linked to desired learning outcomes" (Astin, 1984;Kuh, 2009). Engagement is considered to be the "holy grail" of learning (Sinatra et al., 2015) which plays a critical role in the development of deep and effective self-regulated learning. While early research was mainly concerned with reducing drop-out (Finn and Voelkl, 1995), there is a shift towards understanding and describing engagement as a "process" and its relation to achievement. There are several models or frameworks aiming at conceptualization of engagement. Previous studies (Marks, 2000) conceptualized student engagement as a unidimensional construct which can be measured by observable behaviors: i.e., attendance, participation, psychological effort. However, current literature favors a multidimensional conceptualization of engagement, e.g., Reschly and Christenson (2012) defined engagement, both as a product and a process, with four dimensions: academic, behavioral, affective, and social. Fredricks et al. (2004) suggested a three-component framework: behavioral, cognitive, and emotional engagement. Behavioral engagement refers to active physical participation, i.e., attending classes, completing the course, note-taking or asking questions. Cognitive engagement describes mental involvement in understanding, willingness to master a difficult task and task management approaches, such as focused attention and problem-solving strategies. Emotional engagement includes affective factors, i.e., being interested in a topic, enjoying the learning experience, flow and sense of belonging to a community.
Considering the positive relationship between engagement and learning outcomes, i.e., Morris et al. (2005) reported that it can account for 31% of variance in gain score, several studies consider using engagement as a viable predictor of academic achievements (Lee, 2014). Traditionally, engagement was measured through 1) self-report questionnaires, 2) interviews, 3) observational checklist and 4) teacher's informal observation of gesture and non-verbal features (i.e., boredom or frusteration). There are several validated, empirical surveys to capture students' engagement, i.e., Classroom Survey of Student Engagement (Ouimet and Smallwood, 2005) or Student Course Engagement Questionnaire (Handelsman et al., 2005). Majority of survey studies show when students have more frequent interaction with instructor, content, and peers, they report higher satisfaction, more meaningful participation and higher-order learning (Eom et al., 2006). However, survey data mostly suffer from validity issues, i.e., social desirability, unstable moods, response biases, etc. Collecting interview data is also excessively costly, time-consuming and impractical in large courses. Although observational checklists, mostly conducted by an external observer, contain some objective measures of engagement (i.e., student-initiated questions), they demand a great deal of skill, effort and time from observers. Lastly, teacher's informal observation, although an important source of data to make decision "on the fly," is criticized due to its unreliability and subjectivity of judgement.

STUDENTS' ENGAGEMENT IN ONLINE ENVIRONMENT
Advances in technology paved the way to the automatic generation, collection and analysis of educational data. The advent of digital or virtual learning environment (VLE), such as Learning Management Systems (i.e,Moodle or Blackboard), Massive Open Online Courses (MOOCs), Open Course Ware (OCW), learning applications, Intelligent Tutoring Systems (ITS), E-book portals, online games, simulations, social networks (i.e., YouTube, Twitter), Open Educational Resources (OER), as well as wearables, sensors, Internet of the Things (IoT), not only provide the users with advantages such as openness, flexibility, autonomy, accessibility, and collaboration, they also led to the exponential growth of big data: large, heterogeneous, multi-modal, multi-sources educational data. Digital environments and big data generate (new forms of) evidence, so-called "digital traces," about students' learning, performance and engagement activities. These "digital traces" that students leave behind as they are involved in the learning process are anaylzed with technologies such as Educational Data Mining (EDM) and Learning Analytics (LA).
Siemens (2012) defined LA as the "measurement, collection, analysis and reporting of data about learners and their contexts for purposes of understanding and optimizing learning and the environment in which it occurs." LA uses of extracted user data or actual learning activities in real time provides insight into engagement, motivation, performance and learning in an 'unobtrusive and ubiquitous' way (Bodily and Verbert, 2017). Such trace data are better indicators of learners' engagement compared to self-report or observational data. Earlier uses of LAgenerated data on engagement aimed at predicting dropout and failure through early alert systems (i.e., StepUp dashboard). Currently, the focus is more on providing just-in-time, formative feedback to teachers (and students), so that they can adjust the content and personalize the delivery mode of instruction.

ENGAGEMENT METRICS AND ACHIEVEMENT
Several studies discussed the question "what should count as desirable metrics of engagement and participation in VLE?" (DeBoer et al., 2014). Existing literature shows various operationalizations of online engagement. For example, Kovanović et al. (2019) examined different aspects of students' engagement, i.e., course access, course navigation, number of recorded lectures watched, discussion, or completed assignments. Kahan et al. (2017) focused on the "frequency of involvement": how many times video lectures are downloaded, embedded questions answered, exams/quizzes submitted, thread views/ opened, posts/comments written in the discussion forums, etc. Below, we summarize some engagement indicators reported by existing literature (Aluja-Baneta et al., 2019;Deng and Benckendorff, 2017;Deng et al., 2020;Joksimovi et al., 2018;Lee et al., 2021). It should be noted that this list is not exhaustive and although being useful, some metrics are difficult to interpret: i.e., whether an unfinished task or exercise is an evidence of low engagement or low cognitive skill?
• log data • frequency and intensity of access to LMS • mouse click count • time-on-Task (i.e., problem-solving or a particular resource) • reaction time or timeCost (time taken to answer a question or complete an exercise) • use of hints/help/guide (quantity and quality) • artifacts (portfolio, essay, projects, lab reports) • assessment data (test, exam, quiz) • patterns in errors, mistakes, misconceptions about a particular topic/concept • number of videos watched • video-watching frequency and duration (pause, playback, adjusted speed, embedded questions) • number of completed assignments/submission • materials' annotation, highlight, notes, summary • page turning/jumping • discussion forums (viewing, posting, voting, updating) • bookmarks (created, shared, commented) • Library usage • Survey data • Social networks (number of connections and community membership) How could these metrics or indicators be used to support teaching and learning? LA uses such metrics to examine the relationship between online engagement indicators and learning outcomes. Learning outcomes are defined as retention/ completion and final grade or gain score. Although there is a general consensus about a positive correlation between students' engagement and their learning outcomes, the relevant literature seems to report contradictory findings or sometimes failed to show any associations. Macfadyen and Dawson (2010) used learning analytics data to explore the relationship between number of messages a student posted, time spent on educational resources and learning outcomes. Although there was a strong correlation between students active posting and their academic success, there was no significant relation between time spent on educational resources and final grades. These results are at stark contrast with the current LA studies: Yang et al. (2019) reported a positive correlation between reading time spend on e-book learning materials, as measured with time-stamp data, and students' learning outcomes. Lu et al. (2017) asserted that by using online engagement data collected by LA, it is possible to predict academic performances even after only one-third of the semester's duration. In another attempt, Moreno-Marcos et al. (2020) included engagement indicators, such as videowatching behaviors, click counts, number of posts to the discussion forums, and answers to a quiz to predict students' final grades. Participation in discussion forums showed a weak predictive ability, whereas average answer scores had the highest predictive ability.
Studies on MOOCs showed that although active participation in discussion forum (a higher number of posts) was associated with course completion and certification, it was not associated with final achievement (Goldberg et al., 2015). Vu et al. (2015) noted a onedirectional relationship: higher score can predict the number of messages or posts, but participation in forums and the number of posts can not necessarily predict the gain score. It should be noted that majority of research on MOOCs considered 'completion of a course' as the main outcome and academic achievement.

ENGAGEMENT ANALYTICS
LA dashboard is defined as "a single display that aggregates different indicators about learners, learning processes and learning contexts into one or multiple visualizations" (Schwendimann et al., 2017). In online or blended (hybrid) courses, instructors can use LA visualization dashboard to get an effective and automatic "engagement tracking'" and inform their pedagogical decisions. Such dashboards can provide insights to the following questions: Do students access LMS regularly or only before exams/submissions? What proportion of reading materials is covered? What resources are used more frequently/seldom? In what order students navigate through the content? How many items or questions in a quiz are answered correctly/wrong?
There are several LA-based tools to report on educational activities, i.e., SNAPP uses comments and posts from discussion forums to visualize the interaction network among learners; GISMO visualizes online learning activities, or LOCO-Analyst summerizes and reports learning activities to instructors. However, most digital learning platforms do not automatically include the advanced tools required for applying LA.
Additionally, utilizing these tools is too complex and have features (i.e., statistics) beyond the capability of classroom teachers. Moodle, an open sources and freely available LMS, is used extensively in higher education around the world. Therefore, we briefly describe two free LA dashboards for Moodle. a) "MEAP: Moodle engagement Analytics Plugin" allows instructors to track students engagement based on their log activity, assessment data and posting to discussion forum (Liu et al., 2015). b) "LEMO2 CourseExplorer" is an LA dashboard which visualizes students' performance in both standard overview and customized format. Instructors can apply different filters to a specific learning objects and track daily activity flow, distribution of performance (quiz, forum, video-watching), test performance, calendar heatmap, peak activity in terms of time, etc., (Jaakonmäki et al., 2020). c) "Inspire" is a Moodle Analytics API (application programming interface) that provides descriptive and predictive analytics engine by implementing machine learning backends (Romero and Ventura, 2020).

GAPS AND CHALLENGES FACING LA
Notwithstanding the fact that LA has a great potential to support teachers and improve learning, there are a number of issues (and potential measures to ameliorate them) which should be considered in using LA: 1. Using LA to track digital traces and monitor students' behavior raises serious question about data security, privacy, user protection and ethics. Enhancing LMS security, setting authorized access, informed consent of students, as well as clear instructions about how data will be stored, accessed, or used are among potential solutions. 2. LA dashboards provide instructors with a summary of students' engagement. The decision and action to be taken are left to instructors or supporting staff. Drawing actionable knowledge from dashboards needs considerable Pedagogical Content Knowledge (PCK). This mandates extensive investment in Teacher Professional Development, both in pre-service and in-service programs. No matter how powerful or expensive an LA dashboard is, it can fulfill its potentials only through appropriate human judgement, decision and action. 3. Student-facing dashboards are claimed to foster reflection and Self-regulated Learning (SRL). However, visualization of activity per se may not automatically promote awareness in learners. Majority of LA experts believe students are unable to interpret what analytic dashboards are suggesting them or fail to take appropriate action (Winne, 2018). 4. Many scholars warned against too much reliance on teachers' intervention based on LA dashboards due to the risk of reducing students' autonomy and responsibility of taking charge of their own learning (Bodily and Verbert, 2017).
5. Quantitative, easy-to-display measures (i.e., presence, frequency or time spent) cannot fully portray a purposeful and deep engagement in learning. Dawson and Siemens (2014) called for "LA tools and techniques that go beyond surface-level analytics which uses diverse, alternative assessment methods which reflect twenty-first century education, a kind of complex and multimodal learning." More qualitative indicators and multi-modal data types should be considered. For example, if a student spends less time on a task, it might be due to his superior background knowledge, better task management strategy or lack of engagement. LA can exploit Machine Learning and AIbased techniques, i.e., Natural Language Processing (NLP) and Deep Neural Networking (DNN), to distinguish between the quantity (frequently posting shallow comments in a forum) and the quality of engagement (deep, meaningful reasoning in a discussion forum). Furthermore, ML techniques build predictive models which facilitate timely intervention. 6. Finally, although LA tracks and monitors students activities in VLE, a great part of learning may happen offline: the time students spend reading textbooks, reflecting, solving problems, or completing a field project, contributes greatly to learning, especially to self-regulated learning, which can not be captured by LA. Therefore, not being active in online environments cannot be equated with a complete lack of learning. This might explain why some studies failed to find any clear correlational patterns between log data and final scores.

CONCLUSION
Given the importance that existing literature attach to students' engagement and the negative consequence of disengagement (i.e., higher dropout, failure, decreased intrinsic motivation), this study explored how LA can be used to continually and formatively track digital traces of engagement through interaction data, i.e., time-stamped logs. Traditionally, "perceived engagement" was measured through self-reported scales, interview or teacher observation. LA can extract objective engagement features automatically, unobtrusively (without interrupting learning) and ubiquitously. Such data can be used in predicting dropout and failure, giving early warnings to teachers about students who are struggling or losing interest, and support timely teachers' appropriate intervention. LA, as a data-driven approach, has a great potential to give an objective picture of learning by uncovering the relations between the processes of learning (e.g., engagement) and the end-product of learning (e.g., achievement). Metrics/indicators of engagement in VLE as well as freely available LA dashboards were delineated. Based on the potential challenges pointed out, we suggest the followings for the future research: 1. Complementary research approaches which triangulate multifaceted evidence from multiple sources, i.e., students' self-reported data and LA trace data can be combined with sensor-data. Wearables such as eye-movement tracking, noninvasive EEG, fMRI, Augmented Reality (AR), smart glasses can measure neurological status, i.e., alertness, as well as motivational and affective states, i.e., while confusion associates positively with learning, boredom and constant frustration are negatively correlated with learning outcomes. 2. Robust evaluation of LA dashboards as a pedagogical tool, its impact on students' awareness, self-regulated learning, engagement, outcomes, etc. 3. Optimal design of LA dashboards: there should be a shift from "what elements" are displayed towards "how" combining metrics of engagement support future engagement. Furthermore, machine learning algorithms, such as Support Vector Machine (SVM) and Artificial Neural Networks (ANN), can be used to develop more fine-tuned and meaningful classification of engagement's levels. This could support instructors in providing timely and tailored pedagogical interventions.
As Worsley et al. (2021) mentioned: raw data collected by LA or any other tools can seldom give clear, immediate answers to any research questions. The goal of data analysis is to transform data into actionable insights, which requires an in-depth understanding of the local context by researchers (e.g., setting, participants, available resources and analytic techniques). LA research community needs to find ways to encourage more involvement of main stakeholders (e.g., students, teachers, policy-makers) to find solutions to technical, methodological, and practical challenges which are briefly reviewed in this paper Bonafini et al., 2017, Drachsler and Greller, 2012, Gardner et al., 2020, Hill, 2013, Hussain et al., 2018, Lei et al., 2018, Yin et al., 2019

AUTHOR CONTRIBUTIONS
The author confirms being the sole contributor of this work and has approved it for publication.