Learning analytics tools should be useful, i.e., they should be usable and provide the functionality for reaching the goals attributed to learning analytics. We present a short summary of the learning analytics goals, and the importance of evaluation of learning analytics while trying to attain these goals. Furthermore, we present three different case studies of learning analytics evaluation, and in the end provide a short outlook about the necessity of systematic way of learning analytics evaluation.
Over the past two decades learning has been extensively influenced by technology. Learning is a dynamic activity, which should constantly be monitored, evaluated, and adjusted to the demands of changing social contexts and needs of the different involved stakeholders, to ensure quality and the best possible outcomes. The incorporation of educational technologies created new prospects and opportunities to gain insight into student learning [GDS01]. One area of these educational technologies or technologyenhanced learning that is specifically concerned with improving the learning processes is learning analytics.
During the past decade the field of learning analytics (LA) has been defined in several ways; see [DV12], [El11], [JAC12], [Si10]. In the context of our research, we understand learning analytics as development and exploration of methods and tools for visual analysis and pattern recognition in educational data to permit institutions, teachers, and students to iteratively reflect on learning processes and, thus, call for the optimization of learning designs [LD12] on the one hand and aid the improvement of learning on the other [CDS+12]. In our understanding, learning analytics thus subsumes research areas of educational data mining (methods and tools), and teaching analytics, as well as academic (or organizational) analytics, when all are applied to optimize learning opportunities. In this paper we will present a short summary of the learning analytics goals, provide three case studies of learning analytics tools’ evaluation in correlation with the goals, and conclude with the challenges and the outlook of the evaluation of learning analytics tools. 2
Intrinsically, learning analytics has the noble goal of improving learning and has a pedagogical focus. It puts different analytics methods into practice for studying their actual effectiveness on the improvement of teaching and learning (learner-focused analytics) [CLT+15]. As Clow [Cl13] puts it “Learning analytics is first and foremost concerned with learning”. Table 1 presents a collection of the overall goals of LA from various literature published in the respective field. The goals of LA can be divided into goals that: a) b) c) explicitly inform the design of learning analytics tools involve a behavioral reaction of the teacher involve a behavioral reaction of the student b. Educators are supposed to c. Learners are supposed to a. Learning analytics tools are supposed to track user activities capture the interaction of students with resources / the interactions among students gather data of different systems provide educators / students with feedback/information on students’ activities provide an overview highlight important aspects of data provide differentperspectives offer possibilities for (peer) comparison draw the users attention monitor learning process / way of learning / students’ effort explore student data / get to know students’ strategies identify difficulties discover patterns find early indicators for success / poor marks / drop- out draw conclusions about usefulness of certain learning materials and success factors become aware / reflect / selfreflect better understand effectiveness of learning environments intervene / supervise / monitor own activities / interactions / learning process compare own behavior with the whole group / high performing students become aware reflect / self-reflect improve discussion participation / learning behavior / performance become better learners learn to interesting correlations pinpoint problematic issues establish an early warning system provide decision support advice / assist improve teaching / resources / environment Gašević et. al [GDS15] in a recent publication focused on critical goals, topics, and aspects that require immediate attention in order for LA to make sustainable impact on the research and practice of teaching and learning. In their work, they provide a summary of critical points and discuss a growing set of issues that need to be addressed and strongly points out that learning analytics are about learning [GDS15]. Their work focuses on specific points which encompass the LA goals provided in Table 1. One point they provide is that LA resources should be aligned to well-established research on effective instructional practice. To argue this point they point out that observations and analyses suggested that instructors preferred tools and features which offered insights into the learning processes and identified student gaps, rather than simple performance measures. Additionally, teachers should be aware of what their students i.e. learners are doing within a course, reflect and draw conclusions about the quality of the learning content they are providing, how are the students interacting with the learning materials, the pedagogical practices, the level of collaboration and interaction among the students, while supporting them within a course [DLM+13]. Likewise, LA can stimulate and motivate students to self-reflect on their learning behavior, become aware of their actions, learning practices and processes [SDS+14]. This, in turn, could initiate change in the learners’ behavior in order to become better learners, improve their communication skills, improve their performance, etc. [DLM+13]. In order to check whether the learning analytics tools attain and support these goals, we need conclusive evidence. Learning analytics evaluation practice could help in providing conclusive evidence and showing that the tools are not only usable, but also useful for the teaching and learning processes. 3
According to the research, learning analytics provides added value to both learners and educators [DLM+13], [LS11]. However, very little research has been done to actually confirm and reassure that LA studies and the tools have the desired effect and positive impact on the involved parties [SDS+14]. Surprisingly, there are very few publications that report about findings related to the behavioral reactions of teachers and students, i.e. few studies measure the impact of using LA tools. This raises several questions: What are the effects of usage of LA? How do LA systems influence practical learning situations? How does an indicator, or a set of indicators help the user to reflect and change his behavior? If there are behavioral changes, how do we see them? [DLM+13]. In order to answer these questions (and more questions that will arise) we need to implement evaluation techniques and carry out effective evaluation. Effective evaluation is difficult and is problem-prone, but it is essential to support the LA tasks. LA tools (as most information visualization interfaces) are in essence, generative artefacts. They do not have value in themselves, but they generate results in a particular context. In essence, an LA tool is used for a particular reason by a particular user, on a particular dataset. Hence, the evaluation of such tool is very complicated and diverse [CLT+15]. Ellis and Dix [ED06] argue that to look for empirical evaluation of validation of generative artefacts, is methodologically unsound. Any empirical evaluation, cannot tell you, in itself, that the LA tool works, or does not work [CLT+15]. 3.1
In this section, three different case studies will demonstrate different evaluations carried out on LA tools. Early on, the research on evaluation of LA tools focused on functionality and usability issues (comprehensibility, the design of indicators, terminology) and perceived usefulness of specific indicators [DLM+13]. For this purpose, well defined methods from the HCI field had been applied in these three different case studies.
LOCO-Analyst is a learning analytics tool that was developed to provide educators with feedback on the learning activities and performance of students. The researchers have done the evaluation of their tool in two iterations. The first iteration they conducted was a formative evaluation aimed to investigate how educators perceive the usefulness of such a tool to help them improve the content in their courses, and to which extend the user interface of the tool impacted this perceived value. Additionally, they used the evaluation as chance to elicit additional requirements for improvement of the tool. The study design was implemented with focus on collection of quantitative data and perceived qualitative data from a larger sample of educators. During the study, 18 participants from different higher education received the tool, and a questionnaire with guidelines how to evaluate the tool. The researchers analyzed and coded the results in three different categories: Data Visualization, GUI, and Feedback. The results of the first evaluation of the tool influenced the enhancement of the tool’s data visualization, user interface, and supported feedback types [AHD+12].
The second evaluation iteration, conducted on the improved LA tool, was summative evaluation. The main goal of it was to reassess the perceived usefulness of the improved tool, focusing on assessing how the changes influenced the perceived value of the LA
Learning Analytics Evaluation – Beyond Usability 127 tool and determining the extent of interconnection between the variables that characterized this perceived usefulness. The design of the study and the artifacts used, were the same as in the previous iteration i.e. the participants received the tool, a questionnaire and guidelines how to evaluate the tool. Additionally, the participants received video clips which introduced the tool, and described how each individual feature works. The researchers analyzed and coded the results of the second evaluation in the same way as in the first one [AHD+12].
The results of the second evaluation provided information how the implemented improvements to the tool affected the users’ perception of the tool’s value. In the end, the evaluation showed that educators find the kinds of feedback implemented in the tool informative and they valued the mix of textual and graphical representations of different kinds of feedback provided by the tool [AHD+12].
Student Activity Meter (SAM) is a LA tool that visualizes collected data from learning environments. The researches incorporated the evaluation in the development of the tool, i.e. applied design-based research methodology which relied on rapid prototyping in order to evaluate ideas in frequent short cycles. They did four iterations over the course of 24 months. The results of each evaluation iteration were put into two major groups: positive and negative. The results and the provided feedback were incorporated to improve the tool [GVD+12].
The methodology for the first iteration were task based interviews coupled with thinkaloud strategy, and usage of System Usability Scale, and Microsoft Desirability Toolkit on Computer Science students. The negative results were the identification of usability issues, and points for improvement. The positive results revealed that learnability was high, the error rate was low. The user satisfaction and usability were decent, and preliminary usefulness was regarded as positive. The study also revealed which LA indicators were considered as most useful [GVD+12].
The methodology for the second iteration was conducting an online survey with Likert items on teachers in order to assess and evaluate teacher needs, extract information about use and usefulness, and whether SAM can assist them in their everyday work. The most prominent result that was considered negative was that teachers did not find resource recommendation useful. On the positive side, the results showed that SAM provided awareness to the teachers, that all of the indicators were useful, and that 90% of them wanted to continue using SAM [GVD+12].
The third iteration was also an online survey with Likert items, but the demographics was LAK course participants (teachers and visualization experts). The goal was also similar like in the second iteration, to assess the teacher needs, improve the use and usefulness, and enhanced to collect feedback from the experts in the field. The negative results of the evaluation was the failure to find which needs needed more attention. On the positive side, the results from the second iteration were strengthened, with the addition that resource recommendation could be useful [GVD+12].
The fourth iteration was conducted with Computer Science teachers and teaching assistants. The methodology was conducting structured face-to-face interviews with tasks and open ended questions with the goal to assess the user satisfaction, the use and usefulness of SAM. The fourth iteration revealed that there are conflicting visions of what were students who were doing well, or what were student who were at risk. Furthermore, it revealed which indicators were good and useful, provided different insights from the teachers, and also further use cases for SAM were discovered [GVD+12]. Overall, the conducted evaluation discovered that the most important feature that SAM addresses was to help teachers provide feedback to the students. Another important provision was the methodology of the evaluation which could be applied/used from other researchers when creating a visualization tool [GVD+12].
Course Signals is an early intervention solution for higher education faculty, allowing instructors the opportunity to use analytics in order to provide real-time feedback to a student. The development team had closely tracked the student experience from the introduction of the tool (the pilot phase), and at the end of every additional semester. Furthermore, they conducted an anonymous student survey to collect feedback at the end of each semester, and had held focus groups. In general, students reported positive experience with the feedback they received from Course Signals. The students felt supported, and the feedback provided by the system was labeled as motivating. Some students had concerns that the system did over penetration (e-mails, text messages, LMS messages) all of them conveying the same message [AP12].
In general, the faculty and instructors had positive response, but still approached it with caution. The main points the development team extracted from the faculty feedback were that the students might create a dependency on the system, instead of developing their own learning traits. Furthermore, the evaluation discovered that there was a clear lack of best practices how to use Course Signals. This was also confirmed by the students. The most important point here was that this tool with its evaluation provided actual impact on teaching and learning [AP12]. 3.2
The three different evaluation case studies show that there is no standardized approach how to effectively evaluate learning analytics tools. Measuring the impact and usefulness of LA tools is a very challenging task. LA tools try to support both learners and educators in their respective tasks, and fulfil the goals mentioned in section 2. Although much work has been done on visualizing learning analytics results – typically in the form of
Learning Analytics Evaluation – Beyond Usability 129 dashboards, their design and use is far less understood [VDK12]. So far not many have conclusive evaluation and strong proof for beneficial impact on either educators, or students. These LA tools should not only be usable and interesting to use, but also useful in the context of the goals: awareness, self-reflection, and above all, improvement of learning.
In order to do effective evaluation, there are several things that need to be taken into account. First and foremost, one has to consider the purpose and the gains of the evaluation. The evaluators need to carefully design the goals and attempt to meet them with the evaluation. Once the goals are set, the next step is to think about the measures and tasks that will be included in the evaluation. It is very important to define the appropriate indicators and metrics. Wherever possible, the evaluators should also collect qualitative data, and use qualitative methods in pair with the quantitative evaluation. Mixed-method evaluation approach that combines both quantitative and qualitative evaluation methods can be very powerful to capture when, why, and how often a peculiar behavior happens in the learning environment [CLT+15].
While usability is relatively easy to evaluate, the challenge is to investigate how LA could impact learning and how it could be evaluated. Measuring the impact of LA tools is a challenging task, as the process needs long periods of time, as well as, a lot of effort and active participation from researchers and participants [CLT+15]. Moreover, the analysis of the qualitative data from the evaluation is always prone to personal interpretations and biased while making conclusions [DLM+13]. 4
As the research field matures, there is a continuous increase in research about systematic evaluation of learning analytics. Impact remains especially hard to determine in evaluation studies and further research is also required to investigate effective mixed- method evaluation approaches that focus on usability and usefulness of the Learning Analytics tools [VDK12]. To enhance the work in capturing and measuring impact, collaboration with cognitive sciences is necessary in order to develop methods how to attain this qualitative information. This means that asking the right questions, selecting elements of the environment and the tool to examine, and processing, visualizing, and analyzing the data become the challenges for researchers. There already has been literature and community based research that empirically tries to identify quality criteria and quality indicators for LA tools to form an evaluation framework [SDS15]. This evaluation framework has five criteria, and each criteria contains different quality indicators. The main limitation is that the participants who helped create this evaluation framework were more research than practice oriented. More importantly, the researchers should strive to a common goal, which is to unify and standardize the different evaluation methods into a structured tool that can help researchers and developers to build better Learning Analytics tools. 5
In this paper we presented a short summary about the goals in learning analytics. Furthermore, we argued that learning analytics evaluation will provide the necessary and conclusive evidence that LA tools help both teachers and students in their work. We have provided three different case studies from learning analytics evaluation where the respective researchers evaluated their LA tools with successful outcomes. Additionally, we presented a summary of the challenges that are yet to be resolved by the research community in order to do effective evaluation. Finally, we gave concrete directions which need to be investigated into details in order to help in overcoming the evaluation challenges. There is still a lot of work to be done in the direction of standardizing and structuring the evaluation of LA tools, and hence providing enough evidence that LA tools are assisting and valuable asset for the learning and teaching processes. However, the true test for learning analytics is demonstrating a longer term impact on student learning and teaching practices. [AP12] [Cl13] [DV12] [ED06]
Duval, E.; Verbert, K.: "Learning Analytics," Eleed, no. 8, 2012.
Ellis, G; and Dix, A.: "An explorative analysis of user evaluation studies in information visualisation," in Proceedings of the 2006 AVI workshop on BEyond time and errors: novel evaluation methods for information visualization, Venice, 2006.