On Using Learning Analytics to Track the Activity of Interactive MOOC Videos Josef Wachtler, Mohammad Khalil, Behnam Taraghi, and Martin Ebner Educational Technology Graz University of Technology Münzgrabenstraße 35A - 8010 Graz - Austria josef.wachtler@tugraz.at Abstract It is widely known that interaction, as well as communication, are very important parts of successful online courses. These features are considered crucial because they help to improve students’ attention in a very significant way. In this publication, the authors present an innova- tive application, which adds different forms of interactivity to learning videos within MOOCs such as multiple-choice questions or the possibil- ity to communicate with the teacher. Furthermore, Learning Analytics using exploratory examination and visualizations have been applied to unveil learners’ patterns and behaviors as well as investigate the effec- tiveness of the application. Based upon the quantitative and qualitative observations, our study determined common practices behind dropping out using videos indicator and suggested enhancements to increase the performance of the application as well as learners’ attention. 1 Introduction It is a common knowledge that interaction, as well as the communication, are very important influencing factors of students’ attention. This indicates that different possibilities of interaction should be offered at a MOOC 1 in all possible directions. So it is vital to offer some communication channels like e-mail or a discussion forum, and in addition it is suggested that a form of interaction with the content of the course itself is available [9]. [1] The attention is considered as the most crucial resource for human learning [5]. Due to that, it is from high importance to understand and to analyze this factor. The results of such an analysis should be used to further improve the different methods of attention enhancing [6]. Moreover, learning analytics plays a major factor into enhancing learning environments components such as the video indicator of MOOCs and finally acts into reflecting and benchmarking the whole learning process [7]. In this publication, the usage of a web-based information system which provides the possibility to enrich the videos of a MOOC with different forms of interactivity (see Section 3) is presented. This paper covers an experiment on a MOOC named Making - Creative, digital creating with children 2 1 short for Massive Open Online Course 2 http://imoox.at/wbtmaster/startseite en/maker.html (last accessed Jannuary 2016) Copyright © 2016 for the individual papers by the papers' authors. Copying permitted only for private and academic purposes. This volume is published and copyrighted by its editors. SE@VBL 2016 workshop at LAK’16, April 26, 2016, Edinburgh, Scotland 8 and is attended by both, school-teachers as well as people who educate children in non-school settings. It is scheduled in seven weeks with at least one video per week. A detailed analysis of the activity of the attendees at the videos is presented by Section 4. Finally, this work aims to show how learning analytics could be done to monitor the activity of the students within videos of a MOOC. In other words, the research goal of this publication could be summarized to ”using an interactive video platform to support students’ attention and to analyze their participation”. 2 Related work In comparison to the approach shown by Section 3 there are other services pro- viding similar features. First, there is the possibility to use the built-in features of Youtube3 (e.g. text annotations or polls) itself. However, the different methods of analysis are very limited. A further tool is Zaption4 which provides various forms of interactive content for videos (e.g. multiple-choice questions) at planned positions as well as a rich set of analysis possibilities. Unfortunately, it shows the position of the interactions in the timeline of the video. This means that the users are able to jump from interaction to interaction without really watching the video. In comparison to that, a tool named EdTed5 also offers the possibility to enrich a video with questions. However, the questions are not bound to a position in the video and furthermore, they could be accessed every time during the video. The real-world pendant of interactive learning videos is known as ARS 6 , which enables the lecturer to present questions to students during the lecture in a standard classroom situation [10] [4]. Based on that, it offers several possibil- ities of analysis. It is well-known that an ARS has the power to enhance both, students’ attention and participation [2]. This means that the addition of inter- activity to learning videos tries to generate similar benefits as those generated by an ARS. 3 Interactions in Learning Videos To provide interactive learning videos a web-based information system called LIVE 7 first introduced by [3] is developed. It offers the possibility to embed different forms of interaction in videos (e.g. multiple-choice questions). As indi- cated above (see Section 1), the main purpose of these interactions is to support the attention of the students. The functionalities of LIVE could be categorized by the tasks of three different types of users. 3 https://www.youtube.com/ (last accessed Jannuary 2016) 4 http://www.zaption.com/ (last accessed Jannuary 2016) 5 http://ed.ted.com/ (last accessed Jannuary 2016) 6 short for Audience-Response-System 7 short for LIVE Interaction in Virtual learning Environments 9 The first ones are normal users who could be seen as students. They are only allowed to watch the videos and to participate to the interactions. Figure 1 shows a screenshot of a playing video which is currently paused and overlaid by an interaction (1). To resume playing, it is required to respond to the interac- tion which means that the displayed multiple-choice question has to be answered in this example. Furthermore, it can be seen that there are some other control elements on the right side of the videos (2). They could be used to invoke in- teractions manually. For instance, it is possible to ask a question to the teacher. [12] Figure 1. A Screenshot of LIVE shows a video interrupted by a multiple-choice ques- tion. In comparison to that, the users of the second group are equipped with teacher privileges. They are additionally able to embed interactions in the videos as well as to view different forms of analysis. To add an interaction, the teacher has to select its position within the video by using a preview of it or by entering the position. With a dialog, the teacher can embed multiple-choice questions or text-based questions in the video and furthermore, it is possible to add an image to a question. [12] The analysis consists of several parts. At first, there is a list of all students who watched the video and for each student in this list it is shown how much of the videos they watched. In addition, a chart shows the number of users (green) and views (red) across the timeline of the video (see Figure 2). This chart could be used to identify the most interesting part of the video. Furthermore, it is possible to access a detailed analysis of each student. It shows the timeline of the video and marks each watched part of it with a bar (see Figure 3). If such a bar is hovered with the mouse-pointer, additional information is displayed. This consists of the time of the joining and the leaving of this watched timespan in relative as well as the absolute values. [12] 10 Figure 2. A Screenshot of the timeline analysis. Figure 3. A Screenshot of the watched-parts analysis. In comparison to these forms of analysis related to the watching of the stu- dents, there is also a detailed statistic about the answers to the embedded ques- tions. This means that for the multiple-choice questions, the answers of the students as well as their correctness is displayed. For the text-based questions, LIVE displays answers of the students and the teacher has to evaluate them manually because text-based answers are impossible to check automatically. [12] The third group of users are researchers. They are able to download different forms of analysis as raw data. This means that they can select a video and obtain the data as a spreadsheet (CSV 8 ). Finally, the following list aims to give a summarizing overview of the features of LIVE [3] [12]: – only available for registered and authenticated users – different methods of interaction • automatically asked questions and captchas9 • asking questions to the lecturer by the learners • asking text-based questions to the attendees live or at pre-defined posi- tions • multiple-choice questions at pre-defined positions • reporting technical problems – different possibilities of analysis [11] • a detailed logging of the watched time-spans to point out at which time a user watched which part of the video 8 short for Comma-Separated Values 9 short for Completely Automated Public Turing Test to Tell Computers and Humans Apart 11 • a calculation of an attention level to measure the attention of the stu- dents – raw data download for researchers 4 Evaluation This section presents a very detailed analysis of the videos of the MOOC as well as of the multiple-choice questions. For that, the data provided by LIVE is evaluated using visualizations and exploratory analysis. First, the delay of response to the questions provided by LIVE in the MOOC videos during the seven weeks is demonstrated by two figures. Figure 4 visualizes a box plot. The x-axis records MOOC videos during the period of the course, while the y-axis shows students’ delay of response in seconds. This period was limited to 60 seconds. Students are categorized to certified students, who finished the course successfully and applied for a certificate, and non-certified students. In this figure, we tried to study the difference in behavior between both categories. In some of the weeks, certified students took more time to answer the questions such as in week 4 and week 7. For instance, certified students median in week 4 was 15 seconds, while the median for the non-certified students was 13 seconds. Furthermore, there was 3 seconds difference in the median between certified and non-certified students in week 7. Additionally, the median in week 1 and week 5 are typically the same with an insignificant variation between the first and the third quartiles. Figure 4. A box plot showing the reaction delays to multiple-choice questions. In comparison to that, Figure 5 visualizes a violin plot. The x-axis indicates students’ status. This visualization summarizes the students’ status and the 12 delay of response time to the multiple-choice questions in all of the MOOC videos. The thickness of the blue violin shape is slightly wider than the red one in the (8-13) seconds range, which indicates the more time needed to answer the questions. In addition to that, the non-certified violin shape holds more outliers attributes than the certified division. It is believed from the previous two observations, that certified students took less time in answering the questions in general. This case can be explained as the questions were easy to answer if the student were paying enough attention to the video lectures. Figure 5. A violin plot summarizes the reaction delays to multiple-choice questions. Figure 6 displays the timespan division in percentage and the timing of the first multiple-choice question represented as a vertical dashed line. Using this visualization, we can infer the relevance timing of the first question to describe the drop rate during videos. The questions were programmed to pop up after 5% of any MOOC video. Students may watch the first few seconds and make skips or drop out after that [13], and this can be seen in the plot where students are dropping in the early 15% of the videos. To grab the attention of the students and maintain a wise attrition rate, the multiple-choice questions were intended to be shown randomly in the high drop rate scope. Further, week 6 was tested to check the postponed question effect on the retention rate. The data in the figure also shows that students do not drop out a learning video in the range between 20%-80%, unless they replay it on that period and spend time on a particular segment to understand a complex content. The promising outcomes are seen with a stable attrition rate in the last four weeks when students are offered an interactive content during the video indoctrinate process. In Figure 7, the data is displayed in order to trace the video drop ratio of each second in every video. The x-axis displays the percentage of videos. The 13 Figure 6. Timespan division in percentage and the timing of the first multiple-choice question. Figure 7. Trace of the video drop ratio of each second in every video. 14 colored points specify the video name and the watchers count. While the black shadowed points indicate number of views. For instance, it is obvious that the data in the first three weeks shows more views per user which can be explained as an initial interest of the first online course weeks. On the other hand, the views nearly equaled the number of users from week 4 to the last week. Another interesting observation is the slow drop rate during the videos in all of the weeks despite the high drop in the last 2-3% of every video. A clarification of such attitude is due to the closing trailer of every video which most students jump over it. Due to the independency of the examined MOOC, each video of this course does not rely on the previous one. The activity of every video varies in every week. For this reason, Figure 8 shows activity of the total number of stop and play actions in the MOOC videos. The blue points denote the certified students while the orange ones denote the non-certified students. In fact, the first three weeks reflect proper enthusiastic count of actions. We realized that there was a distinct activity by the non-certified students in week 5. A reasonable clarifica- tion is because of the interesting topic of that week which was about 3D-Printing. However, their engagement becomes much less in the last two weeks, as this was proven in other MOOCs’ videos analysis [8]. Figure 8. The activity of the total number of stops and plays in the MOOC videos. 15 5 Conclusion Catching the attention of learners in online videos of MOOCs is an intriguing argument across learning analytics discussions. With this publication, the usage of an interactive video platform presenting videos of a MOOC is shown. It points out the main functionalities of this platform as well as the participation and the activity of the students. Additionally, we demonstrated an evaluation of this system in order to examine its performance and describe the behavior of students. Finally, the results show that the main concept behind latching on the students’ attention becomes attainable through evaluating the questions’ content and the interactions timing. References 1. Carr-Chellman, A., Duchastel, P.: The ideal online course. British Journal of Educational Technology 31(3), 229–241 (2000), http://dx.doi.org/10.1111/ 1467-8535.00154 2. Ebner, M.: Introducing live microblogging: how single presentations can be en- hanced by the mass. Journal of research in innovative teaching 2(1), 91–100 (2009) 3. Ebner, M., Wachtler, J., Holzinger, A.: Introducing an information system for successful support of selective attention in online courses. In: Universal Access in Human-Computer Interaction. Applications and Services for Quality of Life, pp. 153–162. Springer (2013) 4. Haintz, C., Pichler, K., Ebner, M.: Developing a web-based question-driven audi- ence response system supporting byod. J. UCS 20(1), 39–56 (2014) 5. Heinze, H.J., Mangun, G.R., Burchert, W., Hinrichs, H., Scholz, M., Münte, T.F., Gös, A., Scherg, M., Johannes, S., Hundeshagen, H., Gazzaniga, M.S., Hillyard, S.A.: Combined spatial and temporal imaging of brain activity during visual se- lective attention in humans. Nature 372, 543–546 (Dec 1994) 6. Helmerich, J., Scherer, J.: Interaktion zwischen lehrenden und lernenden in medien unterstützten veranstaltungen. In: Breitner, M.H., Bruns, B., Lehner, F. (eds.) Neue Trends im E-Learning, pp. 197–210. Physica-Verlag HD (2007) 7. Khalil, M., Ebner, M.: Learning analytics: principles and constraints. In: Proceed- ings of World Conference on Educational Multimedia, Hypermedia and Telecom- munications. pp. 1326–1336 (2015) 8. Khalil, M., Ebner, M.: What can massive open online course (mooc) stakeholders learn from learning analytics? Learning, Design, and Technology. An International Compendium of Theory, Research, Practice, and Policy. Springer. Accepted, in print. (2016) 9. Lackner, E., Ebner, M., Khalil, M.: Moocs as granular systems: design patterns to foster participant activity. eLearning Papers 42, 28–37 (2015) 10. Tobin, B.: Audience response systems, stanford university school of medicine. http://med.stanford.edu/irt/edtech/contacts/documents/2005-11_AAMC_ tobin_audience_response_systems.pdf (2005), [Online; accessed 2012-10-09] 11. Wachtler, J., Ebner, M.: Attention profiling algorithm for video-based lectures. In: Learning and Collaboration Technologies. Designing and Developing Novel Learn- ing Experiences, pp. 358–367. Springer (2014) 16 12. Wachtler, J., Ebner, M.: Support of video-based lectures with interactions- implementation of a first prototype. In: World Conference on Educational Mul- timedia, Hypermedia and Telecommunications. vol. 2014, pp. 582–591 (2014) 13. Wachtler, J., Ebner, M.: Impacts of interactions in learning-videos: A subjective and objective analysis. In: EdMedia: World Conference on Educational Media and Technology. vol. 2015, pp. 1642–1650 (2015) 17