This article provides a systematic literature review on Learning Analytics methods and applications for video-based learning. For that purpose 33 research articles have been analyzed and described regarding aspects of capturing, measuring, visualizing data that represent user behavior and learning activities. The advent of video in online and blended learning started at the turn of the century and became more and more popular as lecture recording, how-to-videos and screen casts could be easily produced and distributed. Since 2012 video obtained a wide echo in Massive Open Online Courses (MOOCs). Because videos are mainly used in online distance learning teachers can not observe the user behavior, resource usage, and learning activities in a directly manner. Instead methods from Learning Analytics, Educational Data Mining, and Video Usage Mining [MBD06] are required to track and analyze the user activities. This paper ofers a systematic literature review of the state of research in the field that could be summarized as video analytics. Since this is a work in progress paper, the review focuses only on three research question (RQ) concerning data gathering, measurements and visualizations, rather than providing a complete overview on video analytics: RQ1: What data needs to be captured form video players and learning environments in order to perform analytics? RQ2: What measures can be derived from the captured data? RQ3: What data representation are suitable to support visual analytics.
The literature review was conducted in a four-step process: i) search in the selected academic databases by using the proposed search terms; ii) selecting relevant articles from the title and abstract of the search results; iii) identify further articles that were referenced in the selected articles; vi) paper review by following the guidelines of [Ch16].For the review we selected seven academic databases for articles related to technology enhanced learning: ACM Digital Library, IEEE Xplore, SpringerLink, Science Direct, Taylor & Fransis, dblp, and Wiley. Additionally, we queried Google Scholar, Research Gate, and Mendeley in order to embrace potentially relevant “gray literature” such as technical reports or position papers. To perform the search we used combinations of two sets of search terms: i) video, audiovisual media, electure, lecture recording, and ii) analytics, data, user behavior, usage, mining, watch*, click, log. Overall 93 publications were gathered from search and the recognized references. After getting an overview of the field 44 relevant articles could be identified for deeper analysis. Most of these articles were retrieved from the ACM Digital Library. The publication dates range from 1994 to 2017, whereas 14 were published in the year 2014. 14 articles covered analytics about MOOCs. The same amount of articles described studies in a university setting. The remaining papers covered technological or methodological aspects as well as experiments that were not directly related with educational technology. 3 3.1
Analytics about video-based learning mainly gather data form log files. In terms of videobased learning environments dedicated logs from the video player are required to capture the entire user interactions. Currently there is a lack in standardization of log formats and data structures. Only a minority of platform providers published their log format (e.g. edX data API), while existing drafts (e.g. the videoprofiles for the xAPI) have not been recognized by the community yet.
Watching: On of the core question in video analytics is the way of approximating the user’s time spent on watching a video. Whereas modern web application make use of accurate Javascript timeupdate events, some systems still lack the possibility to gather ifne-grained second-by-second data. However, captured playback activity does not imply user engagement. Therefore, playback measures need to be compared with clickstream data in order to ensure minimal engagement indicators. Furthermore, the time that a user needs to watch a particular part of the video also depends on the playback speed [Li15b, GKR14]. As a consequence the length of a video depends on the users’ watching habits.Table 1 summarizes diferent approaches for approximating playback durations by amplifying varying granularities.
Although imprecise measurements may limit statistical inferences, privacy concerns should urged as an important argument. [HGM14, Br11] outlined how the principle of data economy could be applied by factoring playback traces as well as click events into binary data. Video: Learning videos can not be considered as a homogeneous type of media. Today we ifnd various technical representations, formats, and styles. A few researchers focused on Timeupdate Segments Clickstream Heartbeat Section visits Videos assessed Video visits
Tab. 1: Methods for approximating playback duration This HTML5 event is fired when the playing position of a video has changed.
It returns the current position in milliseconds.
Split the video in segments of equal size and write a log as soon the user completes playback of the segment. [KE16b] define segments of 120 seconds, while in most cases fine-grained segments of one second are used (e.g. [ MKB10, Ki14b, Si14, Ki14a]).
Approximates playback duration by comparing time diferences of physical time and playback time of subsequent click events (e.g. pause or timeline navigation) [Se14].
Request the play head position in regular periods of time to approximate the watched segments [Br11, BTG13].
Number of times a specified content section has been visited. The extent of a section can be derived from a table of content [KE16b], quiz [WL15] or the temporal boundaries of presentation slides synchronized to the video [MKB10] Total number of assessed videos [CdBB17] The number of times a video has been accessed poses as loose estimation [KE16b, HGM14, KE16a, LW10, BTG13, BLGS17]. these particularities by using video, audio, and text properties as indicators for analytical investigations (see Tab. 2).
Length Visual transitions Speaking rate Speech Audio Type of video Production style
User: Surprisingly demographic informations about the students did not play a large role in the past studies on video analytics. Only [GR14] related video coverage and inter-video navigation to demographic data (age, country of origin). Possible relations between video usage and demographic factors remain an open research question. [RM02] used clickstream data as depended variable to determine relations to the users’ personality types. Beside that, learners can be further involved by participating in surveys during or after watching the videos. [dBT08] tried to confirm findings from log analysis by esquiring students about their viewing patterns (e.g. ”one-pass”, ”zapping”). [SMP01] requested the intentions for browsing and watching (e.g. ”looking for something”, ”aimless browse”) at random times during playback.
Other: Except the research about MOOCs the majority of the studies are based on a small number of participants. As a consequence particular statistical methods can not be applied or will not return to significant results. Thus, stochastically generated data may be an suitable alternative or addition to real log files. Methods for modeling user behaviors including clickstreams and video playback on basis of existing data are well established. [SMP01] used hidden Markov models of user behavior to generate video previews, whereas [MBD05, Mo07] identified clusters from user behavior data that were modeled by non-hidden Markov models. Similar approaches have been used for making predictions about in-video and course drop outs [HGM14]. 3.2
Measurements are built upon the captured data that was described in the previous subsection. Regarding video-based learning measurements can be classified in three categories: i) video watching behavior, ii) video interactions, and iii) other user input considered as learning results.
Video watching behavior: Analyzing the users’ behavior when watching the video can provide insight regarding in-video drop out rates [Li15a, Ki14c, BLGS17] and most frequent watched segments. [KE16b] even demonstrated on how to derive playback events from the timeupdate events. Table 3 provides an overview on common indicators that can help to describe video usage behaviors.
Viewing duration Replay segments Total watching time Watching ratio Watching threshold Retention rate Coverage Session length Average session length Number of sessions Session views Length threshold Video interactions: Gathering clickstream data is essential for analyzing, modeling and predicting video interactions. Typically the frequency (total and per segment) and the duration of clickstream events is used (see Tab. 4) to perform various analyses. The majority of researchers focus on in-video interactions, rather than inter-video interactions (see [GR14, HGM14, Br11]). The latter consider browsing behavior between multiple videos in a course or database. The analysis of clickstream data has diferent purposes. Basically, the event total events play pause volume full screen show captions speed changes slow forward slow reverse fast forward fast rewind seeks seek forward seek backward seek from seek to [Li15b, KE16b, Si14, GC14] [Li15b, KE16b, Si14, GC14, SJD15] [Dí15] [Dí15] events are used to identify access patterns across courses [HGM14], week days or hours of the day [Br11, Se14]. Considering in-video interactions [Ki14c] identified and analyzed peaks of both frequently watched scenes and the used playback controls during that scenes. So fare the properties of the peaks (width, height, area) have been analyzed and related to possible explanations (e.g. visual transitions or returning to missed content) [Ki14c]. [Ki14a] is looking forward to classify peaks automatically considering visual transitions, speech properties and topic transitions derived from the video transcripts. [SJD15] found significant lower pause and seek back rates when the teacher gaze augmented to the video. [Si14] determined clickstream profiles of students in-order to predict engagement states as well as in-video and course drop outs. [MBD06, dBT08, Ch17, CdBB17], and [Li15b] found diferent in-video viewing and interaction patterns. [ Li15a] demonstrated how the perceived dificulty of the video content correlates with some of the determined video interaction patterns.
Learning results: Learning results in a broader sense include student contributions such as answers to quizzes, forum or wiki entries as well as annotations. These contributions may either being entered during the video playback or separate from the video. [Li15b] distinguished strong from weak students by comparing correct answers in relation to the number of attempts made to pass an assignments. The interaction patterns of strong students significantly difered from the students considered as weak. [ MD13] found correlations between quiz scores and watched portions of lecture recordings, not least because the correct answers were told in the corresponding parts of the video. [KE16b] found significant correlations between the quiz attempts as well as results and the watched video segments. [GMD14] applied quantitative text analysis to evaluate large amounts of video annotations. Measuring word counts related to linguistic and psychometric processes aims to reduce the time for reading and scoring submissions. 3.3
Data charts are essential for visual analytics tasks. Efective visualization are primarily determined by the selected dimensions, rather than the type of chart. Going into detail about the various forms of data visualization would go beyond the scope of this article, but should be considered as a future research direction. The same is true for learning dashboards. According to [Sc16] a “learning dashboard is a single display that aggregates diferent indicators about learner(s), learning process(es) and/or learning context(s) into one or multiple visualizations.” The latest review articles on learning dashboards did not go into detail about visualizations or dashboards representing video-based learning activities [Sc16, Ve14]. Particular dashboards for MOOC instructors or students as reported by [Fr16, Vi17, KE16a] stay on the surface by explaining selected data charts instead of providing a complete overview. Some insights could be gained from edX. However, the advances in visual analytics in terms of data visualization like rewatching graphs [BTG13], forward-backward diagrams [Se14], or interaction peaks [Ki14c] have not been transfered to dashboards yet. Learner-centered social navigation aids along the player timeline are known for many years now [MKB10, Ki14c, WL15, Ch16], but have not spread beyond research prototypes. Potential data representations for visual analytics tasks could be identified in the works of [GKR14, Li15a, HGM14, BET99, LW10, Br13, De14, Co14]. 4
This literature review presented the foundations, current state and potentials of video analytics. However, the review should be enlarged upon efective visualizations for video learning dashboards. Furthermore, the analysis methods that were stated in the literature are worth to be compared considering the available data. The method set ranges from statistics over sequence mining to natural language processing.