Utilizing Gaze Data in Learning: From Reading Patterns Detection to Personalization Robert Moro, Maria Bielikova Slovak University of Technology in Bratislava, Faculty of Informatics and Information Technologies, Ilkovičova 2, 842 16 Bratislava, Slovakia name.surname@stuba.sk Abstract. Although a lot of attention has been dedicated towards improvement of the modeling of learners’ knowledge within learning systems, recommenda- tion, or personalization, there is less attention on improvement of the learning content itself and providing support to learning content creators. In addition, the complexity of learning systems requires utilization of novel sources of implicit feedback, such as gaze data in order to model learners’ interactions in their en- tirety. In this poster paper, we present a framework for collection of gaze data and its utilization in the learning systems environment. We focus on the analy- sis of reading patterns for the detection of problematic parts of text and present results of a preliminary evaluation in a web-based learning system ALEF. Keywords: learning system, personalization, implicit feedback, eye tracking, reading patterns, learning content comprehension, ALEF. 1 Challenges of Using Gaze Data in Learning Much of the work that students do when learning, whether they passively read learn- ing texts or actively solve exercises and assignments, has in recent years gradually shifted to web-based learning environments. These can provide additional support to the learners in the form of content or navigation personalization, which is based on learners’ goals and knowledge modeling, or even automatic assessment and analysis of their assignments (e.g. in the domain of learning programming [9]). The ever-increasing complexity of learning systems makes the analysis of learners’ behavior using solely logs of their actions harder and imprecise at best. For example, it is not atypical for learning systems to enrich their interfaces with different widgets and other elements that can be interacted with. Thus, a learner can work on e.g. a programming assignment and at the same time view the important concepts or hints, chat with his peers, etc. In order to understand and model these complex interactions, eye tracking can be used in combination with the classical action logging approach [7]. The detailed analysis of gaze data can help distinguish novice and advanced learners, or even help discover errors in their assignments [5]. However, not only learners, but also learning content creators are in need of help and support. They could benefit from the information, how the learning texts are used and read, which parts are easy to understand and which need clarification or rewriting. This can be also achieved by the analysis of gaze data; the existing body of work found useful to focus on distinguishing between at least two forms: reading and scan- ning (or skimming) the text [6, 3]. Less focus is given on detecting which parts of the text are hard to comprehend for the learners [2]. This would be especially useful to the learning content creators, because it has potential to help them to improve the learning texts and in turn, improve learning experience of the learners. Although the eye tracking hardware is becoming more and more affordable, the challenges associated with the utilization of the gaze data remain, namely: ─ Limited support of the dynamic web content, which is now prevalent in the modern web-based learning systems, but complicates analysis of the learners’ fixations on the areas of interests. The existing studies often limit their analysis only to the stat- ic content which does not require scrolling and does not change with time. ─ Limited support of reading patterns detection which focuses mainly on the reading vs. scanning, but does not identify the reading comprehensibility problems; identi- fication of these problems is crucial for improvement of the learning content. 2 Framework for Collection and Utilization of Gaze Data In order to address the presented challenges we have developed a framework for col- lection of the gaze data extending the infrastructure described in [8]. It consists of Gaze Monitor, a desktop client that communicates with the eye tracker and sends the gaze data to the server as well as to the web browser where the gaze data are pro- cessed in real-time by a reusable JavaScript component. The component provides: ─ Enrichment of the gaze data by the DOM element of the web page that corresponds to the gaze coordinates; the element is identified by a unique XPath string which enables to unambiguously determine whether the gaze coordinates fall within any of the defined areas of interest (AOIs). The fact that the user looks at a certain web page element triggers a gaze event that can be processed in the real-time. ─ A communication with the Gaze API that enables to retrieve the AOIs defined for a web page (or application), as well as pre-defined gaze events that should be trig- gered if the preconditions are met, such as when the user does not fixate on the specified area of interest over a given period of time. It also enables the users to re- trieve the gaze-based statistics for the areas of interest, e.g. the number of fixations, o dwell time, and most importantly, the identified reading patterns for the AOIs. The main advantage of our framework is that by providing a reusable JavaScript component, it can easily extend any web-based application (learning system) so that it can benefit from the eye tracking. For example, based on a more precise estimation of a student’s active learning time by gaze analysis, we can improve the estimation of the knowledge level related to the particular concepts presented in the learning object. In addition, since the areas of interest are defined as elements of the web page, they are robust to changes of their position or size; we can easily aggregate the gaze data from different web pages of the same web-based system (e.g. navigation menu or widgets that are usually present in all the pages of the system). 3 Learning Content Comprehension Detection Using our framework, we have extended our web-based learning system ALEF [4] with the gaze data collection. We propose a method of reading patterns detection based on [6]. Our modification of the original method in [6] allows to detect not only reading vs. scanning patterns, but also reading comprehension problems. The comprehension problems present a subset of the reading pattern identified by the original method. We label a sequence of fixations (classified by the Tobii I-VT fixation filter [1]) as problematic to comprehend if the number of revisits (recurrent saccades) of the given area is above the threshold value. We also consider the average length of the fixations which is higher in case of a comprehensibility problem. We are interested not only in identifying the individual patterns on the per-user ba- sis which can be used for improvement of the user modeling within the learning sys- tem, but also in providing an aggregate value of area comprehensibility. It serves as a feedback to the learning content creator, suggesting which parts of text need rewriting or rephrasing, thus leading to the improvement of the quality of the learning materials. The proposed visualization of the text comprehensibility can be seen in Fig. 1. Fig. 1. Screenshot of ALEF with the proposed comprehensibility visualization; paragraphs 1 and 4 have no comprehensibility problems, 3 is problematic to some extent and paragraph 4 is the hardest to comprehend. Each paragraph (area of interest) has its font colored with a shade of gray based on the normalized comprehensibility value from the <0; 1> interval, where 0 means no problems with comprehensibility and 1 means very problematic to comprehend. Thus, the most problematic areas of text immediately stand out from the remaining parts. In order to determine the optimal parameters of the proposed algorithm, we started collection of a reading dataset using Tobii TX300 eye tracker. Up to now we have gathered the data from four participants. Each of them read four texts with different tasks that included reading the text thoroughly, scanning and skimming the text to find the required information, omitting fragments of the text or simulating compre- hensibility problems by re-reading parts of the text. Each of the four texts was divided into multiple areas of interest where each paragraph of text corresponded to one area. Additionally, during the post-processing, the recordings were segmented and annotat- ed with one of the following labels: reading, skimming, scanning, and re-reading. As a result, each fixation is assigned one of these labels together with the information whether it falls into any of the areas of interest. We continue to gather a larger dataset. For this purpose, we plan to utilize our UX Lab that is equipped with 20 Tobii X2-60 eye trackers which allow the parallel collection of the gaze data. Acknowledgement. This work was partially supported by grants No. VG 1/0646/15, No. KEGA 009STU-4/2014, and it was created with the support of the Research and Development Operational Programme for the project “University Science Park of STU Bratislava”, ITMS 26240220084, co-funded by the European Regional Devel- opment Fund. References 1. Olsen, A.: The Tobii I-VT Fixation Filter: Algorithm description. (2012). Available online at http://www.tobii.com/eye-tracking-research/global/library/white-papers/the-tobii-i-vt-fix ation-filter/. Cited 6th April 2015. 2. Biedert, R., Dengel, A., Elshamy, M., Buscher, G.: Towards Robust Gaze-Based Objective Quality Measures for Text. In: ETRA '12: Proc. of the Symposium on Eye Tracking Re- search and Applications, pp. 201–204. ACM, New York (2012) 3. Biedert, R., Hees, J., Dengel, A., Buscher, G.: A Robust Realtime Reading-Skimming Classifier. In: ETRA '12: Proc. of the Symposium on Eye Tracking Research and Applica- tions, pp. 123–130. ACM, New York (2012) 4. Bieliková, M. et al. ALEF: From Application to Platform for Adaptive Collaborative Learning. In: Manouselis, N. et al. (eds.): Recommender Systems for Technology En- hanced Learning, pp. 195-225, Springer (2014) 5. Busjahn, T. et al.: Eye Tracking in Computing Education. In: Proc. of the 10th Annual Conf. on Int. Computing Education Research, pp. 3–10. ACM, New York (2014) 6. Campbell, C.S., Maglio, P.P.: A Robust Algorithm for Reading Detection. In: PUI '01: Proc. of the Workshop on Perceptive User Interfaces, pp. 1–7. ACM, New York (2001) 7. Kardan, S., Conati, C.: Comparing and Combining Eye Gaze and Interface Actions for De- termining User Learning with an Interactive Simulation. In: UMAP '13: Proc. of the 21th Int. Conf. on User Modeling, Adaptation, and Personalization, LNCS 7899, pp. 215–227. Springer (2013) 8. Móro, R., Daráž, J., Bieliková, M.: Visualization of Gaze Tracking Data for UX Testing on the Web. In: Hypertext '14 Extended Proc. of the 25th ACM Hypertext and Social Me- dia Conference, vol. 1210. CEUR-WS (2014) 9. Navrat, P., Tvarozek, J.: Online Programming Exercises for Summative Assessment in University courses. In: CompSysTech '14: Proc. of the 15th Int. Conf. on Computer Sys- tems and Technologies, pp. 341–348. ACM, New York (2014)