Understanding Student Attention to Adaptive Hints with Eye-Tracking

Abstract

Prime Climb is an educational game that provides individualized support for learning number factorization skills. This support is delivered by a pedagogical agent in the form of hints based on a model of student learning. Previous studies with Prime Climb indicated that students may not always be paying attentions to the hints, even when they are justified. In this paper we discuss preliminary work on using eye tracking data on user attention patterns to better understand if and how students process the agent's personalized hints, with the long term goal of making hint delivery more effective.

Keywords

• Adaptive help
• educational games
• pedagogical agents
• eye-tracking

1 Introduction

Educational games (edu-games) are one of the most promising media for the development of innovative computer-based pedagogy, however, while there is ample evidence that edu-games are highly engaging, there is less direct support for evidentiary claims about what is learned through play [e.g. 1, 2]. We believe that edu-games effectiveness can be improved by making them more adaptive to the specific needs of individual students, and we are doing so by devising intelligent pedagogical agents that can provide individualized support to student learning during game playing [ 3 ]. Providing this support is challenging because it requires a trade-off between fostering learning and maintaining engagement. Our long-term goal is to enable our agents to achieve this trade-off by relying on models of both student learning and affect [ 3 ]. In this paper, we focus on an issue that has been raised in the context of various user-adaptive learning environments: are interactive, personalized didactic hints effective? Do students pay attention to them [e.g. 11]? We investigate this issue in relation to the user-adaptive hints provided by the pedagogical agent in Prime Climb, an edu-game for number factorization. The current agent’s version provides hints based on a model of student learning [ 3 ]. A previous study showed that the adaptive version of Prime Climb did not perform better than a version with random hints, and provided initial indications that one reason for this outcome is student limited attention to the agent’s adaptive hints. In that study, attention was estimated from how long students had the hints open on the screen. In this paper, we start looking at a more accurate measure of attention, based on eye-tracking data. We present preliminary results from the analysis of one student’s interaction with Prime Climb, as a proof of concept for this methodology.

User-adaptive educational games are receiving increasing attention [e.g. 4, 5] although most of the existing work has not been formally evaluated in terms of how adaptive game components contribute to learning. There has also been increasing interest in using eye-tracking to gain insights on the cognitive and perceptual processes underlying a user’s performance with an interactive system [ 6, 11 ]. In this paper, we contribute to this line of research by using gaze information to understand if/how users attend to a system’s adaptive interventions. Adaptive incremental hints are commonly used in personalized learning environments, but their effectiveness is in question because there are students who ignore them, or use them to extract quick solutions from the system [ 8, 11 ]. Researchers have proposed predictive models of hint processing based on reaction-time (lapsed time between the hint being displayed and the next observable student action) [ 8, 9 ]. Despite encouraging results, these models cannot capture the details of the student’s cognitive reactions to a hint because these are unobservable when using only reaction time. We investigate how to uncover these details by relying on attention patterns captured via eye-tracking. In the rest of the paper, we first describe the Prime Climb edu-game and its personalized agent. We then provide an example of attention analysis and the insights that it can provide. 2

The Prime Climb Game

model predicts that the student doesn’t know how to factorize one of the numbers involved in the current move (regardless of move correctness). The agent starts by reminding the student to evaluate her move in term of number factorization, then it generates a tool hint that encourages the student to use the magnifying glass to see relevant factorizations. If the student needs further help, the agent gives definition hints designed to re-teach what is a factor via explanations and generic examples. There are two different factorization definitions (“Factors are numbers that divide evenly into the number”, “Factors are numbers that multiply to give the number”). The agent alternates which definition to give first, and gives the second the next time it needs to provide a hint. The generic examples that accompany the definitions change for every hint. Finally, the agent provides a bottom-out hint giving the factorization of the two numbers involved in the current move. Students can choose to progress through the various levels by asking. Otherwise, the agent goes through the progression as the student model calls for a new hint. A hint is displayed until the student selects to resume playing or to access the next hint level, if available.

3 Sample gaze analysis

Previous studies with Prime Climb suggested that students may often ignore agent’s hints, even when these hints are well justified (i.e. based on a reliable student model’s assessment) [ 3 ]. Those results were based on hint display time (duration of time a hint stays open on the screen) as a rough indication of attention. However, display time can be unreliable because students may not attend a displayed hint, or be fast readers and thus processing a hint even when display time seems short. For a more precise analysis, we are using a Tobii T120 eye-tracker to capture students’ attention patterns. At the time of writing we have reliable data for only one subject, which we present as an example of the type of analysis that eye-tracking can support.

The agent’s adaptive hints can be divided into two categories: short hints, which are on average 8 words long and include tool or bottom-out hints; long hints, which are on average 25 words long and include all definition hints. The amount of time it would take an average-speed reader to read the text would be 2.3 seconds and 7.3 seconds for the short and long hint respectively. Table 1 shows mean and standard deviation of total fixation time (i.e. total time a student’s gaze rested on a displayed hint) for each hint type. These numbers show that, although this particular student spent more time looking at the longer hints (definition hints), the increase is not proportional to the increased hint length, and in fact there is no statistically significant difference between the reading time for these three hint types (as tested via ANOVA). Furthermore, fixation time is much shorter than the time an average-reader would need to read the hints. The high standard deviation on all three measures indicates a trend of selective attention. Figure 2 visualizes this trend by showing total fixation time on each individual hint, for each hint category. The x-axes show hint number in each category.

It is interesting to see that, for about the first half of the displayed definition hints, there is a pattern of attention being high for one hint, and low for the definition hint given as the next step in the hinting cycle. This pattern suggests that this student tends to ignore the second definition hint, possibly because two subsequent definition hints are perceived as redundant. Student attention then decreases substantially for all of the second half of definition hints provided. In contrast, attention to tool and bottom-out hints reaches its low in the middle of the interaction, but picks up again towards the end. A possible explanation for these trends is that definition hints become less useful overtime, as mountains get more difficult (i.e. include larger numbers), because the student is already familiar with the factorization definitions and the generic examples in the hint don’t help directly with the current moves. However, apparently the student still needs help dealing with the higher numbers, so she does read short hints when they appear and specifically attends to bottom-out hints because they provide the information needed to understand the outcome of the current move. We need of course to collect more data before drawing any firm conclusion. These trends, however, are consistent with previous indications that attention to some the Prime Climb hints can be scarce and start providing specific information on why and how the current hinting strategy needs to be revised to make it more effective. Further insights can be derived from a more detailed analysis of the attention patterns associated with specific hints, e.g. attention shifts between a hint and relevant places of the mountain (or lack thereof). Following the approaches proposed in [ 4 ] and [ 10 ], we plan to apply data mining techniques to discover patterns associated with learning/reasoning vs. confusion or distraction. In the long term, we want to use this information to add to the Prime Climb user model a classifier that can recognize these patterns in real time, and use the information to generate adaptive interventions geared at focusing student attention when needed.

References

[1] de Castell , S. & Jenson , J. ( 2007 ). Digital Games for Education: When Meanings Play . Intermedialities, 9 , 45 - 54

[2] Van Eck , R. , ( 2007 ). Building Artificially Intelligent Learning Games . Games and Simulations in Online Learning: Research and Development Frameworks . D. Gibson , C. Aldrich , and M. Prensky, Editors, Information Science Pub . 271 - 307 .

[3] Conati C and Manske M. ( 2009 ). Evaluating Adaptive Feedback in an Educational Computer Game , Proc. of IVA 2009 , 9th International Conference on Intelligent Virtual Agents, Lecture Notes in Artificial Intelligence 5773 . Springer Verlag, 146 - 158

[4] Peirce , N. , Conlan , O. and Wade , V. ( 2008 ). Adaptive Educational Games: Providing Non-invasive Personalised Learning Experiences . In Second IEEE International Conference on Digital Games and Intelligent Toys Based Education (DIGITEL 2008 ), Banff, Canada. 28 - 35

[5] Johnson , W. L. ( 2007 ). Serious use for a serious game on language learning . In Proc. of the 13th Int. Conf. on Artificial Intelligence in Education , Los Angeles, USA. 67 - 74

[6] Conati , C. and Merten , C. ( 2007 ): Eye-tracking for user modeling in exploratory learning environments: An empirical evaluation . Knowl.-Based Syst . 20 ( 6 ): 557 - 574

[7] Conati C. and Maclaren H. ( 2009 ). Modeling User Affect from Causes and Effects . Proceedings of UMAP 2009 , First and Seventeenth International Conference on User Modeling, Adaptation and Personalization , Springer. 4- 15

[8] de Baker , R.S.J. , Corbett , A.T, Roll, I. and Koedinger , K.R. ( 2008 ): Developing a generalizable detector of when students game the system . User Model. User-Adapt. Interact . 18 ( 3 ): 287 - 314

[9] Shih , B. , Koedinger , K.R. and Scheines , R. ( 2008 ) : A Response Time Model For BottomOut Hints as Worked Examples . EDM 2008 : 117 - 126

[10] Amershi S. and Conati C. ( 2009 ) Combining Unsupervised and Supervised Machine Learning to Build User Models for Exploratory Learning Environments . Journal of Educational Data Mining 1 ( 1 ), 18 - 71 .

[11] Roll , I. , Aleven , V. , McLaren , B.M , Ryu , E., de Baker , R.S.J , and Koedinger , K.R. ( 2006 ): The Help Tutor: Does Metacognitive Feedback Improve Students ' Help-Seeking Actions , Skills and Learning? Intelligent Tutoring Systems 2006 : 360 - 369 .

[12] Muldner K. , Christopherson R. , Atkinson R. , Burleson W. ( 2009 ) : Investigating the Utility of Eye-Tracking Information on Affect and Reasoning for User Modeling . UMAP 2009 : 138 - 149