The impact of feedback on students' a ective states

Abstract

A ective states play a signi cant role in students' learning behaviour. Positive a ective states can enhance learning, while negative a ective states can inhibit it. This paper describes a Wizard-of-Oz study that investigates the impact of di erent types of feedback on students' a ective states. Our results indicate the importance of providing feedback matched carefully to the a ective state of the students in order to help them transition into more positive states. For example when students were confused a ect boosts and speci c instructive feedback seem to be e ective in helping students to be in ow again. We discuss this and other ways to adapt the feedback, together with implications for the development of our system and the eld in general.

-

This paper reports the results of a set of two Wizard-of-Oz studies which explore the e ect of di erent feedback types on students' a ective states.

It is well understood by now that a ect interacts with and in uences the learning process [ 9, 6, 2 ]. While positive a ective states such as surprise, satisfaction or curiosity contribute towards constructive learning, negative ones including frustration or disillusionment at realising misconceptions can lead to challenges in learning. The learning process is indeed full of transitions between positive and negative a ective states and regulating those is important. For example, a student may seem interested in exploring a particular learning goal, however s/he might have some misconceptions and need to reconsider her/his knowledge. This can evoke frustration and/or disappointment. However, this negative a ective state may turn into deep engagement with the task again. D'Mello et al., for example, elaborate on how confusion is likely to promote learning under appropriate conditions [ 6 ].

It is important therefore, to deepen our understanding of the role of a ective states for learning, and to be able to move students out of states that inhibit learning. Pekrun [ 13 ] discusses achievement emotions or a ective states, which arise in a learning situation. Achievement emotions are states that are linked to learning, instruction, and achievement. We focus on a subset of a ective states identi ed by Pekrun: ow/enjoyment, surprise, frustration, and boredom. We also add confusion, which has been identi ed elsewhere as an important a ective state during learning [ 15 ] for tutor support and for learning in general [ 6 ].

As described in Woolf et al. [ 20 ] students can become overwhelmed (very confused or frustrated) during learning, which may increase cognitive load [ 19 ] for low-ability or novice students. However, appropriate feedback might help to overcome such problems. Carenini et al. [ 3 ] describe how e ective support or feedback needs to answer three main questions: (i) when the support should be provided during learning; (ii) what the support should contain; and (iii) how it should be presented.

In this paper we focus on the question of what the support should contain with respect to a ect i.e. the types of feedback that are able to induce a positive a ective state.

In related work students' a ective states have been used to tailor motivational feedback and learning material in order to enhance the learning experience. For example, Santos et al. [ 17 ] show that a ect as well as motivation and selfe cacy impact the e ectiveness of motivational feedback and recommendations. Additionally, Woolf et al. [ 20 ] developed an a ective pedagogical agent which is able to mirror a student's a ective state, or acknowledge a student's a ective state if it is negative. Another example is Conati & MacLaren [ 5 ], who developed a pedagogical agent to provide support according to the a ective state of the students and the user's personal goal. Also, Shen et al. [ 18 ] recommend learning material to the student based on their a ective state. D'Mello et al. [ 7 ] developed a system that is able to respond to students via a conversation that takes into account the a ective state of the student.

In contrast, in this paper, we investigate the impact of di erent types of feedback on students' a ective state and how and whether they can help students regulate their a ect and thus improve learning. In what follows we present two sets of Wizard-of-Oz studies where feedback was provided to students interacting with an exploratory learning environment designed to learn fractions. From these studies, the a ective states of the students were carefully annotated in order to address our research questions. 2 2.1

The Wizard-of-Oz studies

Aims

One of our research aims is to develop intelligent support that enhances the learning experience by taking into account the student's a ective state. We were speci cally interested in identifying how di erent feedback types modify a ective states.

In order to address this question we conducted two sets of ecologically valid Wizard-of-Oz studies (e.g. [ 11, 8 ]) which investigated the e ect of a ective states on di erent feedback types at di erent stages of the task. 2.2

Participants and Procedure

In total, 26 Year-5 (9 to 10-year old) students took part in the Wizard-of-Oz studies. Each session lasted on average 20 minutes. Each student participated in one Wizard-of-Oz session.

The sessions were run in an ordinary classroom with multiple computers, where additional children were working with the learning platform (not wizarded) in order to support ecological validity. This was important particularly as in early settings we identi ed that children would not speak that much to the platform if they felt that they were monitored [ 10 ]. Figure 1 shows the setup of the studies. Wizards followed a script with pre-canned messages to send messages to the students through the learning platform and deliberately limited their communication capacity in order to simulate the actual system. To achieve that wizards were only able to see students' screen. An assistant was able to hear students' reactions to re ections or talk-aloud prompts (as prompted by the `system') and provide recommendations to the wizard with respect to the detected a ective state. Any feedback provided was both shown on screen and read aloud by the system to students. 2.3

Feedback types

Di erent types of feedback were presented to students at di erent stages of their learning task. The feedback provided was based on interaction via keyboard and mouse, as well as speech.

We explore di erent types of feedback that are known from the literature to support students in their learning and t our context. The following di erent feedback types were provided: { AFFECT BOOSTS - a ect boosts. As described in [ 20 ] a ect boosts can help to enhance student's motivation in solving a particular learning task. These included prompts that acknowledged for example that a task is di cult or that the student may be confused but they should keep trying. { INSTRUCTIVE FEEDBACK - instructive task-dependent feedback. This feedback provided detailed instructions, what subtask or action to perform in order to solve the task. { OTHER PROBLEM SOLVING FEEDBACK - task-dependent feedback. This support was centred on helping students to solve a particular problem that they are facing during their interaction by providing either questions to challenge their thinking or speci c hints designed to help them identify the next step themselves. { TALK ALOUD PROMPTS - talking aloud. With respect to learning in particular, the hypothesis that automatic speech recognition (ASR) can facilitate learning is based mostly on educational research that has shown bene ts of verbalization for learning (e.g., [ 1 ]). { REFLECTIVE PROMPTS - re ecting on task performance and learning. Self-explanation can be viewed as a tool to address students' own misunderstandings [ 4 ] and as a `window' into students' thinking. { TALK MATHEMATICS PROMPTS - using particular domain speci c mathematics vocabulary. The aim of this prompt was to encourage students to use mathematical vocabulary in order continually revise their interpretations. In early studies [ 10 ] we found that students' re ections were often procedural and pragmatic (e.g. talking about the user interface) rather than mathematical. { TASK SEQUENCE PROMPTS - moving to the next task. This feedback is centred on providing support regarding what action to perform next in order to change the task, such as clicking the `Next' button.

Annotation of a ective states and feedback reactions

From the Wizard-of-Oz studies we recorded the students' screen display and their voices. From this data, we annotated a ective states (e.g. screen interaction and what the students said) before and after feedback was provided.

As described earlier, for the a ective state detection we discriminated between ve di erent a ective types: enjoyment, surprise, confusion, frustration, and boredom. For the annotation of those a ective states we used a similar strategy to that described in [ 15 ], where a dialogue between a teacher and a student was annotated retrospectively by categorising utterances in terms of di erent feedback types. Also, [ 2 ] describe how they coded di erent a ective states based on observations of students interacting with a learning environment. Similarly, we annotated student's a ective states for each type of feedback provided. In addition to the student's voice we also used the video of the screen capture to support the annotation process. Students' a ective states were annotated as follows: { FLOW: Engagement with the learning task. Statements like `I am enjoying this task' or `This is fun'. Sustained interaction with the system. Feedback type Example AFFECT BOOSTS You're working really hard! Keep going! INSTRUCTIVE FEEDBACK Use the comparison box to compare your fractions.

OTHER PROBLEM SOLVING FEEDBACK If you add fractions, they need to have the same denominators rst.

REFLECTIVE PROMPTS What do you notice about the two fractions? TALK ALOUD PROMPTS Remember to talk aloud, what are you thinking? TALK MATHEMATICS PROMPTS Can you explain that again using the terms denominator, numerator? TASK SEQUENCE PROMPTS Well done. When you are ready click `next' for the next task. { SURPRISE: Gasping. Statements like `Huh?' or `Oh, no!'. { CONFUSION: Failing to perform a particular task. Statements such as `I'm confused!' or `Why didn't it work?'. Uncertain interaction with the system. { FRUSTRATION: Tendency to give up, repeatedly clicking or deleting of objects in the system or repeatedly failing to perform a particular task, sighing, statements such as, `What's going on?!'. { BOREDOM: Inactivity or statements such as `Can we do something else?' or `This is boring'. 4

Results

In total 396 messages were sent to 26 students. The video data in combination with the sound les were analysed independently by three researchers (one was independent of the project) who categorised the a ective states of students before and after the feedback messages were provided.

The data is combined from two sets of Wizard-of-Oz studies. We use kappa statistics to measure the degree of the agreements of the annotations for reliability. Kappa was .46, p<.001. This is generally expected from retrospective annotation of naturalistic a ect experiences [ 14 ]. We consolidated the annotations based on discussion between the annotators and the rest of the authors of the paper in order to agree upon the annotations that did not match originally. In the second set we had resources to introduce the Baker-Rodrigo Observation Method Protocol (BROMP) and the HART mobile app that facilitates the coding of students a ective states in the classroom [ 12 ]. Kappa based on the retrospective annotation was still .56, p<.001. We rst consolidated the data with the same approach as before and then compared against the eld annotations. Kappa between the consolidated annotation and the HART data was .71, p<.05 (note that is may appear low but we did not expect the retrospective annotation to get surprise and frustration accurately). We used the HART data to improve the annotation by mapping feedback actions against the observation for 20 seconds prior to the delivery of the feedback to 20 seconds after the student had closed the corresponding feedback window. We marked the changes for an independent annotator to revisit the rst set of annotations.

The student's a ective states, that occurred before and after the di erent types of feedback was provided, can be seen in gure 2. Each block shows an a ective state before feedback was provided. The colour within the bars indicates the type of a ective states that occurred after the feedback was provided. The number within the bars indicate the number of times the a ective state occurred.

In order to investigate whether there was an e ect of the feedback on the learning experience, we looked at whether a student's a ective state was enhanced, stayed the same or worsened. An a ective state was enhanced for example, when it was changed from confusion to ow, or (given the ndings about confusion [ 6 ]) from frustration to confusion, frustration to ow, boredom to ow etc. An a ective state was worsened if it moved for example, from ow to frustration or confusion, or from confusion to frustration.

As the data is categorical [ 16 ], we apply chi-square tests to investigate statistical signi cant di erences between the groups. We present them below and discuss in more detail in the next section.

Flow When students were in ow, there was no signi cant di erence between the feedback types on whether the a ective state stayed in the same ow state (X2(6, N=169) = 4.31, p>.05) or worsened (X2(6, N=169) = 4.89, p>.05). As ow is the most positive a ective state, the a ective state in this sub-sample cannot be enhanced.

Confusion When students were confused, there was a signi cant e ect of the feedback type on whether students' a ective state was enhanced into a ow state (X2(6, N=181) = 13.65, p<.05). The most e ective feedback types were a ect boosts with 68% of the cases, followed by guidance feedback with 67%, and task sequence prompts with 63%. Re ective prompts resulted in a ow state in 48% of the cases, talk aloud prompts 38%, and problem solving support with 34%. Talk maths prompts were the least e ective with only 25% of the cases.

There was also a signi cant e ect of the feedback type and whether the a ective state stayed the same (X2(6, N=181) = 14.34, p<.05). Talk maths prompts were highest associated with a continuing confused state with 75% of the cases. This was followed by problem solving support with 66%, talk aloud prompts with 59%, re ective prompts with 52%, task sequence prompts with 37%, a ect boosts with 32%, and the least feedback type that was associated with a continuing confused state were guidance feedback with 29% of the cases.

There was no signi cant association between the feedback type and whether the a ective state worsened (X2(6, N=181) = 4.65, p>.05). Frustration, boredom and surprise There was not su cient data available when students were frustrated (36 cases), nor when they were bored (9 cases), or surprised (3 cases) to run a statistical test across the di erent a ective states and feedback types.

However, the data indicates that some of the provided feedback types were better able to change the a ective state of the student when they were frustrated, bored or surprised, as can be seen in gure 2. For example, 60% of the a ect boosts were able to change frustration into ow, followed by re ective prompts 33% and problem solving support 20%. 5

Discussion

The results presented in the previous section show that feedback can enhance students' a ective states, and that the impact of the various feedback types mostly depends on the students' a ective state before the feedback was provided.

When students were in ow there was no signi cant di erence between the feedback types on whether or not the a ective state stayed the same or worsened. This suggests that, when students are in ow, challenging feedback can be provided without negative implications.

However, when students were confused there was a di erence between the feedback types on whether the a ective state was enhanced, stayed the same or worsened. The feedback types that most e ectively moved the student out of a confusion state were a ect boosts, instructive, and task sequence prompts. When they were struggling to overcome problems, a ect boosts appeared to encourage some students to redouble their e orts without the need for task speci c support. We can hypothesise that this enabled students to self-regulate their a ect and move forward. As expected, instructive feedback appears to have given the students the next steps that they needed, whereas other problem solving was less successful. Other problem solving feedback seems to have led students to be more confused because of the increased cognitive load caused by them having to understand the hint or the question provided.

While talk aloud prompts and talk maths, encouraged them to vocalize what they are trying to achieve, they appear not to have helped the students address their confusions. Instead, when they were confused, students appeared to have welcomed a new task (the opportunity to abandon the cause of their confusion). While as a strategy this can be pedagogically debatable, there is scope to provide tasks aimed to help them at the same concepts in a di erent, simpler way or to allow them to practice rst some skills in a practice-based rather than exploratory task.

Although there was insu cient data to analyse the impact of the di erent feedback types on students' a ective state when they were frustrated, some tentative observations can be made. For example, it was evident that the a ect of students who were frustrated was enhanced whatever the feedback they were provided with. However, it is notable that the frustrated students who were provided a ect boosts were most likely to move to a ow. We have other anecdotal evidence in the same scenario with di erent students that suggest that explicitly addressing a ect and helping students to think of their emotions during learning can help them move to confused or to ow state without need for immediate problem solving support.

It is worth noting that compared to other research we may have been unable to detect more negative states, especially boredom, because of the nature of the environment that the students were using { an exploratory learning environment that encouraged them to speak. The combination of unstructured learning and speech might prevent students from becoming bored. 6

Conclusion and future work

The a ective state of students can be modi ed with feedback. There is a di erence in the impact of di erent feedback types according to the a ective state the student is in before the feedback was provided. Although there seems not to be too much of a di erence when students are in ow, when students were confused di erent feedback types seem to matter more. While, for example, affect boosts and instructive feedback were able to change confusion into ow, prompting students to use mathematical vocabulary or providing other problem solving support, were associated with the same confused state or even lead to frustration.

In the light of ndings like D'Mello et al. [ 6 ] for example of the importance of confusion under appropriate conditions in learning, our ndings have important implications for learning and teaching in general, and AIED in particular. Problem solving support speci cally in exploratory learning environments is di cult to achieve successfully, particularly when students are in a situation that was not previously encountered during a system's design. However, detecting a ect may be relatively easier in certain contexts particularly in speech-enabled software like in our case and therefore a ective support matters as much, if not more than, problem solving support. In addition, the exact type of support provided when students are frustrated is important. To understand this better we need to investigate more the di erent types of problem solving support and their combination with a ective feedback that can act both as a way to self-regulate a ect and take student into a more positive state like confusion or ow.

In our current study we are implying that learning performance is enhanced when students are in a positive a ective state. In the future we are planning to evaluate if learning performance will be enhanced when students are moved out of a negative into a positive a ective state. Our next step is to train an intelligent system that is able to tailor the type of feedback according to the a ective state of the student in order to enhance the learning experience.

References

1. Askeland , M. : Sound-based strategy training in multiplication . European Journal of Special Needs Education 27 ( 2 ), 201 { 217 ( 2012 )

2. Baker , R.S.J.d. , DMello, S.K. , Rodrigo , M.T. , Graesser , A.C. : Better to be frustrated than bored: The incidence, persistence, and impact of learners cognitivea ective states during interactions with three di erent computer-based learning environments . Int. J. Hum.-Comput. Stud . 68 ( 4 ), 223 { 241 ( 2010 )

3. Carenini , G. , Conati , C. , Hoque , E. , Steichen , B. , Toker , D. , Enns , J.: Highlighting interventions and user di erences: Informing adaptive information visualization support . In: Proceedings of CHI 14 . pp. 1835 { 1844 ( 2014 )

4. Chi , M. : Self-explaining expository texts: The dual processes of generating inferences and repairing mental models . In: Glaser, R . (ed.) Advances in instructional psychology , pp. 161 { 238 . Mahwah , NJ: Lawrence Erbaum Associates ( 2000 )

5. Conati , C. , MacLaren , H.: Empirically building and evaluating a probabilistic model of user a ect. User Modeling and User-Adapted Interaction ( 2009 )

6. DMello , S.K. , Lehman , B. , Pekrun , R. , Graesser , A.C. : Confusion can be bene cial for learning . Learning & Instruction 29 ( 1 ), 153 { 170 ( 2014 )

7. DMello , S. , Craig , S. , Gholson , B. , Franklin , S. , Picard , R. , Graesser , A. : Integrating a ect sensors in an intelligent tutoring system . In: A ective Interactions: The Computer in the A ective Loop Workshop at IUI 2005 . pp. 7 { 13 ( 2005 )

8. Eynon , R. , Davies , C. , Holmes , W. : Supporting older adults in using technology for lifelong learning: the methodological and conceptual value of wizard of oz simulations . In: Proceedings of NLC 2012 . pp. 66 { 73 ( 2012 )

9. Kort , B. , Reilly , R. , Picard , R. : An a ective model of the interplay between emotions and learning . In: Proceedings of ICALT 2001 . No. 43 { 46 ( 2001 )

10. Mavrikis , M. , Grawemeyer , B. , Hansen , A. , Gutierrez-Santos , S. : Exploring the potential of speech recognition to support problem solving and re ection . In: ECTEL 2014 . pp. 263 { 276 ( 2014 )

11. Mavrikis , M. , Gutierrez-Santos , S. : Not all wizards are from Oz: Iterative design of intelligent learning environments by communication capacity tapering . Computers & Education 54 ( 3 ), 641 { 651 ( 2010 )

12. Ocumpaugh , J. , Baker , R.S.J.d. , Rodrigo , M.M.T.: Baker-Rodrigo Observation Method Protocol (BROMP) 1.0 . Training Manual version 1.0. Tech. rep. , New York, NY: EdLab. Manila, Philippines: Ateneo Laboratory for the Learning Sciences . ( 2012 )

13. Pekrun , R.: The control-value theory of achievement emotions: Assumptions, corollaries, and implications for educational research and practice . J. Edu. Psych. Rev . pp. 315 { 341 ( 2006 )

14. Porayska-Pomsta , K. , Mavrikis , M. , DMello , S. , Conati , C. , de Baker , R.S.J.: Knowledge elicitation methods for a ect modelling in education . I. J. Arti cial Intelligence in Education 22 ( 3 ), 107 { 140 ( 2013 )

15. Porayska-Pomsta , K. , Mavrikis , M. , Pain , H. : Diagnosing and acting on student a ect: the tutors perspective . UMUAI 18 ( 1 ), 125 - 173 ( 2008 )

16. Rosenthal , R. , Rosnow , R.: Essentials of Behavioral Research: Methods and data analysis . McGraw Hill , 3rd edn . ( 2008 )

17. Santos , O. , Saneiro , M. , Salmeron-Majadas , S. , J.G. , B. : A methodological approach to elicit a ective educational recommendataions . In: Proceedings of ICALT 2014 ( 2014 )

18. Shen , L. , Wang , M. , Shen , R.: A ective e-learning: Using emotional data to improve learning in pervasive learning environment . Educational Technology & Society 12 ( 2 ), 176 { 189 ( 2009 )

19. Sweller , J., van Merrienboer, J.G. , Paas , G.W. : Cognitive Architecture and Instructional Design . Educational Psychology Review 10 , 251 { 296 + ( 1998 )

20. Woolf , B. , Burleson , W. , Arroyo , I. , Dragon , T. , Cooper , D. , Picard , R.: A ectaware tutors: recognising and responding to student a ect . Int. J. Learning Technology 4 ( 3-4 ), 129 { 164 ( 2009 )