=Paper= {{Paper |id=Vol-1446/amadl_pap2 |storemode=property |title=The impact of feedback on students' affective states |pdfUrl=https://ceur-ws.org/Vol-1446/amadl_pap2.pdf |volume=Vol-1446 |dblpUrl=https://dblp.org/rec/conf/edm/GrawemeyerMHHLS15 }} ==The impact of feedback on students' affective states== https://ceur-ws.org/Vol-1446/amadl_pap2.pdf

The impact of feedback on students’ affective
states

Beate Grawemeyer1 , Manolis Mavrikis2 , Wayne Holmes2 , Alice Hansen2 ,
Katharina Loibl3 , and Sergio Gutiérrez-Santos1
1
London Knowledge Lab, Dep of Computer Science and Information Systems,
Birkbeck, London, UK
beate@dcs.bbk.ac.uk, sergut@dcs.bbk.ac.uk
2
London Knowledge Lab, UCL Institute of Education,
University College London, London, UK
m.mavrikis@ioe.ac.uk, w.holmes@ioe.ac.uk, a.hansen@ioe.ac.uk
3
Institute of Educational Research, Ruhr-Universität Bochum, Germany
katharina.loibl@rub.de

Abstract. Affective states play a significant role in students’ learning
behaviour. Positive affective states can enhance learning, while negative
affective states can inhibit it. This paper describes a Wizard-of-Oz study
that investigates the impact of different types of feedback on students’
affective states. Our results indicate the importance of providing feedback
matched carefully to the affective state of the students in order to help
them transition into more positive states. For example when students
were confused affect boosts and specific instructive feedback seem to be
effective in helping students to be in flow again. We discuss this and
other ways to adapt the feedback, together with implications for the
development of our system and the field in general.

1 Introduction

This paper reports the results of a set of two Wizard-of-Oz studies which explore
the effect of different feedback types on students’ affective states.
It is well understood by now that affect interacts with and influences the
learning process [9, 6, 2]. While positive affective states such as surprise, satis-
faction or curiosity contribute towards constructive learning, negative ones in-
cluding frustration or disillusionment at realising misconceptions can lead to
challenges in learning. The learning process is indeed full of transitions between
positive and negative affective states and regulating those is important. For ex-
ample, a student may seem interested in exploring a particular learning goal,
however s/he might have some misconceptions and need to reconsider her/his
knowledge. This can evoke frustration and/or disappointment. However, this
negative affective state may turn into deep engagement with the task again.
D’Mello et al., for example, elaborate on how confusion is likely to promote
learning under appropriate conditions [6].
It is important therefore, to deepen our understanding of the role of affective
states for learning, and to be able to move students out of states that inhibit
learning. Pekrun [13] discusses achievement emotions or affective states, which
arise in a learning situation. Achievement emotions are states that are linked to
learning, instruction, and achievement. We focus on a subset of affective states
identified by Pekrun: flow/enjoyment, surprise, frustration, and boredom. We
also add confusion, which has been identified elsewhere as an important affective
state during learning [15] for tutor support and for learning in general [6].
As described in Woolf et al. [20] students can become overwhelmed (very
confused or frustrated) during learning, which may increase cognitive load [19]
for low-ability or novice students. However, appropriate feedback might help to
overcome such problems. Carenini et al. [3] describe how effective support or
feedback needs to answer three main questions: (i) when the support should be
provided during learning; (ii) what the support should contain; and (iii) how it
should be presented.
In this paper we focus on the question of what the support should contain
with respect to affect i.e. the types of feedback that are able to induce a positive
affective state.
In related work students’ affective states have been used to tailor motivational
feedback and learning material in order to enhance the learning experience.
For example, Santos et al. [17] show that affect as well as motivation and self-
efficacy impact the effectiveness of motivational feedback and recommendations.
Additionally, Woolf et al. [20] developed an affective pedagogical agent which is
able to mirror a student’s affective state, or acknowledge a student’s affective
state if it is negative. Another example is Conati & MacLaren [5], who developed
a pedagogical agent to provide support according to the affective state of the
students and the user’s personal goal. Also, Shen et al. [18] recommend learning
material to the student based on their affective state. D’Mello et al. [7] developed
a system that is able to respond to students via a conversation that takes into
account the affective state of the student.
In contrast, in this paper, we investigate the impact of different types of
feedback on students’ affective state and how and whether they can help students
regulate their affect and thus improve learning. In what follows we present two
sets of Wizard-of-Oz studies where feedback was provided to students interacting
with an exploratory learning environment designed to learn fractions. From these
studies, the affective states of the students were carefully annotated in order to
address our research questions.

2 The Wizard-of-Oz studies
2.1 Aims

One of our research aims is to develop intelligent support that enhances the
learning experience by taking into account the student’s affective state. We were
specifically interested in identifying how different feedback types modify affective
states.
In order to address this question we conducted two sets of ecologically valid
Wizard-of-Oz studies (e.g. [11, 8]) which investigated the effect of affective states
on different feedback types at different stages of the task.

2.2 Participants and Procedure
In total, 26 Year-5 (9 to 10-year old) students took part in the Wizard-of-Oz
studies. Each session lasted on average 20 minutes. Each student participated in
one Wizard-of-Oz session.
The sessions were run in an ordinary classroom with multiple computers,
where additional children were working with the learning platform (not wiz-
arded) in order to support ecological validity. This was important particularly
as in early settings we identified that children would not speak that much to the
platform if they felt that they were monitored [10]. Figure 1 shows the setup of
the studies. Wizards followed a script with pre-canned messages to send mes-

Fig. 1. The layout. The Wizard-of-Oz studies took place on the central isle while the
rest of the students worked on a version of the system which only sequences tasks and
provides minimal support (W=wizard, S=student).

sages to the students through the learning platform and deliberately limited
their communication capacity in order to simulate the actual system. To achieve
that wizards were only able to see students’ screen. An assistant was able to
hear students’ reactions to reflections or talk-aloud prompts (as prompted by
the ‘system’) and provide recommendations to the wizard with respect to the
detected affective state. Any feedback provided was both shown on screen and
read aloud by the system to students.

2.3 Feedback types
Different types of feedback were presented to students at different stages of their
learning task. The feedback provided was based on interaction via keyboard and
mouse, as well as speech.
We explore different types of feedback that are known from the literature to
support students in their learning and fit our context. The following different
feedback types were provided:
– AFFECT BOOSTS - affect boosts. As described in [20] affect boosts
can help to enhance student’s motivation in solving a particular learning
task. These included prompts that acknowledged for example that a task is
difficult or that the student may be confused but they should keep trying.
– INSTRUCTIVE FEEDBACK - instructive task-dependent feed-
back. This feedback provided detailed instructions, what subtask or action
to perform in order to solve the task.
– OTHER PROBLEM SOLVING FEEDBACK - task-dependent feed-
back. This support was centred on helping students to solve a particular
problem that they are facing during their interaction by providing either
questions to challenge their thinking or specific hints designed to help them
identify the next step themselves.
– TALK ALOUD PROMPTS - talking aloud. With respect to learning
in particular, the hypothesis that automatic speech recognition (ASR) can
facilitate learning is based mostly on educational research that has shown
benefits of verbalization for learning (e.g., [1]).
– REFLECTIVE PROMPTS - reflecting on task performance and
learning. Self-explanation can be viewed as a tool to address students’ own
misunderstandings [4] and as a ‘window’ into students’ thinking.
– TALK MATHEMATICS PROMPTS - using particular domain
specific mathematics vocabulary. The aim of this prompt was to en-
courage students to use mathematical vocabulary in order continually revise
their interpretations. In early studies [10] we found that students’ reflections
were often procedural and pragmatic (e.g. talking about the user interface)
rather than mathematical.
– TASK SEQUENCE PROMPTS - moving to the next task. This
feedback is centred on providing support regarding what action to perform
next in order to change the task, such as clicking the ‘Next’ button.
Table 1 shows examples of the different feedback types.

3 Annotation of affective states and feedback reactions
From the Wizard-of-Oz studies we recorded the students’ screen display and their
voices. From this data, we annotated affective states (e.g. screen interaction and
what the students said) before and after feedback was provided.
As described earlier, for the affective state detection we discriminated be-
tween five different affective types: enjoyment, surprise, confusion, frustration,
and boredom. For the annotation of those affective states we used a similar strat-
egy to that described in [15], where a dialogue between a teacher and a student
was annotated retrospectively by categorising utterances in terms of different
feedback types. Also, [2] describe how they coded different affective states based
on observations of students interacting with a learning environment. Similarly,
we annotated student’s affective states for each type of feedback provided. In
addition to the student’s voice we also used the video of the screen capture
to support the annotation process. Students’ affective states were annotated as
follows:
– FLOW: Engagement with the learning task. Statements like ‘I am enjoying
this task’ or ‘This is fun’. Sustained interaction with the system.
Feedback type Example
AFFECT BOOSTS You’re working really hard! Keep
going!
INSTRUCTIVE FEEDBACK Use the comparison box to compare
your fractions.
OTHER PROBLEM SOLVING FEEDBACK If you add fractions, they need to
have the same denominators first.
REFLECTIVE PROMPTS What do you notice about the two
fractions?
TALK ALOUD PROMPTS Remember to talk aloud, what are
you thinking?
TALK MATHEMATICS PROMPTS Can you explain that again using
the terms denominator, numerator?
TASK SEQUENCE PROMPTS Well done. When you are ready click
‘next’ for the next task.

Table 1. Examples of feedback types

– SURPRISE: Gasping. Statements like ‘Huh?’ or ‘Oh, no!’.
– CONFUSION: Failing to perform a particular task. Statements such as
‘I’m confused!’ or ‘Why didn’t it work?’. Uncertain interaction with the
system.
– FRUSTRATION: Tendency to give up, repeatedly clicking or deleting
of objects in the system or repeatedly failing to perform a particular task,
sighing, statements such as, ‘What’s going on?!’.
– BOREDOM: Inactivity or statements such as ‘Can we do something else?’
or ‘This is boring’.

4 Results
In total 396 messages were sent to 26 students. The video data in combination
with the sound files were analysed independently by three researchers (one was
independent of the project) who categorised the affective states of students before
and after the feedback messages were provided.
The data is combined from two sets of Wizard-of-Oz studies. We use kappa
statistics to measure the degree of the agreements of the annotations for reli-
ability. Kappa was .46, p<.001. This is generally expected from retrospective
annotation of naturalistic affect experiences [14]. We consolidated the annota-
tions based on discussion between the annotators and the rest of the authors of
the paper in order to agree upon the annotations that did not match originally.
In the second set we had resources to introduce the Baker-Rodrigo Observa-
tion Method Protocol (BROMP) and the HART mobile app that facilitates the
coding of students affective states in the classroom [12]. Kappa based on the
retrospective annotation was still .56, p<.001. We first consolidated the data
with the same approach as before and then compared against the field annota-
tions. Kappa between the consolidated annotation and the HART data was .71,
p<.05 (note that is may appear low but we did not expect the retrospective an-
notation to get surprise and frustration accurately). We used the HART data to
improve the annotation by mapping feedback actions against the observation for
20 seconds prior to the delivery of the feedback to 20 seconds after the student
had closed the corresponding feedback window. We marked the changes for an
independent annotator to revisit the first set of annotations.
The student’s affective states, that occurred before and after the different
types of feedback was provided, can be seen in figure 2. Each block shows an
affective state before feedback was provided. The colour within the bars indicates
the type of affective states that occurred after the feedback was provided. The
number within the bars indicate the number of times the affective state occurred.

In order to investigate whether there was an effect of the feedback on the
learning experience, we looked at whether a student’s affective state was en-
hanced, stayed the same or worsened. An affective state was enhanced for ex-
ample, when it was changed from confusion to flow, or (given the findings about
confusion [6]) from frustration to confusion, frustration to flow, boredom to flow
etc. An affective state was worsened if it moved for example, from flow to frus-
tration or confusion, or from confusion to frustration.
As the data is categorical [16], we apply chi-square tests to investigate sta-
tistical significant differences between the groups. We present them below and
discuss in more detail in the next section.

Flow When students were in flow, there was no significant difference between
the feedback types on whether the affective state stayed in the same flow state
(X 2 (6, N=169) = 4.31, p>.05) or worsened (X 2 (6, N=169) = 4.89, p>.05). As
flow is the most positive affective state, the affective state in this sub-sample
cannot be enhanced.

Confusion When students were confused, there was a significant effect of the
feedback type on whether students’ affective state was enhanced into a flow state
(X 2 (6, N=181) = 13.65, p<.05). The most effective feedback types were affect
boosts with 68% of the cases, followed by guidance feedback with 67%, and task
sequence prompts with 63%. Reflective prompts resulted in a flow state in 48%
of the cases, talk aloud prompts 38%, and problem solving support with 34%.
Talk maths prompts were the least effective with only 25% of the cases.
There was also a significant effect of the feedback type and whether the
affective state stayed the same (X 2 (6, N=181) = 14.34, p<.05). Talk maths
prompts were highest associated with a continuing confused state with 75% of
the cases. This was followed by problem solving support with 66%, talk aloud
prompts with 59%, reflective prompts with 52%, task sequence prompts with
37%, affect boosts with 32%, and the least feedback type that was associated
with a continuing confused state were guidance feedback with 29% of the cases.
There was no significant association between the feedback type and whether
the affective state worsened (X 2 (6, N=181) = 4.65, p>.05).
Fig. 2. Students’ affective states before and after feedback was provided. Each block
shows an affective state before feedback was provided. The colour within the bars
indicates the type of affective states that occurred after the feedback was provided.
The number within the bars indicate the number of times the affective state occurred.
Frustration, boredom and surprise There was not sufficient data available
when students were frustrated (36 cases), nor when they were bored (9 cases),
or surprised (3 cases) to run a statistical test across the different affective states
and feedback types.
However, the data indicates that some of the provided feedback types were
better able to change the affective state of the student when they were frustrated,
bored or surprised, as can be seen in figure 2. For example, 60% of the affect
boosts were able to change frustration into flow, followed by reflective prompts
33% and problem solving support 20%.

5 Discussion
The results presented in the previous section show that feedback can enhance
students’ affective states, and that the impact of the various feedback types
mostly depends on the students’ affective state before the feedback was provided.
When students were in flow there was no significant difference between the
feedback types on whether or not the affective state stayed the same or wors-
ened. This suggests that, when students are in flow, challenging feedback can be
provided without negative implications.
However, when students were confused there was a difference between the
feedback types on whether the affective state was enhanced, stayed the same
or worsened. The feedback types that most effectively moved the student out
of a confusion state were affect boosts, instructive, and task sequence prompts.
When they were struggling to overcome problems, affect boosts appeared to
encourage some students to redouble their efforts without the need for task
specific support. We can hypothesise that this enabled students to self-regulate
their affect and move forward. As expected, instructive feedback appears to
have given the students the next steps that they needed, whereas other problem
solving was less successful. Other problem solving feedback seems to have led
students to be more confused because of the increased cognitive load caused by
them having to understand the hint or the question provided.
While talk aloud prompts and talk maths, encouraged them to vocalize what
they are trying to achieve, they appear not to have helped the students address
their confusions. Instead, when they were confused, students appeared to have
welcomed a new task (the opportunity to abandon the cause of their confusion).
While as a strategy this can be pedagogically debatable, there is scope to pro-
vide tasks aimed to help them at the same concepts in a different, simpler way
or to allow them to practice first some skills in a practice-based rather than
exploratory task.
Although there was insufficient data to analyse the impact of the different
feedback types on students’ affective state when they were frustrated, some ten-
tative observations can be made. For example, it was evident that the affect of
students who were frustrated was enhanced whatever the feedback they were
provided with. However, it is notable that the frustrated students who were pro-
vided affect boosts were most likely to move to a flow. We have other anecdotal
evidence in the same scenario with different students that suggest that explicitly
addressing affect and helping students to think of their emotions during learning
can help them move to confused or to flow state without need for immediate
problem solving support.
It is worth noting that compared to other research we may have been unable
to detect more negative states, especially boredom, because of the nature of the
environment that the students were using – an exploratory learning environment
that encouraged them to speak. The combination of unstructured learning and
speech might prevent students from becoming bored.
6 Conclusion and future work
The affective state of students can be modified with feedback. There is a differ-
ence in the impact of different feedback types according to the affective state
the student is in before the feedback was provided. Although there seems not
to be too much of a difference when students are in flow, when students were
confused different feedback types seem to matter more. While, for example, af-
fect boosts and instructive feedback were able to change confusion into flow,
prompting students to use mathematical vocabulary or providing other problem
solving support, were associated with the same confused state or even lead to
frustration.
In the light of findings like D’Mello et al. [6] for example of the importance of
confusion under appropriate conditions in learning, our findings have important
implications for learning and teaching in general, and AIED in particular. Prob-
lem solving support specifically in exploratory learning environments is difficult
to achieve successfully, particularly when students are in a situation that was not
previously encountered during a system’s design. However, detecting affect may
be relatively easier in certain contexts particularly in speech-enabled software
like in our case and therefore affective support matters as much, if not more
than, problem solving support. In addition, the exact type of support provided
when students are frustrated is important. To understand this better we need to
investigate more the different types of problem solving support and their combi-
nation with affective feedback that can act both as a way to self-regulate affect
and take student into a more positive state like confusion or flow.
In our current study we are implying that learning performance is enhanced
when students are in a positive affective state. In the future we are planning to
evaluate if learning performance will be enhanced when students are moved out
of a negative into a positive affective state. Our next step is to train an intelligent
system that is able to tailor the type of feedback according to the affective state
of the student in order to enhance the learning experience.
References
1. Askeland, M.: Sound-based strategy training in multiplication. European Journal
of Special Needs Education 27(2), 201–217 (2012)
2. Baker, R.S.J.d., DMello, S.K., Rodrigo, M.T., Graesser, A.C.: Better to be frus-
trated than bored: The incidence, persistence, and impact of learners cognitive-
affective states during interactions with three different computer-based learning en-
vironments. Int. J. Hum.-Comput. Stud. 68(4), 223–241 (2010)
3. Carenini, G., Conati, C., Hoque, E., Steichen, B., Toker, D., Enns, J.: Highlight-
ing interventions and user differences: Informing adaptive information visualization
support. In: Proceedings of CHI 14. pp. 1835–1844 (2014)
4. Chi, M.: Self-explaining expository texts: The dual processes of generating infer-
ences and repairing mental models. In: Glaser, R. (ed.) Advances in instructional
psychology, pp. 161–238. Mahwah, NJ: Lawrence Erbaum Associates (2000)
5. Conati, C., MacLaren, H.: Empirically building and evaluating a probabilistic model
of user affect. User Modeling and User-Adapted Interaction (2009)
6. DMello, S.K., Lehman, B., Pekrun, R., Graesser, A.C.: Confusion can be beneficial
for learning. Learning & Instruction 29(1), 153–170 (2014)
7. DMello, S., Craig, S., Gholson, B., Franklin, S., Picard, R., Graesser, A.: Integrat-
ing affect sensors in an intelligent tutoring system. In: Affective Interactions: The
Computer in the Affective Loop Workshop at IUI 2005. pp. 7–13 (2005)
8. Eynon, R., Davies, C., Holmes, W.: Supporting older adults in using technology for
lifelong learning: the methodological and conceptual value of wizard of oz simula-
tions. In: Proceedings of NLC 2012. pp. 66–73 (2012)
9. Kort, B., Reilly, R., Picard, R.: An affective model of the interplay between emotions
and learning. In: Proceedings of ICALT 2001. No. 43–46 (2001)
10. Mavrikis, M., Grawemeyer, B., Hansen, A., Gutiérrez-Santos, S.: Exploring the
potential of speech recognition to support problem solving and reflection. In: EC-
TEL 2014. pp. 263–276 (2014)
11. Mavrikis, M., Gutiérrez-Santos, S.: Not all wizards are from Oz: Iterative design of
intelligent learning environments by communication capacity tapering. Computers
& Education 54(3), 641–651 (2010)
12. Ocumpaugh, J., Baker, R.S.J.d., Rodrigo, M.M.T.: Baker-Rodrigo Observation
Method Protocol (BROMP) 1.0. Training Manual version 1.0. Tech. rep., New
York, NY: EdLab. Manila, Philippines: Ateneo Laboratory for the Learning Sci-
ences. (2012)
13. Pekrun, R.: The control-value theory of achievement emotions: Assumptions, corol-
laries, and implications for educational research and practice. J. Edu. Psych. Rev.
pp. 315–341 (2006)
14. Porayska-Pomsta, K., Mavrikis, M., DMello, S., Conati, C., de Baker, R.S.J.:
Knowledge elicitation methods for affect modelling in education. I. J. Artificial
Intelligence in Education 22(3), 107–140 (2013)
15. Porayska-Pomsta, K., Mavrikis, M., Pain, H.: Diagnosing and acting on student
affect: the tutors perspective. UMUAI 18(1), 125-173 (2008)
16. Rosenthal, R., Rosnow, R.: Essentials of Behavioral Research: Methods and data
analysis. McGraw Hill, 3rd edn. (2008)
17. Santos, O., Saneiro, M., Salmeron-Majadas, S., J.G., B.: A methodological ap-
proach to elicit affective educational recommendataions. In: Proceedings of ICALT
2014 (2014)
18. Shen, L., Wang, M., Shen, R.: Affective e-learning: Using emotional data to improve
learning in pervasive learning environment. Educational Technology & Society 12(2),
176–189 (2009)
19. Sweller, J., van Merrienboer, J.G., Paas, G.W.: Cognitive Architecture and In-
structional Design. Educational Psychology Review 10, 251–296+ (1998)
20. Woolf, B., Burleson, W., Arroyo, I., Dragon, T., Cooper, D., Picard, R.: Affect-
aware tutors: recognising and responding to student affect. Int. J. Learning Tech-
nology 4(3-4), 129–164 (2009)