When designing adaptive tutoring systems, a myriad of psychological theories must be taken into account. Popular notion follows cognitive theory in supporting multi-channel processing, while working under assumptions that pedagogical agents and affect detection are of the utmost significance. However, motivation and affect are complex human characteristics that can muddle human-computer interactions. The following study considers the promotion of the growth mindset, as defined by Carol Dweck, within middle school students using an intelligent tutoring system. A randomized controlled trial comprised of six conditions is used to assess various delivery mediums of growth mindset oriented motivational messages. Student persistence and mastery speed are examined across multiple math domains, and self-response items are used to gauge student mindset, enjoyment, and perception of system helpfulness upon completion of the assignment. Findings, design limitation, and suggestions for future analysis are discussed.
The optimal design of adaptive tutoring systems is a continuous
debate for researchers in the Learning Sciences. Decisions when
authoring content can be immense, including not only the user
interface and tutor material, but also the presence of adaptive
feedback strategies such as hints or scaffolding, the use of affect
detectors, and in growing popularity, the use of pedagogical
agents. While many adaptive tutors share designs rooted in
cognitive theory, creators should also incorporate elements that
improve student motivation, engagement, persistence,
metacognition, and self-regulation skills. These elements aid in
the promotion of active learning, an experience that has been
shown to heighten the creation of mental connections [
Human motivation has historically been explained and argued
by an array of theories, as intrinsic or as extrinsic, as static or as
the constant flow of needs, emotions, and cognitions [
An expert in his own right, Richard Mayer has devoted much
of his career to promoting a series of multi-media learning
principles that enhance e-learning design. These principles call
for learning environments to be driven by active learning
processes while considering the cognitive load and working
memory of users [
Considering the optimal design of adaptive tutoring systems
and the incorporation of hypermedia and pedagogical agents to
engage students in active learning, the current study seeks to
analyze the promotion of Dweck’s growth mindset theory within
ASSISTments, an adaptive mathematics tutor. The following
research questions were derived from themes relevant to Dweck’s
[
It is hypothesized that students randomly assigned to a messaging condition will be more likely to show continued, persistent effort than those in the control condition. Similarly, regardless of the delivery medium, researchers expect students who receive mindset messages to show improved mastery speed, with fewer items, on average, required to complete a problem set. In the assessment of message delivery, it is hypothesized that motivational messages delivered using an animated version of Jane, a learning companion that originates from partnering tutor Wayang Outpost, will have a stronger effect on student persistence and learning efficiency than alternative message mediums.
To determine appropriate math content for this study, the tutor’s database was queried to compile a historical record of usage data for a variety of problem sets that fit within Common Core State Standards across various grade levels. All observed problem sets were of a style unique to the ASSISTments tutor, requiring students to answer three consecutive questions correctly in the same day in order to complete the assignment. If the student were to reach a preset ‘daily limit’ (i.e., ten problems) while attempting to solve three consecutive questions, they are prompted to consult with their teacher and try again tomorrow.
Five problem sets were chosen based on high usage, with math content spanning grades four through seven. The skill topics assessed by these problem sets included finding missing values using percent on a circle graph, equivalent fractions, multiplying decimals, rounding, and order of operations. The goal in designing multiple problem sets was three-fold: to increase data collection, to determine any significant effect for student skill level, and to determine if content was linked to student motivation, perhaps due to difficulty level. Six conditions were then established for each problem set, as defined in Table 1.
These conditions were designed following the principles set forth
by Mayer [
Math content was isomorphic across conditions, and was thus
considered comparable in difficulty. A test drive of the student
experience for each problem set can be found at [
Motivational message content, as depicted in Table 2, was
matched across conditions to reduce confounding. These
messages were validated in and derived from [
Therefore, each student’s experience of motivational messaging may have differed slightly, even within each condition. This design was established to reduce persistent message delivery and to avoid inundating students with messages on each question, with the goal of optimizing the effects of motivational messages while retaining a primary focus on math content. All visual motivational messages appeared within the tutor and remained until the student completed the problem; audio messages were played once upon loading the problem or tutoring strategy.
Did you know that when we learn something new our brain actually changes? It forms new connections inside that help us solve problems in the future. Pretty amazing, huh? Did you know that when we practice to learn new math skills our brain grows and gets stronger? That is so cool! Hey, I found out that people have myths about math… like that only some people are “good” at math. The truth is we can all be successful in math if we give it a try.
I think the most important thing is to have an open mind and believe that one can actually do math! I think that more important than getting the problem right is putting in the effort and keeping in mind the fact that we can all be good at math if we try.
Making a mistake is not a bad thing. It’s what learning is all about! When we realize we don’t know why that was not the right answer, it helps us understand better what we need to practice.
We may need to practice a lot, but our brains will develop with what we learn.
At the end of each problem set, students were asked to partake in
a series of four survey questions developed based on previously
validated content from [
All students received these questions regardless of condition. All
survey content can be accessed at [
Teachers in the state of Massachusetts who frequently use ASSISTments with their students were approached with a brief presentation explaining the study and providing examples of the conditions, motivational messages, and math content. Teachers assigned one or more of the problem sets to their students in accordance with the teachers’ usual use of the tutoring system (i.e., as either classwork or homework). Material was assigned as current content and/or review, for a total of 765 student assignments. Log data was compiled for each student’s performance. Prior to analysis of persistence and mastery speed, students were removed if they had noted experiencing technical difficulties or if they failed to log enough progress to enter one of the six conditions. Additional students were removed prior to survey analysis due to incompletion. Students remaining after each step are examined across problem sets in Table 3. A1 = Assigned. MA = Math Analysis. SA = Survey Analysis. *Students were removed prior to math analysis due to technical difficulties or failure to initiate a condition. **Additional students were removed prior to survey analysis due to incompletion.
An ex post facto judgment of student gender was determined for 570 students within the sample remaining for math content analysis. Due to incompletion rates within this subset of students, gender was determined for 554 students within the sample remaining for survey content analysis.
Analyses of student persistence and mastery speed were performed at the condition level for each problem set, as well as for an aggregate of the five sets to serve as a composite analysis of the conditions across math content. To determine if an effect existed within a particular processing channel, similar conditions were compiled based on delivery elements. For example, all conditions utilizing audio were compiled to assess the effect of audio (i.e., audio, animation, static image with audio). Similar analyses were performed to determine the effect of textual messages and the effect of the pedagogical agent’s presence.
Researchers also compared a compilation of all conditions containing motivational messages to the control condition in order to determine the effectiveness of motivational messages in general. Initial findings suggested that in general, the sample was too advanced for the math content as students were found to be at ceiling across many of the problem sets. Thus, secondary analyses examined gender differences and assessed the aforementioned variables for a subset of students operationally defined as “strugglers,” or those requiring more than three questions to complete their assignment.
When considering student persistence, as defined by continuing until reaching completion, ANOVA results suggested null results (p > .05) across all problem sets except for multiplying decimals F (5, 42) = 2.57, p < .05, η2 = 0.23. No significant results were observed when the problem sets were compiled or when specific delivery elements were isolated, and there was no significant difference between messaging conditions and the control. For the full sample, gender was found to differ significantly on persistence, F (1, 568) = 3.84, p = 0.051, η2 = 0.01, with girls showing significantly more persistence (M = 0.99, SD = 0.12) across conditions than boys (M = 0.96, SD = 0.20). While girls were found to be approaching completion in all conditions (p < .05), boys showed lower completion overall, with the lowest performance apparent in the control condition.
When considering mastery speed, as defined by the number of questions required for problem set completion, ANOVA results suggested null results (p > .05) across all problem sets analyzed individually. Further, no significant results were observed when problem sets were compiled or when specific delivery elements were isolated, and there was no significant difference between messaging conditions and control. Although there was no significant difference in mastery speed across genders, trends suggested that girls had faster mastery speed in general, requiring consistently fewer questions to complete problem sets regardless of condition (M = 4.25, SD = 2.65) than boys (M = 4.43, SD = 2.86). Means and standard deviations for the full sample are presented in Table 4.
ANOVA comparisons of the survey measures of mindset, enjoyment, and system helpfulness similarly conveyed null results within the full sample. The “mindset” variable was established from an average of two binary survey questions, with a composite score scaled from 0-2 representing the spectrum from fixed mindset (0) to growth mindset (2). The “enjoyment” variable was based on one question with Likert scale scores from 0-3, representing how much the student enjoyed their assignment. The “helpfulness” variable is represented in the same manner, based on the student’s perception of how helpful the tutoring system was in completing their assignment. Null results were found for all three measures across problem sets when analyzed individually, and no significant differences were observed between conditions when problem sets were compiled or when specific delivery elements were isolated. Further, there was no significant difference between all messaging conditions and the control group. Gender was found to have a significant effect on enjoyment, regardless of condition F (1, 552) = 19.50, p < .001, η2 = 0.03, with girls measuring more enjoyment on average (M = 1.84, SD = 0.81) than boys (M = 1.52, SD = 0.90). As shown by Table 4, the Control was found to be the most enjoyable condition, while WordArt was enjoyed significantly less (p < .10). Gender was also approaching significance on the mindset measure, F (1, 552) = 3.31, p = 0.069, η2 = 0.01, with boys exhibiting a lower mindset in general (M = 0.93, SD = 0.78) than girls (M = 1.05, SD = 0.77). Gender was not found to have a significant effect on student’s perception of tutor helpfulness.
In an attempt to answer our third research question, elements within message delivery were collapsed based on similarity to better understand if a certain processing channel (i.e., audio) was providing the main effect for messaging results. As noted briefly in results for persistence, mastery speed, and survey measures, researchers were not able to isolate any significant differences among delivery elements (p > .05).
While few significant findings were observed in the full sample, it became clear that many students were at ceiling in the math content and therefore showing high persistence (completion) in minimum mastery speed (three consecutive correct questions).
When we reassessed the sample for students operationally defined as ‘struggling,’ or those who required more than three questions to complete their assignments, our analysis became a bit more informative. Among 253 student assignments, no significant differences were found among conditions in persistence or mastery speed (p > .05). However, findings suggested that it took struggling students less questions on average to reach mastery when in the audio condition (M = 5.59, SD = 2.00) compared to all other conditions, as shown in Table 5.
When considering gender, struggling boys exhibited lower mastery in conditions including audio (p < .05) yet were found to persevere more when an image of Jane was present, while girls persevered less with the female presence (p < .05). Survey results for struggling students suggested that boys exhibited the lowest mindset measures after experiencing the control condition (p < .05), and trends suggested that regardless of condition, girls exhibited the growth mindset more consistently (M = 1.00, SD = 0.79) than boys (M = 0.91, SD = 0.75). As with the primary analysis, trends suggested that boys exhibited the growth mindset after experiencing the animation condition (p < .10). It was also found that regardless of condition, girls enjoyed their assignments (M = 1.72, SD = 0.87) significantly more than boys (M = 1.42, SD = 0.92), p < .05, and that girls consistently found the tutoring system more helpful in completing their assignment (M = 2.10,
SD = 0.83) than did boys (M = 1.92, SD = 0.90).
M 0.95 4.74 1.06 1.83 1.99
SD 0.21 3.35 0.81 0.80 0.85
M 0.98 4.32 0.96 1.67 1.94
SD 0.14 2.67 0.78 0.89 0.86
M 0.97 4.24 1.01 1.74 1.86
SD 0.17 2.69 0.80 0.87 0.89
M 0.97 4.62 0.96 1.66 2.01
SD 0.16 2.83 0.77 0.90 0.89
M 0.98 4.32 1.02 1.77 2.01
SD 0.13 2.42 0.77 0.82 0.77
Word Art (90a,b) M
SD
M
SD 0.18 1.91 0.75 0.96 0.95 aSample size for Persistence and Mastery Speed. bSample size for Mindset, Enjoyment, and Helpfulness.
Note. “Mindset” is measured by two questions (0 = Fixed Mindset, 1 = Growth Mindset) and scores are compiled. “Enjoyment” is measured by one question (Likert Scale, 0-3). “Helpfulness” is measured by one question (Likert Scale, 0-3).
M
SD
M
SD
M
SD
M
SD
M
SD M
SD Approximately 60% of students in the full sample exhibited the growth mindset in their survey responses, regardless of condition.
Noting Table 4, students in the control condition actually reported the highest levels of growth mindset (M = 1.06, SD = 0.81), with those in the audio condition reporting the lowest levels (M = 0.78, SD = 0.75). Among struggling students, the highest levels of growth mindset were reported by students in the static image with audio condition (M = 1.04, SD = 0.82), while those in the word art condition reported the lowest levels (M = 0.82, SD = 0.86). Responses to measures of enjoyment and helpfulness followed normal distributions, with approximately 60% finding the assignments at least “somewhat” enjoyable, and approximately 78% finding the tutoring system at least “somewhat” helpful.
Within the current study, the addition of motivational messaging to the ASSISTments tutor did not significantly affect the likelihood of student persistence or mastery speed. Further, there was little evidence that the motivational messages had the intended effect on mindset within the full sample. Trends suggested that those in messaging conditions experienced a slight increase in persistence and a decrease in mastery speed in comparison to those in the control condition. However, students in the messaging conditions also exhibited consistently lower levels for measures of mindset, enjoyment of the assignment, and perception of system helpfulness. A larger student population would be required to discern a truly significant effect within these trends.
Interestingly, struggling students appeared to benefit from the presence of messages, showing an increase in persistence, a decrease in mastery speed, and slightly increased measures of the growth mindset. It can be argued that struggling students, or those facing a challenge, are most in need of motivational interventions, and that they are more likely to respond to messaging, regardless of condition. Motivational messages produced distinctly higher adoption of the growth mindset in struggling students who experienced the static image with audio condition. Thus when designing motivational content for struggling students, current findings promote the addition of audio as an alternative processing channel to assist students. Researchers were not able to pinpoint an optimal processing channel for the delivery of growth mindset messages when targeting the general population.
One participating teacher requested that her students use a
feature within the tutoring system to comment on their experience
while completing their assignment. Feedback was predominantly
negative, with students citing the messages as distracting or
confusing. One student specifically questioned why the animated
learning companion simply repeated messages rather than helping
to solve the problems. This suggests that students are familiar
with systems that utilize pedagogical agents, and that they have
developed expectations for characters that are associated with
learning. This echoes the argument set forth by Kapoor, et al. [
This study had a variety of limitations. The ASSISTments math content chosen due to popular usage lead to a high percentage of ceiling effects within the sample. Teachers assigned multiple problem sets to their students, often as review. Thus, many students easily mastered the content intended for lower grades and thereby skewed rates of persistence and mastery speed. Further, the null effects found in the full sample raise important questions regarding the generalizability of mindset interventions outside of struggling student populations. Within the context of an adaptive mathematics tutor, students who appear to be at ceiling in math content may not require motivational messaging, and it may become detrimental to the learning process.
We also note that approximately 18.8% of students reported having technical difficulties and were removed prior to analysis. The incompatibility of simple HTML files serves as a reminder that many classrooms struggle to maintain up-to-date technological resources. Students are often required to share computers or iPads that come equipped with outdated software and generally slow internet connections. Future research should incorporate allowance for these issues within the experimental design, as incompatibilities may lead to selection bias.
It is also difficult to justify whether or not students
consistently attended to the motivational messages. As students
were simply presented the messages and were not asked to
respond in any manner, the levels of message internalization may
be broad. We also note that the duration of the intervention may
have been too short to observe reliable differences among
messaging conditions. In much of her work, Dweck has provided
longer interventions upfront, coupled with ‘reminders’ such as the
messages used in the current study [
Finally, it should be noted that researchers relied on the tutoring system to perform random assignment. While prior research has suggested that this practice is sound, assignment for this study appears to have favored the static image with audio condition. Future research using ASSISTments should take this bias into consideration.
Future iterations of this study should focus on struggling
students, or those undertaking challenging academic tasks. Future
research should also seek to assess these conditions in an even
more adaptive environment. It seems as though students were not
reaping the benefits of the "persona effect" found in prior research
[
We acknowledge funding from NSF (#1316736, 1252297, 1109483, 1031398, 0742503), ONR's “STEM Grand Challenges,” and IES (#R305A120125, R305C100024). Thanks to S. & L.O.