Promoting Growth Mindset Within Intelligent Tutoring Systems Korinn S. Ostrow Sarah E. Schultz Ivon Arroyo Worcester Polytechnic Institute Worcester Polytechnic Institute Worcester Polytechnic Institute 100 Institute Road 100 Institute Road 100 Institute Road Worcester, MA 01609 Worcester, MA 01609 Worcester, MA 01609 ksostrow@wpi.edu seschultz@wpi.edu iarroyo@wpi.edu ABSTRACT adaptive tutors are designed under the assumption that students When designing adaptive tutoring systems, a myriad of are ideal learners, driven and motivated, ready to employ a full psychological theories must be taken into account. Popular notion range of self-regulation skills coupled with technological prowess follows cognitive theory in supporting multi-channel processing, [1]. Thus, researchers have recently undertaken a more thorough while working under assumptions that pedagogical agents and examination of how to universally encourage and motivate affect detection are of the utmost significance. However, students while still promoting self-regulated learning skills and motivation and affect are complex human characteristics that can optimizing system design [3, 8]. muddle human-computer interactions. The following study Human motivation has historically been explained and argued considers the promotion of the growth mindset, as defined by by an array of theories, as intrinsic or as extrinsic, as static or as Carol Dweck, within middle school students using an intelligent the constant flow of needs, emotions, and cognitions [13]. In a tutoring system. A randomized controlled trial comprised of six somewhat similar sense, recent research promoting affect conditions is used to assess various delivery mediums of growth detection within educational technology suggests that affect plays mindset oriented motivational messages. Student persistence and a primary role in learning success [2]. How can researchers mastery speed are examined across multiple math domains, and incorporate deeply rooted human characteristics like motivation self-response items are used to gauge student mindset, enjoyment, and affect into the design of an adaptive tutoring system? A and perception of system helpfulness upon completion of the renowned leader in the field of psychology, Carol Dweck has assignment. Findings, design limitation, and suggestions for helped to establish theories of intelligence that marry these future analysis are discussed. complex constructs within the confines of learning studies [5]. Her research has shown that students approach learning tasks Keywords largely with one of two ‘mindsets.’ The fixed mindset is characterized by the notion that intelligence is somehow innate or Motivational messages, growth mindset, pedagogical agents, immutable. Students who live within this fixed realm generally multi-media learning principles, e-learning design. emit lower learning and performance outcomes as well as higher attrition rates based in the notion that effort will not lead to 1. INTRODUCTION intellectual advancement [6]. Much of American society is rooted The optimal design of adaptive tutoring systems is a continuous in this view; strong emphasis is placed on standardized testing and debate for researchers in the Learning Sciences. Decisions when zero sum competition, with the goal of comparing student authoring content can be immense, including not only the user intelligence rather than promoting learning. Alternatively, interface and tutor material, but also the presence of adaptive students with a growth mindset believe that intelligence is feedback strategies such as hints or scaffolding, the use of affect malleable and that effort and persistence can lead to success. detectors, and in growing popularity, the use of pedagogical While Dweck [7] argues that neither mindset is necessarily agents. While many adaptive tutors share designs rooted in ‘correct,’ she promotes the notion that mindset can be altered, and cognitive theory, creators should also incorporate elements that explains the growth mindset as offering a healthier mental improve student motivation, engagement, persistence, lifestyle. Altering mindset is best achieved by varying the type of metacognition, and self-regulation skills. These elements aid in praise students receive and by realigning their definition of the promotion of active learning, an experience that has been successful learning. By highlighting the learning process rather shown to heighten the creation of mental connections [10]. than the student’s intelligence or performance, ‘process praise’ However, successful adaptive tutoring systems are not just a and the promotion of malleable intelligence has led to positive, random conglomeration of these learning goals. All too often, long-term learning gains [5]. Students trained in the growth mindset show increased enjoyment in difficult learning tasks as well as higher overall achievement and performance [6]. An expert in his own right, Richard Mayer has devoted much of his career to promoting a series of multi-media learning principles that enhance e-learning design. These principles call for learning environments to be driven by active learning processes while considering the cognitive load and working memory of users [4]. As such, those authoring adaptive tutors should utilize audio, animation, graphics, video, and other decimals, rounding, and order of operations. The goal in hypermedia elements to appease multiple sensory channels and designing multiple problem sets was three-fold: to increase data thereby reduce the user’s overall cognitive load. It is important to collection, to determine any significant effect for student skill note that powerful design requires a fine balance of these level, and to determine if content was linked to student resources, as exorbitance may serve to distract or disrupt learners. motivation, perhaps due to difficulty level. Six conditions were The evolution of pedagogical agents and learning companions then established for each problem set, as defined in Table 1. within adaptive tutoring systems has served as a primary way to These conditions were designed following the principles set forth incorporate both multi-media elements and non-cognitive support. by Mayer [4], to test matched content messages across a variety of As guidelines for the design of human-computer interaction have processing channels. followed those set forth by human-human interaction, the art of appropriating the cognitive and affective responses of pedagogical Table 1. Motivational messaging conditions. agents has been of major concern [9]. Agents are typically Control ASSISTments as usual; no messages added designed with the premise that they should respond happily to Animation Jane, a female pedagogical agent, delivers student successes and with a shared disappointment upon failures messages with motion and sound [9]. Considering the optimal design of adaptive tutoring systems Static Image with Text The agent is presented as a static image, and the incorporation of hypermedia and pedagogical agents to with a speech bubble to deliver motivational text messages engage students in active learning, the current study seeks to analyze the promotion of Dweck’s growth mindset theory within Static Image with Audio The agent is presented as a static image, ASSISTments, an adaptive mathematics tutor. The following supplemented by audio files to deliver research questions were derived from themes relevant to Dweck’s motivational messages [6] work, in combination with adaptive tutoring structures unique Word Art A speech cloud shows motivational text to ASSISTments: messages, with no agent involvement 1. Does the addition of motivational messaging within the Audio The agent’s voice delivers motivational tutoring system affect the likelihood of student persistence or messages with no graphical changes to tutor attrition? content 2. Does the presence of motivational messaging within the tutoring system affect mastery speed as defined by how many The student experience for each problem set was formatted in the items, on average, it takes for students to complete the same manner. An introductory ‘question’ explained the format of problem set? the problem set and alerted the student to turn on their computer 3. Can specific elements within message delivery be pinpointed volume and to use headphones if necessary. The second ‘question’ as significantly powerful? That is, can researchers isolate an tested whether or not the student was able to see and hear the element (e.g., the presence of a pedagogical agent, the audio pedagogical agent Jane as she introduced herself as a problem- component, static images, or a combination of these elements) solving partner. This question was included to test the that is responsible for the majority of variance in persistence compatibility of the HTML files that supported the pedagogical and learning efficiency? agent’s animation and sound conditions, thus serving as It is hypothesized that students randomly assigned to a confirmation of fair random assignment. Researchers then relied messaging condition will be more likely to show continued, on a randomization feature unique to ASSISTments that randomly persistent effort than those in the control condition. Similarly, assigned students to one of the six conditions depicted in Table 1. regardless of the delivery medium, researchers expect students Math content was isomorphic across conditions, and was thus who receive mindset messages to show improved mastery speed, considered comparable in difficulty. A test drive of the student with fewer items, on average, required to complete a problem experience for each problem set can be found at [12]. set. In the assessment of message delivery, it is hypothesized that Motivational message content, as depicted in Table 2, was motivational messages delivered using an animated version of matched across conditions to reduce confounding. These Jane, a learning companion that originates from partnering tutor messages were validated in and derived from [1]. Each problem Wayang Outpost, will have a stronger effect on student set was designed to randomly select questions from a pool of persistence and learning efficiency than alternative message approximately 100 problems, containing two types of mediums. motivational message delivery: general attributions, in which the motivational message was presented with the primary question, 2. METHODS and incorrect attributions, in which the motivational message was To determine appropriate math content for this study, the tutor’s presented alongside content feedback if the student responded database was queried to compile a historical record of usage data incorrectly or employed a tutoring strategy. Following this design for a variety of problem sets that fit within Common Core State structure, students saw general attributions on approximately half Standards across various grade levels. All observed problem sets of the questions, with the remaining half displaying incorrect were of a style unique to the ASSISTments tutor, requiring attributions only to students who answered a problem correctly. students to answer three consecutive questions correctly in the Therefore, each student’s experience of motivational messaging same day in order to complete the assignment. If the student were may have differed slightly, even within each condition. This to reach a preset ‘daily limit’ (i.e., ten problems) while attempting design was established to reduce persistent message delivery and to solve three consecutive questions, they are prompted to consult to avoid inundating students with messages on each question, with with their teacher and try again tomorrow. the goal of optimizing the effects of motivational messages while Five problem sets were chosen based on high usage, with retaining a primary focus on math content. All visual motivational math content spanning grades four through seven. The skill topics messages appeared within the tutor and remained until the student assessed by these problem sets included finding missing values completed the problem; audio messages were played once upon using percent on a circle graph, equivalent fractions, multiplying loading the problem or tutoring strategy. Table 2. Motivational message item content. General Attributions 1. Did you know that when we learn something new our brain actually changes? It forms new connections inside that help us solve problems in the future. Pretty amazing, huh? 2. Did you know that when we practice to learn new math skills our brain grows and gets stronger? That is so cool! 3. Hey, I found out that people have myths about math… like that only some people are “good” at math. The truth is we can all be successful in math if we give it a try. 4. I think the most important thing is to have an open mind and believe that one can actually do math! 5. I think that more important than getting the problem right is putting in the effort and keeping in mind the fact that we can all be good at math if we try. Incorrect Attributions 1. Making a mistake is not a bad thing. It’s what learning is all about! 2. When we realize we don’t know why that was not the right answer, it helps us understand better what we need to practice. 3. We may need to practice a lot, but our brains will develop with what we learn. At the end of each problem set, students were asked to partake in An ex post facto judgment of student gender was determined for a series of four survey questions developed based on previously 570 students within the sample remaining for math content validated content from [11], to assess student mindset, goal analysis. Due to incompletion rates within this subset of students, orientation, and perceptions of enjoyment and system helpfulness. gender was determined for 554 students within the sample All students received these questions regardless of condition. All remaining for survey content analysis. survey content can be accessed at [12]. 4. RESULTS 3. PROCEDURE Analyses of student persistence and mastery speed were Teachers in the state of Massachusetts who frequently use performed at the condition level for each problem set, as well as ASSISTments with their students were approached with a brief for an aggregate of the five sets to serve as a composite analysis presentation explaining the study and providing examples of the of the conditions across math content. To determine if an effect conditions, motivational messages, and math content. Teachers existed within a particular processing channel, similar conditions assigned one or more of the problem sets to their students in were compiled based on delivery elements. For example, all accordance with the teachers’ usual use of the tutoring system conditions utilizing audio were compiled to assess the effect of (i.e., as either classwork or homework). Material was assigned as audio (i.e., audio, animation, static image with audio). Similar current content and/or review, for a total of 765 student analyses were performed to determine the effect of textual assignments. Log data was compiled for each student’s messages and the effect of the pedagogical agent’s presence. performance. Prior to analysis of persistence and mastery speed, Researchers also compared a compilation of all conditions students were removed if they had noted experiencing technical containing motivational messages to the control condition in order difficulties or if they failed to log enough progress to enter one of to determine the effectiveness of motivational messages in the six conditions. Additional students were removed prior to general. Initial findings suggested that in general, the sample was survey analysis due to incompletion. Students remaining after too advanced for the math content as students were found to be at each step are examined across problem sets in Table 3. ceiling across many of the problem sets. Thus, secondary analyses Table 3. Explanation of Students Remaining After Removals. examined gender differences and assessed the aforementioned variables for a subset of students operationally defined as Problem Set A1 MA* SA** “strugglers,” or those requiring more than three questions to complete their assignment. Percent on a Circle Graph 87 69 62 When considering student persistence, as defined by continuing until reaching completion, ANOVA results suggested Equivalent Fractions 255 208 205 null results (p > .05) across all problem sets except for multiplying decimals F (5, 42) = 2.57, p < .05, η2 = 0.23. No significant Multiplying Decimals 62 48 47 results were observed when the problem sets were compiled or when specific delivery elements were isolated, and there was no Rounding 253 208 205 significant difference between messaging conditions and the control. For the full sample, gender was found to differ Order of Operations 108 88 86 significantly on persistence, F (1, 568) = 3.84, p = 0.051, η2 = 0.01, with girls showing significantly more persistence (M = 0.99, REMAINING 765 621 605 SD = 0.12) across conditions than boys (M = 0.96, SD = 1 A = Assigned. MA = Math Analysis. SA = Survey Analysis. 0.20). While girls were found to be approaching completion in all *Students were removed prior to math analysis due to technical conditions (p < .05), boys showed lower completion overall, with difficulties or failure to initiate a condition. the lowest performance apparent in the control condition. **Additional students were removed prior to survey analysis due to When considering mastery speed, as defined by the number of incompletion. questions required for problem set completion, ANOVA results suggested null results (p > .05) across all problem sets analyzed individually. Further, no significant results were observed when In an attempt to answer our third research question, elements problem sets were compiled or when specific delivery elements within message delivery were collapsed based on similarity to were isolated, and there was no significant difference between better understand if a certain processing channel (i.e., audio) was messaging conditions and control. Although there was no providing the main effect for messaging results. As noted briefly significant difference in mastery speed across genders, trends in results for persistence, mastery speed, and survey measures, suggested that girls had faster mastery speed in general, requiring researchers were not able to isolate any significant differences consistently fewer questions to complete problem sets regardless among delivery elements (p > .05). of condition (M = 4.25, SD = 2.65) than boys (M = 4.43, SD = While few significant findings were observed in the full 2.86). Means and standard deviations for the full sample are sample, it became clear that many students were at ceiling in the presented in Table 4. math content and therefore showing high persistence (completion) ANOVA comparisons of the survey measures of mindset, in minimum mastery speed (three consecutive correct questions). enjoyment, and system helpfulness similarly conveyed null results When we reassessed the sample for students operationally defined within the full sample. The “mindset” variable was established as ‘struggling,’ or those who required more than three questions to from an average of two binary survey questions, with a composite complete their assignments, our analysis became a bit more score scaled from 0-2 representing the spectrum from fixed informative. Among 253 student assignments, no significant mindset (0) to growth mindset (2). The “enjoyment” variable was differences were found among conditions in persistence or based on one question with Likert scale scores from 0-3, mastery speed (p > .05). However, findings suggested that it took representing how much the student enjoyed their assignment. The struggling students less questions on average to reach mastery “helpfulness” variable is represented in the same manner, based when in the audio condition (M = 5.59, SD = 2.00) compared to on the student’s perception of how helpful the tutoring system was all other conditions, as shown in Table 5. in completing their assignment. Null results were found for all When considering gender, struggling boys exhibited lower three measures across problem sets when analyzed individually, mastery in conditions including audio (p < .05) yet were found to and no significant differences were observed between conditions persevere more when an image of Jane was present, while girls when problem sets were compiled or when specific delivery persevered less with the female presence (p < .05). Survey results elements were isolated. Further, there was no significant for struggling students suggested that boys exhibited the lowest difference between all messaging conditions and the control mindset measures after experiencing the control condition (p < group. Gender was found to have a significant effect on .05), and trends suggested that regardless of condition, girls enjoyment, regardless of condition F (1, 552) = 19.50, p < .001, η2 exhibited the growth mindset more consistently (M = 1.00, SD = = 0.03, with girls measuring more enjoyment on average (M = 0.79) than boys (M = 0.91, SD = 0.75). As with the primary 1.84, SD = 0.81) than boys (M = 1.52, SD = 0.90). As shown by analysis, trends suggested that boys exhibited the growth mindset Table 4, the Control was found to be the most enjoyable after experiencing the animation condition (p < .10). It was also condition, while WordArt was enjoyed significantly less (p < found that regardless of condition, girls enjoyed their assignments .10). Gender was also approaching significance on the mindset (M = 1.72, SD = 0.87) significantly more than boys (M = 1.42, measure, F (1, 552) = 3.31, p = 0.069, η2 = 0.01, with boys SD = 0.92), p < .05, and that girls consistently found the tutoring exhibiting a lower mindset in general (M = 0.93, SD = 0.78) than system more helpful in completing their assignment (M = 2.10, girls (M = 1.05, SD = 0.77). Gender was not found to have a SD = 0.83) than did boys (M = 1.92, SD = 0.90). significant effect on student’s perception of tutor helpfulness. Table 4. Means and Standard Deviations for Persistence, Mastery Speed, and Survey Measures Across Control and Messaging Conditions for All Students. Static Image Static Image Control All Messaging Animation with Text with Audio Word Art Audio (104a, 99b) (517a, 506b) (106a, 103b) (116a, 113b) (117a, 115b) (90a,b) (88a, 85b) Analysis M SD M SD M SD M SD M SD M SD M SD Persistence 0.95 0.21 0.98 0.14 0.97 0.17 0.97 0.16 0.98 0.13 1.00 0.00 0.97 0.18 Mastery Speed 4.74 3.35 4.32 2.67 4.24 2.69 4.62 2.83 4.32 2.42 4.28 3.33 4.09 1.91 Mindset 1.06 0.81 0.96 0.78 1.01 0.80 0.96 0.77 1.02 0.77 1.00 0.79 0.78 0.75 Enjoyment 1.83 0.80 1.67 0.89 1.74 0.87 1.66 0.90 1.77 0.82 1.49 0.91 1.67 0.96 Helpfulness 1.99 0.85 1.94 0.86 1.86 0.89 2.01 0.89 2.01 0.77 1.82 0.79 1.95 0.95 a Sample size for Persistence and Mastery Speed. b Sample size for Mindset, Enjoyment, and Helpfulness. Note. “Mindset” is measured by two questions (0 = Fixed Mindset, 1 = Growth Mindset) and scores are compiled. “Enjoyment” is measured by one question (Likert Scale, 0-3). “Helpfulness” is measured by one question (Likert Scale, 0-3). Table 5. Means and Standard Deviations for Persistence, Mastery Speed, and Survey Measures Across Control and Messaging Conditions for Struggling Students. Static Image Static Image Control All Messaging Animation with Text with Audio Word Art Audio (46a , 45b) (207a, 204b) (42a, 41b) (49a, 47b) (49a,b) (28a,b) (39a,b) Analysis M SD M SD M SD M SD M SD M SD M SD Persistence 0.98 0.15 0.99 0.12 0.98 0.15 0.96 0.20 1.00 0.00 1.00 0.00 1.00 0.00 Mastery Speed 7.07 3.95 6.34 3.32 6.17 3.48 6.84 3.24 6.14 2.88 7.11 4.95 5.59 2.00 Mindset 0.93 0.75 0.95 0.78 1.00 0.81 0.89 0.73 1.04 0.82 0.82 0.86 0.92 0.70 Enjoyment 1.60 0.86 1.58 0.94 1.76 0.92 1.45 1.00 1.71 0.79 1.43 1.07 1.51 0.97 Helpfulness 1.98 0.92 2.01 0.87 1.98 0.94 1.98 0.82 2.04 0.87 2.00 0.82 2.05 0.94 a Sample size for Persistence and Mastery Speed. b Sample size for Mindset, Enjoyment, and Helpfulness. Note. “Mindset” is measured by two questions (0 = Fixed Mindset, 1 = Growth Mindset) and scores are compiled. “Enjoyment” is measured by one question (Likert Scale, 0-3). “Helpfulness” is measured by one question (Likert Scale, 0-3). Approximately 60% of students in the full sample exhibited the struggling students, current findings promote the addition of audio growth mindset in their survey responses, regardless of condition. as an alternative processing channel to assist students. Noting Table 4, students in the control condition actually reported Researchers were not able to pinpoint an optimal processing the highest levels of growth mindset (M = 1.06, SD = 0.81), with channel for the delivery of growth mindset messages when those in the audio condition reporting the lowest levels (M = 0.78, targeting the general population. SD = 0.75). Among struggling students, the highest levels of One participating teacher requested that her students use a growth mindset were reported by students in the static image with feature within the tutoring system to comment on their experience audio condition (M = 1.04, SD = 0.82), while those in the word art while completing their assignment. Feedback was predominantly condition reported the lowest levels (M = 0.82, SD = negative, with students citing the messages as distracting or 0.86). Responses to measures of enjoyment and helpfulness confusing. One student specifically questioned why the animated followed normal distributions, with approximately 60% finding learning companion simply repeated messages rather than helping the assignments at least “somewhat” enjoyable, and to solve the problems. This suggests that students are familiar approximately 78% finding the tutoring system at least with systems that utilize pedagogical agents, and that they have “somewhat” helpful. developed expectations for characters that are associated with learning. This echoes the argument set forth by Kapoor, et al. [9] 5. DISCUSSION regarding the necessity for tutors to provide appropriate cognitive Within the current study, the addition of motivational messaging and affective responses, and aids in the design of tutoring systems to the ASSISTments tutor did not significantly affect the hoping to incorporate learning companions. likelihood of student persistence or mastery speed. Further, there This study had a variety of limitations. The ASSISTments was little evidence that the motivational messages had the math content chosen due to popular usage lead to a high intended effect on mindset within the full sample. Trends percentage of ceiling effects within the sample. Teachers assigned suggested that those in messaging conditions experienced a slight multiple problem sets to their students, often as review. Thus, increase in persistence and a decrease in mastery speed in many students easily mastered the content intended for lower comparison to those in the control condition. However, students grades and thereby skewed rates of persistence and mastery speed. in the messaging conditions also exhibited consistently lower Further, the null effects found in the full sample raise important levels for measures of mindset, enjoyment of the assignment, and questions regarding the generalizability of mindset interventions perception of system helpfulness. A larger student population outside of struggling student populations. Within the context of would be required to discern a truly significant effect within these an adaptive mathematics tutor, students who appear to be at trends. ceiling in math content may not require motivational messaging, Interestingly, struggling students appeared to benefit from the and it may become detrimental to the learning process. presence of messages, showing an increase in persistence, a We also note that approximately 18.8% of students reported decrease in mastery speed, and slightly increased measures of the having technical difficulties and were removed prior to analysis. growth mindset. It can be argued that struggling students, or The incompatibility of simple HTML files serves as a reminder those facing a challenge, are most in need of motivational that many classrooms struggle to maintain up-to-date interventions, and that they are more likely to respond to technological resources. Students are often required to share messaging, regardless of condition. Motivational messages computers or iPads that come equipped with outdated software produced distinctly higher adoption of the growth mindset in and generally slow internet connections. Future research should struggling students who experienced the static image with audio incorporate allowance for these issues within the experimental condition. Thus when designing motivational content for design, as incompatibilities may lead to selection bias. It is also difficult to justify whether or not students learning technologies for mathematics. Journal of consistently attended to the motivational messages. As students Educational Psychology. 105, 4, 957-969. were simply presented the messages and were not asked to [2] Baker, R., Walonoski, J., Heffernan, N., Roll, I., Corbett, A. respond in any manner, the levels of message internalization may & Koedinger, K. 2008. Why students engage in "Gaming the be broad. We also note that the duration of the intervention may System" behavior in interactive learning environments. have been too short to observe reliable differences among Journal of Interactive Learning Research. 19, 2, 185-224. messaging conditions. In much of her work, Dweck has provided [3] Bernacki, M. L., Nokes-Malach, T. J., & Aleven, V. 2013. longer interventions upfront, coupled with ‘reminders’ such as the Fine-grained assessment of motivation over long periods of messages used in the current study [7]. Further, her studies often learning with an intelligent tutoring system: Methodology, run longitudinally across the course of a school year or more. advantages, and preliminary results. In International Still, regardless of condition, the majority of students in our handbook of metacognition and learning technologies. sample exhibited the growth mindset. Future research should Springer New York. 629-644. include a pretest mindset survey to determine if these results can [4] Clark, R.C. & Mayer, R. E. 2003. e-Learning and the science be credited solely to the motivational messages provided of instruction: proven guidelines for consumers and throughout the learning experience. designers of multimedia learning. San Francisco, CA: Finally, it should be noted that researchers relied on the Pfeiffer. tutoring system to perform random assignment. While prior [5] Dweck, C.S. 2002. Messages that motivate: how praise research has suggested that this practice is sound, assignment for molds students’ beliefs, motivation, and performance (in this study appears to have favored the static image with audio surprising ways). Improving Academic Achievement: Impact condition. Future research using ASSISTments should take this of Psychological Factors in Education. Ed. Joshua Aronson. bias into consideration. New York. Future iterations of this study should focus on struggling [6] Dweck, C.S. 2006. Mindset: The new psychology of success. students, or those undertaking challenging academic tasks. Future Random House. research should also seek to assess these conditions in an even [7] Dweck, C.S. 2013. Mindsets: Helping Students Fulfill Their more adaptive environment. It seems as though students were not Potential. Smith College Lecture Series, North Hampton, reaping the benefits of the "persona effect" found in prior research MA. September 19. [1], due to a lack of bonding with the agent. A truly adaptive [8] Graesser, A., Chipman, P., King, B., McDaniel, B., & agent, one consistently present and building rapport, may be more D'Mello, S. 2007. Emotions and learning with autotutor. effective in message delivery. Rather than repeating the same Frontiers in Artificial Intelligence Applications, 158, 569. select set of general and incorrect attributions, struggling students [9] Kapoor, A., Burleson, W., & Picard, R. W. 2007. Automatic may require motivational messages linked with the tutor content prediction of frustration. International Journal of Human- and their progress. Perhaps just as a pedagogical agent, these Computer Studies. 65, 724-736. messages must be fine-tuned to a student’s cognitive and affective [10] Mayer, R.E. 2014. Incorporating motivation into multimedia states. Alternative message delivery methods, including video learning. Learning and Instruction. Volume 29, 171-173. feedback with human tutors used as hints, scaffolding, and [11] Mueller, C. & Dweck, C. 1998. Praise for Intelligence Can misconception messages, should also be considered in future Undermine Children's Motivation and Performance. Journal research. of Personality and Social Psychology, Vol. 75, No. 1, 33-52. [12] Ostrow, K.S. 2013. Motivational Message Study. Accessed 6. ACKNOWLEDGEMENTS 12/12/2013. Student Experience, RCT & All Data: We acknowledge funding from NSF (#1316736, 1252297, https://sites.google.com/site/korinnostrow/research 1109483, 1031398, 0742503), ONR's “STEM Grand Challenges,” [13] Reeve, J. 2009. Understanding motivation and emotion. (5th and IES (#R305A120125, R305C100024). Thanks to S. & L.O. ed.). Hoboken, NJ: Wiley. 7. REFERENCES [1] Arroyo, I., Burleson, W., Tai, M., Muldner, K., Woolf, B.P. 2013. Gender differences in the use and benefit of advanced