The Nature of Achievement Goal Motivation Profiles: Exploring Situational Motivation in An Algebra-Focused Intelligent Tutoring System Leigh Harrell-Williams1, Christian Mueller1, Stephen Fancsali2, Steven Ritter2, Xiaofei Zhang1, Deepak Venugopal1 1 University of Memphis {leigh.williams, cemuellr, xzhang12, dvngopal}@memphis.edu 2 Carnegie Learning {sfancsali, sritter}@carnegielearning.com ABSTRACT situational (i.e., in-the-moment, [8], [9]) motivation differentially Building on recent work related to measuring situational, in-the- impact student learning outcomes. moment motivation and the stability of motivation profiles, this In the present study, we examined archived situational motivational study explores the nature of situational motivation profiles data to generate motivation profiles. Specifically, adaptive and constructed with measurements of achievement goals during maladaptive achievement goal profiles were generated in order to middle and high school students’ algebra-focused intelligent potentially predict where students disengage during ITS math tutoring system (ITS) learning during an academic semester. The learning. The ultimate aim of our broader research agenda is to results of multi-level profile analyses nesting multiple timepoints explore where adaptive and maladaptive motivational patterns within students indicates the presence of four distinct profiles, with emerge during in-the-moment math learning, how these patterns similar characteristics to those found in previous studies on influence student behavior, and perhaps most importantly, discern dispositional achievement goals in mathematics for similar-aged if adaptive and maladaptive motivational profiles can pinpoint students. Present findings have potential implications for designing where students disengage with learning so that interventions can be effective motivation interventions during ITS learning. implemented (by teachers or tutors) before problems arise. Keywords 1.2 Current Study Achievement goals, intelligent tutoring system, multilevel profile The current study seeks to explore the nature of situational analysis achievement goal profiles that emerge across an academic year as students employ an algebra-focused ITS in the classroom. 1. INTRODUCTION 1.1 Background 2. METHOD Measurement of motivation constructs in education has been 2.1 Data Source rightly criticized for over-reliance on student self-report measures The study presents a secondary analysis of a dataset collected in an [12] and treating motivation as a static process during student algebra-focused online intelligent tutoring system, Cognitive Tutor learning (i.e., pre/post). Schunk & DiBenedetto [19] and others [4, [18] that was made available to the first two authors through the 10] suggest that technological and measurement advances offer a Carnegie Mellon University DataShop [7] as part of all authors’ much-needed opportunity to understand how motivation and self- participation in Learning Data Institute (LDI) collaborations regulation under a social cognitive framework [3] function across (https://sites.google.com/view/learnerdatainstitute). Data were time, context, and task. Although some researchers have attempted collected across an academic year from middle and high schoolers to address some of these noted limitations by measuring motivation in a school district in the Northeast United States. At the end of processes more precisely (e.g., fine-grained at task and domain every unit in the ITS, students completed a short survey that level [5]) and dynamically [3], additional work is needed [17]. alternated in content between self-efficacy and achievement goal Furthermore, more recently, researchers have also attempted to items. These items were worded so that they referenced each unit, distinguish how dispositional motivation (i.e., person-level) and making them situational in nature as opposed to dispositional (i.e., trait-focused). Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License 2.2 Participants Participants were 355 middle and high school students enrolled in Attribution 4.0 International (CC BY 4.0). a suburban school district in western Pennsylvania. These students were taking pre-algebra, algebra and geometry courses and used the ITS in the classroom. The student population was primarily White (97%) and closely balanced in terms of sex. Specific student-level demographics were not available. 2.3 Measure Profile 4, which had the largest membership at 45.7%, had the Students’ achievement goals were assessed using an adapted subset highest MAP and PAP means across all profiles. We might label of items from the Achievement Goals Questionnaire - Revised this profile as the “very high approach” profile. Students in Profile (AGQ-R; [11]). Only the three items from each of the mastery 2 had the next highest MAP and PAP subscale score means, their approach (MAP), performance approach (PAP), and performance PAV scores had a similar mean, and the means were similar to the avoidance (PAV) subscales were used. The items were worded in means of the overall subscale scores for the entire sample. Hence, terms of the algebra unit, such as “In this unit, my goal is to learn we might label this profile as the “average motivation” profile. as much as possible,” as to measure situational motivation at the Table 1. Descriptive Statistics for AGQ-R Subscales completion of each unit. Students responded using a 7-point Likert- type scale from 1 (not at all true of me) to 7 (very true of me). AGQ-R Subscale Statistic 2.4 Analysis MAP PAP PAV As traditional latent profile analysis assumes that observations are independent of one another, multilevel latent profile analysis was Mean 17.40 16.84 16.86 implemented in Mplus version 8 [15] to identify latent profiles that best described the patterns of motivation constructs. Specifically, SD 4.23 4.58 4.86 the survey responses recorded within the ITS at the end of the algebra units (level 1, n = 2905) were nested within students (level Q1 15 14 13 2, n = 355). Models containing one through six latent profiles were estimated. Similar to Dietrich et al. [18], for the models with three Median 19 18 18 or more latent profiles, the random means of the three AGQ-R subscale on the between-level were correlated with one another and Q3 21 21 21 a common factor approach to modeling these correlations was used to minimize computational load. Skew -1.15 -1.04 -1.02 Several criteria were used to decide on the number of latent profiles. Both the Akaike’s Information Criteria (AIC; [1], [2]) and Kurtosis .88 .58 .40 the Bayesian Information Criteria (BIC; [20]) were used with the smallest value indicating the best fitting model. The Voung-Lo- Mendell likelihood ratio test (VLMR) and Lo-Mendell-Rubin Table 2. Latent Profile Model Selection Results Adjusted likelihood ratio test (LMR) were also used [14]. These tests evaluate whether a model with k latent profiles has better Model observed fit than a model with one less profile. A non-significant result indicates no model improvement with the additional latent Statistic 2 3 4 5 profile. Entropy, an indicator of classification certainty, was also Profiles Profiles Profiles Profiles considered, with values closer to 1 indicating better distinction of profiles [16]. Entropy 0.90 0.96 0.92 0.93 3. RESULTS AIC 46475 42757 41500 40734 3.1 Descriptive Statistics Due to the differences in time to completion for units and variability BIC 46534 42853 41626 40889 in classroom usage time for the ITS, there was variability in the number of times that each student completed the surveys. Only SABIC 46503 42802 41559 40806 students with at least 1 complete set of AGQ-R scores were included, resulting in 329 students retained in the sample Adj. LMR 4544 2038 1236 758 (minimum number of attempts = 1, maximum number of attempts Test Stat = 25, mean number of attempts = 8.19, median mean number of attempts = 8). Adj. LMR 4 5 5 5 df Table 1 summarizes descriptive statistics for the three AGQ-R subscales scores averaged over all 2905 observations. The Adj. LMR 0.12 0.21 0.02 0.44 distributions for each subscale score had a negatively skewed p-value distribution with peaks at the maximum score of 21, which is noticeable via the third quartile values for each subscale at 21. VLMR 4687 2089 1267 777 Test Stat 3.2 Multilevel Profile Analysis Results While the information-based fit indices and the BLRT indicated VLMR df 4 5 5 5 that an increasing number of profiles was best, the VLMR LRT and the LMR LRT results indicated that the 4-profile model was best VLMR (see Table 2). As models beyond the 5 profiles contained multiple 0.11 0.20 0.02 0.44 p-value latent classes with less than 5% of the sample, this also supported the use of the 4-class model. Note: The 2-profile model has 4 degrees of freedom, instead of the 5 like the other models, because there is no correlation As visible in Figure 1, the three AGQ-R subscale means for each between estimated between the latent class means in this model. profile, when considered together, create distinguishable profiles. and achievement emotions, could provide more unique profiles that Similarly, students in Profile 3 had similar means across all 3 lead to better understanding of students’ overall motivation when subscales, and these means were somewhat below the means of working with the algebra ITS. Lastly, a larger sample size and more each subscale. So, we might call this profile the “below average” standardized measurement timepoints could potentially allow for profile. Lastly, the smallest profile, Profile 1, had the lowest means the use of other techniques, such as latent transition analyses. of all profiles, but their MAP scores were higher than their PAP and PAV scores. So, they might be labeled the “low MAP - lower 4.4 Conclusion performance” profile. Despite some noted limitations, results from the present study offer a promising step in the evolution of understanding how student motivation profiles impact choices and behavior during ITS learning. As noted, adding additional motivation constructs (e.g., self-efficacy, emotions) can improve efficacy as teacher and tutor intervention strategies are designed. Present results are important as a review of the literature yielded no studies utilizing latent profile analysis while students were engaged with ITS math learning. 5. ACKNOWLEDGEMENT This research was sponsored by the National Science Foundation under the awards The Learner Data Institute (award# 1934745). The opinions, findings, and results are solely the authors and do not reflect those of the funding agency. 6. REFERENCES [1] Akaike, H. 1973. Information theory and an extension of the maximum likelihood principle. In B. N. Petrov & F. Caski (Eds.), Figure 1. Plot of Means by Latent Profiles Proceedings of the Second International Symposium on Information Theory (pp. 267-281). Budapest: Akademiai Kiado. 4. DISCUSSION [2] Akaike, H. 1987. Factor analysis and AIC. Psychometrika. 52, 4.1 Implications 317-332. Several characteristics found in the multi-level-based profiles in this study were also observed in a single timepoint study focusing [3] Bandura, A. 2001. Social cognitive theory: An agentic on dispositional math motivation in a sample of urban grade 7-12 perspective. Annual review of psychology. 52,1, 1-26. DOI= students in the United States [13]. Specifically, the mastery scores 10.1146/annurev.psych.52.1.1 were highest in the amotivated group (similar to Profile 1 in this study) and the prevalence of multiple approach goals being [4] Bernacki, M., Aleven, V., and Nokes-Malach, T. J. 2014. simultaneously endorsed in those with higher motivation. A review Stability and change in adolescents’ task-specific achievement of literature yielded little insight into the nature of these findings goals and implications for learning mathematics with intelligent with relation to an ITS context. tutors. Computers in Human Behavior. 37, 73–80. DOI= 10.1016/j.chb.2014.04.009 4.2 Limitations Two limitations for this study stem from using existing data that [5] Bernacki M.L., Nokes-Malach T.J., Aleven V. 2013. Fine- was collected in a naturalistic intelligent tutoring setting in one grained assessment of motivation over long periods of learning district. As district teachers select the order of course content and with an intelligent tutoring system: Methodology, advantages, and choose which modules that the students complete in the ITS, not all preliminary results. In: Azevedo R., Aleven V. (eds) International of the students were completing the same module during the same Handbook of Metacognition and Learning Technologies. Springer time of year. Additionally, since the system is focused on mastery, International Handbooks of Education, vol 28. Springer, New the time-to-completion that each student takes in each unit differs. York, NY. DOI=10.1007/978-1-4419-5546-3_41 These differences in classrooms likely lead to differences in results rather than if the data were collected on content covered in the same [6] Bernacki, M., Nokes-Malach, T. J., & Aleven, V. 2015. order. Examining self-efficacy during learning: variability and relations to behavior, performance, and learning. Metacognition and Additionally, the students in the district were rather homogeneous Learning. 10, 1, 99–117. DOI= 10.1007/s11409-014-9127-x with regards to ethnicity. Lastly, only three of the four subscales of the AGQ-R were used in profile construction, as items on the [7] Bernacki, M., & Ritter, S. (2013). Hopewell 2011-2012. Dataset mastery avoidance subscale were not included. 613 in DataShop. Retrieved from https://pslcdatashop.web.cmu.edu/DatasetInfo?datasetId=613 4.3 Future Research Future research in this area should include dispositional motivation [8] Dietrich, J., Moeller, J., Guo, J., Viljaranta, J., and Kracke, B. profiles, using start of year and end of year motivation profiles, to 2019. In-the-Moment profiles of expectancies, task values, and assess the relationships between dispositional and situational costs. Frontiers in Psychology. 10, 1662. DOI= profiles and metacognitive behaviors, such as glossary usage and 10.3389/fpsyg.2019.01662 hint-seeking, within the ITS. Additionally, including other motivation constructs, such as self-efficacy as in Bernacki et. al [6] [9] Dietrich, J., Viljaranta, J., Moeller, J., and Kracke, B. 2017. students’ effort. Learning and Instruction. 47, 53–64. DOI= Situational expectancies and task values: Associations with 10.1016/j.learninstruc.2016.10.009 [15] Muthén, L. K., and Muthén, B. O. 2017. Mplus: Statistical [10] Dumas, D., McNeish, D., & Greene, J. A. 2020. Dynamic Analysis with Latent Variables: User’s Guide (Version 8). Los measurement: A theoretical–psychometric paradigm for modern Angeles, CA: Authors. Retrieved from educational psychology. Educational Psychologist. 55, 2, 88-105. https://www.statmodel.com/download/usersguide/MplusUserGuid DOI=10.1080/00461520.2020.1744150 eVer_8.pdf [11] Elliot, A.J., & Murayama, K. 2008. On the measurement of [16] Nagin, D.S. 2005. Group-based modeling of development. achievement goals: critique, illustration, and Harvard: Harvard University Press. application. Journal of Educational Psychology. 100, 3, 613-628. DOI=10.1037/0022-0663.100.3.613 [17] Perez, T., Cromley, J. G., & Kaplan, A. 2014. The role of identity development, values, and costs in college STEM retention. [12] Fancsali, S., Bernacki, M., Nokes-Malach, T., Yudelson, M., Journal of Educational Psychology. 106, 1, 315–329. DOI= & Ritter, S. 2014. Goal orientation, self-efficacy, and “online 10.1037/a0034027 measures” in intelligent tutoring systems. In Proceedings of the Annual Meeting of the Cognitive Science Society, 36. Retrieved [18] Ritter, S., Anderson, J.R., Koedinger, K.R., & Corbett, A. from https://escholarship.org/content/qt0p53v46r/qt0p53v46r 2007. The Cognitive Tutor: Applied research in mathematics education. Psychonomics Bulletin & Review. 14, 2, 249-255. [13] Holden, S. M., Harrell-Williams, L. M., Mueller, C. E., Ford, J. M., and Jones, M. H. Comparison of motivational latent profiles [19] Schunk, D. H., & DiBenedetto, M. K. 2020. Motivation and using the PALS and AGQ-R. Manuscript in revision. social cognitive theory. Contemporary Educational Psychology, 60, 101832. DOI=10.1016/j.cedpsych.2019.101832 [14] Muthén, B., and Asparouhov, T. 2012. Using Mplus TECH11 and TECH14 to test the number of latent classes. Retrieved from [20] Schwarz, G. 1978. Estimating the dimension of a model. http://statmodel2.com/examples/webnotes/webnote14.pdf Annals of Statistics. 6, 461-464. DOI=10.1214/aos/1176344136