The Nature of Achievement Goal Motivation Profiles: Exploring Situational Motivation in An Algebra-Focused Intelligent Tutoring System

Abstract

Building on recent work related to measuring situational, in-themoment motivation and the stability of motivation profiles, this study explores the nature of situational motivation profiles constructed with measurements of achievement goals during middle and high school students' algebra-focused intelligent tutoring system (ITS) learning during an academic semester. The results of multi-level profile analyses nesting multiple timepoints within students indicates the presence of four distinct profiles, with similar characteristics to those found in previous studies on dispositional achievement goals in mathematics for similar-aged students. Present findings have potential implications for designing effective motivation interventions during ITS learning.

Keywords

• Achievement goals
• intelligent tutoring system
• multilevel profile analysis

-

2 Carnegie Learning

1. INTRODUCTION

1.1 Background

Measurement of motivation constructs in education has been rightly criticized for over-reliance on student self-report measures [12] and treating motivation as a static process during student learning (i.e., pre/post). Schunk & DiBenedetto [ 19 ] and others [ 4, 10 ] suggest that technological and measurement advances offer a much-needed opportunity to understand how motivation and selfregulation under a social cognitive framework [ 3 ] function across time, context, and task. Although some researchers have attempted to address some of these noted limitations by measuring motivation processes more precisely (e.g., fine-grained at task and domain level [ 5 ]) and dynamically [ 3 ], additional work is needed [ 17 ]. Furthermore, more recently, researchers have also attempted to distinguish how dispositional motivation (i.e., person-level) and Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). situational (i.e., in-the-moment, [ 8 ], [ 9 ]) motivation differentially impact student learning outcomes.

In the present study, we examined archived situational motivational data to generate motivation profiles. Specifically, adaptive and maladaptive achievement goal profiles were generated in order to potentially predict where students disengage during ITS math learning. The ultimate aim of our broader research agenda is to explore where adaptive and maladaptive motivational patterns emerge during in-the-moment math learning, how these patterns influence student behavior, and perhaps most importantly, discern if adaptive and maladaptive motivational profiles can pinpoint where students disengage with learning so that interventions can be implemented (by teachers or tutors) before problems arise.

1.2 Current Study

The current study seeks to explore the nature of situational achievement goal profiles that emerge across an academic year as students employ an algebra-focused ITS in the classroom.

2. METHOD

2.1 Data Source

The study presents a secondary analysis of a dataset collected in an algebra-focused online intelligent tutoring system, Cognitive Tutor [ 18 ] that was made available to the first two authors through the Carnegie Mellon University DataShop [ 7 ] as part of all authors’ participation in Learning Data Institute (LDI) collaborations (https://sites.google.com/view/learnerdatainstitute). Data were collected across an academic year from middle and high schoolers in a school district in the Northeast United States. At the end of every unit in the ITS, students completed a short survey that alternated in content between self-efficacy and achievement goal items. These items were worded so that they referenced each unit, making them situational in nature as opposed to dispositional (i.e., trait-focused).

2.2 Participants

Participants were 355 middle and high school students enrolled in a suburban school district in western Pennsylvania. These students were taking pre-algebra, algebra and geometry courses and used the ITS in the classroom. The student population was primarily White (97%) and closely balanced in terms of sex. Specific student-level demographics were not available.

2.3 Measure

Students’ achievement goals were assessed using an adapted subset of items from the Achievement Goals Questionnaire - Revised (AGQ-R; [11]). Only the three items from each of the mastery approach (MAP), performance approach (PAP), and performance avoidance (PAV) subscales were used. The items were worded in terms of the algebra unit, such as “In this unit, my goal is to learn as much as possible,” as to measure situational motivation at the completion of each unit. Students responded using a 7-point Likerttype scale from 1 (not at all true of me) to 7 (very true of me).

2.4 Analysis

As traditional latent profile analysis assumes that observations are independent of one another, multilevel latent profile analysis was implemented in Mplus version 8 [ 15 ] to identify latent profiles that best described the patterns of motivation constructs. Specifically, the survey responses recorded within the ITS at the end of the algebra units (level 1, n = 2905) were nested within students (level 2, n = 355). Models containing one through six latent profiles were estimated. Similar to Dietrich et al. [ 18 ], for the models with three or more latent profiles, the random means of the three AGQ-R subscale on the between-level were correlated with one another and a common factor approach to modeling these correlations was used to minimize computational load.

Several criteria were used to decide on the number of latent profiles. Both the Akaike’s Information Criteria (AIC; [ 1 ], [ 2 ]) and the Bayesian Information Criteria (BIC; [ 20 ]) were used with the smallest value indicating the best fitting model. The Voung-LoMendell likelihood ratio test (VLMR) and Lo-Mendell-Rubin Adjusted likelihood ratio test (LMR) were also used [14]. These tests evaluate whether a model with k latent profiles has better observed fit than a model with one less profile. A non-significant result indicates no model improvement with the additional latent profile. Entropy, an indicator of classification certainty, was also considered, with values closer to 1 indicating better distinction of profiles [ 16 ].

3. RESULTS

3.1 Descriptive Statistics

Due to the differences in time to completion for units and variability in classroom usage time for the ITS, there was variability in the number of times that each student completed the surveys. Only students with at least 1 complete set of AGQ-R scores were included, resulting in 329 students retained in the sample (minimum number of attempts = 1, maximum number of attempts = 25, mean number of attempts = 8.19, median mean number of attempts = 8).

3.2 Multilevel Profile Analysis Results

While the information-based fit indices and the BLRT indicated that an increasing number of profiles was best, the VLMR LRT and the LMR LRT results indicated that the 4-profile model was best (see Table 2). As models beyond the 5 profiles contained multiple latent classes with less than 5% of the sample, this also supported the use of the 4-class model.

As visible in Figure 1, the three AGQ-R subscale means for each profile, when considered together, create distinguishable profiles.

Statistic

Mean SD Q1 Q3

Median

Skew Kurtosis

Statistic

Entropy

AIC

BIC

SABIC Adj. LMR Test Stat Adj. LMR

df Adj. LMR p-value VLMR Test Stat VLMR df VLMR p-value Profile 4, which had the largest membership at 45.7%, had the highest MAP and PAP means across all profiles. We might label this profile as the “very high approach” profile. Students in Profile 2 had the next highest MAP and PAP subscale score means, their PAV scores had a similar mean, and the means were similar to the means of the overall subscale scores for the entire sample. Hence, we might label this profile as the “average motivation” profile. Note: The 2-profile model has 4 degrees of freedom, instead of the 5 like the other models, because there is no correlation between estimated between the latent class means in this model.

MAP 17.40 4.23 15 19 21 -1.15 .88 2

Profiles 3 Profiles

0.90 46475 46534 46503 4544 4 4 0.12 4687 0.11 0.96 42757 42853 42802 2038 5 5 0.21 2089 0.20 PAP 16.84 4.58 14 18 21 -1.04 .58

Model

4 Profiles 0.92 41500 41626 41559 1236 5 5 0.02 1267 0.02

PAV 16.86 4.86 13 18 21 -1.02 .40

5 Profiles 0.93 40734 40889 40806 758 5 0.44 777 5 0.44 Similarly, students in Profile 3 had similar means across all 3 subscales, and these means were somewhat below the means of each subscale. So, we might call this profile the “below average” profile. Lastly, the smallest profile, Profile 1, had the lowest means of all profiles, but their MAP scores were higher than their PAP and PAV scores. So, they might be labeled the “low MAP - lower performance” profile.

4. DISCUSSION

4.1 Implications

Several characteristics found in the multi-level-based profiles in this study were also observed in a single timepoint study focusing on dispositional math motivation in a sample of urban grade 7-12 students in the United States [13]. Specifically, the mastery scores were highest in the amotivated group (similar to Profile 1 in this study) and the prevalence of multiple approach goals being simultaneously endorsed in those with higher motivation. A review of literature yielded little insight into the nature of these findings with relation to an ITS context.

4.2 Limitations

Two limitations for this study stem from using existing data that was collected in a naturalistic intelligent tutoring setting in one district. As district teachers select the order of course content and choose which modules that the students complete in the ITS, not all of the students were completing the same module during the same time of year. Additionally, since the system is focused on mastery, the time-to-completion that each student takes in each unit differs. These differences in classrooms likely lead to differences in results rather than if the data were collected on content covered in the same order.

Additionally, the students in the district were rather homogeneous with regards to ethnicity. Lastly, only three of the four subscales of the AGQ-R were used in profile construction, as items on the mastery avoidance subscale were not included.

4.3 Future Research

Future research in this area should include dispositional motivation profiles, using start of year and end of year motivation profiles, to assess the relationships between dispositional and situational profiles and metacognitive behaviors, such as glossary usage and hint-seeking, within the ITS. Additionally, including other motivation constructs, such as self-efficacy as in Bernacki et. al [ 6 ] and achievement emotions, could provide more unique profiles that lead to better understanding of students’ overall motivation when working with the algebra ITS. Lastly, a larger sample size and more standardized measurement timepoints could potentially allow for the use of other techniques, such as latent transition analyses.

4.4 Conclusion

Despite some noted limitations, results from the present study offer a promising step in the evolution of understanding how student motivation profiles impact choices and behavior during ITS learning. As noted, adding additional motivation constructs (e.g., self-efficacy, emotions) can improve efficacy as teacher and tutor intervention strategies are designed. Present results are important as a review of the literature yielded no studies utilizing latent profile analysis while students were engaged with ITS math learning.

5. ACKNOWLEDGEMENT

This research was sponsored by the National Science Foundation under the awards The Learner Data Institute (award# 1934745). The opinions, findings, and results are solely the authors and do not reflect those of the funding agency.

References

[1] Akaike , H. 1973 . Information theory and an extension of the maximum likelihood principle . In B. N. Petrov & F. Caski (Eds.), Proceedings of the Second International Symposium on Information Theory (pp. 267 - 281 ). Budapest: Akademiai Kiado.

[2] Akaike , H. 1987 . Factor analysis and AIC . Psychometrika. 52 , 317 - 332 .

[3] Bandura , A. 2001 . Social cognitive theory: An agentic perspective . Annual review of psychology. 52 , 1 , 1 - 26 . DOI= 10 .1146/annurev. psych.52.1.1

[4] Bernacki , M. , Aleven , V. , and Nokes-Malach , T. J. 2014 . Stability and change in adolescents' task-specific achievement goals and implications for learning mathematics with intelligent tutors . Computers in Human Behavior . 37 , 73 - 80 . DOI= 10 .1016/j.chb. 2014 . 04 .009

[5] Bernacki M.L. , Nokes-Malach T.J. , Aleven V. 2013 . Finegrained assessment of motivation over long periods of learning with an intelligent tutoring system: Methodology, advantages, and preliminary results . In: Azevedo R., Aleven V . (eds) International Handbook of Metacognition and Learning Technologies. Springer International Handbooks of Education , vol 28 . Springer, New York, NY. DOI= 10 .1007/978-1- 4419 -5546-3_ 41

[6] Bernacki , M. , Nokes-Malach , T. J. , & Aleven , V. 2015 . Examining self-efficacy during learning: variability and relations to behavior, performance, and learning . Metacognition and Learning . 10 , 1 , 99 - 117 . DOI= 10 .1007/s11409-014-9127-x

[7] Bernacki , M. , & Ritter , S. ( 2013 ). Hopewell 2011-2012 . Dataset 613 in DataShop . Retrieved from https://pslcdatashop.web.cmu.edu/DatasetInfo?datasetId= 613

[8] Dietrich , J. , Moeller , J. , Guo , J. , Viljaranta , J. , and Kracke , B. 2019 . In-the-Moment profiles of expectancies, task values, and costs . Frontiers in Psychology. 10 , 1662. DOI= 10 .3389/fpsyg. 2019 .01662

[9] Dietrich , J. , Viljaranta , J. , Moeller , J. , and Kracke , B. 2017 . Situational expectancies and task values: Associations with

[10] Dumas , D. , McNeish , D. , & Greene , J. A. 2020 . Dynamic measurement: A theoretical-psychometric paradigm for modern educational psychology . Educational Psychologist . 55 , 2 , 88 - 105 . DOI= 10 .1080/00461520. 2020 . 1744150 students' effort. Learning and Instruction . 47 , 53 - 64 . DOI= 10 .1016/j.learninstruc. 2016 . 10 .009

[15] Muthén , L. K. , and Muthén , B. O. 2017 . Mplus: Statistical Analysis with Latent Variables: User's Guide (Version 8) . Los Angeles, CA: Authors. Retrieved from https://www.statmodel.com/download/usersguide/MplusUserGuid eVer_8.pdf

[16] Nagin , D.S. 2005 . Group-based modeling of development . Harvard: Harvard University Press.

[17] Perez , T. , Cromley , J. G. , & Kaplan , A. 2014 . The role of identity development, values, and costs in college STEM retention . Journal of Educational Psychology . 106 , 1 , 315 - 329 . DOI= 10 .1037/a0034027

[18] Ritter , S. , Anderson , J.R. , Koedinger , K.R. , & Corbett , A. 2007 . The Cognitive Tutor: Applied research in mathematics education . Psychonomics Bulletin & Review. 14 , 2 , 249 - 255 .

[19] Schunk , D. H. , & DiBenedetto , M. K. 2020 . Motivation and social cognitive theory . Contemporary Educational Psychology , 60 , 101832 . DOI= 10 .1016/j.cedpsych. 2019 .101832

[20] Schwarz , G. 1978 . Estimating the dimension of a model . Annals of Statistics . 6 , 461 - 464 . DOI= 10 .1214/aos/1176344136