The Impact of a Personalization Intervention for Mathematics on Learning and Non-Cognitive Factors

Abstract

Personalization of learning environments to the background characteristics of learners, including non-cognitive factors, has become increasingly popular with the rise of advanced technology systems. We discuss an intervention within the Cognitive Tutor ITS where mathematics problems were personalized to the out-ofschool interests of students in topic areas such as sports, music, and movies. We found that relative to a control group receiving normal problems, personalization had benefits for interest and learning measures. However, personalization that included deeper connections to students' interests seemed to be more effective than surface-level personalization. Personalization; interest; mathematics; intelligent tutoring systems

1. INTRODUCTION

The question of how to enhance the interest and motivation of adolescents has gained increasing prominence [1] especially in secondary mathematics [ 2 ]. Students often find mathematics, especially the math in middle and high school, to be disconnected from their interests, everyday lives, and typical ways of thinking about relationships and quantities [ 3 ]. At the same time, young people are using increasingly sophisticated and technology-driven ways to pursue and learn about their non-academic interests, and have become accustomed to a high level of customization, interaction, and control when seeking knowledge [ 4 ].

As a result, the idea of designing and advancing highly personalized systems for student learning has become a central focus for educational stakeholders [ 5 ]. Technology systems that enact personalized learning in the classroom have the potential to intelligently adapt to students’ prior knowledge, interests, preferences, and goals [ 4 ]. In mathematics, these systems can make explicit connections between the interests students pursue outside of school – like sports, video games, or social networking – and the academic concepts they are learning. Algebra in particular is a rich space for such connections to be made [ 6 ] – students experience mathematical concepts like rate of change as they gain points in their favorite video game, track their pace in cross country, or accumulate followers on Instagram. As Algebra is often considered to be a gatekeeper to higher-level mathematics [ 7 ], and a subject that adolescents struggle to see as relevant [ 3 ], it may be a particularly important area for the development of interventions for personalized learning. We posit that 1) using a technology-based system for personalization that grounds algebra problems in students’ out-of-school interests has the potential to elicit students’ interest in the mathematics content to be learned, and 2) that personalization to well-developed individual interests can have a long-term effect on students’ learning of algebraic concepts and their motivation to learn mathematics.

2. THEORETICAL FRAMEWORK

Interest has been defined as being both the state of engaging and the predisposition to re-engage with particular activities, events, and ideas over time [ 8 ]. Researchers have defined two types of interest. Situational interest is a state of heightened attention and increased engagement elicited by elements of an environment that are surprising, salient, evocative, or personally relevant. Situational interest can be triggered in response to stimuli, and becomes maintained over time as a learner engages further with the stimuli [ 8 ]. Individual interest is an enduring preference for certain objects or activities that persists over time and involves knowledge, value, and enjoyment; individual interest can be emerging or well-developed.

Situational interest can also be subdivided into interest based on enjoyment of the activity and interest based on valuing of the activity with respect to other things the learner values. Valuebased situational interest has also been referred to as utility value – a learner’s awareness of the usefulness of a topic to their life and goals [ 9 ]. Interventions that are intended to trigger students’ situational interest are sometimes called “catch” interventions – the idea is to immediately grab students’ attention through salient, evocative, relevant, or surprising characteristics of the instructional materials. Interventions that are designed to promote maintained situational interested as sometimes called “hold” interventions – they often reveal the value of the content to students’ lives and goals, seeking to empower students [ 10-12 ]. For example, Mitchell [ 4 ] proposed that activities involving group work, computers, and puzzles function as “catch” mechanisms in the secondary mathematics classroom, while meaningfulness and involvement “hold” situational interest. Research has shown that when individuals are interested in a task or activities, they engage in more productive learning behaviors and have improved learning outcomes [e.g., 13].

An important question, then, is how to elicit and develop learners’ interests for academic content areas. Personalization is a particular kind of intervention that can be used in learning environments to accomplish this goal. Personalization interventions identify topics for which learners have emerging or well-developed individual interest, and then connect these topics to academic content topics they are learning about in school (like algebra), for which they may have a lower level of interest. For example, consider a student who has a well-developed individual interest in music, but is not interested in Algebra. In their Algebra I class, they may engage with a variety of problems and projects that explore the mathematics behind musical pieces. Over time, the connection between these two areas might support her in developing situational interest based on her enjoyment of the incorporation of music as a context and the value perceived for music-themed problems, ultimately leading to the development of individual interest in Algebra [ 14 ]. By making explicit connections to students’ interests, personalization interventions are hypothesized to trigger situational interest in the academic content being learned, which can be maintained over time and eventually develop into individual interest in that content area. Personalization can increase students’ engagement in the math task, improve their performance on personalized math tasks and future math tasks that are not personalized [ 15 ], and may even increase students’ interest in the math they now see as relevant to their personal interests. However, little research has investigated the mechanisms by which personalization promotes these learning outcomes. In this study, we test this situational interest hypothesis by monitoring students’ interest in math units via embedded selfreport surveys and examining whether personalization induces higher levels of situational interest, and whether this situational interest transforms into individual interest. Thus we test whether increased situational interest is an important mechanism through which personalization may gain its effect.

In addition to possessing enjoyment and value components, Renninger, Ewen, and Lasher [ 16 ] accentuate that interest also involves knowledge. Learners tend to possess useful prior knowledge related to their areas of interest, but this knowledge may be intuitive and informal with respect to underlying principles, making connections to concepts being learned in school (like algebra) difficult to acknowledge or articulate. In addition to possessing the potential to spur enjoyment and valuedriven reactions to an academic content area, personalization is advantageously positioned to formalize students’ intuitive prior knowledge about their interests by explicitly connecting it to a concept learned in school. For example, a learner with substantial knowledge of musical composition may have implicit understandings of the mathematical or numerical underpinnings of music, and this knowledge can potentially act as a support when they are learning formal algebra. In mathematics education, this follows a “funds of knowledge” perspective [ 17 ], which accentuates that students bring with them to the classroom powerful quantitative ways of reasoning from their home and community lives. These informal, interest-based funds of knowledge are potential strengths that can be leveraged through thoughtful instructional approaches like personalization to develop students’ algebraic knowledge. In this study, we test the funds of knowledge hypothesis by examining whether solving personalized problems that incorporate deeper features of one’s interest (e.g., mechanics of a popular video game) elicit stronger effects on learning than problems personalized based on shallower features of a learner’s interest (e.g. passing reference to a game title in a problem about snacking) or non-personalized problems. Thus we test whether increased activation of prior knowledge is an important mechanism through which personalization gains its effect. Whereas outside interests can be leveraged by personalization, initial interest in mathematics may moderate the effectiveness of personalization interventions. Durik and Harackiewicz [ 10 ] found that an intervention designed to “catch” (i.e., trigger [ 8 ]) student interest (adding colorful, vivid decorations to instructional materials) was most effective for learners with low individual interest in mathematics (IIM), but hampered learners with high IIM. Conversely, they found that an intervention designed to “hold” (i.e., maintain based on value [ 8 ]) student interest (informing students of the value of the content being learned) was beneficial for high IIM students, and detrimental for low IIM students.

In order for personalized instructional materials to successfully activate knowledge, trigger interest, and enhance perceptions of value, Walkington and Bernacki [ 14 ] identified three key features designers must consider. First is the depth of the intervention – whether the personalization draws upon surface level aspects of a learners’ interest (e.g., simply inserting familiar objects or names into an already-designed task), or whether the personalization involves deep, authentic connections to actual experiences the learner has pursuing an interest like music. Second is the grain size of the intervention – whether the personalization is targeted to the specific experiences of an individual, or to the generic experiences of an entire group. When considering grain size, it is important to remember that some topics will tend to tap into the interests of larger groups of students more than others – for example, a problem about the specifics of football may match the fine-grained interests of more ninth graders than a problem about field hockey. Use of these topics that relate to many students’ experiences may be a productive way to allow materials to be personalized at a finer grain size. Third is the ownership of the personalization – whether the students themselves take a role in generating the connections between the academic content area and their interests, or if teachers or curriculum developers control the personalization. In this study, we examined students’ interest in mathematics and algebra learning when exposed to a personalization intervention of medium grain size (i.e., personalized for local users based on interest interviews conducted at the same school in a prior year) versus a standard set of problems (i.e., broad grain size written by curriculum developers for all Algebra I students who use the curriculum). In the fourth unit of the intervention, we also varied the depth of problems by personalizing on surface or deep features of the problem to examine the effects of depth on interest and learning (i.e. the funds of knowledge hypothesis). No manipulation of problem ownership was conducted.

In the present study, we pursue the following research questions by implementing a personalization intervention for Algebra I: 1) What is the immediate impact of a personalization intervention on students’ situational interest in algebra instructional units? 2) What long-term effect does personalization have on students’ individual interest in algebra? 3) What is the impact of a personalization intervention on students’ learning of algebra concepts? 4) How does depth influence the impact of personalization on interest and learning? Based on prior work examining the effects of personalization on learning [ 15 ] and theoretical assumptions about the development of interest [ 8 ] including the situational interest hypothesis, we hypothesize that 1) Personalized problems should trigger greater situational interest in algebra units than standard problems; 2) Students completing personalized problems that incorporate out of school interests will report greater individual interest in algebra; and 3) Students who complete personalized problem solving units will achieve greater increases in their algebra performance than students completing standard problem solving units. In accordance with the funds of knowledge hypothesis, we expect 4) that students who complete problems that are personalized based on deeper features of their interest area should outperform those completing problems personalized on surface features of the problems and standard problems.

3. METHODS

3.1 Participants and Environment

Total participants included N = 152 ninth grade Algebra I students in the classes of two Algebra I teachers. Students attended a rural Northeastern school that was 96% Caucasian with 21% of students eligible for free or reduced price lunch. In 2012, 71% of students passed the state standardized test in Mathematics, which is administered in the 11th grade. The sample was 51% female. Because one teacher at the school site did not administer the pretest before students began using the Cognitive Tutor, eightythree students completed pretest, posttest and all questionnaires delivered in the CTA software and compose the primary sample for this study.

The school at which the study took place used the Cognitive Tutor Algebra (CTA) curriculum [ 18 ]. CTA is an intelligent tutoring system for Algebra I that uses model-tracing approaches to relate the students’ actions back to the domain model to provide individualized error feedback. CTA also uses knowledge-tracing approaches to track learning from one problem to the next, using this information to identify strengths and weakness in terms of production rules. CTA presents learners with algebra story problems where they must navigate tabular, graphical, and symbolic representations of functions (Figure 1). Students in schools that use CTA typically use the software 2 days per week.

4. Personalization Intervention

Before entering the first unit in CTA (Unit 1), all participants were given an interests survey where they would rate their level of interest in 10 topic areas – music, art, cell phones, food, computers, games, stores, TV, movies, and sports. Participants were then assigned to one of two main conditions: (1) a Control Condition that received the standard algebra story problems in all units in CTA including Units 1, 3, 7, and 9 covering linear equations, (2) an Experimental Condition that received versions of these same problems with the same underlying structure that were matched to the interests they indicated on the interests survey for Units 1, 3, 7, and 9 (i.e. Personalization Condition). In unit 9, we tested the funds of knowledge hypothesis by further subdividing learners in the Personalization condition to (A) a Deep Personalization condition where they received personalized problems with greater depth – i.e., the personalized problems the Deep Personalization group received in Unit 9 were written to better correspond to ways that adolescents might actually use linear functions when pursuing their interests, and were intended to draw upon “funds of knowledge” more explicitly. The remaining students were assigned to (B) a Surface Personalization Condition where they received problems that contained stories with only superficial references to their identified interests. These problems should elicit situational interest, but not draw upon knowledge about one’s interests.

In the first sample Control problem in Table 1, students must identify the relationship between dosage and weight. This relationship is grounded in a story that provides a context that likely to be of limited relevance to the student. In the Surface Personalization problem the structure of the problem remains consistent, but a topic that corresponds to the learners’ personal interests has been applied. In the Deep Personalization version, the personal interest is applied more intentionally. Like the surface-level personalization problem, The Clash of Clans problem matches students’ reported interest in games. However it is also intended to draw upon the learner’s knowledge of the game’s architecture to frame the underlying algebraic relationship to be learned in a deeply relevant context (i.e. it is actually useful to keep track of the relationship between elapsed time and how goals are accomplished, and this quantity is explicitly tracked and displayed for the player within the game interface). We consider this to be a deeper level of personalization compared to the Surface Personalization condition, as it seems less likely that despite an interest in games, a teen would care about or track exactly how frequently they consume snacks during play. Personalized problems were written based on surveys (N = 45) and interviews (N = 23) with Algebra I students at the school where they discussed their out-of-school interests.

Deep Personalization problems were written to more closely correspond to quantitative information given by students in the interviews and open-ended surveys about their out-of-school interests, including interviews with Algebra I students at the school where the study was conducted. In these interviews, students discussed how they consider rate of change as they play video games, participate in sports, track their rate of texting and battery usage on their cell phone, engage in cooking, work at parttime jobs, activities, and so on. (see [ 6 ] for a full analysis of student interviews).

Problems across the 3 conditions were written to hold constant factors like order of information given, numbers, sentence structure and length, mathematical vocabulary, readability, pronoun use, and distractor information. The personalized problems did not require that students have additional knowledge of specific numerical mathematical information in their interest area (e.g., knowing how many points a field goal is worth) – all information given was matched across problem types. All instructional units involved in the study involved linear functions. Of the core sample comprising most of our analyses, 31 participants were assigned to the Control, 34 were assigned to Surface Personalization, and 27 were assigned to Deep Personalization.

4.1 Measures

We collected the following measures from all participants:

4.1.1 Paper-Based Pre/Post Assessments

At the beginning of the school year, prior to entering the tutor, all students completed a paper-based pre-test on linear functions. The test contained 4 story problems where a linear function was described that either had a slope and intercept (2 problems) or had only a slope (2 problems). Participants first were given an x value in the linear function and asked to solve for y, then they were given a y value in the linear function and asked to solve for x. Finally, they were asked to write the linear function using algebra symbols. A post-test was administered to all students around the midterm of their ninth grade year (i.e., four months later). The post-test contained 4 matched items containing slightly different wording and numbers. Students’ responses to each part of each problem were scored as correct or incorrect.

4.1.2 Domain-Level Motivational Surveys

Prior to entering Unit 1 (pre-) and Unit 10 (post-) in CTA, the software presented students with a survey asking them to rate their attitudes about algebra. Specifically, they rated their individual interest in mathematics (IIM), as well as their maintained situational interest–enjoyment and maintained situational interestvalue for mathematics. Subscales were adopted from a larger set of scales from Linnenbrink-Garcia et al. [ 19 ]. Sample items for each scale appear in Table 2.

4.1.3 Unit-Level Motivational Surveys

After each unit impacted by the personalization intervention (Figure 2; Units 1, 3, 7, and 9), participants were also given a unit-level motivational survey that assessed the degree to which that unit triggered their situational interest and maintained their situational interest in the CTA unit. These scales were adapted based on measures from Linnenbrink-Garcia et al. [ 19 ] with the math unit as the referent. Sample items for each scale appear in Table 2, as do Cronbach’s alphas for the initial administration of each survey. An overview of the survey measures and CTA units completed by participants in this study is provided in Figure 2.

5.1 What is the impact of personalization on students’ situational interest in algebra units?

To assess the effect of the personalization interventions on students’ situational interest, we conducted a series of analyses of covariance examining students’ reported triggered and maintained interest in CTA units. All students were given unit-level surveys assessing their level of interest in the instructional unit after each of the units impacted by the personalization treatment (Units 1, 3, 7, and 9). We controlled for initial individual interest in mathematics (IIM) as indicated on the domain survey before Unit 1 (Figure 2).

Students in the two Personalization conditions (i.e., Surface Personalization and Deep Personalization are identical in Units 1, 3, and 7) consistently reported significantly higher levels of triggered situational interest than students assigned to the Control condition (Table 3; Unit 1 F(1,80) = 5.19, MSe = .96, p = .03, Unit 3 F(1,80) = 5.31, MSe = .98, p = .02; Unit 7 F(1,80) = 3.82, MSe = .91, p = .05).

Significant differences between any of the 3 groups in triggered situational interest were not obtained in Unit 9. The level of triggered situational interest reported by the Deep Personalization was consistent with prior units with the triggered interest for the Surface Personalization group was slightly lower. The Control group, however, reported greater triggered situational interest, and the inclusion of three groups (two with smaller Ns) further diminished the statistical power available to detect effects. No significant differences in maintained situational interest were found between groups on any of the four units observed, Fs < 3.73, ps = ns. Directionally, measures of maintained situational interest generally favored the personalization groups.

5.2 What effect does personalization have on students’ individual interest in algebra?

All students were given domain-level surveys assessing their interest towards learning algebra prior to the intervention and after the final personalized unit (i.e., Unit 9). A repeated measures analysis of variance examining change in Individual Interest in Mathematics (i.e., Post-Pre) between the two Personalization conditions (i.e., Deep & Surface) versus Control was conducted to examine the main effect of Time and Interaction between Time X Condition. Results indicated a significant main effect of Time, F (1, 81) = 5.39, MSe = 1.75, p = .023. Overall, students’ individual interest in mathematics declined from pretest to posttest. Analyses also indicated a marginally significant interaction between Time and Condition, F (1, 81) = 3.73, p = .057. Students in the control group significantly reduced their rating of individual interest in algebra an average of 0.37 points over the 10-unit span (Table 3; t(29) = 3.21, p < .01), while students in the Deep and Surface Personalization groups maintained their individual interest in algebra (M = 0.04 decline). Thus personalization had a positive effect in that it preserved students’ individual interest in algebra. Within the Personalization condition, no differences were found between students who received Surface versus Deep Personalization.

5.3 What is the impact of personalization on students’ learning of Algebra I concepts?

The pre- and post- test scores on the algebra learning measures for each of the three conditions is shown in Table 4. A linear regression model predicting amount of absolute gain from pre- to post-test (i.e., post-test score minus pre-test score) was fit to the data, with students’ class period as a random effect. Adding a predictor for Condition significantly improved the fit of the model (χ2(2) = 6.39, p = 0.04), as did a control variable for students’ initial level of individual interest in mathematics (IIM) prior to the intervention (χ2(1) = 4.07, p = 0.04). The interaction of Condition and IIM also significantly improved the fit of the model (χ2(2) = 14.43, p < .001). The regression output is shown in Table 5. The reference category is the Control Group, and we interpret all significant simple effects regardless of whether they are displayed in the table. The IIM control measure was dichotomized to separate students with high IIM (average rating of 3 or more) from low IIM (average rating less than 3) to aid interpretability and to be consistent with prior work [e.g., 14]. As can be seen from Table 5, for students with low individual interest in math, Deep Personalization was significantly more effective than Control (p < 0.05). Additional contrasts not shown in the table compared Surface Personalization to Deep Personalization, and found that for students with low IIM, Deep Personalization was significantly more effective than Surface Personalization (B = 0.24, SE (B) = 0.07, p < 0.001). Finally, within the Deep Personalization condition, students with high IIM gained significantly less than students with low IIM (B = .17, SE(B) = .07, p = .01).

6. DISCUSSION & CONCLUSION

This study examined whether personalizing algebra problems to students’ out-of-school interests would increase their situational interest in CTA algebra problems, increase their interest in mathematics, and improve their acquisition of algebra knowledge (i.e., the situational interest hypothesis). It additionally tested whether solving problems that incorporated deep features of an interest into problems would produce greater benefits that solving problems that incorporated interests superficially or standard problems (i.e. the funds of knowledge hypothesis). Students who received problems personalized to their out-of school interests reported significantly higher triggered situational interest for CTA math units. Compared to a Control group that experienced a drop in their individual interest in mathematics, Personalization also had a preserving effect on students’ interest in mathematics. After accounting for students’ initial individual interest in mathematics, significant differences in learning gains were found between groups of students in the Deep Personalization, Surface Personalization and Control Conditions. These findings are next discussed in light of prior theory and research.

6.1 Personalization and Situational Interest

Students who completed algebra problems personalized to their interests reported greater triggered situational interest compared to students who completed standard CTA problems, however students who solved personalized problems did not report significantly greater maintained interest resulting from enjoyment or perceptions of value. The finding that personalization was effective in triggering situational interest is encouraging as we consider the Control condition to be a considerably strong control. That is, the standard problems included in tutor units might be considered to be personalized to student interests at a very broad grain size [ 11 ] – they were generally written by teachers and curriculum writers with this student population in mind (i.e., adolescent algebra learners). The personalized problems in the intervention, on the other hand, had a medium grain size – they were written for and provided to subsets of the student population that had particular topic interests (e.g., sports, video games). The change from a large to a medium grain size was sufficient to elicit changes in triggered situational interest, though additional effort may be necessary to elicit sufficient enjoyment or perception of p .07 .18 .05 .94 .41 .09 value to maintain students’ situational interest. Indeed, in another personalization study [ 20 ], we found that a personalization intervention with a much smaller grain size where students wrote and solved problems that incorporated features of their personal interests produced increases in students’ maintained situational interest associated with perceived value. This intervention also involved a higher level of ownership of the personalization on the part of the students [ 14 ], which suggests that personalization at a medium grain size may successfully trigger situational interest, but a personalization at a smaller grain size with some level of ownership may be necessary to achieve more enduring situational interest in math units. This type of intervention may be especially important given that it takes the burden of generating fine-grained instructional materials away from teachers and curriculum developers and places it on students.

6.2 Personalization and Individual Interest

Despite a failure to elicit maintained situational interest, the Personalization intervention did have a significant effect on students’ individual interest in mathematics. Importantly, the individual interest items assessed how students felt about the domain of mathematics as a whole, rather than how they felt about the particular math class they were enrolled in or the particular units they were working on. This preservation of individual interest in algebra over half a year of high school coursework is a desirable outcome, given research that documents declines in interest in math over adolescence [ 21, 22 ]. In sum, the findings from the first two research questions support the situational interest hypothesis. We consider this finding in light of theory on interest development in section 6.4.

6.3 Deep Personalization and Algebra Learning

Walkington [ 12 ] found that a one-unit personalization intervention improved students’ long-term learning of algebra concepts within the CTA environment, relative to a control condition. This study extends that work and indicates that, when personalization incorporates deep features of students’ out-ofschool interests, it can also induce learning gains that transfer outside of an intelligent tutoring environment (i.e. to delayed, paper-based tests). However, these effects are moderated by students’ initial level of individual interest in mathematics, with Deep Personalization being beneficial mainly for low IIM students. Walkington [ 15 ] did not collect such interest measures in her study, but did find that personalization was most effective for students who were making slower progress through CTA– a variable known to track closely with interest in math [ 23 ]. We consider these findings in light of proposed hypotheses that personalization may obtain effects on learning by activating students’ funds of knowledge in their out-of-school interest, and that personalization may trigger greater situational interest in math tasks. The current study showed that Deep Personalization was significantly less effective for learners with high IIM, compared to learners with low IIM. This, along with the results that personalization triggers but does not maintain situational interest, suggests that even Deep Personalization may achieve its effects on learning as a “catch” intervention, immediately eliciting triggered situational interest. That is, solving personalized problems triggered students’ interests, but did not maintain them. This provides some promise as prior research has shown catch interventions that trigger interest to be beneficial primarily for learners with low IIM [ 10 ]. This is contrasted with a “hold” intervention that maintains situational interest, often by communicating the value of the content being learned. In this study personalization did not increase students’ perceptions that algebra problems had value, but additional interventions aimed at boosting perceived value and relevance [ 11, 12 ] could potentially be incorporated to ITSs to also obtain this effect and its benefits for learning.

Although we termed our Condition “Deep” Personalization, the connections made to learners’ actual experiences may not have been uniformly deep depending on students more specific interests within a topic area, and thus may not have elicited valuebased reactions from some students. This stems from issues with the grain size of the intervention – students merely indicated their level of interest in a broad topic (e.g., “sports”), and were then given problems that could cover the entire space of activities that fell within that topic (e.g., basketball, hockey, football), without considering students more specific interest in a subtopic (e.g., just hockey). Although attempts were made to use the “high-leverage” interest sub-topics that many students would have specific knowledge of (i.e., football rather than field hockey) this approach likely allowed for the personalization to have highly variable level of correspondence to students’ exact interests. The level of correspondence depended on the overlap between a student’s interest and the commonly reported interests by peers in surveys and interviews prior to problem development. Walkington and Bernacki [ 20 ] found significant increases in maintained situational interest (value) for students who authored problems about their specific interests, suggesting that the smaller grain size and increased ownership of the personalization intervention in that study allowed it to function more as a “hold” intervention. Finally, the current study showed that Deep Personalization was significantly more effective than Surface Personalization for students with low IIM. This suggested that personalization may need to have at least a moderate level of depth for it to be effective at all for supporting learning outcomes for any subgroup of students. Indeed, a number of recent personalization interventions that employed relatively surface-level personalization have reported null findings [ 24, 25 ]. Thus we conclude from all of these analyses that a personalization intervention with a moderate depth and grain size can potentially have long-term effects on student learning for students who begin with limited interest in mathematics. However, increasing depth and personalizing at an even smaller grain size may have more powerful effects, especially for students with higher IIM for whom value-based connections may be most critical.

Although learning gains were produced for low IIM students who received Deep Personalization (rather than Surface Personalization), these students did not show differences in situational or individual interest measures within Unit 9 compared to the Surface Personalization group. There were also no differences between Surface and Deep in individual interest over the course of the entire intervention. This suggests that Deep Personalization may gain its effectiveness over Surface Personalization by connecting to students’ prior knowledge (funds of knowledge hypothesis) rather than triggering and maintaining differing levels of situational interest (situational interest hypothesis). However, ultimately comparisons between these two groups are of limited usefulness given the relatively small sample sizes. Thus we find limited but promising support for the funds of knowledge hypothesis.

6.4 Theoretical Implications

When viewed through the lens of interest development theory [ 8 ], the findings regarding personalization and interest development are somewhat puzzling. Per Hidi and Renninger’s [ 8 ] theory, interest is 1) triggered by environmental stimuli and 2) maintained when engagement in the environment is enjoyable or confers value through consistent or repeated situational interest. This supports 3) the emergence of an individual interest, which 4) becomes well developed over time. In this study, analyses reveal a triggering of situational interest among students in the Surface and Deep Personalization conditions, no reported maintenance of situational interest via enjoyment or value, but a significant effect of Personalization on individual interest. Thus individual interest developed without being maintained during learning; this requires that we consider alternate explanations by which such effects on individual interest may have been obtained.

One potential explanation is that the way instructors used Cognitive Tutor in the math classes may have reproduced some of the behaviors expected when students’ situational interest is maintained. In their model, Hidi and Renninger [ 8 ] describe that those who maintain interest in a topic tend to repeatedly engage with content involving the topic (e.g., a student who is interest in dolphins may seek more opportunities to learn about them by reading books about them in school or choose “dolphins” as a topic for school assignments). While students’ did not report that personalized Cognitive Tutor Algebra units maintained their interest to a degree that we would expect them to voluntarily seek out opportunities to learn using Cognitive Tutor, the compulsory use of the Cognitive Tutor in math class twice a week for many months effectively ensured repeated engagement in (personalized) problem solving via CTA use. Thus we could conclude that the continued exposure to math content personalized to one’s out-ofschool interests approximated behavioral outcomes of maintained situational interest and created an alternate pathway by which individual interest was preserved in Personalization conditions (i.e., no drop in interest), but not in the Control condition where there was no initially triggered interest. Much like the typical adolescent whose interest in math declines over time, students in the Control condition were required to complete math units that did not trigger situational interest and subsequently reported declines in their interest in mathematics.

6.5 Conclusion

The results obtained in this study provide important insight about the ways depth and grain size of personalization may impact the development of students’ interests in their math course, the domain of mathematics, and ultimately their long-term learning of algebra concepts. In future analyses, we will analyze additional data from students participating in this study, and look for difference in in behavior and performance within intervention and subsequent CTA units, including analyses of learning behaviors using log-files and automated detectors.

7. ACKNOWLEDGMENTS

Both authors contributed equally to this manuscript. The authors thank Steve Ritter, Susan Berman, Tristan Nixon and Steve Fancsali (Carnegie Learning), Gail Kusbit (Carnegie Mellon University & LearnLab) and participating teachers. Funding for the study was provided by a subgrant of National Science Foundation Award # SBE-0354420. Additional funding as provided by IES Award # R305B100007.

8. REFERENCES

[1] Hidi, S., & Harackiewicz, J. (2000). Motivating the academically unmotivated: A critical issue for the 21st century. Review of Educational Research, 70, 151–179.

References

[2] Mitchell, M. ( 1993 ). Situational interest: Its multifaceted structure in the secondary school mathematics classroom . Journal of Educational Psychology , 85 , 424 - 436 .

[3] McCoy L. P. ( 2005 ). Effect of demographic and personal variables on achievement in eighth-grade algebra . Journal of Educational Research , 98 ( 3 ), 131 - 135 .

[4] Collins , A. , & Halverson , R. ( 2009 ). Rethinking Education in the Age of Technology: The Digital Revolution and Schooling in America . New York: Teachers College Press.

[5] U.S. Department of Education, Office of Educational Technology, Transforming American Education: Learning Powered by Technology , Washington, D.C., 2010 . http://www.ed.gov/sites/default/files/netp2010.pdf

[6] Walkington , C. , Sherman , M. , & Howell , E. (in press). Connecting Algebra to sports, video games, and social networking: How personalized learning makes ideas “stick .” Mathematics Teacher.

[7] Moses , R. , & Cobb , C. ( 2001 ). Radical Equations: Math Literacy and Civil Rights . Boston: Beacon Press.

[8] Hidi , S. , & Renninger , K. ( 2006 ). The four-phase model of interest development . Educational Psychologist , 41 ( 2 ), 111 - 127 .

[9] Eccles , J. ( 1983 ). Expectancies, values and academic behaviors . In R. C. Atkinson, G. Lindzey, & R. F. Thompson (Eds.), Achievement and achievement motives: Psychological and sociological approaches (pp. 75 - 146 ). San Francisco: W.H. Freeman & Co

[10] Durik , A. , & Harackiewicz , J. ( 2007 ). Different strokes for different folks: How individual interest moderates effects of situational factors on task interest . Journal of Educational Psychology , 99 ( 3 ), 597 - 610 .

[11] Hulleman , C. S. , Godes , O. , Hendricks , B. L. , & Harackiewicz , J. M. ( 2010 ). Enhancing interest and performance with a utility value intervention . Journal of Educational Psychology , 102 , 880 - 895 .

[12] Hulleman , C. S. , & Harackiewicz , J. M. ( 2009 ). Promoting interest and performance in high school science classes . Science , 326 ( 5958 ), 1410 - 1412 .

[13] Harackiewicz , J. , Durik , A. , Barron , K. Linnenbrink , E. , & Tauer , J. ( 2008 ). The role of achievement goals in the development of interest: Reciprocal relations between achievement goals, interest, and performance . Journal of Educational Psychology , 100 ( 1 ), 105 - 122 .

[14] Walkington , C. , & Bernacki , M. (in press). Motivating students by “personalizing” learning around individual interests: A consideration of theory, design, and implementation issues . In S. Karabenick & T. Urdan (eds.) Advances in Motivation and Achievement, Emerald Group Publishing.

[15] Walkington , C. ( 2013 ). Using learning technologies to personalize instruction to student interests: The impact of relevant contexts on performance and learning outcomes . Journal of Educational Psychology , 105 ( 4 ), 932 - 945 .

[16] Renninger , K. , Ewen , L. , & Lasher , A. ( 2002 ). Individual interest as context is expository text and mathematical word problems . Learning and Instruction , 12 , 467 - 491 .

[17] Civil , M. ( 2007 ). Building on community knowledge: An avenue to equity in mathematics education . In N. Nassir. and P. Cobb (Eds.) Improving access to mathematics: Diversity and equity in the classroom (pp. 105 - 117 ). Teachers College Press.

[18] Carnegie Learning ( 2013 ). Cognitive Tutor Algebra [software] . Carnegie Learning , Inc. Pittsburgh, PA, USA.

[19] Linnenbrink-Garcia , L. , Durik , A. , Conley , A. , Barron , K. , Tauer , J. , Karabenick , S. , & Harackiewicz , J. ( 2010 ). Measuring situational interest in academic domains . Educational Psychological Measurement , 70 , 647 - 671 .

[20] Walkington , C. , & Bernacki , M. ( 2014 ). Students authoring personalized “algebra stories”: Problem-posing in the context of out-of-school interests . Presentation at the 2014 Annual Meeting of American Educational Research Association.

[21] Fredricks , J. A. , & Eccles , J. ( 2002 ). Children's competence and value beliefs from childhood through adolescence: Growth trajectories in two male-sex-typed domains . Developmental Psychology , 38 , 519 - 533 .

[22] Frenzel , A. C. , Goetz , T. , Pekrun , R. , & Watt , H. M. G. ( 2010 ). Development of mathematics interest in adolescence: Influences of gender, family, and school context . Journal of Research on Adolescence , 20 , 507 - 537 .

[23] Bernacki , M. L. , Nokes-Malach , T.J. , Aleven , V. , & Glick , J. ( 2014 ). Intelligent tutoring systems promote achievement in middle school mathematics, especially for students with low interest. Presentation at the 2014 Annual Meeting of the American Educational Research Association .

[24] Bates , E. , & Wiest , L. ( 2004 ). The impact of personalization of mathematical word problems on student performance . The Mathematics Educator , 14 ( 2 ), 17 - 26 .

[25] Caker , O. , & Simsek , N. ( 2010 ). A comparative analysis of computer and paper-based personalization on student achievement . Computers & Education , 55 , 1524 - 1531 .