=Paper=
{{Paper
|id=Vol-1183/ncfpal_paper02
|storemode=property
|title=The Impact of a Personalization Intervention for Mathematics on Learning and Non-Cognitive Factors
|pdfUrl=https://ceur-ws.org/Vol-1183/ncfpal_paper02.pdf
|volume=Vol-1183
|dblpUrl=https://dblp.org/rec/conf/edm/BernackiW14
}}
==The Impact of a Personalization Intervention for Mathematics on Learning and Non-Cognitive Factors==
https://ceur-ws.org/Vol-1183/ncfpal_paper02.pdf
The Impact of a Personalization Intervention for
Mathematics on Learning and Non-Cognitive Factors
Matthew Bernacki Candace Walkington
University of Nevada, Las Vegas Southern Methodist University
4505 S. Maryland Parkway 3011 University Blvd. Ste. 345
Las Vegas, NV 89012, USA Dallas, TX, 75205, USA
1-702-895-4013 1-214-768-3072
matt.bernacki@unlv.edu cwalkington@smu.edu
ABSTRACT 2. THEORETICAL FRAMEWORK
Personalization of learning environments to the background Interest has been defined as being both the state of engaging and
characteristics of learners, including non-cognitive factors, has the predisposition to re-engage with particular activities, events,
become increasingly popular with the rise of advanced technology and ideas over time [8]. Researchers have defined two types of
systems. We discuss an intervention within the Cognitive Tutor interest. Situational interest is a state of heightened attention and
ITS where mathematics problems were personalized to the out-of- increased engagement elicited by elements of an environment that
school interests of students in topic areas such as sports, music, are surprising, salient, evocative, or personally relevant.
and movies. We found that relative to a control group receiving Situational interest can be triggered in response to stimuli, and
normal problems, personalization had benefits for interest and becomes maintained over time as a learner engages further with
learning measures. However, personalization that included deeper the stimuli [8]. Individual interest is an enduring preference for
connections to students’ interests seemed to be more effective certain objects or activities that persists over time and involves
than surface-level personalization. knowledge, value, and enjoyment; individual interest can be
emerging or well-developed.
Keywords
Situational interest can also be subdivided into interest based on
Personalization; interest; mathematics; intelligent tutoring systems enjoyment of the activity and interest based on valuing of the
activity with respect to other things the learner values. Value-
1. INTRODUCTION based situational interest has also been referred to as utility value
The question of how to enhance the interest and motivation of – a learner’s awareness of the usefulness of a topic to their life
adolescents has gained increasing prominence [1] especially in and goals [9]. Interventions that are intended to trigger students’
secondary mathematics [2]. Students often find mathematics, situational interest are sometimes called “catch” interventions –
especially the math in middle and high school, to be disconnected the idea is to immediately grab students’ attention through salient,
from their interests, everyday lives, and typical ways of thinking evocative, relevant, or surprising characteristics of the
about relationships and quantities [3]. At the same time, young instructional materials. Interventions that are designed to promote
people are using increasingly sophisticated and technology-driven maintained situational interested as sometimes called “hold”
ways to pursue and learn about their non-academic interests, and interventions – they often reveal the value of the content to
have become accustomed to a high level of customization, students’ lives and goals, seeking to empower students [10-12].
interaction, and control when seeking knowledge [4]. For example, Mitchell [4] proposed that activities involving group
As a result, the idea of designing and advancing highly work, computers, and puzzles function as “catch” mechanisms in
personalized systems for student learning has become a central the secondary mathematics classroom, while meaningfulness and
focus for educational stakeholders [5]. Technology systems that involvement “hold” situational interest. Research has shown that
enact personalized learning in the classroom have the potential to when individuals are interested in a task or activities, they engage
intelligently adapt to students’ prior knowledge, interests, in more productive learning behaviors and have improved
preferences, and goals [4]. In mathematics, these systems can learning outcomes [e.g., 13].
make explicit connections between the interests students pursue An important question, then, is how to elicit and develop learners’
outside of school – like sports, video games, or social networking interests for academic content areas. Personalization is a
– and the academic concepts they are learning. Algebra in particular kind of intervention that can be used in learning
particular is a rich space for such connections to be made [6] – environments to accomplish this goal. Personalization
students experience mathematical concepts like rate of change as interventions identify topics for which learners have emerging or
they gain points in their favorite video game, track their pace in well-developed individual interest, and then connect these topics
cross country, or accumulate followers on Instagram. As Algebra to academic content topics they are learning about in school (like
is often considered to be a gatekeeper to higher-level mathematics algebra), for which they may have a lower level of interest. For
[7], and a subject that adolescents struggle to see as relevant [3], it example, consider a student who has a well-developed individual
may be a particularly important area for the development of interest in music, but is not interested in Algebra. In their Algebra
interventions for personalized learning. We posit that 1) using a I class, they may engage with a variety of problems and projects
technology-based system for personalization that grounds algebra that explore the mathematics behind musical pieces. Over time,
problems in students’ out-of-school interests has the potential to the connection between these two areas might support her in
elicit students’ interest in the mathematics content to be learned, developing situational interest based on her enjoyment of the
and 2) that personalization to well-developed individual interests incorporation of music as a context and the value perceived for
can have a long-term effect on students’ learning of algebraic music-themed problems, ultimately leading to the development of
concepts and their motivation to learn mathematics. individual interest in Algebra [14]. By making explicit
connections to students’ interests, personalization interventions designers must consider. First is the depth of the intervention –
are hypothesized to trigger situational interest in the academic whether the personalization draws upon surface level aspects of a
content being learned, which can be maintained over time and learners’ interest (e.g., simply inserting familiar objects or names
eventually develop into individual interest in that content area. into an already-designed task), or whether the personalization
Personalization can increase students’ engagement in the math involves deep, authentic connections to actual experiences the
task, improve their performance on personalized math tasks and learner has pursuing an interest like music. Second is the grain
future math tasks that are not personalized [15], and may even size of the intervention – whether the personalization is targeted to
increase students’ interest in the math they now see as relevant to the specific experiences of an individual, or to the generic
their personal interests. However, little research has investigated experiences of an entire group. When considering grain size, it is
the mechanisms by which personalization promotes these learning important to remember that some topics will tend to tap into the
outcomes. In this study, we test this situational interest hypothesis interests of larger groups of students more than others – for
by monitoring students’ interest in math units via embedded self- example, a problem about the specifics of football may match the
report surveys and examining whether personalization induces fine-grained interests of more ninth graders than a problem about
higher levels of situational interest, and whether this situational field hockey. Use of these topics that relate to many students’
interest transforms into individual interest. Thus we test whether experiences may be a productive way to allow materials to be
increased situational interest is an important mechanism through personalized at a finer grain size. Third is the ownership of the
which personalization may gain its effect. personalization – whether the students themselves take a role in
generating the connections between the academic content area and
In addition to possessing enjoyment and value components, their interests, or if teachers or curriculum developers control the
Renninger, Ewen, and Lasher [16] accentuate that interest also personalization. In this study, we examined students’ interest in
involves knowledge. Learners tend to possess useful prior mathematics and algebra learning when exposed to a
knowledge related to their areas of interest, but this knowledge
personalization intervention of medium grain size (i.e.,
may be intuitive and informal with respect to underlying personalized for local users based on interest interviews
principles, making connections to concepts being learned in conducted at the same school in a prior year) versus a standard set
school (like algebra) difficult to acknowledge or articulate. In of problems (i.e., broad grain size written by curriculum
addition to possessing the potential to spur enjoyment and value- developers for all Algebra I students who use the curriculum). In
driven reactions to an academic content area, personalization is the fourth unit of the intervention, we also varied the depth of
advantageously positioned to formalize students’ intuitive prior problems by personalizing on surface or deep features of the
knowledge about their interests by explicitly connecting it to a problem to examine the effects of depth on interest and learning
concept learned in school. For example, a learner with substantial
(i.e. the funds of knowledge hypothesis). No manipulation of
knowledge of musical composition may have implicit problem ownership was conducted.
understandings of the mathematical or numerical underpinnings of
music, and this knowledge can potentially act as a support when In the present study, we pursue the following research questions
they are learning formal algebra. In mathematics education, this by implementing a personalization intervention for Algebra I:
follows a “funds of knowledge” perspective [17], which 1) What is the immediate impact of a personalization
accentuates that students bring with them to the classroom intervention on students’ situational interest in algebra
powerful quantitative ways of reasoning from their home and instructional units?
community lives. These informal, interest-based funds of 2) What long-term effect does personalization have on
knowledge are potential strengths that can be leveraged through students’ individual interest in algebra?
thoughtful instructional approaches like personalization to develop 3) What is the impact of a personalization intervention on
students’ algebraic knowledge. In this study, we test the funds of students’ learning of algebra concepts?
knowledge hypothesis by examining whether solving personalized 4) How does depth influence the impact of personalization
problems that incorporate deeper features of one’s interest (e.g., on interest and learning?
mechanics of a popular video game) elicit stronger effects on
learning than problems personalized based on shallower features of Based on prior work examining the effects of personalization on
a learner’s interest (e.g. passing reference to a game title in a learning [15] and theoretical assumptions about the development
problem about snacking) or non-personalized problems. Thus we of interest [8] including the situational interest hypothesis, we
test whether increased activation of prior knowledge is an important hypothesize that 1) Personalized problems should trigger greater
mechanism through which personalization gains its effect. situational interest in algebra units than standard problems; 2)
Students completing personalized problems that incorporate out of
Whereas outside interests can be leveraged by personalization, school interests will report greater individual interest in algebra;
initial interest in mathematics may moderate the effectiveness of and 3) Students who complete personalized problem solving units
personalization interventions. Durik and Harackiewicz [10] found will achieve greater increases in their algebra performance than
that an intervention designed to “catch” (i.e., trigger [8]) student students completing standard problem solving units. In
interest (adding colorful, vivid decorations to instructional accordance with the funds of knowledge hypothesis, we expect 4)
materials) was most effective for learners with low individual
that students who complete problems that are personalized based
interest in mathematics (IIM), but hampered learners with high IIM. on deeper features of their interest area should outperform those
Conversely, they found that an intervention designed to “hold” (i.e., completing problems personalized on surface features of the
maintain based on value [8]) student interest (informing students of problems and standard problems.
the value of the content being learned) was beneficial for high IIM
students, and detrimental for low IIM students. 3. METHODS
In order for personalized instructional materials to successfully 3.1 Participants and Environment
activate knowledge, trigger interest, and enhance perceptions of Total participants included N = 152 ninth grade Algebra I students
value, Walkington and Bernacki [14] identified three key features in the classes of two Algebra I teachers. Students attended a rural
Northeastern school that was 96% Caucasian with 21% of Surface Personalization condition, as it seems less likely that
students eligible for free or reduced price lunch. In 2012, 71% of despite an interest in games, a teen would care about or track
students passed the state standardized test in Mathematics, which exactly how frequently they consume snacks during play.
is administered in the 11th grade. The sample was 51% female. Personalized problems were written based on surveys (N = 45)
Because one teacher at the school site did not administer the and interviews (N = 23) with Algebra I students at the school
pretest before students began using the Cognitive Tutor, eighty- where they discussed their out-of-school interests.
three students completed pretest, posttest and all questionnaires
delivered in the CTA software and compose the primary sample Deep Personalization problems were written to more closely
for this study. correspond to quantitative information given by students in the
interviews and open-ended surveys about their out-of-school
The school at which the study took place used the Cognitive Tutor interests, including interviews with Algebra I students at the
Algebra (CTA) curriculum [18]. CTA is an intelligent tutoring school where the study was conducted. In these interviews,
system for Algebra I that uses model-tracing approaches to relate students discussed how they consider rate of change as they play
the students’ actions back to the domain model to provide video games, participate in sports, track their rate of texting and
individualized error feedback. CTA also uses knowledge-tracing battery usage on their cell phone, engage in cooking, work at part-
approaches to track learning from one problem to the next, using time jobs, activities, and so on. (see [6] for a full analysis of
this information to identify strengths and weakness in terms of student interviews).
production rules. CTA presents learners with algebra story
problems where they must navigate tabular, graphical, and
symbolic representations of functions (Figure 1). Students in
schools that use CTA typically use the software 2 days per week.
4. Personalization Intervention
Before entering the first unit in CTA (Unit 1), all participants
were given an interests survey where they would rate their level of
interest in 10 topic areas – music, art, cell phones, food,
computers, games, stores, TV, movies, and sports. Participants
were then assigned to one of two main conditions: (1) a Control
Condition that received the standard algebra story problems in all
units in CTA including Units 1, 3, 7, and 9 covering linear
equations, (2) an Experimental Condition that received versions of
these same problems with the same underlying structure that were
matched to the interests they indicated on the interests survey for
Units 1, 3, 7, and 9 (i.e. Personalization Condition). In unit 9, we
tested the funds of knowledge hypothesis by further subdividing
learners in the Personalization condition to (A) a Deep
Personalization condition where they received personalized
problems with greater depth – i.e., the personalized problems the
Deep Personalization group received in Unit 9 were written to
better correspond to ways that adolescents might actually use
linear functions when pursuing their interests, and were intended
to draw upon “funds of knowledge” more explicitly. The
remaining students were assigned to (B) a Surface Personalization
Condition where they received problems that contained stories
with only superficial references to their identified interests. These
problems should elicit situational interest, but not draw upon
knowledge about one’s interests.
In the first sample Control problem in Table 1, students must
identify the relationship between dosage and weight. This
relationship is grounded in a story that provides a context that
likely to be of limited relevance to the student. In the Surface
Personalization problem the structure of the problem remains
consistent, but a topic that corresponds to the learners’ personal
interests has been applied. In the Deep Personalization version,
the personal interest is applied more intentionally. Like the
surface-level personalization problem, The Clash of Clans
problem matches students’ reported interest in games. However it
is also intended to draw upon the learner’s knowledge of the
game’s architecture to frame the underlying algebraic relationship
to be learned in a deeply relevant context (i.e. it is actually useful
to keep track of the relationship between elapsed time and how
goals are accomplished, and this quantity is explicitly tracked and
displayed for the player within the game interface). We consider Figure 1. Screenshot of Cognitive Tutor Algebra environment
this to be a deeper level of personalization compared to the with answer key superimposed
Table 1. Study Conditions 4.1.2 Domain-Level Motivational Surveys
Control Surface Deep Prior to entering Unit 1 (pre-) and Unit 10 (post-) in CTA, the
Personalization Personalization software presented students with a survey asking them to rate their
attitudes about algebra. Specifically, they rated their individual
The correct While playing When playing interest in mathematics (IIM), as well as their maintained
dosage of a cards a person Clash of Clans a situational interest–enjoyment and maintained situational interest-
certain medicine typically eats player can build value for mathematics. Subscales were adopted from a larger set
is two two snacks for two barracks for of scales from Linnenbrink-Garcia et al. [19]. Sample items for
milligrams per every 25 every 25 minutes each scale appear in Table 2.
GAMES
25 pounds of minutes of of playing time.
body weight. playing time in 4.1.3 Unit-Level Motivational Surveys
a card game. After each unit impacted by the personalization intervention
Three out of Three out of Three out of five (Figure 2; Units 1, 3, 7, and 9), participants were also given a
every five five people have free throws are unit-level motivational survey that assessed the degree to which
people in a attended a successful for that unit triggered their situational interest and maintained their
recent survey Pittsburgh NBA players. situational interest in the CTA unit. These scales were adapted
SPORTS
supported the Steelers game in based on measures from Linnenbrink-Garcia et al. [19] with the
President's their lifetime. math unit as the referent. Sample items for each scale appear in
Health Plan. Table 2, as do Cronbach’s alphas for the initial administration of
Directions for a Looking In a family each survey. An overview of the survey measures and CTA units
swimming pool through a recipe you use completed by participants in this study is provided in Figure 2.
chemical that collection of six drops of hot
Table 2. Interest Measures
controls the online recipes, pepper oil for
growth of algae there are six every 500 Interest Measure Sample item α
state that you recipes that ounces of chili
should use six require that is being Individual Interest in Thinking mathematically .92
fluid ounces of powdered sugar cooked. Mathematics is an important part of 9
chemical for for every 500 who I am.
FOOD
every 500 recipes that you Maintained Situational What we are studying in .92
gallons of water. find online. Interest in Math- Value math class is useful for me
to know.
Maintained Situational I really enjoy the math we .89
Problems across the 3 conditions were written to hold constant Interest in Math- Enjoyment do in this class.
factors like order of information given, numbers, sentence
structure and length, mathematical vocabulary, readability, Triggered Situational Interest The topics in this unit .84
pronoun use, and distractor information. The personalized in Math grabbed my attention.
problems did not require that students have additional knowledge
of specific numerical mathematical information in their interest Maintained Situational The math in this unit is .90
area (e.g., knowing how many points a field goal is worth) – all Interest in Unit - Value useful for me to know.
information given was matched across problem types. Maintained Situational In this unit, I really .84
All instructional units involved in the study involved linear Interest in Unit - Enjoyment enjoyed the math.
functions. Of the core sample comprising most of our analyses, 31
participants were assigned to the Control, 34 were assigned to
Surface Personalization, and 27 were assigned to Deep
Personalization.
4.1 Measures
We collected the following measures from all participants:
4.1.1 Paper-Based Pre/Post Assessments
At the beginning of the school year, prior to entering the tutor, all
students completed a paper-based pre-test on linear functions. The
test contained 4 story problems where a linear function was
described that either had a slope and intercept (2 problems) or had
only a slope (2 problems). Participants first were given an x value Figure 2. Measures
in the linear function and asked to solve for y, then they were
given a y value in the linear function and asked to solve for x.
Finally, they were asked to write the linear function using algebra 5. RESULTS
symbols. A post-test was administered to all students around the We report results as they address the first three research questions
midterm of their ninth grade year (i.e., four months later). The in section 2. We do not provide a separate section for research
post-test contained 4 matched items containing slightly different question 4 (impact of depth of personalization), and instead
wording and numbers. Students’ responses to each part of each discuss the results for depth of personalization within each of the
problem were scored as correct or incorrect. other three sections.
5.1 What is the impact of personalization on data, with students’ class period as a random effect. Adding a
predictor for Condition significantly improved the fit of the model
students’ situational interest in algebra units? (χ2(2) = 6.39, p = 0.04), as did a control variable for students’
To assess the effect of the personalization interventions on initial level of individual interest in mathematics (IIM) prior to the
students’ situational interest, we conducted a series of analyses of intervention (χ2(1) = 4.07, p = 0.04). The interaction of Condition
covariance examining students’ reported triggered and maintained and IIM also significantly improved the fit of the model (χ2(2) =
interest in CTA units. All students were given unit-level surveys 14.43, p < .001).
assessing their level of interest in the instructional unit after each
of the units impacted by the personalization treatment (Units 1, 3, Table 3. Estimated Marginal Means Controlling for Individual
7, and 9). We controlled for initial individual interest in Interest in Math
mathematics (IIM) as indicated on the domain survey before Unit
1 (Figure 2). Variable Personalizationa Controlb
Unit EMM SE EMM SE
Students in the two Personalization conditions (i.e., Surface Triggered
Personalization and Deep Personalization are identical in Units 1, 1 2.86 0.13 2.33 0.19 *
Situational
3, and 7) consistently reported significantly higher levels of Interest 3 2.82 0.13 2.27 0.19 *
triggered situational interest than students assigned to the Control
7 2.69 0.13 2.25 0.18 *
condition (Table 3; Unit 1 F(1,80) = 5.19, MSe = .96, p = .03,
c
Unit 3 F(1,80) = 5.31, MSe = .98, p = .02; Unit 7 F(1,80) = 3.82, 9 D 2.82 0.18 2.55 0.19
MSe = .91, p = .05). d
S 2.56 0.20
Significant differences between any of the 3 groups in triggered
Maintained
situational interest were not obtained in Unit 9. The level of 1 2.95 0.13 2.77 0.19
Situational
triggered situational interest reported by the Deep Personalization 3 3.07 0.13 2.74 0.18
Interest -
was consistent with prior units with the triggered interest for the
Value 7 2.76 0.13 2.76 0.18
Surface Personalization group was slightly lower. The Control
group, however, reported greater triggered situational interest, and 9 D 2.84 0.19 2.82 0.18
the inclusion of three groups (two with smaller Ns) further
diminished the statistical power available to detect effects. S 2.70 0.17
No significant differences in maintained situational interest were Maintained
1 2.76 0.12 2.46 0.17
found between groups on any of the four units observed, Fs < Situational
3.73, ps = ns. Directionally, measures of maintained situational Interest - 3 2.81 0.13 2.40 0.18
interest generally favored the personalization groups. Enjoyment 7 2.66 0.12 2.35 0.17
5.2 What effect does personalization have on 9 D 2.62 0.19 2.50 0.18
students’ individual interest in algebra? S 2.33 0.17
All students were given domain-level surveys assessing their Individual Pre 2.87 .14 3.34 .20
interest towards learning algebra prior to the intervention and after Interest in
the final personalized unit (i.e., Unit 9). A repeated measures Math Post 2.83 .16 2.94 .22
analysis of variance examining change in Individual Interest in Notes. *- p < .05, EMM = Estimated Marginal Mean, SE =
Mathematics (i.e., Post-Pre) between the two Personalization Standard Error, D = Deep personalization, S = Surface
conditions (i.e., Deep & Surface) versus Control was conducted to Personalization, a - N = 55, b - N = 28, c - N = 24, d - N = 31
examine the main effect of Time and Interaction between Time X
Condition. Results indicated a significant main effect of Time, F
Table 4. Scores on Knowledge tests by Condition
(1, 81) = 5.39, MSe = 1.75, p = .023. Overall, students’ individual
interest in mathematics declined from pretest to posttest. Analyses Pretest Posttest
also indicated a marginally significant interaction between Time Condition N M SD M SD
and Condition, F (1, 81) = 3.73, p = .057. Students in the control Control 32 0.68 0.2 0.83 0.12
group significantly reduced their rating of individual interest in Surface
algebra an average of 0.37 points over the 10-unit span (Table 3; 29 0.73 0.15 0.82 0.15
Personalization
t(29) = 3.21, p < .01), while students in the Deep and Surface Deep personalization 32 0.63 0.22 0.84 0.18
Personalization groups maintained their individual interest in
algebra (M = 0.04 decline). Thus personalization had a positive
effect in that it preserved students’ individual interest in algebra. The regression output is shown in Table 5. The reference category
Within the Personalization condition, no differences were found is the Control Group, and we interpret all significant simple
between students who received Surface versus Deep effects regardless of whether they are displayed in the table. The
Personalization. IIM control measure was dichotomized to separate students with
high IIM (average rating of 3 or more) from low IIM (average
5.3 What is the impact of personalization on rating less than 3) to aid interpretability and to be consistent with
students’ learning of Algebra I concepts? prior work [e.g., 14]. As can be seen from Table 5, for students
The pre- and post- test scores on the algebra learning measures for with low individual interest in math, Deep Personalization was
each of the three conditions is shown in Table 4. A linear significantly more effective than Control (p < 0.05). Additional
regression model predicting amount of absolute gain from pre- to contrasts not shown in the table compared Surface Personalization
post-test (i.e., post-test score minus pre-test score) was fit to the to Deep Personalization, and found that for students with low IIM,
Deep Personalization was significantly more effective than value to maintain students’ situational interest. Indeed, in another
Surface Personalization (B = 0.24, SE (B) = 0.07, p < 0.001). personalization study [20], we found that a personalization
Finally, within the Deep Personalization condition, students with intervention with a much smaller grain size where students wrote
high IIM gained significantly less than students with low IIM (B = and solved problems that incorporated features of their personal
.17, SE(B) = .07, p = .01). interests produced increases in students’ maintained situational
interest associated with perceived value. This intervention also
involved a higher level of ownership of the personalization on the
Table 5. Regression Output for Pre/Post Learning Gains part of the students [14], which suggests that personalization at a
B SE (B) t p medium grain size may successfully trigger situational interest,
but a personalization at a smaller grain size with some level of
ownership may be necessary to achieve more enduring situational
(Intercept) .13 .07 1.81 .07 interest in math units. This type of intervention may be especially
Control (ref.) important given that it takes the burden of generating fine-grained
Surface Personalization -.10 .08 -1.33 .18 instructional materials away from teachers and curriculum
Deep Personalization .14 .07 1.97 .05 developers and places it on students.
Low IIM (ref.)
High IIM .00 .07 -.07 .94 6.2 Personalization and Individual Interest
Surface Personalization ×
.08 .10 .82 .41 Despite a failure to elicit maintained situational interest, the
High Initial Individual Interest
Deep Personalization ×
Personalization intervention did have a significant effect on
-.17 .10 -1.71 .09
High Initial Individual Interest students’ individual interest in mathematics. Importantly, the
individual interest items assessed how students felt about the
domain of mathematics as a whole, rather than how they felt about
the particular math class they were enrolled in or the particular
6. DISCUSSION & CONCLUSION units they were working on. This preservation of individual
This study examined whether personalizing algebra problems to interest in algebra over half a year of high school coursework is a
students’ out-of-school interests would increase their situational desirable outcome, given research that documents declines in
interest in CTA algebra problems, increase their interest in interest in math over adolescence [21, 22]. In sum, the findings
mathematics, and improve their acquisition of algebra knowledge from the first two research questions support the situational
(i.e., the situational interest hypothesis). It additionally tested interest hypothesis. We consider this finding in light of theory on
whether solving problems that incorporated deep features of an interest development in section 6.4.
interest into problems would produce greater benefits that solving
problems that incorporated interests superficially or standard 6.3 Deep Personalization and Algebra Learning
problems (i.e. the funds of knowledge hypothesis). Students who Walkington [12] found that a one-unit personalization
received problems personalized to their out-of school interests intervention improved students’ long-term learning of algebra
reported significantly higher triggered situational interest for CTA concepts within the CTA environment, relative to a control
math units. Compared to a Control group that experienced a drop condition. This study extends that work and indicates that, when
in their individual interest in mathematics, Personalization also personalization incorporates deep features of students’ out-of-
had a preserving effect on students’ interest in mathematics. After school interests, it can also induce learning gains that transfer
accounting for students’ initial individual interest in mathematics, outside of an intelligent tutoring environment (i.e. to delayed,
significant differences in learning gains were found between paper-based tests). However, these effects are moderated by
groups of students in the Deep Personalization, Surface students’ initial level of individual interest in mathematics, with
Personalization and Control Conditions. These findings are next Deep Personalization being beneficial mainly for low IIM
discussed in light of prior theory and research. students. Walkington [15] did not collect such interest measures in
her study, but did find that personalization was most effective for
6.1 Personalization and Situational Interest students who were making slower progress through CTA– a
Students who completed algebra problems personalized to their variable known to track closely with interest in math [23]. We
interests reported greater triggered situational interest compared to consider these findings in light of proposed hypotheses that
students who completed standard CTA problems, however personalization may obtain effects on learning by activating
students who solved personalized problems did not report students’ funds of knowledge in their out-of-school interest, and
significantly greater maintained interest resulting from enjoyment that personalization may trigger greater situational interest in math
or perceptions of value. The finding that personalization was tasks. The current study showed that Deep Personalization was
effective in triggering situational interest is encouraging as we significantly less effective for learners with high IIM, compared to
consider the Control condition to be a considerably strong control. learners with low IIM. This, along with the results that
That is, the standard problems included in tutor units might be personalization triggers but does not maintain situational interest,
considered to be personalized to student interests at a very broad suggests that even Deep Personalization may achieve its effects
grain size [11] – they were generally written by teachers and on learning as a “catch” intervention, immediately eliciting
curriculum writers with this student population in mind (i.e., triggered situational interest. That is, solving personalized
adolescent algebra learners). The personalized problems in the problems triggered students’ interests, but did not maintain them.
intervention, on the other hand, had a medium grain size – they This provides some promise as prior research has shown catch
were written for and provided to subsets of the student population interventions that trigger interest to be beneficial primarily for
that had particular topic interests (e.g., sports, video games). The learners with low IIM [10]. This is contrasted with a “hold”
change from a large to a medium grain size was sufficient to elicit intervention that maintains situational interest, often by
changes in triggered situational interest, though additional effort communicating the value of the content being learned. In this
may be necessary to elicit sufficient enjoyment or perception of study personalization did not increase students’ perceptions that
algebra problems had value, but additional interventions aimed at interest is 1) triggered by environmental stimuli and 2) maintained
boosting perceived value and relevance [11, 12] could potentially when engagement in the environment is enjoyable or confers
be incorporated to ITSs to also obtain this effect and its benefits value through consistent or repeated situational interest. This
for learning. supports 3) the emergence of an individual interest, which 4)
becomes well developed over time. In this study, analyses reveal a
Although we termed our Condition “Deep” Personalization, the
triggering of situational interest among students in the Surface and
connections made to learners’ actual experiences may not have Deep Personalization conditions, no reported maintenance of
been uniformly deep depending on students more specific situational interest via enjoyment or value, but a significant effect
interests within a topic area, and thus may not have elicited value- of Personalization on individual interest. Thus individual interest
based reactions from some students. This stems from issues with developed without being maintained during learning; this requires
the grain size of the intervention – students merely indicated their that we consider alternate explanations by which such effects on
level of interest in a broad topic (e.g., “sports”), and were then individual interest may have been obtained.
given problems that could cover the entire space of activities that
fell within that topic (e.g., basketball, hockey, football), without One potential explanation is that the way instructors used
considering students more specific interest in a subtopic (e.g., just Cognitive Tutor in the math classes may have reproduced some of
hockey). Although attempts were made to use the “high-leverage” the behaviors expected when students’ situational interest is
interest sub-topics that many students would have specific maintained. In their model, Hidi and Renninger [8] describe that
knowledge of (i.e., football rather than field hockey) this approach those who maintain interest in a topic tend to repeatedly engage
likely allowed for the personalization to have highly variable level with content involving the topic (e.g., a student who is interest in
of correspondence to students’ exact interests. The level of dolphins may seek more opportunities to learn about them by
correspondence depended on the overlap between a student’s reading books about them in school or choose “dolphins” as a
interest and the commonly reported interests by peers in surveys topic for school assignments). While students’ did not report that
and interviews prior to problem development. Walkington and personalized Cognitive Tutor Algebra units maintained their
Bernacki [20] found significant increases in maintained situational interest to a degree that we would expect them to voluntarily seek
interest (value) for students who authored problems about their out opportunities to learn using Cognitive Tutor, the compulsory
specific interests, suggesting that the smaller grain size and use of the Cognitive Tutor in math class twice a week for many
increased ownership of the personalization intervention in that months effectively ensured repeated engagement in (personalized)
study allowed it to function more as a “hold” intervention. problem solving via CTA use. Thus we could conclude that the
continued exposure to math content personalized to one’s out-of-
Finally, the current study showed that Deep Personalization was school interests approximated behavioral outcomes of maintained
significantly more effective than Surface Personalization for
situational interest and created an alternate pathway by which
students with low IIM. This suggested that personalization may individual interest was preserved in Personalization conditions
need to have at least a moderate level of depth for it to be (i.e., no drop in interest), but not in the Control condition where
effective at all for supporting learning outcomes for any subgroup there was no initially triggered interest. Much like the typical
of students. Indeed, a number of recent personalization adolescent whose interest in math declines over time, students in
interventions that employed relatively surface-level the Control condition were required to complete math units that
personalization have reported null findings [24, 25]. Thus we did not trigger situational interest and subsequently reported
conclude from all of these analyses that a personalization declines in their interest in mathematics.
intervention with a moderate depth and grain size can potentially
have long-term effects on student learning for students who begin 6.5 Conclusion
with limited interest in mathematics. However, increasing depth The results obtained in this study provide important insight about
and personalizing at an even smaller grain size may have more the ways depth and grain size of personalization may impact the
powerful effects, especially for students with higher IIM for development of students’ interests in their math course, the
whom value-based connections may be most critical. domain of mathematics, and ultimately their long-term learning of
Although learning gains were produced for low IIM students who algebra concepts. In future analyses, we will analyze additional
received Deep Personalization (rather than Surface data from students participating in this study, and look for
Personalization), these students did not show differences in difference in in behavior and performance within intervention and
situational or individual interest measures within Unit 9 compared subsequent CTA units, including analyses of learning behaviors
to the Surface Personalization group. There were also no using log-files and automated detectors.
differences between Surface and Deep in individual interest over
the course of the entire intervention. This suggests that Deep 7. ACKNOWLEDGMENTS
Personalization may gain its effectiveness over Surface Both authors contributed equally to this manuscript. The authors
Personalization by connecting to students’ prior knowledge (funds thank Steve Ritter, Susan Berman, Tristan Nixon and Steve
of knowledge hypothesis) rather than triggering and maintaining Fancsali (Carnegie Learning), Gail Kusbit (Carnegie Mellon
differing levels of situational interest (situational interest University & LearnLab) and participating teachers. Funding for
hypothesis). However, ultimately comparisons between these two the study was provided by a subgrant of National Science
groups are of limited usefulness given the relatively small sample Foundation Award # SBE-0354420. Additional funding as
sizes. Thus we find limited but promising support for the funds of provided by IES Award # R305B100007.
knowledge hypothesis.
6.4 Theoretical Implications 8. REFERENCES
When viewed through the lens of interest development theory [8], [1] Hidi, S., & Harackiewicz, J. (2000). Motivating the
the findings regarding personalization and interest development academically unmotivated: A critical issue for the 21st
are somewhat puzzling. Per Hidi and Renninger’s [8] theory, century. Review of Educational Research, 70, 151–179.
[2] Mitchell, M. (1993). Situational interest: Its multifaceted [14] Walkington, C., & Bernacki, M. (in press). Motivating
structure in the secondary school mathematics classroom. students by “personalizing” learning around individual
Journal of Educational Psychology, 85, 424–436. interests: A consideration of theory, design, and
[3] McCoy L. P. (2005). Effect of demographic and personal implementation issues. In S. Karabenick & T. Urdan (eds.)
variables on achievement in eighth-grade algebra. Journal of Advances in Motivation and Achievement, Emerald Group
Educational Research, 98(3), 131–135. Publishing.
[4] Collins, A., & Halverson, R. (2009). Rethinking Education in [15] Walkington, C. (2013). Using learning technologies to
the Age of Technology: The Digital Revolution and personalize instruction to student interests: The impact of
Schooling in America. New York: Teachers College Press. relevant contexts on performance and learning outcomes.
Journal of Educational Psychology, 105(4), 932-945.
[5] U.S. Department of Education, Office of Educational
Technology, Transforming American Education: Learning [16] Renninger, K., Ewen, L., & Lasher, A. (2002). Individual
Powered by Technology, Washington, D.C., 2010. interest as context is expository text and mathematical word
http://www.ed.gov/sites/default/files/netp2010.pdf problems. Learning and Instruction, 12, 467-491.
[6] Walkington, C., Sherman, M., & Howell, E. (in press). [17] Civil, M. (2007). Building on community knowledge: An
Connecting Algebra to sports, video games, and social avenue to equity in mathematics education. In N. Nassir. and
networking: How personalized learning makes ideas “stick.” P. Cobb (Eds.) Improving access to mathematics: Diversity
Mathematics Teacher. and equity in the classroom (pp. 105-117). Teachers College
Press.
[7] Moses, R., & Cobb, C. (2001). Radical Equations: Math
Literacy and Civil Rights. Boston: Beacon Press. [18] Carnegie Learning (2013). Cognitive Tutor Algebra
[software]. Carnegie Learning, Inc. Pittsburgh, PA, USA.
[8] Hidi, S., & Renninger, K. (2006). The four-phase model of
interest development. Educational Psychologist, 41(2), 111- [19] Linnenbrink-Garcia, L., Durik, A., Conley, A., Barron, K.,
127. Tauer, J., Karabenick, S., & Harackiewicz, J. (2010).
Measuring situational interest in academic domains.
[9] Eccles, J. (1983). Expectancies, values and academic Educational Psychological Measurement, 70, 647-671.
behaviors. In R. C. Atkinson, G. Lindzey, & R. F. Thompson
(Eds.), Achievement and achievement motives: Psychological [20] Walkington, C., & Bernacki, M. (2014). Students authoring
and sociological approaches (pp. 75-146). San Francisco: personalized “algebra stories”: Problem-posing in the context
W.H. Freeman & Co of out-of-school interests. Presentation at the 2014 Annual
Meeting of American Educational Research Association.
[10] Durik, A., & Harackiewicz, J. (2007). Different strokes for
different folks: How individual interest moderates effects of [21] Fredricks, J. A., & Eccles, J. (2002). Children’s competence
situational factors on task interest. Journal of Educational and value beliefs from childhood through adolescence:
Psychology, 99(3), 597-610. Growth trajectories in two male-sex-typed domains.
Developmental Psychology, 38, 519–533.
[11] Hulleman, C. S., Godes, O., Hendricks, B. L., &
Harackiewicz, J. M. (2010). Enhancing interest and [22] Frenzel, A. C., Goetz, T., Pekrun, R., & Watt, H. M. G.
performance with a utility value intervention. Journal of (2010). Development of mathematics interest in adolescence:
Educational Psychology, 102, 880-895. Influences of gender, family, and school context. Journal of
Research on Adolescence, 20, 507–537.
[12] Hulleman, C. S., & Harackiewicz, J. M. (2009). Promoting
interest and performance in high school science classes. [23] Bernacki, M. L., Nokes-Malach, T.J., Aleven, V., & Glick, J.
Science, 326(5958), 1410–1412. (2014). Intelligent tutoring systems promote achievement in
middle school mathematics, especially for students with low
[13] Harackiewicz, J., Durik, A., Barron, K. Linnenbrink, E., & interest. Presentation at the 2014 Annual Meeting of the
Tauer, J. (2008). The role of achievement goals in the American Educational Research Association.
development of interest: Reciprocal relations between
achievement goals, interest, and performance. Journal of [24] Bates, E., & Wiest, L. (2004). The impact of personalization
Educational Psychology, 100(1), 105-122. of mathematical word problems on student performance. The
Mathematics Educator, 14(2), 17-26.
[25] Caker, O., & Simsek, N. (2010). A comparative analysis of
computer and paper-based personalization on student
achievement. Computers & Education, 55, 1524-1531.