=Paper=
{{Paper
|id=Vol-1183/ncfpal_paper01
|storemode=property
|title=The Impact of Cognitive and Non-Cognitive Text-Based Factors on Solving Mathematics Story Problems
|pdfUrl=https://ceur-ws.org/Vol-1183/ncfpal_paper01.pdf
|volume=Vol-1183
|dblpUrl=https://dblp.org/rec/conf/edm/WalkingtonCRNF14
}}
==The Impact of Cognitive and Non-Cognitive Text-Based Factors on Solving Mathematics Story Problems==
https://ceur-ws.org/Vol-1183/ncfpal_paper01.pdf
The Impact of Cognitive and Non-Cognitive Text-Based
Factors on Solving Mathematics Story Problems
Candace Walkington Virginia Clinton Steven Ritter
Southern Methodist University University of Wisconsin - Madison Carnegie Learning, Inc.
3011 University Blvd. Ste. 345 1025 West Johnson Street 437 Grant Street, Suite 918
Dallas, TX, 75205 Madison, WI 53706 Pittsburgh, PA 15219
1-214-768-3072 1-608-890-4259 1-888-851-7094 x122
cwalkington@smu.edu vclinton@wisc.edu sritter@carnegielearning.com
Mitchell Nathan Stephen E. Fancsali
University of Wisconsin - Madison Carnegie Learning, Inc.
1025 West Johnson Street 437 Grant Street, Suite 918
Madison, WI 53706 Pittsburgh, PA 15219
1-608-262-0831 1-888-851-7094 x219
mnathan@wisc.edu sfancsali@carnegielearning.com
ABSTRACT domain model and student model [1]. It is through the
Intelligent tutoring systems (ITSs) that personalize construction of the student model and its contribution to the
instruction to individual learner background and tutoring model that ITSs can enact personalization where
preferences have emerged in K-16 classroom settings all they adapt to the needs and backgrounds of individual
over the world. In mathematics instruction, ITSs may be learners. Here we explore cognitive and non-cognitive
especially important for tracking mathematical skill factors related to how students react to and understand the
development over time. However, recent research has text of mathematics story problems. We argue that these
pointed to the importance of text-based measures when non-mathematical factors may be an important element to
solving mathematics word problems, suggesting that in consider for an ITS in secondary mathematics. In
order to accurately model the student it is important to particular, we provide evidence suggesting that both the
understand how they respond to text characteristics. We students’ reading level (a cognitive factor) and the students’
investigate the impact of text-based factors (readability and interests, preferences, and motivational outlooks (non-
problem topic) on the solving of mathematics story cognitive factors) have the potential to influence how they
problems using a corpus of N = 3394 students working respond to text-based mathematics problems situated in
through an ITS for algebra, Cognitive Tutor Algebra. We “real world” contexts.
leverage recent advances in computerized text-mining to Cognitive Tutor Algebra (CTA; [2]) is a prominent
automate fine-grained text analyses of many different word mathematics ITS used in many schools across the United
problems. We find that several elements of the text of States. CTA uses model-tracing approaches to relate
mathematics word problems matter for performance – student actions to the domain model and provides
including the concreteness of the problem’s topic, the individualized error feedback. CTA also uses knowledge-
length and conciseness of the story’s text, and the words tracing approaches to track students’ learning from one
and phrases used. problem to the next, using this information to identify the
students’ strengths and weakness in terms of production
Keywords rules (i.e., knowledge components or skills). The software
Intelligent tutoring system, readability, mathematics, word then uses this analysis to individualize the selection of
problems, personalization problem tasks. However, missing from this tutoring model
is a consideration of other non-mathematical characteristics
1. INTRODUCTION of the story problem texts – including the reading difficulty
Since the 1980s, Intelligent Tutoring Systems (ITSs) have of the text respective to students’ reading ability and
risen as an important instructional tool to support student preferences, and the real-world topic of the text respective
learning in classrooms, especially in middle and high to students’ interests and preferences.
school. ITSs typically consist of at least three components: For example, a learner presented with a mathematics
(1) the domain model of the appropriate steps needed to word problem that is difficult to read – with high-level
correctly solve each problem, (2) the student model, which vocabulary, complex sentence structure, etc. - may lack the
captures the evolution of an individual student’s cognitive reading ability to appropriately comprehend that problem.
states as they relate to the domain model, and (3) the This cognitive element of the problem’s difficulty is not
tutoring model which selects tutor actions based on the typically monitored by ITSs for mathematics learning. In
addition, such a problem may inhibit the students’ because of their relatedness to learner prior knowledge. In
motivation – a non-cognitive factor. In particular, even if related work [5], they also identified the prevalence of
the learner is technically able to read the problem, they may issues with verbal interpretation of mathematics story
be intimidated by the problem text, and request a hint problems, finding that even high school students struggle to
instead of putting forth the effort of understanding the text understand difficult vocabulary words and construct an
of the problem. ITSs also do not typically monitor the accurate propositional textbase and situation model from a
learner motivation for reading and understanding text-based story problem’s text.
problems.
2.2 Non-Cognitive Factors
Another non-mathematical element of the text of An important precursor to students’ motivation is their
mathematics story problems is the real world topic – level of interest – defined as the state of engaging and the
whether the story is about working at a part-time job or predisposition to re-engage with particular topics, ideas, or
harvesting a field of grain. The way in which students react activities [6]. Two types of interest have been described in
to the topic of the story problem is also based on both the literature. First, situational interest is an immediate,
cognitive and non-cognitive factors. Students may be temporary state of heightened attention and affective
unfamiliar with elements of the context that are important engagement that stems from elements of a learning
for fully comprehending the problem – for example, in a environment that are surprising, salient, evocative,
banking context, they may not know what “break even” challenging, personally relevant, etc. Situational interest
means. In this way, they may lack the prior knowledge can be triggered in response to a stimuli within a learning
needed to interpret the story. Similarly, different real world environment, and then may or may not become maintained
topics may differ in the motivation they elicit from students over time [6]. A second type of interest is individual
– students may experience greater motivation when solving interest – learners’ enduring predispositions to engage with
a problem about a familiar, interesting context than about a certain activities or topics over time.
context they find boring or unfamiliar. Elements of a story problem’s text have the potential to
We next provide a theoretical framework that provides both trigger and maintain situational interest. In particular,
an explanation of how students comprehend story problems story problems that are accessible, easy to read, and
and how cognitive and non-cognitive factors may interact situated within the topics and contexts that a particular
as they solve story problems. learner finds relevant and interesting may trigger and
maintain interest. In the other hand, difficult reading
2. THEORETICAL FRAMEWORK passages disconnected from a learner’s experiences and
interests may not trigger interest and may cause
2.1 Cognitive Factors disengagement if interest has previously been triggered.
Nathan and colleagues [3] proposed a model of
mathematics story problem solving where students navigate 2.3 Research Purpose
three levels of representation as they comprehend and solve If text-based measures like readability and problem
story texts: (1) a textbase containing the propositional topic matter for student performance, these might be
statements made in the story problem, (2) a situation important elements to add to future systems for
model, a qualitative representation of the actions and events personalized learning in mathematics. For example, an ITS
in the story, and (3) a problem model, containing the formal might present weak readers with problems with simplified
mathematical equations, variables, and operands. Because verbal language as these learners are initially mastering a
mathematics word problems are stated in verbal language new mathematical skill. As the student gains expertise with
(rather than mathematics notation), we hypothesize that the the mathematics by mastering skills, additional levels of
reading difficulty and topic of the problem matters for the verbal difficulty could be layered on by the ITS. Similarly,
construction of the situation model and its successful learners that lack motivation may be presented with story
coordination with the problem model. problems that are less intimidating to read and situated
Various aspects of the reading difficulty, including within their interests, with this support faded out over time.
readability measures, may be important in situation model By neglecting to model this aspect of the user’s experience
construction. Readability measures often include the kinds in the ITS, the system may be generating inferences about
of words used, the length of the story, and the structure of learner knowledge states that are inaccurate.
the sentences. These elements of the text’s structure may
make it more difficult to comprehend, especially for 3. LITERATURE REVIEW
students with weaker reading skills. 3.1 The Impact of Reading Difficulty on
Another aspect of reading difficulty is the topic of the
problem – whether it is about, for example, farming or
Solving Mathematics Story Problems
banking. Walkington and colleagues [4] proposed that story Recent research has found that reading ability is especially
contexts that are related to topics that are familiar and important as students solve mathematics word problems
accessible to students are easier for them to solve because [7]. Studies examining the association of reading difficulty
these contexts can facilitate situation model construction of mathematics word problems and U.S. student
performance on large-scale assessments has found that evidence points to the importance of considering the real
problems that use words with multiple meanings, complex world topic of mathematics story problems and its
verbs, and mathematics vocabulary words are more relationship to students’ interests and experiences.
difficult [8]; the effect is especially pronounced for students However, more research is needed to determine which
who speak English as a second language [9]. A small study topics may be more or less likely to trigger and maintain
of students working in CTA found that extraneous text that students’ interest.
provided a real world context for the problem, as well as
references to concrete people, places, and things, were 3.3 Research Questions
associated with less concentration and more confusion in In the present study, we investigate the relationship
the tutor [10]. However, a similar study found that the between readability and topic measures and student
extraneous text was also associated with fewer performance on mathematics story problems. We examine
unproductive “gaming the system” behaviors in the tutor these issues within an ITS for Algebra I, Cognitive Tutor
[11]. Converging evidence suggests text characteristics Algebra (CTA), that tracks student hint requests in addition
relating to reading difficulty are important when solving to whether they get problems correct or incorrect. We
mathematics word problems, but studies are needed that investigate two research questions: (1) How are readability
address which elements of reading difficulty are most and topic measures associated with correct answers and
important. hint requests when students label independent and
dependent quantities in stories in CTA? (2) How are
3.2 The Impact of Problem Topic on Solving readability and topic measures associated with correct
answers and hint requests when students write algebraic
Mathematics Story Problems expressions from stories in CTA? Answers to these
The topic of mathematics story problems also has an
questions could inform the design of future ITSs for
important relationship to students’ prior knowledge and
personalized instruction.
motivation. A study of high school students solving either
standard story problems or story problems personalized to 4. METHOD
topics they were interested in (e.g., sports, video games, Data from N = 3394 students with active CTA accounts
social networking) within one unit of CTA found that were collected from 9 high schools and 1 middle school
personalized stories were associated with higher that were diverse in terms of their socio-economic, racial,
performance. This performance gain was present in two and achievement background (Table 1). Data were
tasks – labeling independent and dependent quantities collected for students solving 151 distinct word problems
given in algebra story problems, and writing algebraic accross the first 8 units of CTA; later units were not
expressions from the story scenarios [12]. It was included because many students did not advance beyond
hypothesized that during these two tasks, students are these units. We collapsed for all analyses (i.e., treat as
working closely with the problem text, constructing their identical) problems containing an identical story but using
situation model and coordinating it with a problem model. slightly different numbers. On average, each problem had
This study also found that students receiving problems in been solved by 742 students (SD = 495). Each problem
the context of their out-of-school interests were less likely included a story scenario that outlined one or more linear
to game the system – to exploit regularities in hints and functions within a real world situation (Figure 1). The
feedback provided by CTA in order to avoid productive student was asked to complete steps in which they
learning behaviors. Further, students who received identified the independent and dependent quantities in the
personalization had stronger performance in future units story, wrote a linear algebraic expression for the story, and
where the problems were no longer personalized. solved their expression for different x and y values; we
In a recent follow-up study [13], story problems in four consider only the first two skills.
units of CTA were personalized to topics students were
CTA log data from students in the selected schools
interested in, and students solving personalized problems
were uploaded to DataShop (pslcdatashop.web.cmu.edu),
were compared to a control group solving normal
an online repository of detailed student interaction data.
problems. Results showed that personalized problems both
These logs contained information on whether the student
triggered students’ situational interest and enhanced
got each problem correct, incorrect, or requested a hint on
students’ individual interest for learning algebra.
their first attempt; because requesting a hint is a distinct
Personalization was associated with greater learning gains
outcome, correct and incorrect are not completely repetitive
than a control condition only when the personalization was
measures. Thus, for each problem, we compiled the
matched to deep features of the students’ interest area. This
percentage of students who had gotten the problem correct
was contrasted with personalization that was only matched
on the first attempt, incorrect, or requested a hint. This
surface features of the learners’ interests – i.e.,
percentage was our dependent measure in three distinct
modifications to the problems that simply involved
regression models.We analyzed the text of the introduction
inserting familiar pop-culture words rather than considering
to each story problem (i.e., the initial text that gives the
how learners might actually use relationships between
linear rate of change and intercept; see Figure 1) with the
quantities in their everyday activities. Thus converging
Coh-Metrix and LIWC text-mining programs. Coh-Metrix
[14] measures a large number of aspects of text readability,
including the amount semantic overlap between sentences,
the number of verbs, use of concrete versus abstract words,
the average sentence length, and others.
Table 1. Demographic characteristics of schools in study
ID Math State School School
Prof % Prof % Enrollment Type
1 88% 70% 797 Middle
2 81% 47% 1,482 High
3 95% 84% 2,163 High
4 55% 46% 708 High
5 27% NA 1,875 High
6 68% 59% 986 High
7 2% 31% 602 High
8 76% 84% 1,333 High
9 19% 39% 397 High
10 68% 79% 800 High
ID White Black Hispanic F/R Lunch
1 72% 7% 15% 21%
2 90% 4% 2% 4%
3 84% 10% 3% 6%
4 99% 1% 1% 41%
5 20% 4% 72% 77%
6 9% 2% 88% 41%
7 1% 99% 1% 82%
8 36% 60% 2% 48%
9 100% 0% 0% 45%
10 38% 51% 11% 62%
Because some of our story introductions had only one
sentence, measures that pre-supposed multiple sentences
Figure 1. Screenshot of algebra story problem in CTA with
were ommitted. LIWC [15] was used to determine the topic answer key superimposed
of the story problems – this program counts how many
words in the story fall into various word categories,
including social processes (family, friends, people), For each category in Coh-Metrix and LIWC, the
affective processes (positive emotions and negative correlation was computed between the list of each
emotions), biological processes (body, health, ingestion), problem‘s score on that category, and the percentage of
cognitive processes (insight, causation, discrepancy, students who got each problem correct, incorrect, or
tentativeness, certainty, inhibition, requested a hint. Correlations that were significantly
inclusive/exclusiveness), perceptual processes (see, hear, different from 0 were tested for inclusion as fixed effects in
feel), relativity processes (motion, space, time), and regression models predicting the performance measures
personal concerns (work, achievement, leisure, home, (hints, corrects, incorrects). These models included random
money, religion). If a story contained any words that fell effects that described various aspects of the problem’s
into one of these topic categories, that story was coded as a mathematical structure, including the unit and section it
1 for that category; otherwise it was coded as a 0. came from in CTA, and the numbers it used. Models were
initially fit using the lmer() command in R including all
potential fixed and random effects. Then we used the step()
command in R to perform backwards elimination on fixed
and random effects, leaving a model with only the effects abstract physics quantities – like distance and speed – may
that significantly improved the fit of the model. These have been more difficult for students than using quantities
analyses were carried out separately for a dataset that relating to specific concrete objects (e.g., accumulating
included only instances of students labeling independent cards, toys, or money). Finally, inhibition words were
and dependent quantities, and a dataset that included only associated with more hint requests. Inhibition words were
instances of students writing algebraic expressions. often included in story problems that discussed safety
issues or saving money. Students may have persieved these
5. RESULTS less concrete, finance- or safety-oriented contexts as less
accessible, making them more likely to request a hint rather
5.1 Labeling Independent and Dependent than attempt to write the labels. These problems often
Variables involved money as the dependent variable, but the label for
Regression results showing the relationship between this variable may have been complex because the actor in
performance measures (% incorrect, hint, and correct) and the story might have already saved or spent some money
readability and topic measures for labeling quantities in when the story started. Thus a label of simply money may
story problems are provided in Table 2. Table 2 shows that not be appropriate, and the student would have to enter a
problems that use adverbial phrases (DRAP) were label that captured that it was total money or net money
associated with fewer incorrect answers. Adverbial phrases saved or spent.
are phrases that add on to verbs, answering the questions
where, when, or how? In the present data set, adverbial 5.2 Writing the Algebraic Expression
phrases mostly answered when the action occured, and Regression results showing the relationship between
often included words like currently, already, next, first, performance measures and readability and topic measures
every day/week, and not yet. However, some of these for writing the expression are shown in Table 3. We again
adverbs also answered the how question, relying see that inhibition words – often associated with financial
information about quantities that might be useful to cue contexts – are more difficult for students – they are
students to the constraints of the problem – examples of associated with more incorrect answers, more hint requests,
words used in this manner included only, completely, and and fewer correct answers. The conceptual difficulty of this
evenly. These words may have given important details topic area might become especially important as students
about how the quantities involved in the story were move from formulating their situation model to
changing as the action in the story proceeded. coordinating their situation model with a problem model.
Table 2. Regression tables relating performance Table 3. Regression tables relating performance
measures on labeling quantities to readability/topic measures on writing expressions to readability/topic
categories categories
Estimate Std. Err t value Pr(>|t|) Estimate Std. Err t value Pr(>|t|)
% Incorrect % Incorrect
(Intercept) 0.182 0.032 5.63 0.00018 *** (Intercept) 0.195 0.060 3.26 0.00167 **
DRAP -0.0008 0.0003 -2.34 0.02104 * WRDPOLc 0.0494 0.013 3.91 0.00014 ***
motion 0.036 0.0137 2.65 0.00899 ** inhibition 0.086 0.034 2.52 0.01286 *
% Hint % Hint
(Intercept) 0.045 0.014 3.21 0.01407 * (Intercept) 0.055 0.014 3.95 0.00050 ***
One sentence (ref.)
inhibition 0.023 0.008 2.83 0.00543 **
Two sentences -0.045 0.016 -2.82 0.00548 **
% Correct
Three Sentences -0.057 0.017 -3.48 0.00067 ***
(Intercept) 0.784 0.044 17.87 0.00000 ***
4 + Sentences -0.033 0.019 -1.77 0.07868
motion -0.042 0.0182 -2.29 0.02370 *
RDL2 0.002 0.001 3.51 0.00061 ***
family 0.030 0.015 2.05 0.04282 *
Stories that involve motion words (e.g., go, move, ran, inhibition 0.052 0.011 4.74 0.00001 ***
arrive, come, enter, threw) are associated with more
incorrect answers and fewer correct answers. These stories motion 0.025 0.009 2.77 0.00637 **
often incldued contexts where people were walking, biking, % Correct
hot-air-balooning, driving, or actively constructing (Intercept) 0.334 0.17478 1.91 0.05778
something. In terms of the quantities used, there was often
a rate of change (e.g., per hour, per minute, a day) that LDTTRc 0.428 0.169 2.53 0.01242 *
involved this motion, and students had to identify the two WRDPOLc -0.041 0.01469 -2.78 0.00609 **
quantities that made up this rate of change. Using more
inhibition -0.128 0.03909 -3.28 0.00132 ** measure (RDL2) was associated with greater hint-seeking
when writing expressions. This measure is calculated
through measures of word frequency (with words that occur
Another factor that stands out in the regression results more frequently in the English language yielding higher
is word polysemy (WRDPOLc) – or the number of different scores), sentence syntax similarity (with sentences that
meanings that a word has (for example, in English, mine have similar grammatical structures yielding higher scores),
can be something you own or an explosive device). The and word overlap (with words that share semantic meaning
results show that stories that contain words with more yielding higher scores; [16]). Given that a higher second
potential meanings are associated with more incorrect language readability score is typically associated with
answers and fewer correct answers. Polysemous words greater ease in comprehending the text [17], it is suprising
have been found to make mathematics word problems more that stories that score higher on this measure would be
difficult to interpret accross other studies [8-9]. associated with students seeking more hints. The
Results also showed that higher type-token ratios explanation of this finding may be similar to that for our
(LDTTRc) are associated with more correct answers. As finding with type-token ratio; story problems that use
type-token ratio increases, more unqiue words are being similar words and sentence structures often use a lot of
used in the story problem, and fewer words are being reptition as a way to present complex ideas. Stories that are
repeated. These results suggest that students have an easier simple and concise may be easier for students to solve.
time writing the expression in a story that is relatively
concise with little reptition of ideas. While it makes sense 6. DISCUSSION
that this type of story may be more amenable to translation Results indicate that readability and topic measures have
into mathematics notation, this result contrasts with important associations with students‘ performance when
research in text comprehension in reading tasks [14] which solving mathematics word problems in an ITS. In
generally finds that repitition and lower type-token ratios particular, it was more difficult for students to name the
facilitate reading comprehension. However, the story independent and dependent quanitities in problems relating
problems with high levels of word repetition frequently to motion (physics) and inhibition (saving and safety),
discuss complex topics of which students may lack while adverbial cues facilitated this skill. When writing
familiarity, including operating capital, business inventory, algebraic expressions, we again see that motion and
and wholesale prices. In this way, a high type-token ratio inhibition topics are difficult, but also find other important
may be indicative of a complex topic rather than increased readability measures that matter. Words with multiple
readability in these story problems. meanings make story problems more difficult, which
Students‘ tendency to seek hints when writing the corresponds to previous findings in both mathematics and
algebraic expression is associated with a number of reading education.
different readability factors. First, we see an effect for the However, mathematics stories that use concise
length of the story text; students are more likely to seek language with little repitition, which in terms of their
hints for one sentence story problems, compared to readability level makes them technically less readable, are
problems that have two or more sentences. Having only one actually easier for students to solve. Thus measures of
single sentence in a story problem might not be enough to readability that stem from research on reading
ground or fully describe a linear rate of change as it arises comprehension may need to be considered differently when
in a real-world situation, and these overly-sparse stories working with mathematics problems. Results also suggest
might consequently inhibit performance. that while a story problem that includes only a single
In addition to greater difficulty of inhibition words, sentence is concise, it might present difficulty for students
stories with family words and motion words were by not providing necessary context and information for
associated with greater hint-seeking. Only 13 of the them to feel they can respond without needing a hint.
problems involved family words, and these were often Overall, our results suggest that mathematics story
complex scenarios where multiple actors (e.g., a main problems that have story texts that are more accessible to
character and his brother) were each contributing to the students have several characteristics: (1) they are concise
algebraic rate of change in their own way (e.g., with little repetition, but not a single sentence only, (2) they
saving/earning/splitting money together). Motion words use only a single actor performing actions, (3) they use
often involved physics contexts (e.g., traveling in a car or simple words with clear meanings, (4) they avoid more
plane) in which students had to track distance, rate, and abstract physics or financial contexts, instead focusing on
time. This suggests that keeping track of multiple familiar contexts involving accumulation or loss of
individuals engaging in mathematical actions and solving concrete physical objects, and (5) they make use of
problems with physical distances and rates may be adverbial cues. Story problems with these characteristics
significant difficulty factors when writing expressions. may allow students to more easily construct a situation
Finally, the regression results showed that scoring model from a propositional textbase. They may promote
higher on Coh-Metrix’s second language readability situation-model construction by both increasing students‘
ability to comprehend the semantics of the problem, and by situation-based reasoning with algebraic representation.
increasing students‘ interest in working on the problem. Journal of Mathematical Behavior, 31(2), 174-195.
[6] Hidi, S., & Renninger, K. 2006. The four-phase model of
7. CONCLUSION interest development. Educational Psychologist, 41(2), 111-
Future adaptive ITSs will be designed to model student 127.
characteristics at an extremely fine-grained level, as [7] Vilenius-‐Tuohimaa, P. M., Aunola, K., Nurmi, J. E. 2008.
technology for personalized learning continues to advance. The association between mathematical word problems and
Here we argue that an important element of these future reading comprehension. Educational Psychology, 28(4), 409-
adaptive systems will be a consideration of the non- 426.
mathematical text-based characteristics of the problem [8] Shaftel, J., Belton-Kocher, E., Glasnapp, D., Poggio, J. 2006.
tasks they present to students. Making inferences about The impact of language characteristics in mathematics test
students‘ current level of mathematical knowledge or items on the performance of English language learners and
motivation without considering these characteristics may students with disabilities. Educational Assessment, 11(2),
105-126.
lead to misspecifications.
[9] Wolf, M. K., Leon, S. 2009. An investigation of the language
Readability and topic measures may be an important demands in content assessments for English language
consideration for ITSs to model in a variety of domains, learners. Educational Assessment, 14(3-4), 139-159.
including when considering tasks from history, social
[10] Doddannara, L. S., Gowda, S. M., Baker, R. S., Gowda, S.
studies, and science. Future research should focus on the
M., De Carvalho, A. M. 2011. Exploring the relationships
readability and topic measures that are most important for between design, students’ affective states, and disengaged
students of different age groups in different subject behaviors within an ITS. In: Proceedings of the 16th
domains, and narrow down which characteristics are most International Conference on Artificial Intelligence and
critical to include in student and domain models as we Education, pp. 31-40.
build future ITSs. In current work, we are analyzing the [11] Baker, R. S., de Carvalho, A. M. J. A., Raspat, J., Aleven, V.,
mathematics problems on the National Assessment of Corbett, A. T., Koedinger, K. R. 2009. Educational software
Educational Progress (NAEP) and Trends in International features that encourage and discourage “gaming the system.”
Mathematics and Science Study (TIMSS) to examine how In: Proceedings of the 14th International Conference on
readability and topic measures impact the performance of Artificial Intelligence in Education, pp. 475-482.
4th and 8th graders in the United States, and how these [12] Walkington, C. 2013. Using learning technologies to
factors interact with cognitive and non-cognitive student personalize instruction to student interests: The impact of
background characteristics. relevant contexts on performance and learning outcomes.
Journal of Educational Psychology, 105(4), 932-945.
8. REFERENCES [13] Bernacki, M. & Walkington, C. 2014. The Impact of a
[1] Padayachee, I. 2002. Intelligent tutoring systems: Personalization Intervention for Mathematics on Learning
Architecture and characteristics. University of Natal, Durban, and Non-Cognitive Factors. Submitted to the 2014
Information Systems & Technology, School of Accounting International Conference of Educational Data Mining,
& Finance. London.
[2] Ritter, S., Anderson, J. R., Koedinger, K. R., Corbett, A. [14] Graesser, A. C., McNamara, D. S., Louwerse, M. M., Cai, Z.:
2007. Cognitive Tutor: Applied research in mathematics Coh-Metrix: Analysis of text on cohesion and language.
education. Psychonomic Bulletin & Review, 14(2), 249-255. Behavior Research Methods, Instruments, & Computers,
[3] Nathan, M. J., Kintsch, W., Young, E.: A theory of algebra- 36(2), 193-202 (2004)
word-problem comprehension and its implications for the [15] Pennebaker, J. W., Chung, C. K., Ireland, M., Gonzales, A.,
design of learning environments. Cognition and Instruction, Booth, R. J.: The development and psychometric properties
9(4), 329-389 (1992) of LIWC2007. Austin, TX, LIWC. Net. (2007)
[4] Walkington, C., Petrosino, A., Sherman, M. 2013. [16] Crossley, S., Allen, D., McNamara, D. 2011. Text readability
Supporting algebraic reasoning through personalized story and intuitive simplification: A comparison of readability
scenarios: How situational understanding mediates formulas. Reading in a Foreign Language, 23(1), 84-101.
performance and strategies. Mathematical Thinking and [17] Crossley, S. A., Greenfield, J., McNamara, D. S. 2008.
Learning, 15(2), 89-120.
Assessing text readability using cognitively based
[5] Walkington, C., Sherman, M., & Petrosino, A. 2012. indices. TESOL Quarterly, 42(3), 475-493.
‘Playing the game’ of story problems: Coordinating