We investigate whether student comprehension and knowledge retention can be predicted from textbook annotations, speci cally the material that students choose to highlight. Using a digital open-access textbook platform, Openstax, students enrolled in Biology, Physics, and Sociology courses read sections of their introductory text as part of required coursework, optionally highlighted the text to ag key material, and then took brief quizzes as the end of each section. We nd that when students choose to highlight, the speci c pattern of highlights can explain about 13% of the variance in observed quiz scores. We explore many di erent representations of the pattern of highlights and discover that a low-dimensional logistic principal component based vector is most e ective as input to a ridge regression model. Considering the many sources of uncontrolled variability a ecting student performance, we are encouraged by the strong signal that highlights provide as to a student's knowledge state.
Digital textbooks have become increasingly available with the popularity of
ereaders and the advent of open-access learning resources such as Openstax. Like
other researchers in AI and education, we see valuable opportunities to
observing students as they interact with their textbooks and become familiar with new
material. For mathematics or physics courses, where students can demonstrate
their understanding by working through exercises, researchers have long had the
opportunity to observe students' problem-solving skills and to suggest hints and
guidance to remediate knowledge gaps [
In this article, we are interested in highlighting|the yellow marks and
underlines that students make in a textbook in order to emphasize material that
they perceive as particularly pertinent. Although there is a well-established
research literature in educational psychology examining whether highlighting
bene ts student learning [
Recently, our team conducted two studies that provide preliminary evidence
in support of the hypothesis that highlights provide insight into comprehension.
In a laboratory experiment, Winchell et al. [
In contrast, Waters et al. [
Neither of these studies is completely satisfying. The Winchell et al. study was was conducted via Mechanical Turk with 200 participants with unknown motivation levels. It involved just three passages and twelve quiz questions (formulated either as multiple choice or ll-in-the-blank) and took place over 40 minutes. Consequently, its application to authentic digital learning environments is unclear. The Waters et al. study was on a much larger scale in the context of actual coursework, but their predictive models were limited in scope: the models considered only the highlighting of a critical sentence, whereas Winchell et al. constructed predictive models based on the pattern of highlights in the section. It's possible that the strongest predictor of subsequent recall by a student may not be whether the critical sentence was highlighted but by the highlighting of material the precedes or follows the critical sentence.
In this article, we aim to integrate the focus of these two previous studies, in order to determine the e ectiveness of models that leverage the pattern of highlights a student produces to predict quiz performance in an authentic digital learning environment. We utilize the Openstax Tutor corpus of Waters et al. and construct a model for each section of text, predicting mean quiz performance for that section based on the entire set of highlights in that section. 2
Our analyses use data collected from Openstax Tutor [
The digital textbook environment provided an annotation facility that allows students to highlight critical material in the text by click-and-dragging the mouse over the material they wished to emphasize. Highlights could also be undone. Highlighting was optional. Some students never used the facility, others used it for only some sections. Figure 1 shows highlighting patterns of two di erent students for the same section of material.
Our analyses were all conducted by section. For each student and each section, we grouped together all questions answered by the student, both core questions and space-practice questions. The student's score is the mean proportion correct of all questions that are associated with the section. For each score, we recovered the highlighted character positions in the section. These positions consist of the complete list of indices of characters in the section that the student highlighted. The rst character in the section is indexed as position 1, and so forth. From the highlighted character positions, one can recover the exact pattern of highlights marked by the student. Because of a quirk in the raw data base, we had to recover the highlighted positions from the unindexed collection of literal words, phrases, and sentences that the student highlighted. In almost all cases, the highlighted positions could be recovered unambiguously. In a few cases, such as when the student highlighted a single word which appeared multiple times in a section, we assumed that the index was its rst occurrence in the section. This ambiguity arose very rarely.
We grouped the data by section. Each section is analyzed independently, and we report mean results across sections. Because the textbooks were electronic, they were revised during the time period in which we obtained data. As a result, some sections have multiple versions. We collapsed these revisions together since typically only a few words changed from one version to the next, and it was easy to align the highlighted fragments.
Table 1 presents an overview of the data set. There are a total of 4,851 students, 1,157 distinct sections, and 479,879 sessions, where a session consists of a particular student reading a particular section. Students answered one or more quiz questions in only 328,575 sessions, and students highlighted portions of the text on only 8,846. One surprising observation is the relative scarcity of highlights during reading, given that students consider highlighting to be a fruitful study strategy. However, highlighting in an electronic text may be awkward or unfamiliar to students.
Nonetheless, we have adequate data to consider with the 8,000+ sessions with highlights. We focus on the sections which had a critical mass of students who highlighted. We identi ed 28 such sections, with the largest section having 142 highlighters and the smallest section having 31 highlighters. Across the highlighted sessions, the mean quiz score is 75% with a standard deviation 10%.
3.1
Is highlighting associated with higher quiz scores?
We rst report on some simple analyses showing that highlighting is associated
with higher quiz scores. Although we cannot ascertain a causal relationship, we
can eliminate some confounders. Waters et al. [
In a rst analysis, we divided sessions by course topic and by whether students had made highlights during that session. In Figure 2a, we show mean scores with 1 SEM bar by course topic. Across all topics, highlighted sessions are associated with higher quiz scores than non-highlighted sessions (Table 2).
This analysis of course does not indicate that highlighting has a causal e ect on performance. Possible non-causal explanations include: { More diligent students may tend to highlight and more diligent students study hard and therefore perform better on quizzes. { Whether or not a student highlights may be correlated with the di culty of the material. For example, a student who is struggling to understand material may not feel con dent to highlight, leading to lower scores for nonhighlighted sections.
To address these explanations, we conducted further analyses. To rule out differences in student diligence being responsible for the e ect, we performed a within-student comparison of highlighted versus non-highlighted sections. We consider only students who have both highlighted and nonhighlighted sections, and we compute the mean score by student when they highlight and when they do not highlight. This within-student comparison is shown in Figure 2b for each of the three course topics as well as a mean across topics. We nd the same pattern as before that mean student scores are higher for highlighted than for non-highlighted sections (Table 3). However, the di erence for Sociology is not statistically reliable.
To rule out the possibility that the decision to highlight is in some way
contingent on the di culty of the section, we used item-response theory, speci cally
the Rasch model [
Although we haven't de nitely ruled out non-causal explanations for the
relationship between highlighting and scores, our results are suggestive that in a
digital textbook setting, highlighting may serve to increase engagement which is
re ected in improved scores. This nding goes somewhat counter to the research
with traditional printed textbooks that fails to nd value for highlighting as a
study strategy [
Can we predict scores from the speci c pattern of highlighting? The analysis in the previous section simply considered whether or not a student highlighted a section, but ignored a rich information source|the speci c words, phrases, and sentences that were highlighted. Our goal is to determine whether highlighting patterns help explain scores. Here we use only sections of the Biology text, which had the greatest number of student highlighters. We model each section independently and we include only students who highlighted one or more words in the section. Our models predict a speci c student's quiz score from the speci c pattern of highlighting that student made. The pattern of highlighting is encoded in a vector representation, and we explore a range of representations which we explain shortly. We use the simplest possible model|a linear regression model with the vector representation as the regressor and quiz score as the regressand. Figure 3 shows an excerpt of text from one section, whose aim is to summarize the steps of the scienti c method and the nature of scienti c reasoning. The words in the text are color coded to indicate whether highlighting that word raises (red) or lowers (blue) the model's prediction of quiz score. Notice that material pertaining to hypothesis testing is associated with better performance and the sentence pertaining to E. coli bacteria is associated with worse performance. Although the E. coli sentence is substantive, it is not the focus of this section of text. Figure 1 shows the highlighting patterns of two students for this section. The top and bottom patterns are for students who score 54% and 100% on the quiz. The model correctly predicts the ranking of the two students.
We express accuracy of models in terms of the proportion of variance in quiz score explained by the highlighting pattern. Any non-zero value indicates some explanatory power. Figure 4 shows results from a variety of representations, which we will explain shortly. The bottom-line nding is that speci c highlighting patterns can explain about 13% of the variance in scores. This is a fairly impressive e ect considering the very large number of factors and in uences on a student's performance. For instance, there is some intrinsic variability due to the fact that questions were sampled randomly, and some of the responses were collected immediately after reading while others were collected after a retention period. There is further variability due to the student's momentary state of engagement, the conditions under which they study, and their prior background with the material.
We built models to predict quiz score from a representation of the highlighting pattern. Separate models were constructed for each section using only the data from students who highlighted at least some words in the section. For this research, we decided to stick to a simple linear model|ridge regression|and to focus on how the highlighting pattern is represented. The results we report are obtained via 10-fold cross validation. The L2 penalty term was weighted with a coe cient of 0.01, chosen by brief manual experimentation to produce models only slightly di erent than straight-up linear regression.
In our earlier modeling work using laboratory data [
In our laboratory study [
We were concerned about over tting, considering that the smallest data set had only 31 students and the number of model parameters could be greater than the number of data points. To address this concern, we performed logistic principal components analysis (LPCA) to reduce the dimensionality of the highlighting representation. We formed binary vectors with one element per word in a section. Element i of the vector for a given student was set to 1 if the student had highlighted word i in the section. Feeding these word-level vectors into LPCA, we obtained the LPCA decomposition of the vector space and LPCA representation of the highlights for each student. We constructed models using the top k components for various k.
To address the over tting issue, we varied the number of components to be proportional to the size of our data set. With S being the number of students, we expressed k as a proportion of S, k = S= , for 2 f1; 2; 3; : : : ; 20g. In Figure 4, the blue points labeled pca(S/ ) indicate that increasing leads to better score predictions.
In a nal series of simulations, we selected k not based on the size of our data set but on the word length of the section. With W being the number of words in a section, we chose a percentage as the dimensionality of the reduced representation, i.e., k = 100 W . The purple points in Figure 4 labeled pca( ) show the bene t of decreasing to obtain the surprising nding that with 30%, we see a signi cant boost in the model's predictive power over previous models. It is reassuring that the precise choice of does not seem to matter, suggesting that the result is robust.
In all of the above approaches, we nd that decreasing the dimensionality of the highlighting representation is bene cial. This nding could either be due to over tting issues, as we have speculated, or to the fact that there is lowdimensional structure in the highlighting patterns. We suspect it is the former, and plan to conduct further investigations optimizing the number of LPCA components based both on S and W . Of course, any results we obtain by the present cross validation methodology will need to be con rmed by tests using another data set; at this point, we cannot entirely trust that true model performance will be as good as is suggested by the best of our cross validation scores. 4
We nd that with a suitable representation of a student's highlighting pattern, we can explain about 13% of the variance in their test performance. While 13% is not on an absolute scale a large fraction of the variance, one must consider the many factors that play into a student's learning and retention, including their interaction with course materials outside of the textbook (e.g., in class, homework, etc.), their prior knowledge, conditions in which they are reading the text, and their degree of engagement with the current material and past sections. Given these highly in uential factors, it's remarkable that as much as 13% of variance can be explained by highlighting patterns.
We found that choice of highlighting representation was critical in determining how useful highlights are to predict quiz performance. Without theoretical justi cation for the representation which yielded the best predictions (the top 10% of principal components), we require additional empirical validation to argue convincingly that this representation will also serve us well for other students and other texts. Nonetheless, the fact that the PCA(10%) representation was superior across all three courses provides some reason for optimism. The fact that it is a fairly compact encoding of the myriad possible highlighting patterns also o ers promise that we may be able to interpret the relationship between these components and course content.
In past work using data produced by laboratory participants [
Our research is important and novel in three particular respects. First, our
results extend across a large sample of students, course topics, and speci c
content. Second, we move outside a laboratory setting (e.g., [
Our research has several potential limitations of this research. First, due to the fact that Openstax Tutor selects questions aimed to be at an appropriate level for students, there is some possibility of a confound that yields an optimistic estimate of the utility of highlights. For instance, it's possible that more motivated students tend both to highlight and to attain a certain level of performance that drives the speci c questions being selected. Second, we have not used all the potential information in the highlighting patterns: in principle, we could leverage dynamical information about the order in which highlights are made, the time lags between highlights (which indicate the pace of reading), and the deletion of highlights (which presently do not register in our analyses). Third, we do not consider individual di erences among students except insofar as their highlighting pattern is concerned. Because students will use Openstax resources over the duration of a course semester, we have opportunity to make multiple observations from the same student and to assemble a pro le of that student which ought to provide additional information for interpreting their textbook annotations. Future research will address these issues.
In this article, we've focused on using highlights to model student comprehension, but highlighting is a rich data source for inferring student interests and foci. We might leverage this fact by, for example, clustering students into interest groups based on similarity of patterns of highlighting, or even group students who show disparate highlighting patterns in order to provoke discussions of what material is important. There is also potential to leverage population highlights as a means of feedback to textbook authors and instructors. If students are highlighting unimportant material or failing to highlight important material from the author's or instructor's perspective, perhaps the textbooks should be rewritten or students should be guided to the material that is deemed to be most important. 5
This research is supported by NSF awards DRL-1631428 and DRL-1631556. We thank Christian Plagemann and three anonymous reviewers for their helpful feedback on earlier drafts of this manuscript.