While past research has shown that learning outcomes can be influenced by the amount of e↵ort students invest during the learning process, there has been little research into this question for scenarios where people use search engines to learn. In fact, learning-related tasks represent a significant fraction of the time users spend using Web search, so methods for evaluating and optimizing search engines to maximize learning are likely to have broad impact. Thus, we introduce and evaluate a retrieval algorithm designed to maximize educational utility for a vocabulary learning task, in which users learn a set of important keywords for a given topic by reading representative documents on diverse aspects of the topic. Using a crowdsourced pilot study, we compare the learning outcomes of users across four conditions corresponding to rankings that optimize for di↵erent levels of keyword density. We find that adding keyword density to the retrieval objective gave significant learning gains on some topics, with higher levels of keyword density generally corresponding to more time spent reading per word, and stronger learning gains per word read. We conclude that our approach to optimizing search ranking for educational utility leads to retrieved document sets that ultimately may result in more ecient learning of important concepts.
The Web has become a primary source of online
information for learning-related tasks [
Exploring new topic areas and learning important domain
Search as Learning (SAL), July 21, 2016, Pisa, Italy
The copyright for this paper remains with its authors. Copying permitted
for private and academic purposes.
vocabulary is one popular instance of a learning task [
Toward that goal, the main contributions of this work are a novel search algorithm that re-ranks for optimized educational utility using keyword density as a proxy for eo↵rt, and a study that evaluates the e↵ectiveness of this approach on actual learning outcomes. 2.
While research on ranking algorithms to maximize the
relevance of generic or personalized search results is
wellestablished, few studies have focused on algorithms that can
optimize results with utility for an educational goal as the
retrieval objective. Researchers have recognized the
importance of going beyond traditional retrieval evaluation
measures to consider user progress over time [
Our retrieval approach has three stages: (1) given a topic expressed as a query, selecting appropriate aspects to be learned for each topic, with each aspect represented by a keyword, (2) for each aspect (keyword), determining the total frequency with which the keyword should occur in the retrieval results, and (3) developing a retrieval algorithm for vocabulary learning that finds documents to ‘cover’ the selected keywords eciently by including the keyword density of the documents as an adjustable sub-objective. 3.1
For each topic in our study, we manually collected a set of exemplar Web documents D⇤ that were deemed to be representative of useful knowledge about that topic. We then represented the vocabulary learning goal for a given topic as a weighted set K = {k1, . . . , kN } of keywords, which we call the target keywords, derived from the topic’s exemplar set. For this study, we chose the top N = 10 most representative keywords for each topic, using a measure of term frequency weighted by inverse term log-frequency in a global corpus. As di↵erent aspects of a topic may have greater or less relevance in understanding the topic, each keyword is assigned an associated weight wi, where w are the parameters of a multinomial distribution estimated from the frequency counts of the keywords in the representative set D⇤ . Table 1 shows the top 5 out of 10 keywords generated for each topic, along with their relative weight wi (in parentheses). 3.2
We assume that a student’s knowledge of each topic keyword ki monotonically increases with each instance of it that they read. Now let T be the total keywords the learner reads. The distribution of T among the N keywords will be proportional to the importance of each keyword, given by wi. Then, if si is the total instances of ki the learner reads, we have: si = T · wi.
Ideally, a student would learn the most with unlimited instances of each keyword (T = 1 ). However, in reality a student’s time and e↵ort will limit the amount of training they experience, so the T value for each topic was manually chosen to produce small document sets (less than 15 documents). 3.3
As a baseline retrieval algorithm, we used the intrinsic
diversity algorithm developed by Raman et al. [
With this extension, setting ↵ = 0 recovers the original intrinsic diversity results, while higher values of ↵ result in document sets with increasingly dense keyword coverage.
More specifically, ✏i is the normalized contribution that document di o↵ers in terms of how much closer it brings the student towards reading the total required number of keyword instances (the sj counts). Let CD represent the cumulative keyword counts the student has seen so far from documents higher in the ranking, and Ci represents the keyword frequency distribution of di. Then we have: ✏i = 1 |di| j=1
XN ⇢ Cij max(0, sj
Cij + CDj sj CDj ) otherwise The term ✏i e↵ectively is a measure of the keyword density in di with respect to the target keywords for the topic. By rewarding documents that have higher density, via the choice of a higher ↵ setting, we enable the learner to reach the target sj counts faster.
Our implementation of the intrinsic diversity algorithm determines the base query’s sub-topics by analyzing the corresponding Wikipedia article on that query’s topic. It generates sub-topic queries by extracting the main header topics in the article and appending them to the base query. For example, for the query “DNA”, some sub-topic queries were: “DNA Properties” and “DNA Biological functions”. We then fetch the top 70 Google search results for the base query and the top 70 results for each of the sub-topics queries and run optimization problem (1).
We intend to refine our subtopic extraction methods to generalize beyond those available in Wikipedia topics in future work. In general, many di↵erent variables can simultaneously influence learning. Some students may learn better with multimedia aids, some will learn better with pure text documents, some will benefit from more technically-worded documents and so on. In this paper, we will specifically evaluate only Web documents that contain only text and, at most, supplementary pictures. 4.
To assess the potential e↵ect on learning outcomes of
retrieved documents optimized using die↵rent levels of
keyword density (choices of ↵ ), we ran a crowdsourced user
study that involved a vocabulary learning task: learning the
target keywords. Participants first completed a
multiplechoice pre-test to assess their existing knowledge of the
keywords, then based on the condition, read through a
provided retrieval set of documents containing the keywords
to be learned, and then completed an immediate post-test
to assess their updated keyword knowledge. We ran five
separate crowdsourced jobs corresponding to five di↵erent
topics selected to cover a range of scientific topics: Igneous
rocks (geology), Tundra (environmental science), DNA
(genetics), Cytoplasm (biology) or GSM (telecommunications).
For each of these topic jobs, a participant was randomly2
assigned to one of four di↵erent keyword density conditions,
2Participants were sorted into conditions based on
Crowdflower’s random assignment to tasks.
corresponding to ↵ settings of [
The pre- and post- vocabulary tests consisted of a series of multiple-choice questions, one for each of the K keywords. Both the pre- and post-reading tests were constructed with identical questions so that we could investigate the participants’ learning gain for each vocabulary term by looking at the di↵erence in scores 3.
We used the Crowdflower platform for this study. Participants were oe↵red US$0.04 per page (the equivalent of US$3.20/hr) for completing the tasks. For quality control, in addition to Crowdflower’s proprietary mechanisms and ‘gold standard’ questions, we limited the participant pool to users from the U.S. and Canada, given the vocabulary-centric nature of the task and reliance on English reading skills. We also o↵ered the tasks only to workers in the highest quality (level 3) pool, and only kept responses from those workers who spent at least four minutes on the task.
The particular set of documents shown to each participant was based on which ↵ condition they were assigned. We gathered data for 35 participants per ↵ condition per topic, resulting in a total of 140 participants per topic and 700 participants overall. After excluding those who didn’t pass the test questions and those who didn’t complete the full task, we ended up with 616 total participants.
Overall, our analysis showed that die↵rent choices of ↵ were in fact associated with di↵erences in learning, as measured by both absolute and normalized gains from pre-test to post-test.
We first analyzed learning gains (sum of learning gains for all K keywords) across the four ↵ conditions. Retrieval results incorporating higher keyword density gave statistically 3In measuring ‘learning gain’, we assume no memory loss so the learning gain is always non-negative.
Keyword Difficulty Low High 0.20 significant mean learning gains for two out of the five topics4 (Table 2). Both of these topics showed a peak learning gain at the ↵ = 80 condition, suggesting that a combination of lowering eo↵rt via the keyword density parameter and rewarding intrinsic diversity in documents o↵ers better learning gains than either factor alone. However, we also found that the setting of ↵ = 120 yielded the worst learning gains in those same topics. This suggests that the learning gains are quite sensitive to the particular choice of ↵ and that choosing an ↵ that combines both the ID objective and the keyword density objective is not always going to improve learning utility. It’s not entirely clear why the specific value of ↵ = 80 oe↵red better performance but we intend to investigate this further and how to algorithmically choose ↵ in future work, using an extended set of topics.
Since the target keywords ranged from more familiar to more technical, and learning gains could be expected to interact with keyword diculty, we faceted the learning gain results by low- and high-diculty keyword categories 5. Figure 1 shows the result of averaging the learning gains for each keyword in the two diculty categories and then averaging the results across the five topics. We see that there were learning gains in all conditions for both low- and highdiculty keywords, but as expected, learning gains were higher for the higher-diculty (and thus initially less familiar) keywords. 4For all ANOVA analysis reported, the same significance ranges were found using bootstrapped ANOVA and the Kruskal-Wallis test. 5Keywords were split into two groups of five keywords according to their age of acquisition (AoA) score in a standard psychometric database. If a keyword didn’t have an AoA score, it was assumed to be maximum diculty.
Next, as a measure of learning eciency, we evaluated absolute learning gain normalized by the total words read. This measure incorporates e↵ort such that, for two students scoring the same absolute gain, the one who achieved this gain with less eo↵rt (reading less text) is rewarded more. ANOVA analysis of the di↵erent ↵ levels shows that most topics had strongly significant di↵erences in means. There was a general trend of increasing gains with increasing ↵ and several topics achieved maximum gains at ↵ = 1 (Table 3).
We note that one topic, Cytoplasm, showed an opposite trend where higher alpha values mostly lead to worse normalized learning gains. We hypothesize that this may be because the total number of words used in each condition for Cytoplasm were significantly lower (almost half as many for ↵ = 0 and ↵ = 80) compared to the four other topics. It is thus possible that the positive impact of choosing higher ↵ values is only e↵ective after passing a certain threshold of minimum reading material. 5.2
When considering learning gains per unit time (Table 4) instead of per word, the results were much less conclusive: for example, two topics showed significant di↵erences in mean learning per time, but with opposite extremes of ↵ values (0 and 1 ). To better understand the factors a↵ecting learning gain per unit time (denoted T imLe ), consider the following decomposition: α = 80 α = 120
Alpha Penalties α = infty
This relationship is visualized in Figure 2, with WT iomrdes on
L the x-axis and W ords on the y-axis. As the plot makes evident, there is a positive correlation (r=.42, p=.06) between these two components. However, while the slope of this approximately linear relationship (which is exactly T imLe , learning per unit time), is relatively stable across conditions, there are very die↵rent tradeo↵ regimes depending on the value of ↵ : the ↵ = 0 condition is characterized by some of the shortest reading times per word and lowest learning gains per word, while the ↵ = 1 condition is characterized by the highest times and learning gains. Thus, while the overall learning gain per unit time (ratio of the two components) may not change dramatically across conditions, the underlying two components, representing the tradeo↵ users choose between reading time and learning eciency, vary greatly as keyword density changes greatly.
To gain more insight into why pages with increased
keyword density might contribute to more ecient learning, we
investigated additional properties of the page content that
might be correlated with keyword density. We found that
while few result documents made use of multimedia such as
animations, audio or video, a number did use images to
supplement the text. Thus, the picture superiority e↵ect [
We consider the possibility that a heavier coverage of images in teaching documents can improve learning outcomes regardless of condition. There is partial evidence of this in that ANOVA analysis of the topics “Igneous rock”, “Tundra” and “DNA” showed no statistical significance in means (Table 2) and these three topics had the top three average image coverage (.0024, .0026 and .0034 respectively). On the other hand, the two topics that showed significant differences (“Cytoplasm” and “GSM”) had the lowest coverage (.0015 and .0006 respectively). As such, it is possible that a higher image coverage can collectively improve or worsen learning gains regardless of conditions. Determining if the presence or absence of images actually has such an ee↵ct warrants further investigation.
We observe informally that pages using a higher density of keywords tend to be those that give an overview of topic for instructional purposes, and thus are more likely to be supplemented with images by the author. We intend to investigate this phenomenon and other content properties that may interact with learning in future work.
We introduced a novel algorithm for optimizing Web search results for a learning-oriented objective – a vocabulary learning task – by extending intrinsically diverse ranking to incorporate a keyword density sub-objective. This keyword density was controlled by a parameter ↵ that rewarded documents containing a high density of topic-relevant keywords. The result was an algorithm that not only gave relevant, diverse results to explore new topics, but also emphasized ecient keyword coverage in the results content, thus allowing learners to potentially expend less e↵ort toward their learning goal. We hypothesized that changing the keyword density ↵ would be associated with positive changes in users’ vocabulary learning outcomes. We tested this hypothesis with a crowdsourced pilot study based on five topics, across four conditions that varied keyword density by using die↵rent values of ↵ . We found that for some topics participants did in fact show stronger learning gains per word with nonzero ↵ settings. Of the four topics that showed significant di↵erences of means, three were maximized at ↵ = 1 . This is an interesting finding as the ↵ = 1 condition only considers the keyword density as its objective which means that our findings suggest that a search algorithm that is blind to the rank or implicit quality of a document is o↵ering better results than an algorithm that explicitly considers such measures. We also examined learning gains per word and per unit time, finding that users showed very di↵erent tradeo↵s between reading time per word and learning gains per word in low- vs high keyword density conditions. In future work we intend to explore criteria for selecting optimal operational settings of ↵ , and to incorporate more personalized components in the retrieval model.
Acknowledgements This research was supported in part by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A140647 to the University of Michigan. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.