Over the last 10 years, the world experienced a rapid increase in volume and diversity of digital learning resources. The abundance of digital resources could support a range of powerful educational scenarios, which were not available before. In this paper, we introduce a novel approach that combines fully automatic knowledge modeling, student modeling, and content recommendation approaches to recommend relevant Wikipedia articles for students working with online electronic textbooks. An assessment of our approach with real classroom data indicated several bene ts of our approach over the baseline and revealed interesting patterns of students' behavior while using the system.
Over the last 10 years, the world experienced a rapid increase in volume and
diversity of digital learning resources. On the one hand, a variety of tutorials,
online textbooks, educational videos, and other open educational resources were
posted online to complement traditional textbooks. On the other hand, almost
all traditional textbooks have migrated to digital format and become available
online [
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
The ideas of this \smart" learning have been explored in early projects
focused on adaptive textbooks [
Following a brief review of related work, this paper introduces the interface of our digital textbook reading system with embedded recommendations. The next four sections introduce the underlying mechanisms of our intelligent textbook: the domain and student modeling approaches, the knowledge graph, and the recommendation approach based on this infrastructure. The following section presents the evaluation of the recommendation approach based on real classroom data. We conclude with a discussion and future work plans. 2
Research on recommendation of related reading sources has deep roots in
research on educational hypertext and hypermedia. Historically, it has been
performed under the name of \intelligent hypertext", since this approach
recommended resources that were not connected by a human-authored link. Research
on intelligent hypertext started in the early days of the educational hypertext
eld and originally focused on linking resources using term-based resource
similarity [
The emergence of MOOCs and the accumulation of large volume of
educational content online encouraged a new wave of research on \intelligent" linking
focused on connecting primary learning content such as textbooks and MOOCs
with several kinds of external learning resources such as videos, Wikipedia pages,
or research papers [
Explainable Wikipedia Recommendations in a Digital
Textbook
We implemented Wikipedia recommendation interface in the context of a
digital textbook system Reading Mirror [
Automatic knowledge-driven linking (recommendation) of Wikipedia articles is one of the newest features of the system. A new set of ve most relevant Wikipedia articles are generated for the target user in two cases. First, when a user starts reading of a new textbook unit (section or subsection), a set of best supportive articles is generated. These articles attempt to provide some alternative reading sources for the knowledge which the unit aims to present as well as prerequisite knowledge which are required to understand the content of this subsection, but not yet mastered by the target user (as evidenced by her knowledge model). Second, when the student answers a question incorrectly a set of best remedial articles is generated. Remedial articles focus on alternative presentation of knowledge that the student failed to master (as evidenced by the wrong answer).
(a) Recommended Item Dialog box (b) Explanations Dialog box
As shown in Figure 1E, the recommended articles are presented on the left side of the interface along with internal table-of-contents links (Figure 1A). The links are ranked by the expected value of each article (importance) in the current context. A colored heat-bar visualizes this importance: here \Green" means more relevant and \Red" means less relevant. When student clicks on a recommended item, the summary of the Wikipedia article will appear rst (Figure 2a). After clicking on \Read the Full Article" button, the complete version of the Wikipedia article is being presented to the user.
To make the recommendation more transparent, we o ered a brief explanation for each recommended item, which could be obtained by clicking on \(Why)" link at the right-hand side of the item. The goal of explanation is helping students to understand the reason for recommending the article. The explanation dialog (Figure 2b) consists of two parts. The rst part lists top three domain model concepts that user learns by reading this article. These concepts are top three items (with highest value) in the list of \useful Knowledge" (see section 5) when this recommendation is generated. The second part explains the reason why the presented concepts are speci cally important for the target user. These reasons are presented as a bullet list and are generated using the current state of the user knowledge re ected in the student model. 4
We built a graph structure to represent the underlying knowledge layer of our system. The entities and relationships in this graph demonstrate the connection between the textbook content, Wikipedia and the student model. The knowledge graph is hosted on a native graph database (Neo4j) and used for both storing the data and generating the recommendations.The overall schema of our knowledge graph is presented in Figure 3.
In the following, we will describe the process of building the knowledge graph.
Question
Related_to
Article
Has_Page
Category
Has_Child Belongs_to Includes
Section Includes Concept
Related_to
Related_to
Knows
User
Wikipedia contains a large number of articles. Only a small number of them are related to the context of any given textbook. To ensure the level of relatedness and to increase the overall performance of our system we generated a subset of Wikipedia articles to be recommended to the students. In order to nd the most relevant articles to the context of a textbook in the domain of computer and information science, we used Wikipedia API and started from a high-level Wikipedia category, namely \Category:Sub elds of computer science" and recursively extracted the subcategories and all the articles associated with them. Since the Wikipedia category structure is not loop-free, we manually stopped the recursion after three steps. For each Wikipedia article, we extracted the following information using the Wikipedia API: { Title: title of the Wikipedia page { Summary : a brief description of the article that appear at the top of the page.
{ Full Text : the complete textual content of the page The total number of 1141 categories and 47772 articles are extracted and added to the graph during this step. We then connect these entities in the graph using \Has Page" (when an article belongs to a category) and \has Child" (when a sub-category belongs to a category) relationships. 4.2
The content of the textbook is represented using three main entities: sections, questions, and concepts. For simplicity, we consider all the variation of the section (i.e, sub-sections and sub-sub-sections) as one entity (Section). Each section or question is associated with a set of concepts that it presents or assesses using \Includes" relationship. Each question is connected to a section with the \Belongs to" relationship. During our calculations, we represent a union of concepts associated with a question and its corresponding section as relevant concepts to the question. Sections and questions are connected to their matched concepts via \Includes" relationship. 4.3
In order to create a relationship between the content of the textbook and extracted Wikipedia articles, we perform a full-text search on the textual representation of the articles using each concept as a query. The graph database (Neo4J) provided us with the full-text indexing functionality which we used to create the index for the combination of article title, summary, and full-text. To nd the most relevant articles for each concept we used the Neo4J internal full-text search algorithm (Lucene). This algorithm provides us with a ranked list of relevant articles as well as a relative score that shows the relevance of each result to the input query. We used this information to connect each concept with the top 100 relevant articles alongside with their relevance score. The \Related to" relationship is representing this connection in the graph schema.
Student models utilize a log of student actions as the input, and predict student
performance with practice activities. To generate and maintain students'
knowledge state for each domain model concept, we used a Comprehension Factor
Analysis framework (CFM ) [
This implementation of the student model allows us to retrieve the students' level of familiarity with the concepts represented in a section/question after each interaction of the user with the system. 5
Our system distinguishes two instances for recommending Wikipedia articles: (1)when student moves to a new section of the book and starts reading and (2) when students fails to answer a question at the end of the section. These instances could appear in any order: the students can move to any given section in the book at any time. Similarly, students can jump right to question section and start answering the questions before reading previous sections. Students are also able to return to a section that they previously read or a question that they already tried. To generate meaningful recommendation that could support this level of freedom (which is natural for reading a paper or electronic book) the use of student knowledge level is essential. In the following, we describe the recommendation approach for both reading and question answering instances.
In order to nd the most relevant Wikipedia articles for a given reading instance, we de ne two overlapping sets of KCs: (1) the knowledge required to fully understand the content of the section (Required Knowledge) and (2) the current level of student mastery that has been predicted by the student model (Obtained Knowledge). The \Required Knowledge" for a given section can be de ned by combining all of the concepts associated with the current and previous sections of the book. This assumption has been made based on the linear organization used in most textbooks (i.e., all the previous sections in the textbook are perquisites of the current section).
Having the set of \Required knowledge" for a given section of the textbook and the set of \Obtained Knowledge" by student while reading that section, we use set di erence to form the \Useful Knowledge" set. The concepts presented in this set are the ones that are required to understand the section but has not, or only partially mastered by the student.
Since the student model predicts the level of student knowledge for each concept as a number between 0 and 1, we consider two conditions for calculating the importance of each concept in the \Useful Knowledge" set.
{ M issingKnowledge: If a concept exists in \Required Knowledge" set but not in \Obtained Knowledge" set, then its important is equal to 1 { P artialKnowledge: If a concept in the \Required Knowledge" set also exist in \Obtained Knowledge" set with the predicted value of s, then its importance is equal to 1-s
As mentioned in section 4.3, we calculated the relevance of each concept to top 100 Wikipedia articles in our graph. In order to nd the most relevant articles for a reading instance, we multiplied the importance of each concept in \Useful Knowledge" set by its relevance score to all Wikipedia articles connected to that concept. Then by aggregating the list for all the concepts presented in \Useful Knowledge" set over the sum of the nal score, we build a ranked list of Wikipedia articles that are both relevant to a given section and take the current level of student knowledge into the account. Finally, we select the top 5 ranked articles in the list and present them as recommendation for that reading instance.
We follow the above approach with a small modi cations in recommendations for question answering instances, which are generated only when the student failed to answer the question correctly. Main di erence is, as mentioned in section 4.2, that the \Required Knowledge" set for a question includes not just concepts directly associated with the question, but also all \Required Knowledge" in its corresponding section. 6
To assess the potential value of our personalized recommendation approach, investigated the impact of considering the current level of knowledge represented in the student-model to generate knowledge-adaptive recommendations. This sections reviews the details of our evaluation design. 6.1
To assess our recommendation approach in a realistic context, we used log data collected from the interaction of students with the reading system in a real semester-long course on Information Retrieval. In this course, the students were required to read 43 sections of the book and answer questions at the end of each section (75 questions in total). The log includes data of 22 students who used the reading system during this course. The students made 9494 interactions with the system (Average: 431.5, Median: 411.5, SD: 108.2). We followed these interactions reconstruct the state of their student models at every recommendation opportunity as described in Section 4.4. 6.2
To highlight the value of using student knowledge in the recommendation
process, we compare our knowledge-adaptive recommendation with a baseline that
only considers the content of a given section/question to generate the
recommendations. This baseline represents the current state of the art for
generating recommendations of external content [
In order to nd the most relevant article with respect to a given section or question, we rst created a list of all articles that are related to the concepts which represent that section or question. We then aggregated that list over the sum of the scores for each concept in the list. Finally we re-rank the list based on the aggregated-sum of scores and selected the top 5 relevant article to each section or question. This connection is illustrated as "Related to" relationship in the graph schema (Figure 3). The \relevance" property of this relationship represents the relatedness of the section/question to the target Wikipedia article. 7
To determine the e ectiveness of our experimental system, we investigated the following key factors: (1) To what extent the recommended items are a ected by involving the student model into the calculations, (2) Whether including the student model improved the quality and coverage of the recommended items, (3) Are there any conspicuous patterns in changes caused by including student model in the process of recommendations and (4) In what ways the proposed approach can facilitate the reading process for the students. 7.1
In order to compare the results of recommendations between our proposed
method (combination of section/question context and the student model) and
the baseline (only the context of section/question) we calculated the Discounted
Cumulative Gain (DCG) [
DCG = i=1 X relevancei n log2(i + 1) (1)
The DCG equation, as it shown in 1, takes into account both the relevance score and the order of items in the recommendations list. The relevance score for each item is calculated by averaging the similarity score (using Lucene search) of all the linked concepts to a given section/question and their corresponding Wikipedia article. This relevance score is being discounted by dividing it with the log of the corresponding position. 7.2
Figure 4 illustrate the overall quality of recommended items for sections and questions in the textbook. The x axis shows the normalized average value of Discounted Cumulative Gain for every given section (Figure 4a) and question (Figure 4b). As the data shows, the average DCG value is always higher when the student model is being involved in the process of recommendation. The proposed approach produced recommendations with in average 23.29% higher DCG value among all sections and 30.27% among the questions. The higher DCG values suggests that more concepts were engaged in the process of recommendation and the recommended items using the proposed approach have higher expected knowledge values.
δ Baseline
It would be expected that adaptive recommendations will suggest di erent sections to di erent students at the start of the same section since their knowledge are likely to be di erent due to di erences in reading paths. To investigate the e ect of including the \student model" in the process of recommendations, 100 75 50 25 100 75 50 25 (a) User-Section Heat-Map.
(b) User-Question Heat-Map.
Fig. 5. The Comparison Between Average DCG values of the Sections/Questions of the Textbook we visualized the di erence between expected knowledge values (DCG) amongst all the students for every section (Figure 5a) and question (Figure 5b) in the textbook. The apparent uctuations in expected knowledge value suggests that every student received a di erent set of recommendations for a given instance of interaction with the reading system. This can be considered as the evidence for the necessity of student-level personalization considering each student has a di erent level of mastery for each concept and required to learn divergent set of concepts at each instance of interaction. 7.3
To examine how well the personalized recommendation could help the users, we examined the \jumping-back" behavior in their reading log. The Reading Mirror system provides students with the ability to jump between sections using the textbook's table of content (Figure 1A). Frequently, this functionality is used by the students to jump back to a certain section of the textbook in order to learn or refresh their memory on a speci c concepts that they need to understand the current section. Data analysis of student navigation behavior in our class shows, this jumping-back behavior was quite frequent taking at average 17.27% of all student navigation steps in the textbook. We believe that in many cases, the adaptive knowledge-based recommendation of Wikipedia articles could prevent this unproductive behavior. Unlike non-adaptive Wikipedia article recommendation (which focuses on the concepts presented in the current page), the adaptive recommendation attempts to proactively o ers information about concepts that might be necessary to understand the current page or question, but are not yet known su ciently by the target user. These recommendation might present the missing information right in place { eliminating the need of jumping-back behavior and helping students to integrate the past and the current knowledge.
To assess to what extent the proactively generated adaptive recommendations could help in this context, we examined each jumping-back case and recorded all concepts presented on the page that the student selected to jump back. We then compared this set of concepts with the concepts covered by adaptive recommendation of Wikipedia articles presented for the given student on the last page visited before jumping back. As the Figure 6 shows, the proactive recommendations cover a remarkable fractions of concepts that were the target of these back-jumps, 86.63% at average. This result indicates that our adaptive recommendation approach can accurately predicts the missing knowledge and considerably reduce the need of jumping through sections to acquire or refresh these knowledge. In contrast, non-adaptive baseline recommendation focused on the current page would cover less than a quarter (24.13%) of student background knowledge needs (Figure 6). This data stresses the importance of considering potentially missing background knowledge in the real classroom context and shows the value of adaptive knowledge-based recommendation.
Proposed 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
User ID In this paper we present a novel approach to generate personalized recommendations of external content for online electronic textbooks. We construct a knowledge graph that represents all three components of \relevant Wikipedia articles", \textbook content" and the \student model". We used this knowledge graph to generate personalized recommendations based on the relevance to a speci c section/question but also taking into account the state of the student model in every instance of interaction with the reading system. The experimental evidence of our data-driven analysis shows that including the student model in the process of generating the recommendation results in higher expected knowledge value in the recommendations. Furthermore, we demonstrate that our proposed approach can accurately predicts the missing knowledge components and potentially reduce the need of jumping-back behaviour amongst students and provides a smoother reading experience. We are aware of possible limitations of our proposed approach and ndings. First, the total number of students in our experimental dataset was limited; including more student data could results in more accurate conclusion. Second, despite of popularity of data-driven studies in the
eld of recommender system, it has been argued that these studies should be complemented by controlled user studies where students could observe and rate generated recommendations. And nally, there are other important components such as the di culty of learning concepts, forgetting factor, etc. that we can potentially incorporate in our recommendation algorithm to produce better and more accurate results. We hope to address these issues in our future work.