=Paper=
{{Paper
|id=Vol-2674/paper05
|storemode=property
|title=Using Knowledge Graph for Explainable Recommendation of External Content in Electronic Textbooks
|pdfUrl=https://ceur-ws.org/Vol-2674/paper05.pdf
|volume=Vol-2674
|authors=Behnam Rahdari,Peter Brusilovsky,Khushboo Thaker,Jordan Barria-Pineda
|dblpUrl=https://dblp.org/rec/conf/aied/RahdariBTB20
}}
==Using Knowledge Graph for Explainable Recommendation of External Content in Electronic Textbooks==
Using Knowledge Graph for Explainable Recommendation of External Content in Electronic Textbooks Behnam Rahdari[0000−0001−6514−912X] , Peter Brusilovsky[0000−0002−1902−1464] , Khushboo Thaker[0000−0003−3619−9376] , and Jordan Barria-Pineda[0000−0002−4961−4818] University of Pittsburgh, Pittsburgh PA 15260, USA ber58,peterb,k.thaker and jab464 @pitt.edu Abstract. Over the last 10 years, the world experienced a rapid in- crease in volume and diversity of digital learning resources. The abun- dance of digital resources could support a range of powerful educational scenarios, which were not available before. In this paper, we introduce a novel approach that combines fully automatic knowledge modeling, stu- dent modeling, and content recommendation approaches to recommend relevant Wikipedia articles for students working with online electronic textbooks. An assessment of our approach with real classroom data in- dicated several benefits of our approach over the baseline and revealed interesting patterns of students’ behavior while using the system. Keywords: Recommender Systems · Personalization · Knowledge Graph · Student Model · Electronic Textbooks · Concept Extraction. 1 Introduction Over the last 10 years, the world experienced a rapid increase in volume and diversity of digital learning resources. On the one hand, a variety of tutorials, online textbooks, educational videos, and other open educational resources were posted online to complement traditional textbooks. On the other hand, almost all traditional textbooks have migrated to digital format and become available online [6]. The abundance of digital resources could support a range of powerful educational scenarios, which were not available before. For example, if a textbook section is challenging for a learner to comprehend, she could be recommended some useful external materials, which explains the same topics in a way that is more adapted to her knowledge and interests. If the student fails to solve problems or answer questions due to the lack of prerequisite knowledge, she could be guided to the readings that introduce or review the missing knowledge. Copyright c 2020 for this paper by its authors. Use permitted under Creative Com- mons License Attribution 4.0 International (CC BY 4.0). 2 B. Rahdari et al. The ideas of this “smart” learning have been explored in early projects fo- cused on adaptive textbooks [4], which demonstrated both the feasibility and the value of knowledge-driven personalized reading support. However, these early at- tempts focused mostly on so-called closed corpus personalization, i.e., guiding readers to most relevant parts of the textbook itself. A few attempts to offer open corpus personalization [3], i.e., recommending most relevant external resources, failed to scale up because it required expensive expert-driven knowledge analysis of every external resource [7]. The goal of the project presented in this paper was to move the idea of open corpus personalization in user-adaptive textbooks closer to reality using fully automatic knowledge modeling, student modeling, and content recommendation approaches. As a test-bed for exploring this idea, we selected the case of recommending relevant Wikipedia pages for a textbook user - both proactively, when she starts reading a new section and remedially following an attempt to answer textbook questions. Following a brief review of related work, this paper introduces the interface of our digital textbook reading system with embedded recommendations. The next four sections introduce the underlying mechanisms of our intelligent textbook: the domain and student modeling approaches, the knowledge graph, and the recommendation approach based on this infrastructure. The following section presents the evaluation of the recommendation approach based on real classroom data. We conclude with a discussion and future work plans. 2 Related Works Research on recommendation of related reading sources has deep roots in re- search on educational hypertext and hypermedia. Historically, it has been per- formed under the name of “intelligent hypertext”, since this approach recom- mended resources that were not connected by a human-authored link. Research on intelligent hypertext started in the early days of the educational hypertext field and originally focused on linking resources using term-based resource sim- ilarity [10]. Simple keyword-based approaches have been gradually replaced by semantic-level similarity based on the Semantic Web ideas and domain ontol- ogy [5, 11] and, later, by modern text-processing approaches such as topic mod- eling and concept extraction [1, 12]. The emergence of MOOCs and the accumulation of large volume of educa- tional content online encouraged a new wave of research on “intelligent” linking focused on connecting primary learning content such as textbooks and MOOCs with several kinds of external learning resources such as videos, Wikipedia pages, or research papers [1, 9]. 3 Explainable Wikipedia Recommendations in a Digital Textbook We implemented Wikipedia recommendation interface in the context of a dig- ital textbook system Reading Mirror [2]. Reading Mirror is an online reading Title Suppressed Due to Excessive Length 3 system specifically focused on supporting student learning from modern digital textbooks (Figure 1). The system supports textbooks in PDF and HTML for- mats augmenting the reading process with a range of advanced features such as self-assessment, student knowledge modeling [13], and reading progress tracking with social comparison (Figure 1D). Fig. 1. Reading Mirror interface. (A) Table of contents (B) Progress bar with mirrored social comparison interface; (C) Navigation bar; (D) Reading area; (E) Recommended Wikipedia articles Automatic knowledge-driven linking (recommendation) of Wikipedia articles is one of the newest features of the system. A new set of five most relevant Wikipedia articles are generated for the target user in two cases. First, when a user starts reading of a new textbook unit (section or subsection), a set of best supportive articles is generated. These articles attempt to provide some alternative reading sources for the knowledge which the unit aims to present as well as prerequisite knowledge which are required to understand the content of this subsection, but not yet mastered by the target user (as evidenced by her knowledge model). Second, when the student answers a question incorrectly a set of best remedial articles is generated. Remedial articles focus on alternative presentation of knowledge that the student failed to master (as evidenced by the wrong answer). (a) Recommended Item Dialog box (b) Explanations Dialog box Fig. 2. The preview of recommended item and explanations dialog boxes. 4 B. Rahdari et al. As shown in Figure 1E, the recommended articles are presented on the left side of the interface along with internal table-of-contents links (Figure 1A). The links are ranked by the expected value of each article (importance) in the current context. A colored heat-bar visualizes this importance: here “Green” means more relevant and “Red” means less relevant. When student clicks on a recommended item, the summary of the Wikipedia article will appear first (Figure 2a). After clicking on “Read the Full Article” button, the complete version of the Wikipedia article is being presented to the user. To make the recommendation more transparent, we offered a brief explana- tion for each recommended item, which could be obtained by clicking on “(Why)” link at the right-hand side of the item. The goal of explanation is helping stu- dents to understand the reason for recommending the article. The explanation dialog (Figure 2b) consists of two parts. The first part lists top three domain model concepts that user learns by reading this article. These concepts are top three items (with highest value) in the list of “useful Knowledge” (see section 5) when this recommendation is generated. The second part explains the reason why the presented concepts are specifically important for the target user. These reasons are presented as a bullet list and are generated using the current state of the user knowledge reflected in the student model. 4 Building the Knowledge Graph We built a graph structure to represent the underlying knowledge layer of our system. The entities and relationships in this graph demonstrate the connection between the textbook content, Wikipedia and the student model. The knowledge graph is hosted on a native graph database (Neo4j) and used for both storing the data and generating the recommendations.The overall schema of our knowledge graph is presented in Figure 3. In the following, we will describe the process of building the knowledge graph. Question Related_to Article Has_Page Category Has_Child Belongs_to Related_to Section Related_to Includes Includes Concept Knows User Fig. 3. Graph Schema representing the entities of the knowledge graph and the rela- tionship between them Title Suppressed Due to Excessive Length 5 4.1 Wikipedia Entities Representation Wikipedia contains a large number of articles. Only a small number of them are related to the context of any given textbook. To ensure the level of relatedness and to increase the overall performance of our system we generated a subset of Wikipedia articles to be recommended to the students. In order to find the most relevant articles to the context of a textbook in the domain of computer and information science, we used Wikipedia API and started from a high-level Wikipedia category, namely “Category:Subfields of computer science” and re- cursively extracted the subcategories and all the articles associated with them. Since the Wikipedia category structure is not loop-free, we manually stopped the recursion after three steps. For each Wikipedia article, we extracted the following information using the Wikipedia API: – Title: title of the Wikipedia page – Summary: a brief description of the article that appear at the top of the page. – Full Text: the complete textual content of the page The total number of 1141 categories and 47772 articles are extracted and added to the graph during this step. We then connect these entities in the graph using “Has Page” (when an article belongs to a category) and “has Child” (when a sub-category belongs to a category) relationships. 4.2 Textbook Entities Representation The content of the textbook is represented using three main entities: sections, questions, and concepts. For simplicity, we consider all the variation of the sec- tion (i.e, sub-sections and sub-sub-sections) as one entity (Section). Each section or question is associated with a set of concepts that it presents or assesses using “Includes” relationship. Each question is connected to a section with the “Be- longs to” relationship. During our calculations, we represent a union of concepts associated with a question and its corresponding section as relevant concepts to the question. Sections and questions are connected to their matched concepts via “Includes” relationship. 4.3 Linking Concepts and Wikipedia Articles In order to create a relationship between the content of the textbook and ex- tracted Wikipedia articles, we perform a full-text search on the textual represen- tation of the articles using each concept as a query. The graph database (Neo4J) provided us with the full-text indexing functionality which we used to create the index for the combination of article title, summary, and full-text. To find the most relevant articles for each concept we used the Neo4J internal full-text search algorithm (Lucene). This algorithm provides us with a ranked list of rel- evant articles as well as a relative score that shows the relevance of each result to the input query. We used this information to connect each concept with the top 100 relevant articles alongside with their relevance score. The “Related to” relationship is representing this connection in the graph schema. 6 B. Rahdari et al. 4.4 Student Model Representation Student models utilize a log of student actions as the input, and predict student performance with practice activities. To generate and maintain students’ knowl- edge state for each domain model concept, we used a Comprehension Factor Analysis framework (CFM ) [13]. CFM incorporates student reading behaviour along with activity performance which has proved to be beneficial in case of learn- ing systems based on online textbooks [14]. At each student practice opportunity CFM provides the probability of student’s success at that point. For our case we require probability on each domain concept associated with that opportunity (reading as well as questions). To generate this opportunity we generate prob- ability of success for each concept at that opportunity ( details in [15]).In the graph representation, student model maintains the level of knowledge of student with the concepts at each interaction. This information is represented by a link (called “knows”) between the “user” node and “concept” node and contains the following properties: – Interaction ID: specifies the interaction which the user gained some level of knowledge with respect to the target concept. – Type: determines the type of activity (reading a section or answering a ques- tion) that lead to learning the concept. – Name: stores the name of the section/question. – Results: if the type is question, represents whether student answer that ques- tion correctly or not. – Level : shows the normalized value of student’s knowledge (between 0 and 1) on a given concept for a specific section or question This implementation of the student model allows us to retrieve the students’ level of familiarity with the concepts represented in a section/question after each interaction of the user with the system. 5 Recommendation Approach Our system distinguishes two instances for recommending Wikipedia articles: (1)when student moves to a new section of the book and starts reading and (2) when students fails to answer a question at the end of the section. These instances could appear in any order: the students can move to any given section in the book at any time. Similarly, students can jump right to question section and start answering the questions before reading previous sections. Students are also able to return to a section that they previously read or a question that they already tried. To generate meaningful recommendation that could support this level of freedom (which is natural for reading a paper or electronic book) the use of student knowledge level is essential. In the following, we describe the recommendation approach for both reading and question answering instances. In order to find the most relevant Wikipedia articles for a given reading instance, we define two overlapping sets of KCs: (1) the knowledge required Title Suppressed Due to Excessive Length 7 to fully understand the content of the section (Required Knowledge) and (2) the current level of student mastery that has been predicted by the student model (Obtained Knowledge). The “Required Knowledge” for a given section can be defined by combining all of the concepts associated with the current and previous sections of the book. This assumption has been made based on the linear organization used in most textbooks (i.e., all the previous sections in the textbook are perquisites of the current section). Having the set of “Required knowledge” for a given section of the textbook and the set of “Obtained Knowledge” by student while reading that section, we use set difference to form the “Useful Knowledge” set. The concepts presented in this set are the ones that are required to understand the section but has not, or only partially mastered by the student. Since the student model predicts the level of student knowledge for each concept as a number between 0 and 1, we consider two conditions for calculating the importance of each concept in the “Useful Knowledge” set. – M issingKnowledge: If a concept exists in “Required Knowledge” set but not in “Obtained Knowledge” set, then its important is equal to 1 – P artialKnowledge: If a concept in the “Required Knowledge” set also exist in “Obtained Knowledge” set with the predicted value of s, then its impor- tance is equal to 1-s As mentioned in section 4.3, we calculated the relevance of each concept to top 100 Wikipedia articles in our graph. In order to find the most relevant articles for a reading instance, we multiplied the importance of each concept in “Useful Knowledge” set by its relevance score to all Wikipedia articles connected to that concept. Then by aggregating the list for all the concepts presented in “Useful Knowledge” set over the sum of the final score, we build a ranked list of Wikipedia articles that are both relevant to a given section and take the current level of student knowledge into the account. Finally, we select the top 5 ranked articles in the list and present them as recommendation for that reading instance. We follow the above approach with a small modifications in recommendations for question answering instances, which are generated only when the student failed to answer the question correctly. Main difference is, as mentioned in section 4.2, that the “Required Knowledge” set for a question includes not just concepts directly associated with the question, but also all “Required Knowledge” in its corresponding section. 6 The Assessment Process To assess the potential value of our personalized recommendation approach, investigated the impact of considering the current level of knowledge represented in the student-model to generate knowledge-adaptive recommendations. This sections reviews the details of our evaluation design. 8 B. Rahdari et al. 6.1 Data Source To assess our recommendation approach in a realistic context, we used log data collected from the interaction of students with the reading system in a real semester-long course on Information Retrieval. In this course, the students were required to read 43 sections of the book and answer questions at the end of each section (75 questions in total). The log includes data of 22 students who used the reading system during this course. The students made 9494 interactions with the system (Average: 431.5, Median: 411.5, SD: 108.2). We followed these inter- actions reconstruct the state of their student models at every recommendation opportunity as described in Section 4.4. 6.2 Baseline To highlight the value of using student knowledge in the recommendation pro- cess, we compare our knowledge-adaptive recommendation with a baseline that only considers the content of a given section/question to generate the recom- mendations. This baseline represents the current state of the art for generat- ing recommendations of external content [1, 9]. In parallel with adaptive recom- mendations, we created a set of baseline recommendations for every reading or question-answering instance. In order to find the most relevant article with respect to a given section or question, we first created a list of all articles that are related to the concepts which represent that section or question. We then aggregated that list over the sum of the scores for each concept in the list. Finally we re-rank the list based on the aggregated-sum of scores and selected the top 5 relevant article to each section or question. This connection is illustrated as ”Related to” relationship in the graph schema (Figure 3). The “relevance” property of this relationship represents the relatedness of the section/question to the target Wikipedia article. 7 Results To determine the effectiveness of our experimental system, we investigated the following key factors: (1) To what extent the recommended items are affected by involving the student model into the calculations, (2) Whether including the student model improved the quality and coverage of the recommended items, (3) Are there any conspicuous patterns in changes caused by including student model in the process of recommendations and (4) In what ways the proposed approach can facilitate the reading process for the students. 7.1 Measure of Ranking Quality - Expected Knowledge Value In order to compare the results of recommendations between our proposed method (combination of section/question context and the student model) and the baseline (only the context of section/question) we calculated the Discounted Title Suppressed Due to Excessive Length 9 Cumulative Gain (DCG) [8] of every set of recommended items for each instances of user interaction with the reading system. i=1 X relevancei DCG = (1) n log2 (i + 1) The DCG equation, as it shown in 1, takes into account both the relevance score and the order of items in the recommendations list. The relevance score for each item is calculated by averaging the similarity score (using Lucene search) of all the linked concepts to a given section/question and their corresponding Wikipedia article. This relevance score is being discounted by dividing it with the log of the corresponding position. 7.2 Overall Expected Knowledge Value of the Recommendations Figure 4 illustrate the overall quality of recommended items for sections and questions in the textbook. The x axis shows the normalized average value of Discounted Cumulative Gain for every given section (Figure 4a) and question (Figure 4b). As the data shows, the average DCG value is always higher when the student model is being involved in the process of recommendation. The proposed approach produced recommendations with in average 23.29% higher DCG value among all sections and 30.27% among the questions. The higher DCG values suggests that more concepts were engaged in the process of recommendation and the recommended items using the proposed approach have higher expected knowledge values. δ Baseline Proposed δ Baseline Proposed 100 100 75 75 50 50 25 25 0 0 1.1 1.4 4.2 4.4 4.6 5.2 6.1 6.3 8.1 8.3 8.5 8.7 9.2 12.4 13.2 13.4 13.6 14.2 14.4 14.6 16.2 16.4 261 263 265 267 269 271 273 275 277 279 281 307 309 311 313 315 325 327 329 331 333 335 337 339 341 387 389 391 393 396 398 400 402 404 408 410 412 414 (a) Average DCG values in Sections (b) Average DCG values in Questions Fig. 4. The Comparison Between Average DCG values of the Sections/Questions of the Textbook It would be expected that adaptive recommendations will suggest different sections to different students at the start of the same section since their knowl- edge are likely to be different due to differences in reading paths. To investigate the effect of including the “student model” in the process of recommendations, 10 B. Rahdari et al. (a) User-Section Heat-Map. (b) User-Question Heat-Map. Fig. 5. The Comparison Between Average DCG values of the Sections/Questions of the Textbook we visualized the difference between expected knowledge values (DCG) amongst all the students for every section (Figure 5a) and question (Figure 5b) in the textbook. The apparent fluctuations in expected knowledge value suggests that every student received a different set of recommendations for a given instance of interaction with the reading system. This can be considered as the evidence for the necessity of student-level personalization considering each student has a different level of mastery for each concept and required to learn divergent set of concepts at each instance of interaction. 7.3 Predicting User’s Knowledge Requirements To examine how well the personalized recommendation could help the users, we examined the “jumping-back” behavior in their reading log. The Reading Mir- ror system provides students with the ability to jump between sections using the textbook’s table of content (Figure 1A). Frequently, this functionality is used by the students to jump back to a certain section of the textbook in order to learn or refresh their memory on a specific concepts that they need to understand the current section. Data analysis of student navigation behavior in our class shows, this jumping-back behavior was quite frequent taking at average 17.27% of all student navigation steps in the textbook. We believe that in many cases, the adaptive knowledge-based recommendation of Wikipedia articles could prevent this unproductive behavior. Unlike non-adaptive Wikipedia article recommenda- tion (which focuses on the concepts presented in the current page), the adaptive recommendation attempts to proactively offers information about concepts that might be necessary to understand the current page or question, but are not yet known sufficiently by the target user. These recommendation might present the missing information right in place – eliminating the need of jumping-back behavior and helping students to integrate the past and the current knowledge. To assess to what extent the proactively generated adaptive recommendations could help in this context, we examined each jumping-back case and recorded all concepts presented on the page that the student selected to jump back. We then compared this set of concepts with the concepts covered by adaptive rec- ommendation of Wikipedia articles presented for the given student on the last Title Suppressed Due to Excessive Length 11 page visited before jumping back. As the Figure 6 shows, the proactive rec- ommendations cover a remarkable fractions of concepts that were the target of these back-jumps, 86.63% at average. This result indicates that our adaptive recommendation approach can accurately predicts the missing knowledge and considerably reduce the need of jumping through sections to acquire or refresh these knowledge. In contrast, non-adaptive baseline recommendation focused on the current page would cover less than a quarter (24.13%) of student background knowledge needs (Figure 6). This data stresses the importance of considering po- tentially missing background knowledge in the real classroom context and shows the value of adaptive knowledge-based recommendation. Proposed Baseline Average Proposed (86.63) Average Baseline (24.13) 100 75 50 25 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 User ID Fig. 6. Percentage of potentially missing previous concepts targeted by jumping-back behavior that are covered by the adaptive and baseline recommendations 8 Summary and Discussion In this paper we present a novel approach to generate personalized recommenda- tions of external content for online electronic textbooks. We construct a knowl- edge graph that represents all three components of “relevant Wikipedia articles”, “textbook content” and the “student model”. We used this knowledge graph to generate personalized recommendations based on the relevance to a specific sec- tion/question but also taking into account the state of the student model in every instance of interaction with the reading system. The experimental evi- dence of our data-driven analysis shows that including the student model in the process of generating the recommendation results in higher expected knowledge value in the recommendations. Furthermore, we demonstrate that our proposed approach can accurately predicts the missing knowledge components and poten- tially reduce the need of jumping-back behaviour amongst students and provides a smoother reading experience. We are aware of possible limitations of our pro- posed approach and findings. First, the total number of students in our experi- mental dataset was limited; including more student data could results in more accurate conclusion. Second, despite of popularity of data-driven studies in the 12 B. Rahdari et al. field of recommender system, it has been argued that these studies should be complemented by controlled user studies where students could observe and rate generated recommendations. And finally, there are other important components such as the difficulty of learning concepts, forgetting factor, etc. that we can potentially incorporate in our recommendation algorithm to produce better and more accurate results. We hope to address these issues in our future work. References 1. Agrawal, R., Gollapudi, S., Kannan, A., Kenthapadi, K.: Study Navigator: An algorithmically generated aid for learning from electronic textbooks. Journal of Educational Data Mining 6(1), 53–75 (2014) 2. Barria-Pineda, J., Brusilovsky, P., He, D.: Reading mirror: Social navigation and social comparison for electronic textbooks. In: First Workshop on Intelligent Text- books at 20th International Conference on Artificial Intelligence in Education. pp. 30–37 (2019), http://ceur-ws.org/Vol-2384/paper03.pdf 3. Brusilovsky, P., Henze, N.: Open corpus adaptive educational hypermedia. In: The Adaptive Web, pp. 671–696. Springer, Berlin (2007) 4. Bursilovsky, P., Eklund, J., Schwarz, E.: Web-based education for all: A tool for developing adaptive courseware. In: 7th International World Wide Web Conference. pp. 291–300 (1998) 5. Carr, L., Hall, W., Bechhofer, S., Goble, C.: Conceptual linking: Ontology-based open hypermedia. In: 10th International World Wide Web conference. pp. 334–342 (2001) 6. Crestani, F., Landoni, M., Melucci, M.: Appearance and functionality of electronic books. International Journal on Digital Libraries 6(2), 192–209 (2006) 7. Henze, N., Nejdl, W.: Adaptation in open corpus hypermedia. International Journal of Artificial Intelligence in Education 12(4), 325–350 (2001) 8. Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems 20(4), 422–446 (2002) 9. Kokkodis, M., Kannan, A., Kenthapadi, K.: Assigning educational videos at appro- priate locations in textbooks. In: Seventh International Conference on Educational Data Mining. pp. 201–204 (2014) 10. Mayes, J.T., et al.: Strathtutor: The development and evaluation of a learning-by- browsing on the macintosh. Computers and Education 12, 221–229 (1988) 11. Milne, D., Witten, I.H.: Learning to link with wikipedia. In: 17th ACM Conference on Information and Knowledge Management. pp. 509–518 (2008) 12. Rihák, J., Pelánek, R.: Measuring similarity of educational items using data on learners’ performance. In: 10th International Conference on Educational Data Min- ing (2017) 13. Thaker, K., Carvalho, P., Koedinger, K.: Comprehension factor analysis: Modeling student’s reading behaviour: Accounting for reading practice in predicting students’ learning in moocs. In: 9th International Conference on Learning Analytics and Knowledge (2019) 14. Thaker, K., Huang, Y., Brusilovsky, P., He, D.: Dynamic knowledge modeling with heterogeneous activities for adaptive textbooks. In: the 11th International Conference on Educational Data Mining. pp. 592–595 (2018) 15. Thaker, K., Zhang, L., He, D., Brusilovsky, P.: Recommending remedial readings using student’s knowledge state. In: 13th International Conference on Educational Data Mining. pp. 233–244 (2020)