Online textbooks with interactive content emerged as a popular medium for learning programming and other computer science topics. While the textbook component supports acquisition of programming concepts by reading, various types of \smart" interactive learning content such as worked examples, code animations, Parson's puzzles, and coding problems allow students to immediately practice and master the newly learned concepts. This paper attempts to automate the time-consuming manual process of augmenting textbooks with \smart" interactive content. We introduce an ontology-based approach that can link fragment of text with \smart" content activities, demonstrate its application to two practical linking cases, and present the results of its pilot evaluation.
3 Electronic textbooks and various kinds of \smart" interactive systems such as
Intelligent Tutoring Systems (ITS) or virtual labs have been traditionally
considered as two opposite ways to leverage the power of computers for human learning.
The research on electronic textbooks attempted to enhance learning-by-reading
supported by traditional textbooks by augmenting them with internal hyperlinks
[
Gradually, the recognition of complementary nature of learning-by-reading
and learning-by-doing encouraged an increasing stream of research on
integrating textbooks with interactive content [
Commons License Attribution 4.0 International (CC BY 4.0).
in computer science domain, such as learning programming languages, where
researchers and practitioners developed and explored a broad range of \smart"
interactive learning content [
Yet, textbooks with \smart" interactive content are still a minority among
other types of learning tools due to considerable problems of integrating
traditional text with \smart content". While technical problems associated with
integration are being gradually addressed by modern interoperability standards,
the conceptual problems related to linking text with interactive content (which
interactive activity is the best match for a section or text?) are not yet resolved.
The current generation of interactive textbook is still developed by manual
allocation of interactive content developed by the textbook authors to textbook
sections. This approach has scaling problems and complicates the reuse of smart
learning content created by other authors. While approaches for automatic
linking of textbook sections with various types of text-based resources such as other
textbooks [
E ective integration of various types of learning content and systems serving
it has been both an important practical problem and a long-standing research
challenge for the developers of educational software. On a more practical side,
several issues have been addresses with di erent degrees of success. For example,
we have a range of standard protocols for reliable identi cation of users across
multiple systems [
From the architecture perspective, there are two primary models for
integrating textbooks with smart interactive content:
{ linking external interactive content into the relevant parts of a textbook;
{ linking relevant fragments of a textbook into an existing interactive
education system;
The former method has been implemented in several successful system. For
example, the classic adaptive system for learning LISP - ELM-ART [
The automated linking of textbooks is another important stream of research.
However, we are not aware of other examples of automatic linking of textbooks to
smart interactive content besides OOPS. Most authors have looked into di erent
ways to either cross-link multiple textbooks within the same domain [
The main platform used in our studies of linking textbooks with interactive
content is the Intextbooks (Intelligent textbooks) system [
The o ine component extracts a semantic model of a textbook using a
rulebased system. Its ruleset captures common conventions and formatting guidelines
for textbook formatting, structuring, and organization. Such elements and tables
of contents and indices play a crucial role. More information can be found in [
To assess the value of automatically extracting concepts from Python textbooks,
we also decided to explore linking the di erent sections of these books with
the learning units presented in the Python Personalized Programming Practice
System (P4), based on their conceptual similarity. P4 is an online personalized
system o ering students in introductory Python programming courses to practice
their skills using several types of interactive learning materials. The system is
designed as a non-mandatory practice and self-assessment tool that each student
could use for individual needs. P4 was developed by using the Mastery Grids
system [
On top of it, personalized guidance is provided, based on a concept-level
model of student's Python knowledge. The conceptual structure of the student
model is driven by an ontology of Python programming concepts 6 (from now on,
4 http://dbpedia.org
5 https://tei-c.org/
6 http://acos.cs.hut. /static/python-parser/ontology.png
the Python ontology), which was homologically created by using a Java ontology
as a template [
The model is built by observing student behavior in the system and represents the probability of students knowing each Python concept. To make this learner model \open" to the student, it is visualized as a bar chart on the bottom part of the activity selection interface (see Fig. 2). Each bar depicts one concept, and the height represents the estimated level of knowledge (i.e., the taller it is, the more the estimation of knowledge). Based on these concept estimations, P4 recommends the three learning activities that are more appropriate for the student at that stage by following a speci c learning goal (e.g., knowledge maximization or misconceptions' remediation). Recommended learning materials are highlighted with stars of di erent sizes within the interface (see Fig. 2). 3.3
Intextbooks/P4 integration The research presented in this paper primarily bene ts from the annotation of the textbook's content with domain terms in Intextbooks and the topic-conceptactivity model in P4. Each content unit (page, sub-chapter, chapter) is annotated with its corresponding domain terms in the resulting knowledge models for a textbook. When those domain terms are linked to the Python programming concepts used in P4, two potential integrations are enabled: (1) learning activities from P4 can be displayed along with the corresponding content units in Intextbooks, and (2) content units from textbooks in Intextbooks can be additional learning activities associated with the most appropriate topics in P4.
We present how this two-way integration for learning programming looks like
in each of the two systems. As a working example, we linked the content from
the \Python for Everybody" textbook [
Figure 1 re ects the addition of learning activities associated with speci c sub-chapters in Intextbooks. Since each sub-chapter is annotated with domain terms, learning activities from P4 that cover the same conceptual terms can be included in Intextbooks. When a user is navigating a sub-chapter linked with one or more learning activities, these are displayed as additional content (see topright panel on Fig. 1). The user can interact directly with the learning activities without leaving the system.
Figure 2 shows how \Textbook readings" were added as an additional type
of learning activity (last row) in P4. When a \Textbook readings" cell is clicked,
P4 directs students to the Reading Mirror system [
Figure 3 shows the elements from both systems (Intextbooks and P4) used in the methodology. Each step is described in the following subsections. 4.1
The rst step in our methodology is to extract the list of all the index terms present in the Python textbooks to create a single uni ed glossary of terms to be mapped with the Python ontology.
First, for each textbook, a knowledge model is extracted and enriched with
semantic information from DBpedia. For the enrichment process, the DBpedia
category Computer programming is used. Then, a glossary of index terms is
created for each textbook. Each term in the glossary has a preferred label (PF),
a set of alternative labels (AL), and an external concept (EC). The rst one
corresponds to the ID of the term, the second to a set of alternative names for
the same term, and the last one, to an external concept from a di erent glossary
or ontology. For example, a glossary term is fPF: if statement; AL: statement
<> if; EC:- g (<> is used for hierarchical index terms). Initially, no term has
an external concept associated with it; this will be done in the next step. The
presented glossary term corresponds to the \id statement" and \statement <>
if" index terms from the \Python for Everybody" textbook [
Since two terms representing the same concept can be written di erently,
we merge all glossaries to identify repeated terms. Terms that are identi ed as
the same are merged, keeping the individual terms as alternative labels.
Besides exact textual matching, three cases are handled: (1) di erent lexical forms
(e.g., branch and branches), (2) acronyms (e.g., API and APIs (Application
Programming Interfaces)), and (3) synonyms (e.g., Turing Completeness and
Turing complete programming language ). The rst case is handle by
comparing the terms using their stems (computed for each word using the snowball
stemming algorithm [
The next step in the methodology is to link the terms from the glossary to the concepts in the Python ontology.
First, the concepts in the Python ontology are extracted. For each concept, both the single name (e.g., while) and the compound name with the parent classes (e.g., while <> iteration statement <> statement <> python language <> python) are retrieved. Then, a linking strategy is applied. First, glossary terms are linked to the Python concepts using exact textual matching between the di erent labels of the terms and the single name of the concepts. Then, stemming is applied to nd more matches. Third, a small list of abbreviations is used. The list was created manually since the Python ontology uses some abbreviated names (e.g. int and div ) that appear with a full name in the glossary (e.g., integer and division). Fourth, the parent classes of the Python concepts are used. Since the class hierarchy in the Python ontology has a deeper granularity than the names used in the textbooks, matches between a complete glossary term and a partial compound concept name are accepted. For example, the glossary term if statement is linked to the concept if <> selection statement <> statement <> python language <> python since the term matches the concept's single name (if ) and one of its parent classes (statement ). Finally, the glossary terms that are not linked to any Python concept are compared to the inner classes of the Python ontology (e.g., Boolean Expression) using the mentioned strategies (textual matching, stemming, and synonyms). The result of this step is that the linked glossary terms have a Python concept as an external concept (e.g., fPF: if statement; AL: statement <> if, statement <> conditional; EC: if g). The number 2 in Fig. 3 illustrates the linking between the terms in the glossary and the concepts from the Python ontology used in P4. 4.3
The nal step is to linking the textbooks from Intextbooks with the learning activities from P4.
Textbook sub-chapters to P4 First, a list of the concepts introduced in each topic is extracted from P4. Then, for each textbook, the linked glossary terms to Python concepts are used to get the individual index terms in the textbook associated with the external concepts. After that, we get from the knowledge models the sub-chapters associated with the index terms. Now, we have subchapters mapped to index terms, index terms mapped to concepts, and concepts mapped to topics. Finally, using the concepts as a bridge, we link each subchapter to the topics in P4 where the Python concepts are introduced. This sub-step in represented with the number 3.1 in Fig. 3.
P4 learning activities to textbook sub-chapters In P4, each learning activity has a set of associated concepts, plus the topic where it is used. We rst extract this list of learning activities and compute the topics that are prerequisites using the associated concepts and the topics where they are introduced. Then, for each sub-chapter linked to a topic in P4, we select as candidates the learning activities that belong to the same topic and have one of the concepts associated with the sub-chapter. Then, we check that the topic prerequisites for each learning activity have been introduced in other previous sub-chapters. If all the prerequisites are met, we create a sub-chapter-learning activity pair. This sub-step is represented with the number 3.2 in Fig. 3. 5
We applied our methodology using 5 di erent Python textbooks ([
Knowledge model extraction and glossary uni cation After creating the knowledge models for the textbooks, we got a variate number of index terms in each one (817, 848, 834, 451, 1105), producing a total of 4055 di erent elements. After merging the terms, we got a uni ed glossary with 3250 elements (a reduction of almost 20%). Additionally, when processing the Python ontology, we got 108 elements: 73 leaf concepts and 35 inner classes.
Glossary - ontology linking After linking the terms in the glossary to the elements in the Python ontology, we got 53 instances. Some terms in the glossary were linked to the same Python concept. Thirty-six concepts and ten classes from the Python ontology were linked to the 53 terms in the glossary. Granular content linking At the nal step of the methodology, we got multiple contents linked in both systems. First, 266 di erent index terms from the ve textbooks were linked to 217 concepts and 49 classes from the Python ontology. Of those 266 terms, only 186 are linked to concepts used in P4, since not all the Python concepts from the ontology are currently a part of the system. Using the linked index terms and the Python concepts, 245 di erent sub-chapters from all the textbooks were mapped to the topics in P4. The number of linked sub-chapters for each topic is as follows: 36 to `Variables and Operations', 35 to `Boolean Expressions', 12 to `If-Else', 3 to `While Loops', 19 to `For Loops', 40 to `Functions', 49 to `Lists', 11 to `Dictionary', 17 to `Strings', 3 to `File Handling', 5 to `Exceptions', and 15 to `Classes Objects'. Regarding linking learning activities to sub-chapters, 2240 possible mappings were analyzed, from which 1790 ful lled the prerequisite restrictions. If we only assign each learning activity once in the whole textbook, instead of to multiple sub-chapters, the average of unique learning activities mapped per textbook is 67, from a total of 157 di erent activities available in P4.
General Analysis The obtained data shows that despite the small number of glossary terms that are matched to Python concepts (53), the number of both linked sub-chapters to topics (245) and the learning activities to sub-chapters (1790) is promising. P4 can bene t from incorporating textual material explaining the concepts used in each topic. Additionally, multiple textbooks enable more personalization: a student can select one or more textbooks to get the textual material recommendations according to their preferences. Intextbooks can present additional interactive content to the learners as they progress and navigate throughout a textbook.
Currently, we link the same learning activity to all the tting sub-chapters
within a textbook, which will cause problems if the textbook is read sequentially.
This approach has been used because Intextbooks could generate personalized
navigation paths for each student. Hence, it is helpful to have an extensive
mapping of learning activities. The system will need to be aware of this situation
and not display learning activities that have been seen already in previous
subchapters. Finally, the order of topics and the association of concepts to learning
activities in P4 allows the comparison of prerequisite-outcome relations with the
textbooks. As we found, the order of topics in four of the ve textbooks was
similar to the one in P4. However, textbook [
In order to assess the automatic concept linking approach described above, two domain experts with experience on teaching Python independently rated the match quality (appropriateness) of the textbook sections conceptually linked to each of the topics presented in the P4 system. The following rating schema was used: 3 as a good match, 2 as a partial match, and 1 as a bad match. 55 sections of the \Python for Everybody" textbook were automatically associated to one of the 17 topics of the Python course. Note that the topical structure of that Python course was de ned by Python instructors from several universities, who used P4 systems in to support Python practice in their courses. After the scoring, the results were examined to determine possible causes for the discovered bad matches. The examination revealed that a number of low matching scores were produced by the sections titled as \Glossary" that were included in each chapter rather than assembled at the end of the book in a more traditional way. The nature of these sections make them poor independent learning resources since they served as a reminder of already learned content. We decided to exlude these sections from matching and evaluation.
After glossaries were removed, inter-reliability between raters was calculated. using weighted Cohen's Kappa. The resulting Kappa 0.63 is considered as moderate inter-reliability. Given that raters did not have made ratings fully independently and that they only had access to a very short and concise description of the rating schema, this result is deemed as positive. The main disagreement was registered in the topics of Boolean Expressions, and Lists/ Strings. The point of disagreement here was that in the textbook some chapters introduce some concepts blended together (e.g., boolean operators and if statement ) while in the P4 course they are clearly separated and taught one after another (i.e., rst boolean expressions, and then if-else). In terms of general mutual agreement, both raters coincided in thinking that there was a good match in 45% of the textbook associations, a partial match in 12% of the cases and a bad match in 14% of the total linkages. Considering the evaluations that lead to disagreement (29%), 17% involved positive evaluations (either one of the two ratings as 3 or 2), while only a 12% lead a bad rating for the match. Finally, in total, 74% of raters' pairs of evaluations were at least either partial or good matches, so we can conclude that the automatic linking approach that we followed lead to acceptable good results which can be used in the context of automatic integration of learning content. 7
In this paper we demonstrated that a two-way linking between textbook sections and smart learning content items could be generated by automatic extraction of concepts from programming textbooks using a combination of textbook metadata and di erent ontologies as underpinning tools for this task (e.g., DBPedia, Python ontology). We consider our work as the rst step to resolving the problem of automatic linking and plan to continue research in this direction.
Given that the textbook concept extraction works by analyzing the textual content, a natural next step would be exploring a more \ ne-grained" association of concepts, e.g., at a paragraph level rather than on section level. This approach will enable us to recommend practice learning material to the users \in context", right after the student reads the corresponding lines where a concept is introduced. In a similar way, given that the textual information presented in the textbook generally focuses on presenting the concepts in an introductory way, a more \ ne-grained" association in terms of textual units will enable us to recommend remedial reading when students fail in certain learning activities which could re ect the learner's misconception(s) on certain concept(s).
One potential problem of the proposed methodology is that the index terms
from the textbooks are not always a high-quality representation of a domain.
They can potentially su er from high subjectivity, poor coverage and
granularity, lack of semantics, and ambiguity. For example, in one of the textbooks used,
the term \tuple" is used both to refer to the Python data type and a database
tuple. Another problem speci c to the programming domain is how to di
erentiate reserved keywords from natural language in the content accurately (e.g.,
for \if" or \or"). Finally, although the used knowledge models have been
created for di erent domains (statistics, history, computer science, literature, and
information retrieval [
To more reliably assess the value of our current approach and its suggested extensions, each linking approach and its interface implementation has to be evaluated in classroom studies from the prospect of pedagogical usefulness and quality of integration between online textbooks and online practice systems.