One of the main directions of increasing the educational value of a digital textbook is its enrichment with interactive content. Such content can come from outside the textbooks - from multiple existing repositories of educational resources. However, finding the right place for such external resources is not always a trivial task. There exist multiple sources of potential problems: from mismatching metadata to mutually contradicting prerequisite-outcome structures of underlying resources, from diferences in granularity and coverage to ontological conlficts. In this paper, we make an attempt to categorize these problems and give examples from our recent experiment on automated assignment of smart interactive learning content to the chapters of an intelligent textbook in a programming domain.
One of the popular directions of augmenting electronic textbooks with
valueadding functionality is extending textbooks with “smart content” - interactive
examples, simulations, and problems [
Copyright © 2022 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)
However, the current platforms for development and delivery of interactive textbooks share the same problem: these textbooks are developed “as a whole” with text and interactive problems created together as a part of the authoring process. This approach allows developing excellent examples of interactive textbooks, but does not support scaling up this process.
We advocate a more scalable approach for developing interactive textbooks, such that can make any electronic textbook interactive by augmenting it with smart learning content from existing repositories. An essential problem of this approach is matching smart content items to most appropriate structural units of a textbook. While originally this process has been performed manually, the ability to extract concept knowledge and other metadata from both textbooks and smart content items ofers an opportunity to automate this process using AI techniques. In this paper, we review existing approaches for augmenting textbooks with reusable learning content from content repositories and present a novel approach based on smart content annotation with domain concepts automatically extracted from a textbook. In the following sections, we explain our current approach, present expert-based evaluation of the produced results, and discuss problems that have been revealed during this evaluation. We believe that an analysis of these problems will help in constructing more eficient content matching approaches for future interactive textbooks. 2 2.1
One of the first domains to embrace textbooks augmented with smart content
was computer science education (CSE) where development of interactive
learning activities from algorithm animations to automatically-assessed programming
exercises was a popular research direction [
At the moment, several CSE research teams develop collections of smart
learning content that could be reusable across multiple courses and textbooks.
Most of these collections use LTI communication standard that enables smooth
connection of smart content items to all kinds of host systems from LMS to
LTI-compatible textbook platforms like OpenDSA [
The problem of allocating learning content from content repositories has been
explored long before smart interactive content became popular. This direction
of research has its roots in similar problems of early learning management
systems. While these systems provided space to be filled with all kinds of learning
content and facilitated its inclusion, it become evident that a single instructor
is not able to create the expected amounts of quality content. To answer the
needs of the instructors, various repositories of learning content (also known as
learning object repositories) were created. Good examples of these repositories
are Ariadne [
The increased popularity of recommender systems encouraged many researchers
to explore the use of recommender technologies to support instructors looking
for relevant content [
The problem of the current generation of content allocation tools is the amount of human labor they require. The process assumes engagement of domain experts who should analyze each target context where new content should be added, formulate a proper query and examine each candidate item. While recommender systems and content visualization tools help in this process, the whole procedure is still slow and expensive.
In response to this problem, a new generation of content matching tools attempted to further automate the process by building some form of a knowledge model for each target context, building comparable knowledge models for each candidate item, and then engage IR and AI approaches for automated matching. While a human instructor or textbook author might still be required to examine and approve matching results, the speed and the cost of the matching process is decreased considerably making it more scalable. Some early examples of this approach could be found in [19, ?,?]. In this paper, we present and evaluate a new approach for automated matching of smart content to textbook sections. A unique feature of this approach is the ability to reconcile diferences between concepts used to index book sections and concepts used to annotate smart content by creating a “bridge” between these concept spaces. 3 3.1
For our experiment on automatic matching of smart content to textbook sections
we have used textbook Python for Everybody - Exploring Data Using Python
3 [
In this work, we use four smart learning content types from several providers
which cover both Python program understanding and construction skills:
Animated Examples (52 items in total), Examples-Challenges (42 items in total),
Tracing Problems (51 items in total) and Parsons Problems (34 items in
total). Each of these items can be individually allocated to a proper place in the
course. Animated Examples show the step-by-step execution of Python code
snippets while making explicit the state of the variables within memory.
ExamplesChallenges are a set of activities that allow students to examine an example
which explains how to solve a problem, and then gives them the option of
solving a similar problem after where they have to drag-and-drop some lines of code
to complete the right solution. Tracing Problems are parameterized problems
that ask students to predict the output or the final value of a variable after
executing a short code snippet. Finally, Parsons Problems are puzzle-like problems
where the lines of code to solve a problem are presented in an unsorted way
so students need to work on putting the lines of code in the right order. More
details about the system with these four type of smart content can be found
in [
From here on, we will refer to Tracing Problems (QuizPET) and Parson Problems as Exercises ; to Animated Examples and Example-Challenges (Program Construction Examples) as Examples; and to the set of concepts from the Python ontology used for annotating the Exercises and Examples as SmartConcepts. In total, the SmartConcepts set contained 48 concepts extracted from smart content items (i.e., Exercises and Example) used in this work. 3.3
Given that all textbook sections are annotated with BookConcepts and all smart content items are annotated with SmartConcepts, it is not possible to match smart content items to book sections directly; rfist, some connections should be made between these two concept spaces. To create this bridge, all Exercises have been manually annotated by the course instructor with corresponding BookConcepts: for each Exercise, the instructor indicated which BookConcepts should be considered as the expected learning outcomes of these problems (i.e., mastery of which BookConcepts could be demonstrated by solving a specific exercise). Unlike the textbook, which provides a very broad account of Python programming, the collection of the used coding Exercises was much more specific. Hence, the -fi nal subset of BookConcepts used for annotation included only 47 out of available 852 concepts. Essentially these were the concepts that the learner can master by practicing with the Exercises. 3.4
The complete allocation procedure included the following steps. Assigning book sections to lectures Since we planned to evaluate the approach in a lecture-based class, content allocation was performed lecture by lecture. The first step of the process was assigning a set of book sections as readings for each lecture. As it is usually done in college classes, this step was performed by the course instructor. By assigning readings to the lectures the instructor implicitly specified the BookConcepts goal for each lecture, as explained in the next step.
Creating a BookConcepts Profile of Each Lecture The goal of this step
was creating a formal BookConcepts profile of each lecture as a set of
BookConcepts that each lecture intends to teach. To achieve this goal we first created a
full concept profile of each lecture by creating a union of BookConcepts from all
textbook sections assigned from this lecture and then performed
prerequisiteoutcome separation process originally suggested in [
of the lecture and the BookConcepts annotation of each Exercise we assign Exercises to the lecture with the highest number for which the following two rules are observed: (1) the Exercise should not run ahead of the lecture: all BookConcepts of an Exercise should be either prerequisites or outcomes of the lecture; (2) the Exercise should contribute to practicing the lecture goals: at least one BookConcept annotating the Exercise should be among the lecture outcomes.
After the first round of automatic exercise assignment, we assessed the results as explained in Section 4.2. The team examined mismatches, determined the sources of problems, resolved problems and repeated the allocation step to reach the 100% correct assignment.
profile of each lecture enabled assignment of Exercises to lectures since all Exercises were manually annotated by BookConcepts. It was not suficient for direct assignment of Examples though, as they were only annotated by SmartConcepts. The solution comes from the fact that Exercises were annotated by both BookConcepts and SmartConcepts, which created a bridge between the two distinct concept spaces and enabled us to create a SmartConcepts profile of each lecture. The process was similar to creating the BookConcepts profile explained above – we extracted SmartConcepts from all Exercises assigned to each lecture and performed the prerequisite-outcome separation process explained above. The result of this process was a list of prerequisite and outcome SmartConcepts for each lecture (in addition to already obtained list of prerequisite and outcome BookConcepts ).
described as a set of BookConcepts and a set of SmartConcepts, we can run an Example allocation process for each lecture by matching SmartConcepts from lecture profiles and SmartConcepts from Example annotations. This process was performed in the same way as Exercise assignment explained before, with the diference that Example assignment used SmartConcepts instead of BookConcepts.
After completing Example assignment, we performed the second round of evaluation explained in Section 4.3. We discovered and classified additional problems. Following this analysis, all problems were xfied. At the end of this process all Examples were assigned to their correct places in the course structure.
This section provides the reader with the necessary information regarding the evaluation procedure (in Sect. 4.1) and problem analysis (in Sect. 4.2 and 4.3). 4.1
As explained in Section 3.4, we perform expert evaluation of the automatic content assignment procedure twice. First round of evaluation was performed to assess automatic Exercise assignment and the second round to assess Example assignment. The evaluation was performed from a perspective of a course instructor examining appropriateness of smart learning content for each of the course lectures. The goal of evaluation was to detect misplaced content items and detect potential aflws of the process that led to those misplacements. Every discovered case of a misplacement was recorded and discussed by the team. Through these discussions, we have connected each case to a specicfi problem which are reviewed below in details since understanding of these problems is critical to improve this and future automatic allocation approaches. It is important to stress that after achieving this evaluation goal in each round, we fixed all discovered problems to ensure that the next round starts with in the correct state. The problems resulting from manual errors were fixed by re-running the allocation procedure. The problems related to conceptual issues of the process were fixed manually by allocating each content item to its agreed proper place.
To facilitate the evaluation process, we placed all allocated content items into the MasteryGrids interface to experience the content in exactly the same way as an instructor or a student would see it in a course. Figure 1 presents the
MasteryGrids interface. The course is shown as a sequence of lectures. Clicking on each lecture brings up a panel that shows all lecture-related content including book sections, Examples, and Exercises. Note that green colors that display individual student knowledge and progress were not used in the examination process. From this interface, clicking on a specific content item, provides access to the selected smart content item - see Figure 2 showing an opened examplechallenge item. Using this interface, the experts analyzed each automaticallyallocated content item assessing its relevance to the assigned lecture. 4.2
The key sources of exercise misallocation were manual annotation “glitches”. Since manual annotation was done by the single course instructor and without clearly specified rules, the annotation was in several cases inconsistent or incomplete. Most interesting among the observed manual annotation problem was the generality problem. The BookConcepts in the textbook form a taxonomy where a number of specific concepts are covered by a more general concept. For example, “for loop” and “while loop” are specific concepts, but both of them are covered by a more general concept “iteration (loop)”. Similarly, Python prescribes several specific cases of indentation in loops, functions, conditional statements, etc, but all of these specific cases are covered by a more general concept of “indentation”.
It is natural that most textbooks introduce general concepts when two or more specific cases were presented. I.e., in the PYTHON book, the rfist code examples with indentations were introduced in Chapter 3, but it explicitly appears as a concept and becomes a part of the BookConcepts profile of a section only in Chapter 4, by which time several indentation cases are presented and the introduction of this general concept becomes meaningful. Due to this role of more general concepts, many cases where the instructor used a general BookConcept (like “iteration (loop)” or “indentation”) lead to exercise misplacement. I.e., if an exercise that really belongs to Chapter 3 was indexed with “indentation” (because, indentation was featured in the exercise code) it will be allocated to Chapter 4 since this is the first place where this BookConcept appears.
We concluded that all observed manual annotation issues could be resolved
by creating more formal rules of annotation assembled in a “codebook”, for
example, as suggested in [
In total, our automatic content allocation procedure assigned 94 Examples to the lectures. Among them, 80 Examples were allocated correctly and 14 had to be manually reallocated. In other words, the automatic content allocation was successful more than 85% of the time. The cases of wrong example allocations were distilled to two main groups of issues, namely coverage-related and parserrelated (imperfect matching) issues which are explained below.
Coverage-related issues Coverage-related issues have their roots in clusters of similar concepts in programming languages, such as a set of arithmetic operation concepts, that are usually learned together in the same lecture. Due to their similarity, instructors rarely pay attention whether or not all of these concepts are covered by lecture Examples or practice problems. This is what happened more than once in our case. Despite several Exercises specifically targeting simple arithmetic operations, all correctly allocated to Lecture 3, none of them included “floor division” operation. As a result, “floor division” concept has not been added to the SmartConcepts profile of Lecture 3. At the same time, “oflor division” appeared as a part of code in one of Lecture 5 problems. While it was a side concept there, it was the first lecture where “floor division” appeared and it was added as one of the outcomes to the Lecture 5 SmartConcepts profile.
However, some of the Examples focusing on practicing arithmetic operations did include “integer division” in their code. (see Figure 2). The parser successfully identified “floor division” concept in this Example’s code (see Lines 12 and 15) and included it in its annotation. As a result, this Example was allocated to Lecture 5 instead of its correct placement in Lecture 3.
In our evaluation study, we observed 4 coverage-related issues which had to be reallocated manually. This reallocation comprised 4.2% of all cases. A possible long-term solution for this category of problems would be using concept grouping techniques: a set of concepts (such as “arithmetic operations” that are usually introduced and practiced together should be combined into an “unbreakable group”. Whenever one of the concepts in such a group is added to a set of concept outcome of a specific lecture, the rest of them should be added too. Parser-related issues (imperfect annotation) Several misallocation cases were traced to the insuficient sensitivity of the parser used to extract SmartConcepts from the code of smart content items. Most of these errors were related to the minimalistic approach to syntax in Python where the same syntactic form is used to express concepts that are semantically or pedagogically diferent. For example, the parser was not able to recognize important file operations open(file) and file.close(), since from syntactic prospect, these cases are not diferent from any other functions or methods. As a result, instead of being assigned to a lecture covering work with files, these Examples were misallocated to a lecture on functions and methods. We had to reallocate three examples of this kind, which comprised 3.2% of Examples.
Similarly, the parser was not able to identify the use of a “class constructor” as a special case since it was also looking just like any other function call (see Figure 3). It resulted in misallocation of two examples which were not placed into a lecture on object-oriented programming where they belong. We had to manually reallocate two Examples of this kind, which comprises 2.1% of all Examples.
Yet another example, is a parser’s failure to recognize a special case of .format() expression. The corresponding Examples should have been allocated to the “string methods and regular expression” lecture, but for our parser it was looking just like any other attribute causing these Examples to be misallocated. To fix this issue, we had to reallocate five Examples from the prior lecture to the corresponding lecture (5.3% of the Examples set).
All of these cases point to the weakness of the current parser version. A
longterm solution to the observed problem is to refine the parser so that concepts that
are similar syntactically but diferent pedagogically are recognized as diferent
concepts. When developing a concept parser for Java [
In this paper, we presented the first evaluation of our automatic smart content allocation procedure based on automatic annotation of a textbook and a collection of smart content for programming. We focused the paper on examining cases of misallocation detected during expert evaluation process. We believe, this analysis is important for the progress of research on automated content allocation and enrichment of textbooks with interactivity. We are currently running a classroom study to explore to what extent the results of our automated allocation were acceptable to students. In our future work, we will focus on resolving the problems discovered in this study through developing a content allocation “codebook” for instructors and a better version of the smart content parser.