The paper presents Intextbooks - the system for automated conversion of PDF-based textbooks into intelligent educational Web resources. The papers focuses on the new component of Intextbooks responsible for transformation of PDF-based content into semanticallyannotated HTML/CSS. The architecture of the system, the design of the client application rendering resulting textbooks and a short validation experiment demonstrating the quality of the transformation workow are presented.
1Digital textbooks have become a standard medium for distributing educational
content, especially, online. The popularity of digital textbooks has been on the
rise. For example, according to DeNoyelles's and Raible's report, the number
of students that used digital textbooks at least once in their college studies
increased from 44% in 2012 to 66% in 2016 [
mons License Attribution 4.0 International (CC BY 4.0).
In our recent work, we have made initial steps to address this problem by
developing an approach towards automated extraction of machine-readable
domain models from PDF textbooks. This approach takes into account common
patterns of textbook formatting and organization and formalizes them as an
extensible set of rules that gradually process the raw content of a textbook, elicit
its underlying structural and formatting elements and, nally, harvest the
semantic model of a textbook [
In this paper, we ll this gap and present Intextbooks (Intelligent textbooks) the system that can perform the complete transformation of PDF textbooks into online intelligent educational resources. After extracting a semantic model from a PDF textbook, it converts it into an HTML/CSS representation with an enriched ne-grained DOM (Document Object Model)2. Every content/layout/structure element of the textbook (words, lines, paragraphs, pages, chapters, index entries, etc.) are uniquely identi able within its DOM. As a result, this implementation is absolutely exible in terms of potential interactivity as virtually any object of a textbook (from a chapter to a keyword) can become an object of targeted interaction. Moreover, each and every element of a textbook's DOM is also identi able within the semantic model extracted from the textbook. As a result, the entire textbook becomes an integrated resource where content elements and pieces of domain knowledge are interlinked on both the presentation and the knowledge levels. Such an organization enables various kinds of semanticallyand adaptively-enhanced interaction with the textbook content.
The rest of this paper is structured as follows. Section 2 gives an overview of related work. Section 3 provides the details of the Intextbooks system. Section 4 presents a validation experiment conducted to test the PDF to HTML conversion component of the system. Finally, section 5 concludes the paper with a discussion and a description of future work.
2.1
Basic conversion from PDF to HTML documents is a well-known task. In 2003,
Rahman and Alam [
A good interaction design is an important factor a ecting user acceptance of
digital textbooks [
In this section we describe Intextbooks, a system that transforms PDF textbooks into interactive intelligent educational resources.
The Intextbooks system consists of two main groups of components. The o ine components perform the tasks of textbook modeling and conversion to HTML, while the online components support students' interaction with the textbooks. Figure 1 presents the overall architecture of the system.
The o ine components take a PDF textbook and, rst, extract its semantic model. The resulting model is represented as an RDF-enriched TEI (Text Encoding Initiative)6 document. Each word, line, text fragment, page, (sub)section, index term, TOC entry, etc. is recognized individually in this model. Then, the PDF textbook is converted into an HTML representation. As the last step, TEI and HTML representations are synchronized, meaning all elements of the TEI model are connected to the DOM elements of the HTML version of the textbook. After that, both the TEI and the synchronized HTML representation of the textbook are stored in the central repository. Together, they play the roles of domain and content models to enable semantic and adaptive access to the textbook content.
The online Web-reader presents processed textbooks to students. Every time a student requests a textbook, the reader displays the synchronized HTML representation of the textbook and supports various kinds of interaction with it. Additionally, the adaptation engine uses a student model and activity logs to generate speci c content and interactions for each student (e.g., tailored navigational aid). Thanks to the extracted model of the textbook, the web interface is aware of various elements of the textbook semantics: the precise beginning and ending of each (sub)section, the relevant terms in every page (based on the index terms) and additional information associated with them (de nitions, links to external resources), etc. 3.2
Textbook model extractor Our textbook model extractor is a rule-based
component that rst extracts a semantic model of a textbook using its symbolic,
formatting, and structural elements. Then, the model is enriched with additional
semantic information using DBpedia and the identi ed index terms in the
textbook. Finally, the enriched semantic model is serialized as an TEI/XML le.
Altogether, 55 unique rules have been de ned to handle the extraction process.
The implementation details of this approach are described in our previous work
[
The resulting TEI textbook model groups all the information extracted from the textbook into three categories: structure, content, and domain knowledge. The structure section contains the hierarchical structure of (sub)sections of the textbook, according to the TOC. The content section provides for every section its textual content organized according to basic textual elements: words, lines, fragments, pages, and subsections. All these elements have unique IDs. Finally,
Fig. 1. Intextbooks architecture the domain knowledge section provides important terms in the textbook according to its Index. Each term is linked to the (sub)sections and pages where it appears, and it can have additional semantic information such as de nition, external resource, classi cation categories, and related terms. This semantic information is incorporated into the TEI model using RDFa7 attributes, which create connections to the Linked Open Data cloud8.
PDF to HTML converter This component converts a PDF textbook into
an HTML representation that preserves its layout and content. Preserving the
layout of a PDF document is a complex task. A PDF document contains
thousands of low-level objects grouped into three layers { text, bitmap images, and
vector graphics { [
While comparing possible libraries, we looked for di erent properties. The most important was the ability of the conversion library to preserve the look of the PDF in the resulting HTML. Therefore, our search was mostly limited to geometrically-based conversion libraries. Another critical factor was the ability to parse the HTML in a structured order to synchronize the HTML with the textbook semantic model. Other factors that were considered were Linux support, performance, and scalability. After analyzing the candidate libraries, we have decided to use pdf2htmlEX.
This library is no longer under active development, but it has an extensive documentation allowing its enrichment. It preserves the layout of the PDF textbook perfectly across di erent types of documents. It can be ran as a standalone process and is quite fast compared to other tools. Furthermore, the HTML output is structured: each page has its own id and is divided into lines. One downside of the library is that tables, graphs, gures, and vector lines are all grouped into one static background image that is loaded for an entire page, which makes it harder to recognize individual elements.
TEI-HTML synchronizer This component modi es the output HTML representation of a textbook to create a ne-grained DOM structure, which is synchronized to the textual elements from the semantic model of the textbook. This process is not straightforward since HTML's structure exists to preserve the layout, but it does not correspond to textual or semantic elements. For example, in the HTML produced by the conversion library, multiple words may share a single span element. After the synchronization process, the nal HTML for a textbook
has a DOM structure that identi es words, lines, and paragraphs with the same IDs from their corresponding elements in the TEI document. The rest of this subsection explains our matching algorithm to synchronize both representations of a textbook.
Internal Representation. The matching takes as an input internal representations for both the TEI model and the HTML le. These representations consist of lists of pages, each of which contains lines (for the HTML) or paragraphs with lines (for the TEI), which themselves contain lists of words. For HTML lines, we also keep extra information, such as their X- and Y-position on a page and font size, which are necessary for the consequent matching. For the words, we keep track of whether they have already been matched. It is important to note that the HTML text is split into DOM's TextNodes. These may contain characters from di erent words (split by a space), parts of a single word, an entire word, or only whitespaces. Di erent parts of a word in the HTML might also be at di erent levels, split by spans. Therefore, we keep a list of the TextNodes that all together form a word in the HTML according to the TEI model.
Matching Words. After the internal representation has been built, the algorithm matches the words between the TEI and HTML representations of a textbook. The di culty comes from the fact that the HTML produced by pdf2htmlEX does not preserve the structure of words. This is also the most important part since the matching of lines and paragraphs depends on this initial matching. For each page, words are matched separately. First, the entire text of a page is extracted separately from the TEI and HTML representations. Then, both texts are compared using the Google's Di Match Patch library13. This library compares the words and determines which words in the HTML belong to which words in the TEI representation. The result is a list containing di erences and matches between the text of both pages. Finally, for each matched TEI word, the HTML is updated to wrap the matching TextNodes as a DOM element with the ID corresponding to the matched word.
There are some special cases when matching words. The rst one is when there is no one-to-one relation between the words from the TEI and HTML representations. When a word in the TEI model corresponds to multiple elements in the HTML, we can match the words by aggregating corresponding elements in the HTML and giving them the same ID. The opposite case is more complex. When an individual element in the HTML corresponds to multiple words in the TEI, the TextNodes for the HTML element are to be split based on the words in the TEI. Then, the TextNodes that contain the characters for each TEI word are wrapped together with the ID of the respective word. Another case occurs when the pdf2htmlEX library inserts special characters that are di erent from the characters in the TEI representation. For example, sometimes the HTML contains orthographic ligatures, rather than Latin characters. The TEI representation contains no ligatures; therefore, in a preprocessing step, we replace 13 https://github.com/google/di -match-patch some special Unicode characters by their correct counterparts. The replacement list is stored is a separate le; we extend it as we observe new cases. Matching Lines and Paragraphs. After the words have been matched, the algorithm starts matching the lines and paragraphs. This process is quite simple using the already matched words. For each line in the TEI representation, the algorithm nds the corresponding words of that line in the HTML, gets their parent element, and wraps it with the right ID. To further improve accuracy, a cross-check is carried out after the line matching. If an HTML line did not get a match, but if the previous and next line did and they have the same ID, the algorithm also wraps the current line with the same ID. A similar process is done to match paragraphs, but in this case, the algorithm keeps track of matched lines in the HTML to detect and wrap the elements that represent paragraphs. 3.3
The web interface of the Intextbooks system will allow students to engage and interact with the textbooks in several ways. Currently, we are working on the design and development of this and the other components of the system, such as the student model. Figure 2 presents the working prototype of the web interface.
The web interface is divided into three parts. The main one is shown in the middle; this is where the textbook is displayed. There are smaller panel on either side of the main part. They provide additional tools for the user to interact with the textbook. Two buttons on the top of the screen can be used to navigate to the main menu and user settings.
The web interface provides a TOC to the reader (panel A in Figure 2). The TOC contains a reference to each (sub)section. The user can click on one of the entries, and the web interface will show to the corresponding content of the section the middle panel. Furthermore, when the teacher sets a path for a speci c textbook or an adapted path is generated for a user, this will be re ected in the TOC. The TOC entry that are not included in the path will be crossed out and have a lighter color. Lastly, the TOC displays annotations in the form of checkmarks and a progress bar to provide navigational cues to the user.
The other feature in the left part of the web interface is the search tool (panel B). When searching a word, the web interface will suggest possible matches prioritizing important terms from the textbook model. The number of matches is also included in the suggestions. After a search, the web interface will show snippets of text where a keyword occurs. The user can make a more precise decision based on these snippets. When clicked on a snippet, the web interface will browse to the corresponding item in the textbook and highlight the corresponding searched term.
An important aspect of creating an intelligent textbook is having interaction with the textbook. The web interface provides interaction by allowing the user to click on certain words that are highlighted in the text. When the user left-clicks on a highlighted word, additional information will be shown in a panel on the right side of the web interface. The additional information can contain de nitions, links, or references to related chapters. This information also comes from the semantic model of the textbook. Panel D in Figure 2 shows the additional information for the histogram term.
Other features are grouped under the same panel using tabs (panel E in Figure 2). When clicked on either tab, the web interface will show the corresponding tool. The di erent tools can be enabled/disabled using the settings button. For the moment, we de ne four tools, but more can be added as necessary. The rst tool is used to interact with other students or teachers. Questions can be asked here to be answered by other students or teachers. The second tool is used to highlight text in the textbook. Di erent colors can be used to categorize highlighted text. The third tool is used to create bookmarks. Bookmarks can be used to improve navigation among di erent pages. Lastly, the user can create notes in a simple text editor, similar to the text editor that most operating systems provide.
The most important component of an textbook to interact with is the its actual content. The textbook is displayed in the middle part of the interface, along with extra options at the bottom (panel C). The user can zoom in and out, skip to the next chapter, download the content as a PDF- le, or bookmark the current page. The user can also right-click on the highlighted terms to display a context menu with additional actions. As our system grows, this part of the interface can provide multiple ways for the users to interact with the textbooks. For example,
Monitoring Engine The web interface communicates directly to the monitoring engine to log every action of a student. All the logs are store in a activity log repository.
be added in the future as the Intextbooks system matures and starts utilizing the domain extracted from the textbooks and the activity of students. 4
We have conducted a validation experiment to test the accuracy of the matching algorithm, which is used to create the ne-grained identi able DOM structure in the HTML textbooks. This DOM structure coupled with the extracted TEI model is required to o er the capability of ne-grained, exible, semanticallyenriched interactions with textbooks. 4.1
We have used a test set of 70 university-level textbooks in several domains: statistics, computer science, web programming, literature, and history. All textbooks are written in English. To estimate the accuracy of the matching algorithm, we have used the percentage of words that were matched between the TEI and HTML representations of each book as our evaluation metric. Such a metric is a good indication for the accuracy of the algorithm since the word matching is the most challenging part, and the line and paragraph matching depend on the number of matched words.
We have run the validation using thee variations of the matching algorithm. Currently, the matching of words is not 100% correct, because sometimes the order of words in the generated HTML is di erent from the one in the extracted TEI model, and because subscripts and superscripts in the HTML are not always placed in the correct position. Therefore, we have implemented a version of the matching algorithm that uses a threshold to merge superscripts and subscripts with either the previous or next line to try to increase matching accuracy. This variant of the algorithm also sorts the words in the HTML based on their Yposition, so they are read in the same order as in the TEI representation. For the validation, we have tested the accuracy of the original matching algorithm (no threshold), an algorithm with a xed threshold, and a variant with a dynamic threshold based on the most frequent distance between two lines on a page. Table 1 shows the results for the validation using the test set and the three algorithms. No Threshold
Fixed Threshold
Dynamic Threshold Mean Median Standard Deviation The results show that at least 87% of all the words are matched for all three methods. The obtained values indicate that the matching algorithm requires some adjustments to match the remainder of the words.
Textbooks that mostly consist of text (without other elements such as
formulae and tables) get a near 100% matching rate. However, the obtained values
are mostly determined by more complex textbooks. Such textbooks get a higher
mismatching rate because of the gures, tables, graphs, and other elements that
are represented as text in the TEI model. However, they are converted to images
by the pdf2htmlEX library. As a result, these elements reduce the matching rate.
For example, one of the textbooks [
We have presented the current state of Intextbooks - a system capable of transforming PDF textbooks into interactive and intelligent educational resources. Currently, we can extract high-quality semantic models of the textbooks, and create HTML representations of the same textbooks that are connected to their semantic models using ne-grained DOM structures. We can match around 88% of all the words in the textbooks to individual elements in the HTML resources, and are working to further improve the matching algorithm. We have designed a web interface that allows to interact with the HTML textbooks in multiple ways. We have plans to extend it with adaptive functionality.
There might be several possible concerns both practical and pragmatic about the scale and applicability of this kind of a system. One question that needs to be addressed is the availability of textbooks and the copyrights issues. In our experience, university libraries can supply enough PDF-based textbooks on a variety of subjects. From the point of copyright protection, if a system provides enhanced access to these books but only to the students of the university holding necessary subscriptions, then publishers do not have a reason to object. In the worst case scenario, many good-quality textbooks are freely available online nowadays in open repositories such as Openstax14/Connections15, Open Textbook Library16, OER-Commons17, etc.
We plan to extend further and improve the components of the system. In particular, it is important to better de ne the semantics of knowledge that is extracted from the textbooks. Using the index terms from the textbooks to precisely identify the sub-domains that are covered in each (sub)section will allow to recommend content to students better. Additionally, we need to test how the automated textbook modeling technology will cope with di erent textbook formatting across less formal domains (e.g., sociology) or domains with con icting viewpoints (e.g., history).
For this system to be as complete as we want, we need to add the missing components to infer the current state of students' knowledge and provide meaningful adaptation. Another direction for future work is to research on how to produce fully interactive HTML elements from the static tables, charts, and formulae in the textbooks. Finally, it is important to evaluate the system's effectiveness in a user study with real students from a target group. Both the user experience and the possible e ect on learning need to be investigated. 14 https://openstax.org 15 https://cnx.org/ 16 https://open.umn.edu/opentextbooks 17 https://www.oercommons.org/