Ensuring accessible textbooks for children with disabilities is essential for inclusive education. However, providing native accessibility for educational content remains a challenge. In the mean time, existing educational materials need to be adapted, for example by providing interactive versions to overcome dificulties caused by disabilities. In this context, our project aims to automatically adapt PDF textbooks to make them accessible to children with disabilities. The first step towards this adaptation involves extracting and structuring the content of textbooks. In this paper, we introduce textbook models, propose an automated extraction pipeline, and conduct preliminary experiments. Our textbook models are based on the various activities involved and provide layout and semantic information. They enable normalized and structured representations of educational content at both document and page levels, facilitating the automatic extraction process and the conversion to popular formats such as TEI and DocBook. In order to automatically extract PDF textbooks structure, our experiments, using a state-of-the-art multimodal transformer for a token classification task, demonstrate promising results. However, these experiments also highlight the dificulty of the task, especially cross-textbook collection generalization. Finally, we discuss the extraction pipeline and the directions of future work. textbook adaptation, inclusive education, modeling textbooks, modeling textbook pages, textbook extraction, interactive textbooks, digital textbooks, PDF processing
The use of e-learning environment and e-textbooks is growing in higher education, yet paper textbooks remain prevalent in elementary and secondary schools in France. Despite the nEvelop-O availability of digital textbooks, the majority of them are not natively accessible due to their ifxed layout and lack of reflowability, which prevents the adjustment of the page layout (font size, line spacing, word spacing, letter spacing, etc.). To ensure inclusive education, there is a pressing need to create accessible textbooks that allow children with disabilities to participate in classroom activities. Inclusive textbooks should take into account students’ dificulties, while preserving the content of the activities and their instructional intent.
Some non-profit organizations have started to produce adapted digital textbooks by doing all the transformations manually. For example, the association Le Cartable Fantastique1 provides the Fantastiques Exercices, a collection of French exercises and their interactive version adapted for children with Developmental Coordination Disorder (DCD). This neurodevelopmental disorder is defined as an impairment in motor coordination which interferes with academic achievement and daily life activities. At school, children with DCD struggle with handwriting. More specifically, they do not automate the handwriting process and continue to pay attention to letter tracing through their school life. In addition, their eye movement disorders may impede their ability to read text that is not presented in an accessible format. Hence, for children with DCD to succeed at school, textbooks must address their dificulties with handwriting and gaze organization. Figure 1 shows an example of a “Fill-in the blank” exercise and its adaptation, allowing children with DCD to complete the sentence by clicking on the correct answer, avoiding the use of handwriting.
6 ** Complète les phrases avec on ou ont. a. Si … allait au cinéma ? b. Ils … vu ce film dix fois. c. … s’installe dans les fauteuils moelleux. d. Mes parents … pris du pop-corn. e. Les enfants … sursauté devant une scène du film.
(a) Original exercise (b) Adapted exercise
Unfortunately, the large variety of textbooks and frequent renewal due to changes in the curriculum make it challenging to adapt them manually. In this context, the MALIN project (MAnuels scoLaires INclusifs, French for Inclusive textbooks) aims to automatically adapt PDF textbooks for children with DCD or visual impairment, and in the long term, for other disabilities. Adapted textbooks facilitate inclusive participation in class, providing students with disabilities 11 * Recopie chaque phrase en rétablissant la withpotnhcetusaatimoneceodmumceatdiaonnsall’ecxoemntpelen.t as their classmates. We do not aim to enrich textbooks with addaiutjioounrda’hluicdoannstelenjatrdoinrlepsrpoevtitsidceochpoenrssoontndaalniszéed assistance to students, but only modify the mode of ina→.tcAeeurmjaocuarttdiin’oheunin, daallnasnlteàjalardigna,rleeslepetrtiotsiscièocmheoncsoocnhtodnoaanustép. uts or enabling connection to external tools (e.g.
by generating alternative texta-cthoe-tsépdeuepcahins.ynthesis or braille displays for blind and low vision readers). Since there are no strubc.tàumrieddidvanesrslei otrnains iolsf’atsesxotitbàocoôktésdt'uhnahtocmomnet.ain suficient semantic information, we must start fro mc. pendant le voyage avec l'homme le petit cochon
matnegxetlbeopaoinks in PDF format. Hence, the first and fundamental step towards our goal is to extrda.àctl'atrhriveéteedxatnbsolaogkasr’e sl'htroumcmteudreonanneddecsobnriqtueenst. This work presents our proposed approach for thisauexpetrtiatccotciohonn pipeline and concentrates on textbook modeling. We examined a large set of
French language study and mathematics textbooks used in elementary classroom to create a template of a textbook, and introduce two complementary textbook models: one that represents the textbook as a whole, and another that operates at the page level, which is essential for the extraction process. This paper also comes as a position paper on the automatic extraction task ahead. In particular, we have already conducted preliminary experiments on the token classification task, using hybrid transformer architecture.
Our main contributions are: (i) a model for structuring textbooks and textbook pages; (ii) a general approach for PDF textbooks extraction, (iii) preliminary token classification experiments.
There are numerous ways to interact with digital textbooks. The simplest digital textbook
includes the content of the paper textbook and can be flipped through like a traditional
ebook. More advanced versions are enriched with additional multimedia material or internal
functionalities such as hyperlinks. Researchers envision the future of textbooks as interactive
learning environments rather than traditional books and promote adaptive, personalized and
collaborative learning. Ou et al. [
Textbook modeling is a fundamental step common to all textbook segmentation research.
Three widely accessible markup schemes cover a range of applications, including textbooks:
HTML, DocBook2, and the Text Encoding Initiative Guidelines (TEI)3. Publishers and authors
customize or combine existing schemes or create their own, tailoring them to their specific
needs and objectives [
2.3.1. Visually-rich document understanding
Emerging deep learning approaches have demonstrated significant potential in addressing
natural language processing (NLP) tasks, particularly those related to visually-rich document
understanding (VrDU). Most rely on the Transformer architecture, with its self-attention
mechanism [
We propose activity-based textbook models with a mix of layout and conceptual features, and a PDF textbook extraction pipeline built upon these models. Our models result from the observation of dozens of elementary school textbooks widely used in French educational institutions, and were inspired by existing models and guidelines for textbook creation.
Most French language study and mathematics textbooks share a basic structure. Textbooks are divided into sub-disciplines (respectively grammar, conjugation, etc., and arithmetic, geometry, etc.), then into learning themes containing several chapters. The bulk of the textbook to be adapted for MALIN is found in the chapter pages. In the majority of textbooks, chapters are structured around a double-page spread and include the following content blocks: the chapter title, an introductory activity to the chapter, a lesson, and a series of exercises. An exercise is then divided into several parts: it always has an instruction, often a statement, and it may
5ScienceParse https://github.com/allenai/science-parse, PaperMage https://github.com/allenai/papermage contain: a number, a title, examples, hints and illustrations, as well as additional indicators such as the level of dificulty. The introductory activity can assume diverse variations (exploratory activity, revision activity, etc.), typically comprising a single statement and several sub-exercises. In some books, opening activities introduce each theme. Depending on the publisher and the collection, other information may be included, such as the skills that are involved in completing an exercise, indicative activity headings, chapter numbers, a reminder of the discipline or the theme in which the chapter is located, various indicators of the modality or interdisciplinarity of an activity, etc. Revision pages are often added at the end of a theme or sub-discipline, with a summary of the learning content from a set of chapters as well as application and integration exercises. Finally, additional tool pages may be found at the beginning and end of the textbook: foreword, directions for use, pedagogical approach, preface, table of contents, index, glossary, bibliography, acknowledgements, pedagogical resources, or other various appendices.
From this inventory of sections, we have developed two models: a whole textbook model and a textbook page model. We created our own Document Type Descriptors (DTDs) to formalize the structure in XML format. This new model precisely aligns with the abstract syntax tree of the document, encompassing all the necessary information for our project.
The first model captures the encapsulation of blocks as described in Section 3.1. Any additional element that is not part of the main content of a block is represented using an “indicator” tag.
We use tokens as the smallest linguistic unit and group them into text segments. In fact, textbooks have a distinct language compared to regular texts. In language study textbooks, tokens are sometimes bound morphemes; in mathematics textbooks, they can be operators, symbols or isolated letters. Besides, text segments usually correspond to grammatical sentences (starting with a capital letter and ending with a strong punctuation mark), bloc titles or labels. However, according to the nature of the activities, we may find ungrammatical and asemantic sequences. Some examples are fill-in-the-blank words ( “c…bat”), sentences (“Manon a perdu … chat.”) or operations (“4 × … = 8”), multiple-choice choices (“(son/sont)”), concatenated words (“cirageâgégéantenfant”), scattered blocks of text (“est une fleur”, “la tulipe” ), list numbers (“a.”,“b.”), etc. As a result, referring to segments instead of sentences is more appropriate to describe such units of text. For the automation process, the use of text segments also allows to easily infer roles through typography, whereas this is not suficiently discriminating at the token level. Moreover, due to the short length of the text blocks in the analyzed grade-level textbooks, the concept of paragraph is not relevant to our project.
The scheme is also extended to lists and tables. In addition, two or more lists can be linked, for instance in an exercise where the instruction asks to match items from diferent lists to each other (e.g., “Match each subject with its predicate”). As needed, we can further refine our model by incorporating sub-elements (e.g., choices in multiple-choice questions) and additional semantic and morpho-syntactic attributes. This refinement is possible despite the fixed general XML structure and layout attributes.
To ensure consistency with previous research on textbook modeling and to guarantee longterm usability, our pivot format can be converted to conform popular formats such as DocBook and TEI. Table 1 shows the correspondence between our elements and their equivalents according to the TEI Guidelines. The documents in the appendix depict the exercise featured in Figure 1, in our format (Listing 1), converted to TEI (Listing 2) and to DocBook (Listing 3).
While modeling the entire textbook seems to be suficient, the content extraction task is performed at the page level. Therefore, we define a diferent scheme to model each textbook page separately. In this second model, each token and segment tags will also be assigned position and style attributes, reflecting the layout and formatting of the textbook page. In addition, the nesting discipline > learning theme > chapter > activity blocks within the chapter is not straightforward. If chapter, theme or discipline titles appear on the page, these titles (along with potential indicators) constitute a block on their own, separate from the activity blocks on the page. As shown in Figure 2, this method enhances the visual representation of the blocks on the page and each text segment can easily be associated with a role (chapter title, lesson heading, lesson content, exercise number, exercise instruction, etc.) for the successful completion of the automatic extraction task.
chapter title exploratory activity heading illustration exploratory activity statement
indicators lesson heading instructions lesson
title number
exercise section heading instruction statement exercise example discipline
title hint
6Note that some text segments (e.g. heading “J’écris” above the last exercise in Figure 2) are integrated into the background images and will require OCR along with our PDF extraction process. @join <pc join=“left”> if preceding token element @sep=“empty”) Attributes @type: <intro type=“open| explo|revision...”> <lesson type= “vocabulary”>
TEI element(s) <pb/> <lb/> <div> <head> <figure> <graphic/> <seg> <w> or <pc> or <number> <table> No corresponding element, converted to an attribute: <cell role=“label”> <head> <row> <cell> <list> @cols @rows @rend=“numbered| lettered|bulleted| inline” <item> @n=“[item number]” No corresponding element, converted to an attribute: <item n=“a.”> No corresponding element, converted to an attribute: <list rend=“inline bulleted”> <list>
Due to the large quantity and diversity of textbook collections, an approach relying solely on rules would not be appropriate for our purposes because it would require extensive manual annotation. Our approach builds upon the model described in Section 3.2 and integrates both rule-based and deep learning methods. Figure 3 depicts the pipeline we are developing to convert a PDF textbook to its structured version.
Each textbook is first parsed to an XML file in ALTO format by pdfalto 7 coupled with MuPDF8. This combination of OpenSource tools enables the extraction of words along with their font style and spatial coordinates, as well as images, in a well-organized structure. The extracted words are tokenized and grouped into text segments using rules on font sizes and styles, character types (numbers, symbols, punctuation marks) and spacing between tokens and characters.
To reconstruct the textbook structure, segments must be labeled according to their role. In order to reduce the manual annotation workload, we utilize an annotation interface designed for MALIN. This web-based interface is supported by a TypeScript and Node.js back-end. The core idea is to map the XML ALTO file to an HTML format, enabling visual representation and annotation. Firstly, the annotator manually tags text segments with roles with just a few clicks. Segments are then semi-automatically labeled based on their dominant font. Secondly, the labeled segments are organized into higher-level categories that reflect the document’s structure (e.g. lesson, exercise, etc.). This organization process leverages geometric features, font types, font sizes, spacing and text patterns. At both stages, the results of the automatic annotation are visually presented in HTML format, allowing for easy examination and potential corrections. This ensures the accuracy of the annotation process.
Once we have collected and processed enough textbook pages, we constitute a dataset to train and evaluate deep learning models to achieve this annotation task automatically. Textbook page structure extraction can be achieved through a token classification task. Preliminary experiments conducted on a few French textbook pages are described in Section 3.4. New pre-processed pages will then directly pass through the extractor.
Upon token classification, textbook pages are formatted into our desired structure. Sections are built using geometric features and logical sequencing. For example, an exercise number following a statement indicates the beginning of a new exercise section. The process also
8https://github.com/ArtifexSoftware/mupdf involves identifying lists and tables, and filtering images in the PDF file. Activity illustrations are matched with the corresponding sections using geometric features, while images used for aesthetic purposes are removed.
After textbook pages are structured, and possibly merged to our document-scale model or its TEI or DocBook equivalent, the resulting data can be used for various artificial intelligence applications in education. For our adaptation purposes and depending on the disability, we would then be able to produce digital textbooks with a custom layout and interactive adaptations.
These preliminary experiments correspond to the training phase of the extraction pipeline depicted in Figure 3. This step involves training a deep-learning model on a token classification task to automatically predict the role of each token in the document, enabling the reconstruction of the document structure. 3.4.1. Experimental setup We constructed a dataset of textbook pages extracted from 1 elementary grade French textbook in PDF format. This constitutes a total of 167 pages, which are then split into 3 subdatasets: training (70%), validation (10%) and test (20%). For evaluation purposes only, we selected an additional 30 pages from a second textbook of the same collection, and 30 pages from a third textbook of a diferent collection. Each token is annotated with a coarse-grained page region label among: discipline, chapter, heading, introductory activity, lesson, exercise, page number. Future experiments will go further by introducing fine-grained labels. Table 2 lists the equivalences between coarse- and fine-grained page region classes.
In our first research on the classification of French textbook exercises according to their
adaptation to DCD with multimodal transformers [
Considering the supported limits of the models, we set the maximum input length to 512.
However, 60% of the pages are longer. These pages are encoded in 2 overlapping segments: once
by truncating the end of the document to the maximum length, and a second time by truncating
the beginning. Inspired by the sliding-window approach [
Our model outperforms the majority class baseline on all test sets. The intra-collection performance is very high. Since layout is the same for textbooks of the same collection, these results highlight the significance of layout features for the model.
The performance scores are lower for the 3rd textbook. Upon closer examination of the
predictions and comparing the pages with those from the textbook used for training, it becomes
evident that the errors are primarily due to layout diferences. Specifically, chapters in the
training collection are typically structured as shown in the example in Figure 2. However,
in the new textbook, exercises can be located before the corresponding lesson, which is
consistently at the bottom of the page. Besides, the distinction between introductory activities
9CamemBERT-BASE’s masked language model is fine-tuned on the following educational texts: pages from 4
French textbooks (apart from the pages of the validation and test subsets), 1293 Fantastiques Exercices, and the 79
original reading texts from the parallel corpus Alector [
10Among the overlapping segments, prediction diferences occur in 1/3 of the pages and impact an average of 5% of the tokens within those segments. exercises and actual exercises within a chapter is not clearly defined. As a result, some parts of lessons are wrongly predicted as exercises, and exercises within exploratory activities are occasionally misclassified as lessons or exercises. With the aim of developing a system with a high generalizability, training and evaluation sets must comprise more books from various collections.
These first experiments do not fully reflect the complexity of the task, since we use coarsegrained labels. For adaptation purposes, it will be necessary to provide a more detailed structure for each activity. Nevertheless, the results obtained with coarse-grained labels already point to a range of adaptation levels, depending on the nature of the blocks identified (e.g. lesson vs. exercise). On one hand, lesson adaptation for children with DCD or any other dyslexia-related disorder is already achievable, as it involves implementing standard accessibility modifications such as adjusting font, size, spacing and colors to enhance readability. On the other hand, activities that necessitate a shift in the mode of interaction require more in-depth extraction, and further processing to accurately identify11 and apply this shift. Regarding the extraction task and given the results obtained in this paper, we can consider a 2-step token classification. Another limitation is that some components do not appear in all textbook collections. For example, pages comprising our dataset do not explicitly indicate the learning theme (this information is exclusive to the index), whereas some textbooks mention it on each chapter page. Finally, our experiments cover only French language study textbooks. Textbooks of diferent subjects may present certain specificities not only in layout but also in content. When applying our models to numerous existing collections, it is imperative to account for these variations in layout and semantics and ensure suficient generalizability.
In this paper, we introduce our approach for automatic textbook structure and content extraction using activity-based textbook models. Our models not only provide layout and conceptual information, but also support the representation of an entire textbook according to widespread standards. We also report preliminary results on the token classification task to automatically identify all the components of a textbook page. These results are promising but reflect the dificulty of the task: generalization to various collections, whose content and layout are very diferent. Future work will address the progression of this automatic extraction task, using new multimodal transformer-based methods and going deeper into fine-grained labeling. Eventually, we will cover the implementation of the whole pipeline to convert a PDF textbook into a structured version according to our models.
This work was supported by the ANR-21-CE38-0014 MALIN project.
11Our previous work introduces a classification task that aims to classify French language study exercises based
on their adaptation type for children with DCD. [
Listing 1: Our format <div type="exercise" n="6", difficulty="**"> <div type="instruction"> <seg> <w>Complète</w> <w>les</w> <w>phrases</w> <w>avec</w> <w>on</w> <w>ou</w> <w>ont</w> <pc join="left">.</pc> </seg> </div> <lb/> <div type="statement"> <list rend="lettered"> <item n="a."> <seg> <w>Si</w> <pc>…</pc> <w>allait</w> <w>au</w> <w>cinéma</w> <pc>?</pc> </seg> </item> <lb/> <item n="b."> <seg> <w>Ils</w> <pc>…</pc> <w>vu</w> <w>ce</w> <w>film</w> <w>dix</w> <w>fois</w> <pc join="left">.</pc> </seg> </item> <lb/> <item n="c."> <seg> <pc>…</pc> <w>s’</w> <w>installe</w> <w>dans</w> <w>les</w> <w>fauteuils</w> <w>moelleux</w> <pc join="left">.</pc> </seg> </item> <lb/> <item n="d."> <seg> <w>Mes</w> <w>parents</w> <pc>…</pc> <w>pris</w> <w>du</w> <w>pop-corn</w> <pc join="left">.</pc> </seg> </item> <lb/> <item n="e."> <seg> <w>Les</w> <w>enfants</w> <pc>…</pc> <w>sursauté</w> <w>devant</w> <w>une</w> <w>scène</w> <w>de</w> <w>film</w> <pc join="left">.</pc> </seg> </item> </list> </div> </div>
Listing 3: DocBook <section role="exercise"> <section role="instruction">
<para>Complète les phrases avec on ou ont.</para> </section> <section role="statement"> <orderedlist numeration="loweralpha"> <listitem>
<para>Si … allait au cinéma ?</para> </listitem> <listitem>
<para>Ils … vu ce film dix fois.</para> </listitem>