An Overview of Recent Developments in Intelligent e- Textbooks and Reading Analytics

Abstract

This paper synthesizes recent developments in intelligent textbooks over the last five years and identifies potential research areas of interest to the AIED community. It characterizes traits that make a textbook intelligent. It discusses hot spots in the AIED community such as a) the prediction of academic performance based on students' reading behaviors, b) the assessment of learner skills based on their reading behaviors, and c) the automatic extraction of concepts taught in textbooks and their interdependencies (e.g., prerequisite, outcome, currency). It highlights key components of adaptivity that lead to full-fledged personalization and advocates the need for intelligent adaptivity as a trade-off between personalized provision of reading/learning materials and development and measurement of self-regulatory traits and grit. It concludes with a proposal to embed observational research methods as part of intelligent e-textbooks to automatically and continually infer causality between reading habits, reading activities, subject-matter competences, and metacognitive competences.

Keywords

• review
• analytics
• adaptivity
• reading
• textbook
• artificial intelligence
• observational data

-

For centuries, textbooks have been the best that technology could offer to extend the teaching experience beyond the mere presence of the human teacher. With the latest development in interactive technologies such as virtual and augmented reality, artificial intelligence (AI) techniques such as deep learning, ubiquitous data sensors through the Internet of Things, big data analytics, and the pervasiveness of mobile devices, the very concept of a textbook demands a reexamination. This paper reviews recent literature and exposes research initiatives in the field of reading analytics and intelligent e-textbooks. It first delineates the data that are currently captured about the reading process. It informs the reader about the datasets that have driven new advances in reading analytics. Based on the literature, it then describes a knowledge map of the entities involved in making a textbook smart. It discusses the prominent role adaptivity plays in the context of such intelligent textbooks and highlights the components of adaptivity that have not yet been addressed as part of the utopian augmented reading experience. Finally, throughout this paper and especially in a section at the end, directions for future work are provided to help newcomers in the field discover the areas that remain underexplored or unexplored, contributing to accelerate the progress and adoption of intelligent textbooks. 2

Data Types

Traditionally, intelligent e-textbooks collect the following pieces of data: the timeline when a reader goes to next/previous page, jumps to another page/section/chapter/document, bookmarks a page, highlights/underlines/marks some text including the color selected, tags a page as “not understood” [ 2 ] or marks/underlines unknown words [ 18 ], adds memo/annotation/comment/note, zooms in/out, opens/launches and exits the ereader application, and searches for keywords (including search jumps).

Some reading actions, such as scrolling/swiping vs. clicking on the next page button, zooming in/out vs. tapping, and portrait vs. landscape viewing mode, are dependent on the type of device (smartphones, tablets, computers, e-readers) used. Some data types are also dependent on the interactive objects embedded within the e-textbook such as links clicked, parameters input when running sample code, etc. According to [ 9 ], interactivity may be viewed as a continuum where at one extreme the e-textbook is identical to a static printed text, while at the other end the e-textbook may embed interactive activities, 3D models, videos, etc.

Collected metadata also include the type of device used and its IP address, the type and version of the web browser/reading tool, the student/reader ID, timestamp of the reading event, and the online status, that is, whether the reading device was connected or not to the Internet when the reading event was generated. Failsafe data collection mechanisms have been developed to support both offline and online reading, storing reading-related actions locally during offline reading episodes and sending those data to backend servers when Internet connectivity becomes available [ 6,12,20,21 ]. These data and metadata have been encoded according to the xAPI specification in order to share and exchange them with other learning analytics systems [ 2 ].

Other data can be collected externally to the e-textbook such as eye gazes (e.g., numbers of fixations, saccades, and blinks; distances of eye movements; and coordinates of eye gazes) through an eye tracker, the reader’s body language/posture through a camera, and whether the reader is reading aloud through a microphone. However, these sensors are intrusive and may introduce further ethical issues by capturing data not related to the reading experience itself. Nevertheless, no matter the reading device used by the learner, the same set of data types can be collected, such as the amount of time spent reading a specific page, reading speed, engagement level, level of attention, the last textbook pages read during a semester, etc.

In the near future, through deep learning and other artificial intelligence techniques, one can expect mechanisms to infer the segments of a page that are currently visible to the student, changes performed on these visible segments, the zoom level or the font size, and the text passages annotated by the student and the spatiotemporal sequence of those annotations. These inferences enable one to predict, the word, phrase, or sentence that the student is currently reading or paying attention to, and the level of comprehension sensed during reading without the use of hardware devices such as eye trackers.

Datasets on Reading

Table 1 lists the datasets of reading interactions found in recent literature generated by readers’ interactions with intelligent textbooks. The number of events by dataset ranges from 65 to 2.8 million, generated by between 9 and 2993 readers, and collected during reading episodes ranging approximately from 30 minutes to one year. The reading experiences occurred in textbook, lecture, research publication, and magazine settings. It is, nevertheless, remarkable that none of these datasets is available to the research community, which calls for the delivery of one or more open benchmark datasets to propel research advances in reading analytics and to allow researchers to replicate results and measure progress in the field. interacting with a textbook or a reading resource. The reading resource is encapsulated within a specific format such as PDF or EPUB and ultimately should include web pages, Word documents, discussion threads, in other words, everything that the student may read, no matter its form. These reading resources are then hosted by a learning resource repository or learning management system and accessed through a reading software application (e.g., Adobe Reader, Kindle, etc.). Readers consume and interact with these reading resources through hardware such as a computer, tablet, or smartphone, potentially enhanced by supplemental equipment such as augmented reality (AR) headsets or eye tracking devices.

Turning a reading resource such as a textbook into a smart one requires at a minimum the following two sources of data: 1) the transactional data resulting from the student’s interactions with the reading tool and its features, and 2) the breaking down of the reading resource’s contents into a knowledge map of prerequisite and outcome knowledge components. Hence, as pointed out in Section 2, Fig. 1 delineates what the research community collectively tracks about device-dependent student interactions with a reading application. These raw data are stored and then transformed to derive more useful pieces of information or metrics, ideally in real time to provide real-time feedback to teachers/students. For example, the reading time is computed by [ 13 ] to determine whether a student has learned or still has to learn a given knowledge component by looking at the amount of time he/she spent reading/skimming the related pages, sections, or chapters of the textbook, measuring also the knowledge level of the student on the underlying concept. The authors employ two techniques: the Knowledge Tracing model and linear regression analysis. To scale their approach and reduce the manual effort needed to extract knowledge components from textbook contents, [ 13 ] uses bagof-words models and latent semantic analysis.

The raw interaction data, in addition to the inferred variables (e.g., reading speed, reading session, etc.), when arranged as a sequence, constitute the reading behavior of the learner. Data about the reading behavior are also collected from self-reported surveys or questionnaires and compared against the quantitative measurements taken from the actual observations of the reading process to estimate the gap in the reader’s perception of the reality [ 21 ]. Furthermore, meaningful reading patterns are extracted through progressive sequential analysis (lag-sequential analysis) by determining the probability that a type of action is followed by another. For example, after having highlighted a part of the text, [ 30 ] found that the likeliest action that the reader is going to do is another highlighting operation.

The bottom of Fig. 1 lists features associated with intelligent textbooks. Among the most cited features are a) the prediction of the student performance or the automated detection of students at risk of dropping out, b) the real-time enhancements to the learning materials based on annotations1 from the readers, c) the provision of teacher annotations within the textbooks to indicate concepts or sections that are particularly important and to provide clarifications or further resources on sections that are especially difficult for students to understand, and d) readers’ level of interest and competence in 1 [ 9,29 ] define the term ‘annotation’ as follows: an explicit expression of knowledge that is attached to a document to reveal the conceptual meanings of an annotator’s implicit thoughts. 6 Fig. 1. Knowledge map (right part) of the various entities involved in intelligent textbooks. the topics/concepts taught in the textbook and their level of interest in making usage of the various annotation and smart features of the textbook.

Other features of interest include a) automated assessment of the difficulty level of a concept, a requirement to prompt teachers to publish annotations within the textbook to address that difficulty; b) automatic correction of the student’s answers to the textbook questions and problems and the underlying solving processes; c) provision of formative feedback fed by the automated assessment of the student’s knowledge and competence of the curriculum’s learning outcomes; d) recommendation of sections explaining concepts that are ready to be learned given the current cognitive profile of the student; e) identification and recommendation of effective and suboptimal learning strategies; f) assessment of the self-regulated learning of the learners; and g) a lecture supporting system that informs teachers in real time of the lecture’s sections requiring more attention and whether students are following the teacher’s explanation [ 24 ]. Interestingly, the prediction of student performance has led researchers to measure and investigate the impact of intermediary variables such as level of engagement2 and gaming behavior3, using both unsupervised (e.g., clustering) and supervised (e.g., binary classification, blocked linear regression) machine learning techniques. Fig. 1 also lists various techniques leveraged to power these smart textbook features as well as methods for effective feature selection and measurement of model performance (accuracy).

Intelligent digital textbooks are envisioned to be interactive, collaborative, adaptive, and as embedding visuals (e.g., video, sketch, animation, diagram) [ 22 ]. In order to turn a textbook into a collaborative tool, previous research works such as [ 27 ] integrated a question and answer forum within the textbook, displaying only those discussion threads related to the page being read by the student and allowing the students to create new questions, vote on provided answers, and tag question contents. On the other side, the interactivity of textbooks is key to the development of reading analytics and to augment students’ reading experiences through a variety of AI-generated insights. More data types can be collected through interactive components, reflecting on the students’ reading behaviors and producing more personalized formative feedback within the etextbooks when and where it is most needed. There is a trend, however, to view intelligent digital textbooks as learning platforms in their own right by incorporating components of the broader learning process such as problems to solve, multiple-choice and open-ended questions, customizable examples (with different sets of parameters), etc. This paper suggests that the discriminatory characteristic of intelligent textbooks or reading analytics should be related to its scope, that is, it should focus on measuring and optimizing reading episodes of students and assist them in their decision-making process related to reading to improve reading’s effectiveness on the overall learning process. The authors of this paper also advocate the need to understand further the role of reading within the overall learning process by analyzing data streams coming from heterogeneous learning activities [ 26 ] (e.g., search, video, listening, discussion, project, 2 [ 3,15 ] define “engagement” as the amount of physical and psychological energy that the student devotes to the academic experience. 3 [ 4,8 ] define “gaming behavior” as the attempt to succeed in an educational environment by exploiting properties of the system rather than by learning the material and trying to use that knowledge to answer correctly. etc.) in addition to the more elementary data on reading and quiz performance [ 13,23,26 ]. For instance, should students start reading before practicing or should they start practicing and read only when necessary (e.g., start coding a program and read the related concepts as they are needed)? This would have the advantage of supporting both the learning by doing and informal learning paradigms. 5

Adaptivity

True adaptivity in online education consists of four components: 1) content model, student model, instructional model, and the adaptive engine [ 25 ]. The content model essentially captures all the knowledge components (viz., topics, concepts, competences) of a learning domain and the interdependencies (prerequisite relationships) among them. For example, the Knowledge Space Theory [ 10 ] leverages combinatorics to model the knowledge space of a learning domain and to identify the knowledge state of a learner in a pool of thousands if not millions of different knowledge states, and this using only a couple of dozens of well-picked assessment questions. This modeling technique enables the tracking of students’ learning paths, with precision, as the students navigate through knowledge states.

Fig. 2 (left) shows the precedence diagram of a knowledge space of 10 topics (Topic A to Topic J), with Topic A being a prerequisite to Topics B, C, and D. This results in a knowledge structure (right of Fig. 2) of 40 distinct knowledge states encapsulating possible learning paths that a student can take toward success. The student model encapsulates the cognitive and metacognitive traits of the learner in addition to many other characteristics such as demographics, socio-economic status, learning style, and learning preferences.

Student modeling is mainly concerned with measuring, assessing, and collecting information about these characteristics. The instructional model constantly compares the student model and the content model to identify any gap in the student’s knowledge and recommend learning resources to fill in that gap, be it another learning activity to consolidate the same concept/topic being learned or which new concept the learner is ready to learn to progress in an optimal learning path toward the most desirable knowledge state. The instructional model is also responsible for the timely delivery of learning resources and “how to present that content to the learner” [ 25 ], that is, which representation (textual vs. graphical; which learning object among those having a graphical representation, for example; collaborative vs. individualized; animation vs. 3D exploration) of the knowledge to be learned should be presented to the learner since each concept can be explained differently. Finally, the adaptive engine applies the rules of adaptivity fed by information coming from the content, student, and instructional models and assigns priorities to the learning objects, delivering to the student the learning object with the highest priority.

The literature highlights the prominence of adaptivity as a feature of intelligent textbooks. However, previous research has mainly focused on content modeling and student modeling, that is, the manual and automatic extraction of the key concepts of textbooks’ contents as well as the assessment of the students’ knowledge level of these concepts by analyzing their reading behaviors and quiz performance. Hence, no research has yet reached full maturity in regard to the four components of adaptivity, laying only the foundation for the delivery of adaptive reading contents to the students.

The instructional model and adaptive engine have received little attention, especially in regard with finding the proper trade-off between full-fledged adaptivity and personalization and development of self-regulatory traits in students, in other words, providing opportunities for both system-driven and user-driven consumption of reading materials [ 28 ]. For example, students could be left on their own to search for the proper resources as part of the strategy they have set to reach their goals. The frequency and points in time of using reading as a learning strategy and the actual contents of the resources they pick to learn will showcase their cognitive and metacognitive profiles, invaluable insights that will further improve adaptivity in intelligent textbooks. In compliance with the open data initiative, benchmark datasets on both reading activities and heterogeneous learning activities should be available to accelerate the development of intelligent textbooks. For example, due to the absence of such datasets or their small size in other cases, it is noticeable that the latest deep learning techniques have not yet been experimented. Moreover, the analysis of the reading process should be made in the broader frame of the learning process by not only analyzing what happens during the reading process but also how the student uses reading as a learning strategy when working toward a learning outcome, taking into account the cognitive and metacognitive traits of the student as s/he resorts to reading strategies.

In parallel to developing smart features for textbooks, researchers have also investigated the impact of e-textbooks with a focus on the medium (paper vs. screen) [ 9 ] and more recently the effectiveness of each individual smart feature (e.g., annotations, interactive animations, annotated code examples, collaborative Q&A) on student performance and motivation [ 9,15,19,27 ]. For example, [ 12 ] assessed the impact of the time of adoption of mark-up features with both digital and printed textbooks on course grades. Reference [ 9 ] investigated the effect of teacher annotations on student learning as measured by multiple-choice and open-ended questions. Others investigated the learning processes and engagement of students interacting with digital textbooks on mobile devices and proposed a framework for learning with digital resources to help students transition from using mobile devices for personal use to effective learning [ 5 ]. References [ 14,23 ] have analyzed the relationship between cognitive states and eye movements and the relations between eye gaze patterns and building of correct graphical causal maps and overall student performance.

Learning analytics has been construed as “an ethics-bound, semi-autonomous, and trust-enabled human-AI fusion that measures and advances knowledge boundaries in human learning” [ 16 ]. This paper hence proposes an experimental design that will turn e-textbooks and learning analytics systems into a research platform that will collect and share observational data among interested parties in education. The proposed experimental design performs causal inferencing based on the Potential-Outcomes framework. It defines the sources of bias and handles these sources of bias by iteratively measuring the level of data imbalance within the observational reading/learning datasets and pruning or weighing those data points introducing most data imbalance within these datasets using techniques such as matching and Inverse Probability Treatment Weighting (IPTW) until trustable levels of data balance and generalizability are reached [ 7,16 ]. This framework assesses the effect size of a reading/learning-related treatment variable on an outcome variable and performs sensitivity tests to estimate the presence of unobserved confounding factors in the analysis. By automating this process, educational stakeholders will be notified when further reading episodes need to be collected to improve the power of conclusions, when to collect data about new groups of the student population to improve generalizability, or when previously unanswered research questions offer new answers based on the available data. Integrating this research platform with intelligent textbooks will provide the mechanisms to conduct meta-analyses by networking research endeavors and connecting evidences together, which will empower the educational community with insights on the reading behavior and learning process observable from learning episodes.

Intelligent textbooks can be extended beyond the traditional digital book to include printed textbooks incorporating diverse interactive components discussed in this paper through augmented reality headsets. Augmented reality observations, along with embedded eye-tracking devices, can capture a rich collection of interaction and physiological data to advance research on optimal reading behaviors and on the effectiveness of merging state-of-the-art technology with traditional media.

Another area of future work is the development of intelligent adaptivity, where the sequencing and presentation of reading and learning contents will be balanced to nurture students’ self-regulatory traits. The level of adaptation should be adjusted to challenge students with good self-regulatory practices and to develop grit. Recommendations from intelligent textbooks should not only target which pages or topics to read next but also target optimal reading behaviors, for example, increasing/decreasing reading speed depending on the level of difficulty of a text passage or reading twice or more times a certain section given the student's knowledge and difficulty level of the concepts to be learned, based on, say, the reading behaviors of previously successful students.

References

1. Akçapınar , G.: Student Modeling for Digital Textbook Reader to Increase Engagement . In: 26th International Conference on Computers in Education. Asia-Pacific Society for Computers in Education, Philippines ( 2018 )

2. Askinadze , A. , Liebeck , M. , Conrad , S. : Predicting Student Test Performance based on Time Series Data of eBook Reader Behavior Using the Cluster-Distance Space Transformation . In: 26th International Conference on Computers in Education. Asia-Pacific Society for Computers in Education, Philippines ( 2018 )

3. Astin , A.W. : Student Involvement: A Developmental Theory for Higher Education . J. Coll. Stud. Dev . 25 , 297 - 308 ( 1984 )

4. Baker , R. , Walonoski , J. , Heffernan , N. , Roll , I. , Corbett , A. , Koedinger , K. : Why Students Engage in “Gaming the System” Behavior in Interactive Learning Environments . J. Interact. Learn. Res . 19 , 185 - 224 ( 2008 )

5. Bikowski , D. , Casal , J.E. : Interactive digital textbooks and engagement: A learning strategies framework . Lang. Learn. Technol . 22 , 119 - 136 ( 2018 )

6. Boratto , L. , Cadeddu , M. , Carta , S. , Deplano , G. , Mereu , F. : A Tool to Analyze the Reading Behavior of the Users in a Mobile Digital Publishing Platform . In: 2nd International Workshop on Knowledge Discovery on the WEB (KDWEB) ( 2016 )

7. Boulanger , D. , Kumar , V. , Fraser , S. : Automating Educational Research Through Learning Analytics: Data Balancing and Matching Techniques ( 2018 ), http://learninganalytics.ca/research/its-2018 - tutorial -on-automating-educational-research/

8. Chau , H. , Li , A. , Lin , Y. -R.: Predicting Students Performance Based on Their Reading Behaviors . In: International Conference on Social Computing, Behavioral-Cultural Modeling & Prediction and Behavior Representation in Modeling and Simulation . Washington, D.C. , USA ( 2017 )

9. Dennis , A.R. , Abaci , S. , Morrone , A.S. , Plaskoff , J. , McNamara , K.O.: Effects of e-textbook instructor annotations on learner performance . J. Comput. High. Educ . 28 , 221 - 235 ( 2016 ). doi: 10 .1007/s12528-016-9109-x

10. Falmagne , J.-C., Doignon , J.-P. : Learning spaces: Interdisciplinary applied mathematics. Springer Science & Business Media ( 2010 )

11. Flanagan , B. , Shimada , A. , Yang , S. , Chen , B.-L. , Shih , Y.-C. , Ogata , H.: Predicting Performance Based on the Analysis of Reading Behavior: A Data Challenge . In: 9th International Conference on Learning Analytics & Knowledge (LAK) . Tempe, Arizona, USA ( 2019 )

12. Van Horne , S. , Russell , J. , Schuh , K.L. : The adoption of mark-up tools in an interactive etextbook reader . Educ. Technol. Res. Dev . 64 , 407 - 433 ( 2016 ). doi: 10 .1007/s11423-016- 9425-x

13. Huang , Y. , Yudelson , M., Han , S. , He , D. , Brusilovsky , P. : A Framework for Dynamic Knowledge Modeling in Textbook-Based Learning . In: 2016 Conference on User Modeling Adaptation and Personalization . pp. 141 - 150 . ACM, Halifax, Canada ( 2016 )

14. Ishimaru , S. , Bukhari , S.S. , Heisel , C. , Kuhn , J. , Dengel , A. : Towards an Intelligent Textbook: Eye Gaze Based Attention Extraction on Materials for Learning and Instruction in Physics . In: 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct . pp. 1041 - 1045 . ACM, Heidelberg, Germany ( 2016 )

15. Junco , R. , Clem , C. : Predicting course outcomes with digital textbook usage data . Internet High. Educ . 27 , 54 - 63 ( 2015 )

16. Kumar , V. , Boulanger , D. , Fraser , S. : Inferring Causal Effects from Learning Analytics: Discovering the Nature of Bias ( 2019 ), http://learninganalytics.ca/research/icsle-2019- tutorial/

17. Labutov , I. , Huang , Y. , Brusilovsky , P. , He , D. : Semi-Supervised Techniques for Mining Learning Outcomes and Prerequisites . In: 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . pp. 907 - 915 ( 2017 )

18. Ogata , H. , Yin , C. , Oi , M. , Okubo , F. , Shimada , A. , Kojima , K. , Yamada , M.: e-Book-based Learning Analytics in University Education . In: 23rd International Conference on Computers in Education. Asia-Pacific Society for Computers in Education, China ( 2015 )

19. Pollari-Malmi , K. , Guerra , J. , Brusilovsky , P. , Malmi , L. , Sirkiä , T. : On the Value of Using an Interactive Electronic Textbook in an Introductory Programming Course . In: 17th Koli Calling International Conference on Computing Education Research . pp. 168 - 172 . ACM, Koli, Finland ( 2017 )

20. Prasad , D. , Totaram , R. , Usagawa , T. : Development of Open Textbooks Learning Analytics System . Int. Rev. Res. Open Distrib. Learn . 17 , ( 2016 ) (a ). doi:10.19173/irrodl.v17i5.2541

21. Prasad , D. , Totaram , R. , Usagawa , T. : A Framework for Open Textbooks Analytics System . TechTrends . 60 , 344 - 349 ( 2016 ) (b ). doi:10.1007/s11528-016-0070-3

22. Quaresma , P. : Towards an intelligent and dynamic geometry book . Math. Comput. Sci . 11 , 427 - 437 ( 2017 )

23. Rajendran , R. , Kumar , A. , Carter , K.E. , Levin , D.T. , Biswas , G.: Predicting Learning by Analyzing Eye-Gaze Data of Reading Behavior . In: 11th International Conference on Educational Data Mining. International Educational Data Mining Society , Raleigh, USA ( 2018 )

24. Shimada , A. , Konomi , S. , Ogata , H.: Real-time learning analytics system for improvement of on-site lectures . Interact. Technol. Smart Educ . 15 , 314 - 331 ( 2018 )

25. Shute , V. , Towle , B. : Adaptive E -Learning. Educ. Psychol . 38 , 105 - 114 ( 2003 ). doi: 10 .1207/S15326985EP3802_ 5

26. Thaker , K. , Huang , Y. , Brusilovsky , P. , Daqing , H.: Dynamic Knowledge Modeling with Heterogeneous Activities for Adaptive Textbooks . In: 11th International Conference on Educational Data Mining . pp. 592 - 595 ( 2018 )

27. Walker , E. , Chakravarthi , R. , Rodriguez , J. , Wylie , R.: Promoting Interaction by Integrating a Question and Answer Forum with a Digital Textbook . In: Exploring the Material Conditions of Learning: The Computer Supported Collaborative Learning (CSCL ) Conference 2015 . International Society of the Learning Sciences, Gothenburg , Sweden ( 2015 )

28. Weber , G. , Brusilovsky , P. : ELM-ART - An Interactive and Intelligent Web-Based Electronic Textbook . Int. J. Artif. Intell. Educ . 26 , 72 - 81 ( 2016 ). doi:10.1007/s40593-015-0066- 8

29. Yang , S.J.H. , Zhang, J., Su , A.Y.S. , Tsai , J.J.P.: A Collaborative Multimedia Annotation Tool for Enhancing Knowledge Sharing in CSCL . Interact. Learn. Environ . 19 , 45 - 62 ( 2011 ). doi: 10 .1080/10494820. 2011 .528881

30. Yin , C.J. , Uosaki , N. , Chu , H.C. , Hwang , G.-J., Hwang , J.J. , Hatono , I. , Tabata , Y. : Learning Behavioral Pattern Analysis based on Students' Logs in Reading Digital Books . In: 25th International Conference on Computers in Education . pp. 549 - 557 . Asia-Pacific Society for Computers in Education ( 2017 )