This study assesses the efficacy of intelligent texts to help students in a computer science class understand and process information about computational thinking and programming. The intelligent texts used in this study were taken from an introductory programming textbook. The texts were ingested into an intelligent text format using the Intelligent Texts for Enhanced Language Learning (iTELL) framework, which converts any type of machine-readable text into an interactive, intelligent text. iTELL asks students to complete constructed response items and summaries, which are scored automatically by large language models (LLMs) specifically trained to generate scores to inform qualitative feedback to students. Survey results indicated that students responded positively to the constructed response and summary items and felt both items helped them learn. An analysis of delta value gain scores between pre-tests and post-tests for students that used iTELL and those that did not use iTELL indicated that iTELL students showed increased learning gains. Regression analyses showed that delta scores for the iTELL students were predicted by the number of scrolls, word scores on summaries, and pre-test proficiency level (low/high). The results indicate that intelligent texts may help computer science students better learn material than traditional texts.
Computational thinking is a critical 21st century skill that
can help students navigate an increasingly digital world. It
describes the ability to express a problem in terms of steps,
such that they could be written out as an algorithm.
Teaching computational thinking allows students to
explore knowledge in concrete ways, and asking students
to code computational thinking into a computer program
can provide students with a quick method to check the
validity of the knowledge [
However, coding is a complex skill
that requires sustained effort, a specialized approach, and
a diverse skill set. Developing these skills is an iterative
process that requires persistence and knowledge well
beyond simple syntax [
© 2024 Copyright for this paper by its authors. Use permitted under
Creative Commons License Attribution 4.0 International (CC BY 4.0).
the dedication required to succeed as a programmer means
that computer science classes suffer from high failure and
dropout rates [
Students in computer science classes are also expected
to learn about computational thinking and programming
approaches, using reading materials. However, printed
books are generally considered ineffective at teaching
computational thinking and the dynamic nature of
programming because they are bound by the static
confines of the text [
The goal of this study is to assess the efficacy of interactive intelligent texts in a computer science class to help students understand and process information about computational thinking and programming. The intelligent texts used in this study are taken from an introduction to computing textbook. They were ingested into an intelligent text format using the Intelligent Texts for Enhanced Language Learning (iTELL) framework. iTELL is a computational framework that converts any type of machine-readable text into an interactive, intelligent text within a web-app. iTELL is based on theories of reading comprehension and provides opportunities for users to generate knowledge about what they read and watch using constructed responses and summary writing. The constructed responses and summaries are scored automatically by large language models (LLMs) specifically trained to generate scores which inform qualitative feedback to students. The feedback from these AI integrations is used in a number of different ways, including to guide learning, correct misconceptions, review missed topics, prepare for upcoming materials, make links between the texts and the real world, and help elaborate on what users have learned. iTELL represents an advanced learning technology based on AI which can be used to improve student learning outcomes in computer science classes. 1.1.
Intelligent textbooks have become more popular as
advancements in Natural Language Processing (NLP) have
made human-machine interaction more accessible [
Digital textbooks have been in production for over 30
years with initial texts using principles of knowledge
engineering wherein domain experts designed and
produced the textbooks [
Research on intelligent textbooks has increased in the
past decade as computational tools have become more
advanced and accessible [
The advent of large language models (LLMs) will further change the developmental landscape for intelligent textbooks. LLMs allow intelligent textbooks to be more interactive and afford more accurate, real-time feedback on student generated responses used to assess text understanding. Additionally, LLMs allow for the integration of AI chatbots that can guide readers through the text and potentially help with any misunderstandings or difficulties the readers may encounter. In short, LLMs enable the creation of intelligent textbooks that can automatically generate content as well as prompt and evaluate reader responses allowing for the automation of scoring and feedback generation. 1.2.
iTELL iTELL is a framework that simplifies the creation and deployment of intelligent texts with integrated interactive features. iTELL includes automated pipelines that leverage Large Language Models (LLMs) with human oversight to generate participatory content like constructed response items and summaries. Additionally, it includes scoring APIs for constructed responses and summaries. iTELL is a domain-agnostic framework that utilizes multiple highly transferable generative LLMs to transform static texts into interactive, intelligent textbooks.
iTELL generates rich clickstream data, allowing for the analysis of user behaviors, particularly related to reading. It uses JavaScript's intersection observer API to determine whether a specific text section is within the user's viewport and logs the observation time for different parts of the text. iTELL also log events within the systems include scrolling and page clicks.
Most importantly, iTELL includes read-to-write tasks
that engage the reader in learning. Read-to-write tasks
require readers to extract and integrate information from
the text into their writing allowing them to construct
knowledge as they read [
The current study examines an iTELL volume deployed in an Introduction to Computing class. A single volume of iTELL was developed that covered a textbook chapter on control structures (Chapter 3). Students within the classroom were given extra credit to use the iTELL volume of which about 25% did. The remaining students depended on a static, digital version of the textbook. At the end of each chapter, the students were given a test to assess their knowledge. The research questions that guide this study are the following: 1. 2.
with, is understandable, helps them learn, and provides accurate feedback? Do students that completed the iTELL volume show gains from the test on chapter two (no iTELL volume) to the test on chapter 3 (iTELL volume) compared to students who did not complete the iTELL volume? Are data points collected from the iTELL volume related to click-stream data, focus time, and summary scores related to differences in test scores from chapter two to chapter three?
2.1.
Data for this research is based on an Introduction to Computing class that was taught in the spring of 2024 at a large technology university in the southeastern United States. The course is one of three courses that can fulfill the computer science requirement for all students at the university and is taken by over a couple of thousand students every academic year in one of two variations: inperson and online. The course covers the basics of computing, presupposing no prior programming ability: it begins with the basics of procedural programming, moves through control structures and data structures, and concludes with brief units on object-oriented programming and algorithms. Throughout the course, students complete several hundred small programming problems through the homework assignments, as well as some live coding problems during four timed, proctored tests. Tests and quizzes comprise 52% of students' grades in the class. 2.2.
The textbook used in the course is called Introduction to
Computing, first edition [
For this study, we ingested the third unit covering Control Structures into an iTELL volume using iTELL’s content management system. The volume comprised an overview page that introduced iTELL and 5 additional pages with each page referencing a chapter from the unit. Each page followed the structure of the chapter in the book including learning objectives, key terms, the chapter prose, and any figures, graphs or tables. However, screenshots of integrated development environments (IDEs) in the textbook that demonstrated Python code and the code output were replaced with a Python interactive sandbox. The sandbox allowed students to enter in their own code and run the code within iTELL. The pages were separated into chunks (i.e., all content under a unique sub-heading) based on related content as selected by the page designer. On average, each page had around 6.6 chunks (SD=1.14). Only the first chunk on a page was visible to a user at the beginning with all subsequent chunks being blurred. Users were required to click on a “chunk reveal” button to unblur the next chunk.
The content management system automatically generates constructed response questions and answers for each chunk with human-in-the-loop. Prior to publishing, the page designer ensured that the questions and answers were accurate. Each chunk had an accompanying constructed response item. There was a 1/3 chance of a constructed response item being presented to a user for each chunk. 2.3.
There was a total of 476 students enrolled in the class. Of those enrolled, 121 students elected to use the iTELL version of the textbook and 356 did not. Students were given 1% extra credit (added to their overall course grade) for participating in the study.
These 121 students first provided consent for their data to be used. If the student did not provide consent or was under the age of 18, they were sent directly to the iTELL volume, but no data was collected. If they provided consent, they then completed an intake survey that collected demographic information and individual difference data including age, sex, race/ethnicity, first language, and reading habits and technology use. They were then sent to the iTELL volume. If the student finished the five pages in the volume, participants were asked to complete an outtake survey to describe their experience of working with the iTELL volume. The outtake survey focused on students’ perceptions of the digital text’s layout, organization, annotation features, and the effectiveness of the summary and short answer tasks.
Of these 121 students, 101 consented to having their data used for analysis. Of those 101 students, 82 completed iTELL including the outtake survey. However, of the 82 students that completed iTELL, 79 reported test scores for units 2 and 3. Of the 356 students that did not use iTELL, 277 reported test scores for units 2 and 3. 2.4. 2.4.1.
Before interacting with iTELL, students completed a short intake survey to collect demographic data such as age, gender, race or ethnicity, and first language background. In this intake survey, students were also asked to provide information about their interactions with technology and they provided input about their reading habits on electronic and traditional texts. Finally, students provided information about the types of features that they have used before in intelligent texts. 2.4.2.
Upon completion of the intelligent texts, users completed an outtake survey which included user feedback questions on each feature of iTELL including annotation and notetaking, the section summary tasks, constructed response items, and overall feedback about the layout and organization of the intelligent text. This survey allowed us to collect data on users’ perceptions of how well each of the features stayed relevant to the text, worked correctly, was easy to interact with, and helped improve the users’ learning. Users were also prompted to provide short text feedback about each of the features. 2.5.
iTELL Data Extraction For this analysis, we extracted data related to participant focus time, click-stream events, constructed responses, and summaries. 2.5.1.
For each page focus time was extracted in two different ways. First, focus time was recorded bey subtracting the time that users opened the page and the time the user moved onto the next page. The focus time included all the time spent on constructed responses and on summary scoring. Second, focus time was recorded by how long each chunk in a page was viewed. From this, a total time for all chunks per page and an average time for all the chunks on a page was derived. 2.5.2.
A number of events are calculated in iTELL that can be instrumented into predictive variables. These include chunk reveal events (for chunks with and without constructed response items), general clicks on items within the system, periods of time when learners are focusing on the page, and when scrolling. The chunk reveals without constructed responses is not the inverse of the chunk reveals after constructed responses because many students reread chapters either as a choice or as a function of needing to reset the chapter because of a bug in the iTELL system. Around 40% of chapters had some type of rereading (i.e., scrolling upwards of more than 3% of the page content) on the part of students. 2.5.3.
As part of the iTELL integration, an accompanying
constructed response item is generated for each chunk
using GPT-3.5-turbo with human-in-the-loop. End users do
not see all constructed response items when reading an
iTELL volume; instead, each chunk has a 1/3 chance of
spawning an accompanying constructed response item,
with a minimum of one constructed response item per
page. Users are required to submit at least one response to
a spawned item before proceeding to the next chunk.
Readers’ constructed responses are scored for correctness
using two separate fine-tuned LLMs, Bilingual Evaluation
Under-study with Representations from Transformers
(BLEURT) [
For each participant, we calculated the number of
constructed responses they produced and the average score
they received for the constructed response on a scale of
13 with a 1 representing when the two LLMs agreed the
answer was incorrect, a 2 representing when one of the
LLMs classified the answer as incorrect and the other
classified as correct, and a 3 representing when the two
LLMs agreed the answer was correct. Figure 1
demonstrates the interface and the feedback returned for a
constructed response scored a 2 by the model. Because of a
bug in the code connecting iTELL to its database, a number
of constructed responses were not logged and were omitted
at random. Of the 395 pages completed by the 79 iTELL
participants, 130 of those pages did not have constructed
response data logged.
After reading each page, students were prompted to write
a summary of what they read. Algorithmic filters in iTELL
ensure that the summaries are between 50 and 200 words
long, are written in English, do not include inappropriate
language. The iTELL interface does not allow copying and
pasting directly from the text. When a student submits a
summary, they receive a score on Language Borrowing by
calculating the proportion of overlapping trigrams between
the summary and the source [
These models, based on the Longformer pretrained
model [
For each participant, we calculated the number of summaries they produced and the average score they received on each summary for Content, Wording, Language Borrowing, and Relevance. 2.5.5.
The iTELL volume of the Introduction to Computing textbook included an integrated development environment (IDE) sandbox that exhibited Python code and allowed students to enter and run their own code. However, logging features for the sandbox data were not implemented at the time of data collection. 2.6.
We ran three different analyses to address our research questions. For the first research question related to whether users think iTELL is easy to interact with, is understandable, helps them learn, and provides accurate feedback, we ran simple descriptive statistics and graphed out the results. For the second research question related to whether students who completed the iTELL volume show gains from chapter 2 to chapter 3 test scores compared to students who did not use the iTELL, we ran statistical tests to assess differences between the two groups’ delta values. Our main statistical metrics are a p value to indicate if an effect exists and a Cliff’s Delta value to examine the strength of the effect. For the third research question related to whether data points collected from the iTELL volume related to click-stream data, focus time, and summary scores are predictive of delta values, we conducted a stepwise linear regression using iTELL data as predictors of the delta values. We included a categorical performance variable on the chapter 2 test based on whether students scored above the mean (high) or below the mean (low). This variable was included as a predictor and as a possible interaction to see if there was an effect of iTELL on lower or higher-performing students.
3.1.
For the user survey data, we were most interested in student responses to the summarization task, the constructed response items, and their overall satisfaction with iTELL. To provide a simpler representation for survey item visualizations, we combined scores of 4 and 5 into a single category (agree) and all scores of 1 and 2 into a single category of disagree. Scores of 3 were labeled neutral. We conducted follow up ANOVAs to examine if any differences were noted across survey responses by ethnicity or reading frequency.
For the summary task, the mean responses were generally positive (M > 4). The lowest responses were for the accuracy of feedback (M = 4.18) while the highest responses were for ease of understanding (M = 4.29). Students felt that the summary tasks helped them learn (M = 4.18). There were no significant differences noted across survey items by race or ethnicity or by reading frequency. Data for this analysis are presented in Figure 3. condition may have improved their test scores after using iTELL more than students who did not use iTELL For our test score difference analysis, we examined the Delta score between test 2 and test 3 for students who used iTELL and those that did not. Descriptive statistics indicated greater gains in learning for test 2 and 3 by the students who used iTELL (M = .032, SD = .304 than the noniTELL students (M = -.018, SD = .248). Visual examinations of the data indicated that it was not normally distributed (see Figure 6 histogram). Thus, we conducted a MannWhitney U test on delta scores across conditions (iTELL vs. non-iTELL). The differences approached significance (p=0.067; U=12,386) indicating that an effect size is likely. The Cliff’s Delta = .132, 95% CI [-0.029, 0.286]) reported a small effect suggesting that students in the iTELL condition scored higher on test 3 (after using iTELL) than they did on test 2 (no iTELL use). * p < .05, ** p < .010 Three variables were significant predictors in our regression model: number of scrolls, Wording scores on summaries, and the categorical variable related to whether the student scored high or low on test 2. The linear model reported r = .501, R2 = .251, F (3, 75) = 8.362, p < .001 (see model parameters summarized in Table 1). The coefficients indicated that higher delta scores between test 2 and 3 were predicted by fewer scrolls, lower Wording scores, and performance on test 2 with lower performers on test 2 that used iTELL showing greater gains between test 2 and 3.
In the modern economy, computing skills are increasingly
important for students to acquire effectively.
Understanding computational thinking and computer
programming enables students to solve important,
realworld problems in a number of complex ways across a
number of domains. However, learning computer skills is
difficult and requires sustained efforts, specialized teaching
environments, and diverse skills, making success difficult.
As a result, many computer science curricula have suffered
from high failure and dropout rates [
Our assessment of the iTELL volume was an A/B test where about 25% of the class volunteered to use the iTELL volume (for extra credit) and the remainder of the class used a plain digital version of the text. We used test scores from the previous chapter where all students used the digital textbook as a baseline measure of student skills. We then examined students’ survey data to better understand their experiences with the iTELL volume. Additionally, we compared delta scores calculated as the difference between the scores on the control structures test and scores on the baseline test to assess potential gains for students who used iTELL. Lastly, we ran a regression model to better understand what features explained gains by students who used the iTELL volume.
The survey results indicated that students’ experiences with the AI tools within iTELL were positive. Overall, students felt that the constructed response items and summary tasks were easy to work with and helped them improve their learning. Students also felt the AI feedback was accurate. Student surveys also indicated that the students were satisfied with the iTELL volume overall.
In terms of learning differences between the iTELL and
non-iTELL students, a Mann-Whitney U test approached
significance and reported a meaningful, but small,
relationship between score differences between the
baseline test and the test on the control structures chapter
(see reported effect sizes). A number of students showed
ceiling effects across both tests and removing these
students led to similar results. While a p value can indicate
whether an effect exists, the Cliff’s Delta size shows the
magnitude of the differences between the iTELL and
noniTELL groups (a small but meaningful effect) and is the
main quantitative consideration of the study [
The regression analysis of the iTELL student data indicated that students who showed a greater number of scrolls showed lower delta scores. This may indicate that students who are non-linear readers or students who scroll in smaller increments performed worse. However, much deeper analysis of scrolling needs to be performed to support any meaningful conclusions. Additionally, the regression model indicated that students who scored higher in Wording for their summaries showed lower delta score gains. This may indicate that students who focused on Wording in their summaries at the expense of content may have performed worse. It may also be that students that are better writers gain less from using iTELL then students who are worse writers. This may be supported by the final feature of the regression model which was testing level. The coefficients for testing level indicated that students that performed lower on the baseline test showed greater gains when using iTELL. This indicates that iTELL may work better for low level students, but, again, much more fine-grained testing is needed to support this notion.
Overall, this study finds evidence that intelligent
textbooks are an advanced learning technology that can
use AI to improve student learning outcomes in an
introduction to computing class. This finding builds on
previous studies that have argued that traditional, static
textbooks may be ineffective in computer science
instruction [
While the learning gains are small (see reported effect sizes), it is unlikely that a single text-based intervention would lead to stronger gains in a computer science class. However, in combination with other AI powered tools help computer science students effectively use debuggers and compilers or guide students through complex, multi-step, open-ended problems, intelligent texts may lead to greater gains.
There were a number of limitations to the analyses
conducted. First, we were looking at a convenience sample
and one in which students were rewarded for using iTELL,
so there is likely a self-selection bias [
Overall, though, the study provides a promising first step in understanding how the use of intelligent texts may lead to learning gains for computer science students. Knowing the importance of computer science skills in the modern economy and the difficulty in obtaining those skills, a variety of educational tools will be needed to address potential learning deficits in the computer science classroom. Intelligent texts may prove to be one of those tools.
This material is based upon work supported by the National Science Foundation under Grant 2112532. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.