The “Doer Efect” is the empirical phenomenon observed as a stronger correlational relationship between students who complete more activities and their course learning outcomes compared to those who complete fewer activities or watch fewer videos. In this paper, we extended prior evidence of a “Doer Efect” to investigate how doing more can be related not only to better learning outcomes but also to motivational ones. Specifically, we investigated persistence as the student's willingness to continue working on course activities. We used secondary analyses of data from MOOC that taught Advanced Placement (AP) Introductory Java Programming to high school students using the digital textbook platform RuneStone. Although we failed to identify a doer efect in learning outcomes, our analyses do suggest that completing more activities is related to longer persistence in the course than reading more pages or watching more videos. This efect does not appear to be limited to highly motivated students.
Several studies have provided empirical evidence from laboratory experiments [
Several researchers have demonstrated the “Doer Efect” using many datasets on diferent domains.
Carvalho and colleagues [
Although to date comparisons between doing more and reading/watching more has been limited to
its relationship to learning outcomes, recent work by Asher and colleagues suggests that practice-only
instructions reduced student motivation to pursue the course in the future [
Inspired by the work by Asher and colleagues [
RQ1: Is there a “Doer Efect” in the context of a digital textbook for programming? RQ2: Does “doing” influence the persistence of students in a CS course? RQ3: Does “doing” influence students’ interest in CS?
Our analyses will help build and design better course materials, as well as MOOCs that support student–driven learning. Additionally, studying the relationship between student motivation and the “Doer Efect” is an important problem that will encourage adaptive personalized sequencing of course content to prevent less–confident students from being discouraged by students who do more activities and achieve success.
Several open digital textbook platforms developed and deployed by other researchers allowed us to investigate “Doer Efect” on anonymized and openly available log data of MOOCs that blend reading, video watching and doing activities. Since the days of early evolution in adaptive hypermedia, digital textbooks have been developed for programming, such as ELM-ART [9]. Walker and colleagues [10] considered student needs in the design of digital textbook platforms integrated with interactive content. A group of researchers has explored the implementation of digital textbooks in Computer Science Education by integrating interactive learning activities [11]. With the evolution of programming languages used, such as Python, Ericson and colleagues [12] developed new kinds of content for these programming digital textbooks, such as Parson’s puzzles, and programming exercises that students can work on in addition to reading through the material. Pollari-Malmi and colleagues [13] investigated the value of adding interactive activities and problems to digital textbooks and found an increase in the usage of digital textbooks by students who persisted until the end of the course.
The “Doer Efect” [
Follow-up studies have worked with larger sample sizes (N=1000) over 2 courses and showed that
students solving more problems learn better than students who re-read to prepare for better performance
at the end of the course [
Studies have also analyzed and demonstrated the efect in other datasets and at scale [ 14, 15]. In these
systems, the VitalSource platform ofers digital textbooks on various topics for students to learn online.
The motivating goal of replication research in these works was to design and implement better curricula
in the format of digital textbooks with high student learning outcomes. Furthermore, in follow-up to
the second original work on causal efects of the doer efect [
Often student motivation at the beginning of the course afects the extent to which they engage in
practicing by solving problems in a course. Asher et al. [
Our study used the data from CSAwesome digital textbook, comprised of approximately 16 content units (see Figure 1). Additional units focus on mock exams to prepare students for the Advanced Placement (AP) level exam on JAVA programming. Students took the exam after completing this course. As students progress through the course, they read texts, watch videos, and work on activities (see Figure 1). The system records all student interactions in a DataShop–specific format [ 16]. In each iteration of the course, the students start by responding to a survey on their confidence in learning JAVA and whether they will pursue a career in computer science, as shown in Figure 21.
The current study included data from 1,060 students who interacted with the CSAwesome digital textbook in the year 2018 - 2019. The students worked on 2 kinds of activities – Programming Activities and Content Activities. Programming activities included Parson’s problems, CodeLens, and ActivityCode [17]. In Parson’s problems, students moved and rearranged blocks of code to generate the correct solution. In CodeLens, students debugged and analyzed the code. In ActiveCode, students wrote code to solve a problem. Content Activities included Multiple Choice Questions (MCQs) and Short Answer Questions (SAQs).
In addition to these activities, students could watch YouTube videos using links embedded in the course material. The average time spent by students watching these videos was 25.65 minutes (SD = 20.73). Students could refer to the reading material while working with activities. The current dataset is a log of time-stamped student interactions with the digital textbook system. The log records for reading activities include open and scroll events; video interactions include click and pause events; activity interactions encompass short answer responses for SAQs, choice selections for MCQs, solution success or failure along with error feedback; notably for ActiveCode, Parson’s Problems, and CodeLens.
Preprocessing As a first step, we filtered students’ interactions that included pre-test scores. For both pre-and post-tests, we considered the students’ first attempts to calculate the scores. We aggregate Multiple sessions of trace log. We winsorized the outliers 2. The final usable version of the dataset had an efective sample size of 694. The logs contained page views, activity completions, and video-watching events. We then sorted the columns using student IDs and timestamps in that order. After sorting the data by student IDs and timestamps, we took the diference between the timestamp of the current row event and the timestamp of the row for the previous row event, which provided us with the duration of the activity or action in each given row. To address our research questions, we filtered the dataset in 3 ways. 694 students completed the pre-test. Ninety-two students completed both the pre-test and the post-test.
Pre- and Post-tests We collected the first attempts of all students who completed the respective tests. We measured the reliability of the pre- and post-test questions using the Cronbach’s , a metric that estimates using a missing data imputation method. Students’ responses to the items in the pre- and post-tests is used to evaluate the reliability. (Cronbach’s = 0.87, = 0.81 respectively).
Resource Use To calculate the total time spent in each resource, we summed the time for all rows for each student in each resource (i.e., text, activities, or videos). To eliminate outliers (possible situations where a student might have started an activity and left their browser open while away from the computer), we replaced the bottom 5% of event durations with the 3rd quantile (75%) value of the distribution of all values of event durations. In addition to calculating the total time spent on each resource, we also calculated the number of pages read, videos watched, and activities completed by each student. We labeled these as “Page Read”, “Video Watched” and “Activity Done” counts. All counts represent repeated increments as present in the student interaction of a given page in the data.
Persistence For us, students’ persistence is a measure of how far a student progresses through the course, that is, the furthest page up to which a student reads, watches videos, or solves problems / activities from the start of the course. As observed in the data and reported in Table 1, it appears that students continue with reading, watching, or doing up to 75.7 pages on average.
Course Engagement Course Engagement is the number of unique activities done / videos watched
/ content read by a student up to the page the student progressed through the course. We calculated this measure by taking the ratios between the number of page read, video watched, and activity done counts, respectively, and dividing them by the maximum of pages, videos and activities that the student could complete up to the page they reach the furthest from the start.
Self-reports of Interest and Students’ Instruction Finally, using the embedded questionnaires, we calculated for each student their career interests and confidence. We summarized the measurements used for testing the hypotheses in Table 1.
Amount of Doing We defined one categorical variable for students who read more pages, watch
more videos, and engage in more activities. Many students read, fewer watched, and even fewer did
activities (see Figure 3). Following the first paper on this topic [
To address our 3 research questions, we used multiple linear regression analyses. For all regression analyses, we normalized all continuous predictors and outcomes by z-scoring the raw values, allowing us to compare regression estimates as efect sizes.
To answer RQ1, we evaluated the “Doer Efect” in the data using the outcome and predictors as described in the original work. We used multiple linear regression analyses with page, video, and activity counts / times as the predictors. The predicted outcome was students’ post-test scores while controlling for their pre-test scores.
To answer RQ2, we studied the relationship between student motivation and doing more activities in a course. We used the standardized resource use and baseline student confidence to learn JAVA (see Table 1) as predictors. The results would help demonstrate that student motivation is associated with “Doer Efect” .
To answer RQ3, we finally analyzed whether students who did more were interested at the end of the course in pursuing a career in computer science (CS) when controlling for their interest to pursue a career in CS at the beginning of the course. We used regression analysis to predict their career interest at the end of the course. We used the baseline confidence to learn JAVA and its interaction with the baseline career interest as predictors.
When using activity, video, and page read times as predictors and post-test scores as the outcome while controlling for the pre-test as a covariate, we could not demonstrate the “Doer Efect” . Similarly, we could not demonstrate the “Doer Efect” with the page, video, and activity counts as predictors of post-test scores. The coeficient for page counts had a small significant negative efect on the post-test scores when controlling for pre-test scores (see Table 2).
We found that both reading a larger percentage of the available pages and completing a larger percentage of the available activities were related to achieving more of the course, that is, persisting longer, controlling for baseline confidence (see Table 3). Notably, the relation between persistence and completing more of the available activities was 2.3 times stronger than the relation between persistence and completing more of the available reading pages, suggesting a “Doer Efect” .
We found that there was a positive relationship between higher baseline confidence in JAVA and posttest interest in following a CS career ( = − 0.76, = 0.12, < .0001), which is not surprising, as Independent Variable Pretest Score Activity Time (in s) Video Time (in s) Read Time (in s) Pretest Score Activity Counts Video Counts Read Counts more confident students are more likely to have more prior knowledge in programming. We found no relationship between the type of interaction the students had with the course and their final interest in pursuing a CS career or any interaction (see Table 4).
Independent Variable Career Interest Confidence to Learn JAVA Confidence × Career
Estimate -0.76 0.03 -0.09
SE 0.12 0.26 0.13
Although the “Doer Efect” has been identified in many courses and datasets before, we failed to identify it in the current paper when predicting learning outcomes. There are multiple possible reasons for this. The number of students contributing data to our analyses is likely too small to identify the efect. Prior work has used larger sample sizes. It is also possible that the use of the text and the video was substantially diferent in the current sample than in previous studies. Regardless, the numeric direction of the data is consistent with prior doer efect results.
Notably, the results presented here extend existing evidence of a doer efect to examine motivational
outcomes rather than learning outcomes. Motivated by previous work [
Despite this greater engagement among students who completed more activities, our third and final analysis found that the type of interaction that students had with the course or their initial confidence in learning JAVA did not afect their interest in pursuing a career in CS at the end of the course. It is not the case that more confident students are more likely to do more or more likely to want to pursue a career. Therefore, the relationships observed between doing more and engagement are unlikely to be limited to highly motivated students, although further research is needed.
Researchers have extensively studied students’ persistence in MOOCs [
Thanks to Brad Miller, DataShop @ CMU team for the dataset. Annie Wang, Ryan Nuqui for LearnSphere integration. This work was supported by the National Science Foundation under Grant Nos. 2418655 and 2418656, to Peter Brusilovsky and Paulo F. Carvalho.
We did not use Generative AI for this work. Grammar correction was done with human interventions. [9] P. Brusilovsky, E. Schwarz, G. Weber, Electronic textbooks on WWW: from static hypertext to interactivity and adaptivity, Educational Technology Publications, Englewood Clifs, New Jersey, 1997, pp. 255–261. [10] E. Walker, R. Wylie, A. Danielescu, J. P. Rodriguez III, E. Finn, Balancing student needs and learning theory in a social interactive postdigital textbook, in: End-user considerations in educational technology design, IGI Global, 2018, pp. 141–159. [11] G. Rößling, T. Naps, M. S. Hall, V. Karavirta, A. Kerren, C. Leska, A. Moreno, R. Oechsle, S. H.
Rodger, J. Urquiza-Fuentes, et al., Merging interactive visualizations with hypertextbooks and course management, in: Working group reports on ITiCSE on Innovation and technology in computer science education, 2006, pp. 166–181. [12] B. J. Ericson, I. YeckehZaare, M. J. Guzdial, Runestone interactive ebooks: A research platform for online computer science learning, in: Proceedings of SPLICE Workshop in the International Computing Education Research Conference (ICER’19), 2019, pp. 1–7. [13] K. Pollari-Malmi, J. Guerra, P. Brusilovsky, L. Malmi, T. Sirkiä, On the value of using an interactive electronic textbook in an introductory programming course, in: Proceedings of the 17th Koli Calling International Conference on Computing Education Research, 2017, pp. 168–172. [14] R. Van Campenhout, B. G. Johnson, J. A. Olsen, The doer efect: Replicating findings that doing causes learning, in: Proceedings of eLmL2021, The Thirteenth International Conference on Mobile, Hybrid, and On-line Learning, 2021, pp. 1–6. [15] R. Van Campenhout, B. Jerome, J. S. Dittel, B. G. Johnson, The doer efect at scale: investigating correlation and causation across seven courses, in: LAK23: 13th International Learning Analytics and Knowledge Conference, 2023, pp. 357–365. [16] K. R. Koedinger, J. Stamper, P. F. Carvalho, P. Pavlik, L. Eglington, Sharing and reusing data and analytic methods with learnsphere, in: Proceedings of the 10th International Conference on Educational Data Mining (EDM) 2017, volume 2, 2019, pp. 501–520. [17] B. J. Ericson, M. J. Guzdial, B. B. Morrison, Analysis of interactive features designed to enhance learning in an ebook, in: Proceedings of the eleventh annual international conference on international computing education research, 2015, pp. 169–178. [18] B. J. Evans, R. B. Baker, T. S. Dee, Persistence patterns in massive open online courses (MOOCs), in: Journal of Higher Education, volume 87, Taylor & Francis, 2016, pp. 206–242.