Using 10 years of grade data from a university computer science department we t a multi-level proportional odds model and nd that students earn a higher grade in an afternoon class at 1.15 times the odds for a morning class, even when controlling for GPA. This nding has implications both for student learning and for experimental studies that compare classes without considering the time of day at which they are taught. We nd that there are no signi cant trends for student performance based on term when looking at the department as a whole, though there are such trends for certain courses in particular.
Multi-levels models have been used in computer science
education, for example, [
As researchers and educators we understand the reasons that larger studies are not undertaken more often: even planning an educational intervention experiment with one experimental and one control section can be very resource intensive! In many situations, especially when rst piloting new eduCopyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). cational techniques, it is completely impractical to expect that researchers will be able to experiment on more than a single section of a course.
Unfortunately, experimenting with one or only a few sections of a course requires the researcher to make the assumption that essentially all things are equal about the students taking the courses and the courses themselves, apart from the intervention.
Despite controlling for as many factors as possible, such as instructor, course assignments, tests, and more, there are still often factors that lie outside the researcher's control, such as the day of the week, time of day, term, and location that their course is scheduled for. Furthermore, students self-select into which section of the course that they want to take! These variations between sections may be introducing a selection bias threatening the validity of these educational experiments.
In this year's SIGCSE Technical Symposium alone, there
were 7 studies which tested new educational practice through
experimenting with either one or just a few experimental
and control sections of the same course, operating either
explicitly or implicitly under the \all things equal" assumption
[
It is has been shown that adolescents struggle to perform
to their fullest potential early in the morning, causing many
school districts to push back school start times [
Marbouti et. al. analyzed data from 15 di erent sections
of an introductory university engineering course and found
that due to lower attendance in morning sections, the early
morning sections of the course signi cantly under-performed
other sections [
Most literature agrees that courses o ered in condensed terms
(such as most universities' summer terms) lead to the
students learning the material equally well or even better than
courses that are taught over a full length term [
Our grade and course scheduling data was acquired from the registrar at the University of Illinois at Urbana-Champaign. Because our primary focus is the relationship between course scheduling and time of day, we removed topics and reading courses that were only taught once, as well as courses that are not scheduled, such as independent study and senior thesis courses. Summer courses were also removed from the data set, to avoid comparing versions of the same course which were taught on an entirely di erent time scale, and sometimes even with a di erent set of instructor expectations.
Drops and withdrawals were also removed from the data set. After cleaning the data, we were left with 72,739 student grades from 24,705 students across 1,938 sections of 101 courses. The grade data consists of letter grades, which we converted to grade points for the purposes of tting the model (A ! 4.0, A- ! 3.67, B+ ! 3.33, etc.). The overall mean grade in the data set is 3.100, and the median is 3.33 (B+).
At the University of Illinois, the Fall term starts in late August and ends in mid-December, and the Spring term starts in mid-January and ends in mid-May. We chose from the beginning to treat time of day as a categorical variable, where courses beginning before 10:00 a.m. were considered \Morning," courses starting between 10:00 a.m. and 2:00 p.m. were considered \Midday," courses starting between 2:00 p.m. and 5:00 p.m. were considered \Afternoon," and courses which started after 5:00 p.m. were considered \Evening" courses. y c n e u q reF 000 0 1 0 0 0 0 2 0 0 0 5 1 0 0 0 5 0 0
F
D−
D 1
D+
C−
C+
B− 4 taught on the same days of the week every time they were taught, and another 30 were only taught on 2 di erent day con gurations (e.g. a class was taught either Monday and Wednesday or Tuesday and Thursday, but not in any other day con gurations). Because of this, we chose to leave day of the week considerations out of our analysis entirely.
We t the data using a three level model of the following form, where students are indexed by i, sections are indexed by j, and coursed are indexed by k, and y represents some grade (e.g. A, A-, B, etc.):
A look at the data set shows that of the 101 courses o ered
in the computer science department, 54 of them were always
Middayjk, Afternoonjk, and Eveningjk are dummy codes
denoting the time of day a course was held, and all of them
0
0
4
0
0
2
0
0
1
0
0
3
5
2
0
2
0
1
5
0
being 0 represents a Morning class. Our model assumes that
the error term on the section level, Ujk, and the error term at
the course level, Wk, are multivariate normal distributions
which are independent of one another. After substituting
and gathering the error terms, we obtain the mixed model:
ln
We used the R (version 3.6.3) package brms [
We also t a version of the model including a dummy code for term (Fall vs. Spring) and found that there was no signi cant general trend for the relationship between term and course. However, tting similar models for some individual courses revealed that some courses do have signi cant di erences in performance between semesters, with some courses having better performance in the Fall and some having better performance in the Spring.
Finally, we t a version of the model including a dummy code for whether or not the section was held in the computer science department building, with the hypothesis that sections held in the computer science department building would be more desirable and would thus ll up with more responsible students who registered on time. We found no signi cant relationship between student performance and which building the course was held in.
The estimated parameters of the nal model are listed in Table 1. This model allows us to estimate the relative probability that a student will receive each letter grade, given their cumulative GPA, the course and section of the course, and the time of day that the course was scheduled. The probability of a higher grade increases for higher values of GPA. The later in day, the probability of a higher grade increases. According to the model, the odds that a student receives a higher grade in an afternoon class (based on model t information from Table 1) are e0:14 = 1:15 times the odds for a student with the same GPA taking the same class in the morning. Additionally, holding all other variable constant, the odds of a higher grade in an evening class are e0:17 = 1:19 times the odds in a morning class.
To allow an interpretation of the e ect size in grade units rather than only as probabilities, we used the model to simulate what the average grade over all data points would be if all courses in the department were o ered at the same time of day. The results, shown in Table 2, show that students perform 0.04 and 0.05 grade points better in afternoon and evening classes, respectively, than they do in morning classes. Figure 4 helps us to visualize that student performance is actually monotonically increasing throughout the Estimate
Est. Error
Credible Interval
Credible Interval
day, with the worst performance in morning classes, and the best performance in evening classes.
It is also important to note that there is a large variance in grades between sections and courses, so in addition to the general trends, it appears there is often large variation between any two sections of a given course. In Figure 5, we visualize the relative amount of uncertainty at the section (Ujk) and course (Wk) levels. It appears that much more variance in course performance comes from the course rather than the section level, but there is still a signi cant amount of unexplained variance between sections in our model.
When planning experiments on multiple sections of the same course, researchers should be aware of the di erences between sections that may have an in uence on the student's grades independent of the instructional techniques used, and plan accordingly. If they must, for some reason or another, conduct an educational experiment between sections that are taught at di erent times of day, or under other di ering circumstances, they should be aware of the typical variance in grades that can be brought on by such circumstances, and ensure that the e ect size of the intervention they are trying to study is signi cantly larger. Alternatively, if one section is expected to have a higher grade due to documented reasons (i.e. being in the afternoon vs. in the morning), they could use the expected-to-be better performing section as the control, and the expected-to-be worse section as the experimental group, counting on the intervention to have a large enough e ect to overcome the small negative impact of scheduling.
Despite what we do know about trends in student performance based on scheduling, it is critical to remember that all the above statistics only show general trends, and can not tell us about the relationship between any particular two course or section instances. Researchers should do all they can to ensure \all things equal" between their experimental and control groups, and should document all the information that they can about their course sections in the interest of good science, i.e. interpretability and reproducibility of their work. They should also be aware of and document the performance trends of the course they are experimenting with in particular, as some courses have much larger di erences term-to-term or based on time of day than others do.
As we have discussed, our study was limited by the data we
were able to receive from the registrar at the University of
Illinois at Urbana-Champaign, and the way that the
computer science department decided to schedule the courses,
making it impossible for us to draw any conclusions about
day-of-week e ects on scheduling. We also did not have
access to attendance data, making it impossible to verify if
morning classes performed more poorly for the same reason
as in [
Additionally, our results should not be interpreted to mean that a particular student will earn higher grades if they register for afternoon classes instead of morning, because we are using observational data where students have self-selected into courses, leading to selection bias. Our study is unable to make any statement about why student performance varies by time of day, but a great area of future work would be to investigate why these performance di erences exist, and what types of interventions may be able to help mitigate them.
We nd that in the computer science department, the odds of a student receiving a higher grade in an afternoon class is 1.15 times the odds of a student with the same GPA in a morning class earning a higher grade. According to simulations run using our model, this di erence amounts to an average grade di erence of 0.04 grade points between morning and afternoon classes. There is also a large unexplained variance in grades between sections of the same course. Based on these ndings and prior work in this area, we assert that the course scheduling information is an important piece of data which should be included in studies that make comparisons between treatments on di erent course sections. Based on our data set, we were not able to investigate trends in student performance based on which days of the week courses were scheduled for, and we found no overall trends for the term a course was o ered in, or for the classroom building in which it was o ered. Replication of our work, as well as work to answer the research questions which were unable to answer given our data set, would be great future contributions to the literature.
We would like to thanks the computer science education research group at the University of Illinois at Urbana-Champaign for useful feedback and suggested references for this work.