Student course reviews are rarely considered as research instruments, yet their ubiquity makes them promising tools for education data science. To illustrate this potential, we use a corpus of student reviews to observe gender differences in how students appraise their own learning and in the advice they give to to future students. We find systematic differences in who submits course reviews, with female and academically high-achieving students more likely to submit. Among submitters, we find (a) females understate their achievement of learning goals relative to males earning the same grades; (b) females offer lengthier written advice to future students than males; (c) advice written by females exhibits more positive tone, even after accounting for grades and course selections.
While a variety of student characteristics of are of interest to
education data scientists, we focus on students’ gender for reasons both
practical and theoretical. For privacy purposes, our case university
has currently granted researcher access to only a few variables
describing review submitters; we utilize those available data here. Yet
we also have two theoretical motivations for focusing on gender.
First, borrowing from social psychology, we recognize that women
tend to under-estimate their own abilities, while men to to
overestimate, conditional on measured accomplishment [
Research on course reviews typically has focused on questions of
their value as an instruments for evaluating the quality of instruction.
Analyses conducted at scale typically focus on whether measures of
learning are correlated with instructional quality [
One potential concern about course reviews from the
pscyhometrics liaterature is the potential for differential item function, a
phenomenon in which respondents of equal ability will exhibit different
responses to a given survey item or question. Studies of differential
item function in course reviews have focused on the quantitative
difficulty of a class, or characteristics of the instructor [
Under experimental conditions where researchers manipulated the
information content and valence of course reviews, researchers
found that these factors had material effects on course enrollment
decisions. Students were more likely to enroll in courses if course
evaluations had positive valence, particularly if there was a large
number of such evaluations [
There is a burgeoning literature on using computational text analysis
and methods to quantify differences in corpora based on
characteristics of the author and the text. These techniques have been quickly
adopted to educational applications but we have seen relatively few
instances of text analysis of course reviews. When text analysis of
course evaluations are done, they are typically focused on keyword
extraction and on predicting Likert item responses as a function of
the text [
We group text analyses methods into the following three categories:
Dictionary-based approaches characterize the words of documents
into groups of predefined categories such as sentiment. The most
popular of these dictionaries is the Linguistic Inquiry and Word
Count (LIWC) dictionary [
Researchers can also create their own custom measures of text. We take advantage of this affordance by capturing mentions of instructors’ names.
Token-based approaches treat every word in a text as input into a
model. These approaches often result in the loss of syntactic
meaning but are often very effective at classifying documents.
Tokenbased approaches have proven effective at detecting socioeconomic
features of authors such as race, gender, and income in college
application essays [
The basic premise behind unsupervised approaches is that texts
include multiple topics, and topics comprise words. Using
unsupervised methods such as Latent Dirichelet Allocation (LDA) , we
can group texts categorically. These same methods have been
augmented recently to allow the distribution of topics to co-vary with
other relevant metadata, a technique known as structural topic
modeling [
Our work has three motivations from prior social-science literature
on higher education and gender. The first is that male and female
students may have different experiences when taking the same courses.
For example, women are less comfortable asking questions and have
less confidence in CS courses than their male peers [
Second, gendered differences in skill perception may influence
how students report their experiences and learning gains in reviews.
While women tend to approach STEM fields with less confidence,
men tend to over-estimate their abilities. Experimental work by
Correll [
Our working hypothesis is that course reviews will exhibit gendered patterns of academic experience, self-perceptions and advice-giving. Specifically: (1) reviews from male students will exhibit stronger professed strong learning gains (2) reviews from female students will exhibit characteristics of care work.
We group our analyses into two parts. The first part examines variation by gender and earned grades on review submission rates and on Likert-scale items on course reviews. The second part examines variation in male and female responses to a qualitative review prompt eliciting advice for future students considering the same courses.
H1: Female students will respond to course evaluations more often than males.
Our care work hypothesis is that female students will be more responsive to institutional requests for reviews. We investigate this hypotheses using an exact binomial-two-sample test. We examine results by gender and grade.
H2: There are systematic differences in response rate by grade. There are many competing theories of how grades might influence response rates to course evaluations. If students have a poor grade, they may be more inclined to view the evaluation as an opportunity to retaliate against the grader. Alternatively, students who receive a low grade may opt to avoid opportunities to reflect on negative experiences. We will investigate this hypothesis utilizing a simple c2 test of response-rates by grade.
H3: Female students will understate their achievements relative to their male counterparts. We hypothesize that men are more likely to see course reviews as a form of positive self-reflection and promotion, and that females are more likely reviews as a form of care work. We believe these differences will have stronger valence in items that focus on a students’ accomplishments rather than other constructs such as student learning. We will model these analyses as a fixed-effect regression model with the following specification: Yi j = b1Malei + Gradesi j + G j + ei j (1) The subscripts i and j correspond to indices for student and course. The Y variable corresponds to our focal outcome variable, in this case, responses to a Likert item. Male corresponds to a student’s self-reported indicator variable of whether the student identifies as male and b1 corresponds to the associated coefficient with this variable. We represent course effects with Gi to control for factors like the difficulty of the course or instructional quality. We also control for grades with an additional fixed-effect for each possible grade a student could receive 2. The error is represented by e. Errors are clustered at the course level.
We pay particular interest to open-response items in course reviews. We suspect that such items are may be the most valuable and least explored element of course reviews. As such, we may be able to detect subtle differences in qualitative responses.
H4: Course evaluations written by females will express more positive and sincere sentiment.
Given our care work hypothesis, we believe that female students will express more positive sentiment in open-response items. We use the same analytical strategy as an equation 1 using LIWC’s tone variable. Specifically, we examine gender differences in these psycho-social variables after controlling for variation that can can be attributed to the course, or to student grade. We report outcomes in standardized effect sizes to facilitate interpretability. We also hypothesize that a corollary to the care work hypothesis is that female students will use more "I" statements and tentative language. This tendency would manifest as reviews written by female students exhibiting more authentic language.
H5: Course evaluations written by male students will express more clout. Based on prior literature pertaining to a confidence gap in CS by gender, we hypothesize this trend should manifest with less expressions of clout and authority in course evaluations by female authors.
H6: Female students will write more on course evaluations and mention the instructor more often.
We hypothesize that care work will manifest in other ways beyond psycho-social variables. Specifically: female submitters will put more effort into reviews by writing more; and they will take a more individualized approach by mentioning the instructor explicitly. 2in our analyses, there are over twenty grade types, including + and - variants as well as credit and nocredit courses. We report A,B,C,D, and not passing grades for simplicity We have crafted two simple measures to facilitate investigation of this hypothesis: the length of each response in number of words, and a capture of each instance of an instructor name.
H7: There will by systematic variation in topics depending on the author’s gender.
Our final analysis is exploratory using structural topic models to identify whether qualitative components of the corpora systematically vary with gender of submitter The goals of this analysis are to develop efficient means of sorting and categorizing qualitative components of course reviews.
Data comprise information describing enrollments in courses offered through the Computer Science (CS) Department of a private research university during the 2015-16 and 2016-17 academic years, and the entire population of formal reviews submitted by students enrolled in those courses. Reviews were administered near the end of the academic term but before the beginning of the term’s official final exam period. As an incentive for submitting reviews, students were given the ability to see their final course grades a bit earlier than non-submitters.
In total these data yield 11,255 student responses from 251 courses. Courses range in character from very large introductory lecture-andlab formats to small advanced seminars. Institutional data made available to us for analysis include each student’s grade, gender, GPA, declared major (if known), and academic year. We combine these data with the corpus of reviews submitted for CS courses during the study period specified above. Approximately one-third of submitted reviews from female students, and approximately half are from undergraduates. We cannot track or identify students enrolling in multiple CS courses during the study period, however we can compute and generate response rates by grade and gender. We limit our analysis to responses in which submitters offered a response to the review’s only open-ended question. That question reads: "What would you like to say about this course to a student who is considering taking it in the future?" The prompt is very well aligned with our care work hypotheses, in that it specifically asks submitters to give advice to a hypothetical future student. Individual responses vary substantially in length: from a single character to over 5,964 characters (the latter equivalent to 1004 words). The mean response length is 132 characters – approximately the length of a tweet. The entire corpus of responses to this question is 300,000 words.
Additionally we analyze responses to two review prompts with five-point Likert responses: 3
How much did you learn from this course?
How well did you achieve the learning goals of this course? 3we will limit this analysis to complete cases due to the fact that one item was not consistently administered across courses. We ignore questions pertaining to quality of instruction and focus on student learning goals. Aggregated responses to these two prompts appear in Figure 1.
Rates of review submission by student gender and earned grade are reported in Figure 2. Two features are notable. First, females are more likely to submit overall. On average, females submit to 78.0% of opportunities to do so; males, 74.5% (p<.001).
Second, those receiving higher grades in a course are more likely to submit reviews Females receiving a grade of "A" are 3.7% more likely to respond to submit than their male counterparts. The gender submission gap is greatest among students receiving a grade of "B," with female "B" recipients 6.5% (p<.001)more likely to submit than males. We do not observe statistically significant gender differences in submission rates for those receiving grades below "B," however such grades represent fewer than 5% of grades in the research sample.
We also examined whether review submission varied systematically grades. Figure 2 indicates a strong positive correlation between grade and likelihood of submission. A student with a grade of A or higher has an 80% chance of responding to the evaluation, while students who do not pass the course or receive credit without a grade responded approximately 50 percent of the time. Effectively, this means that students who fail courses are represented by review submissions about half as often as students who excel in a courses. Differences are statistically significant with a c2 statistic of 395.37 and a p-value of less than 0.001.
We observe the proportion of students reporting having achieved the learning goals of a course Extremely Well by grade in Figure 3. Not surprisingly, we find a strong direct correlation with course grade, such that reported goal achievement declines with grade. What is striking is that at every grade level, there is a clear gap in reported goal achievement, with males more likely to report achievement than females earning the same grade.
We extend this analysis to see if this same pattern occurs with the question of how much students learn. Using the same specification as described in equation 1 in table 1. We see that after controlling for grades and course, males and females exhibit no differences
Male in self-reported measures of ‘how much they learned’ in a class. However, when we look at a similar question about ‘achievement of learning goals’, we see a stark difference. Male students are 8% points more likely than females to state they mastered the learning goals of a course. This finding suggests two concerns. First, given the similarity of these questions, we see that subtle differences in phrasing yield substantial differences in student responses. Second, female students report lower-level of mastery even after controlling for grades. Notably, these surveys are collected before students know their final grades. These perceptions may change after this information is revealed to them. 5.4 5.4.1 graphicx
We report our analyses for H4 and H5 in table 2. We find modest variation by gender in how submitters describe their experience in the same course, conditional on grades. On average, submissions from males evince slightly more negative and slightly less authentic language. While these gender differences are highly significant, their magnitude is modest: on the order of a tenth of a standard deviation. Nevertheless, they are consistent with our care work hypotheses. To wit, men are somewhat more critical and less honest in their reviews than women, suggesting greater empathy and investment among female submitters.
With respect to our hypothesis around clout, we find little evidence that qualitative open-responses exhibit any significant differences in
We report the standardized results of our analysis in table 3. We observe marginally significant differences in the frequency with which submissions from males and females mention instructor name, with women approximately two percent more likely to mention. Submissions from women are also lengthier – about .15 of a standard deviation. While modest in magnitude, these statistically significant findings comport with our care work hypotheses that female submitters approach the task of submitting reviews with more attention to specificity and investment.
We ran structural topic models while allowing submitter gender
to vary with topic prevalence. We tuned the optimal number of
topics from 2 to 50 using an exclusivity measure called FREX
[
F REX = (
+ w E 1 w ) 1
F (2) Using this criterion, We found the locally optimal parameter to be 32 distinct topics. We then hand labelled each topic, observing the ten (10) statements that had the highest probability labels of that topic see Figure 4 (Top). The most common topics were comments about a course being a tutorial, a suggestion to take the course, or a positive review. The least common topics were highly specific suggestions and issues pertaining to course prerequisites. While we did not see substantial gender variation in topics overall, there are exceptions of note. First, submissions from males are more likely to talk about math prerequisites and claims that instruction was poorly organized or of poor quality. They were also more likely to discuss course organization and instruction. Submissions from females were more likely to bear topics pertaining to workload, study practices, and attendance. These patterns provide at least modest evidence that women are offering relatively more specific advice that may be relevant to larger numbers of future students.
We note an important caveat to this analysis, however. In contrast with the above studies results of the topic models presented in this section do not control for grades or course selections, thus reported gender differences in topic prevalence may be an artifact of these other factors. We attempted to model the data with all of these parameters but found the models to be degenerate.
Even while they are controversial for evaluating instructors and instruction, course reviews are ubiquitous features of the US higher education landscape and potentially powerful tools for education data science. In the work presented here we have sought to demonstrate the promise of course reviews as a window into students’ perceptions of their academic experiences and their orientation to the task of submitting evaluations. Taking advantage of archival data that included 11,255 submitted to 251 computer science courses at a single university between 2015-2017 that was linked to administrative information describing submitters’ gender (M/F) and grades, we found patterned variation in who submits course reviews, and how.
In three observational studies we found that (a) women and those earning high grades were disproportionately likely to submit reviews (b) the phrasing of close-ended review prompts influenced patterns of response by gender (c) responses to qualitative review prompts differed subtly but significantly by gender, with women writing somewhat more positive, individualized, and lengthier reviews. These empirical findings comport with theoretical insights from educational social psychology and feminist social science, which suggest gender variation in how men and women perceive their own academic accomplishments and their obligations for the well-being of others.
While the empirical findings presented here are modest, they suggest the promise of leveraging course reviews for cumulative science in at least two ways.
First, we note that the inquiries presented here are based entirely on the premise that course reviews and submitter demographic information are "found" data. To the extent that virtually every US college and university possesses data such as these, we can only imagine the number and variety of insights that might be gained from parallel investigations at other schools. To a nascent field whose promise lies substantially in observing phenomena at scale, course reviews provide exceptionally promising sources of data for education data science.
Second, there is every reason to imagine that education data scientists might collaborate with school administrators to more explicitly and conscientiously instrument reviews for systematic experimental and quasi-experimental research. The basic conditions for such inquiries are already in place and sustained by established administrative rhythms: schools have offices conducting the reviews, students anticipate receiving them, and they take place multiple times a year. It is possible to imagine substantial scientific insight through the linkage review subsmissions with with administrative data describing characteristics of submitters. The initial efforts presented here provide an inkling of this promise.
As with any novel research strategy, pursuing education data science
through course reviews comes with important ethical considerations
regarding participant consent and responsible use. We are grateful
that such discussions are already well underway nationwide [