The purpose of this paper is to use a weighted log-odds-ratio, informative Dirichlet prior method (“bag of words” approach) to analyze student comments and scores posed to MyReviewers, a web-based tool designed for collecting student writing as well as their peers' comments and scores of their colleagues' drafts. Our preliminary findings suggest that students who receive lower scores may also be receiving significantly different kinds of feedback that some in the field of writing studies have suggested may have a negative impact on student learning and motivation. Findings point to the possibility of identifying the the effectiveness of different kinds of feedback on lower and higher performing student writers; evaluating the impact of feedback on student revision and grading practices; and identifying and analyzing symmetries and asymmetries in teacher and student feedback commentary and scores.
Asking students to give writing feedback to their peers is a common
practice and may be introduced to students as early as kindergarten
and continue into graduate school based on the understanding that
evaluating and sharing feedback with peers may accelerate the
learning of the writer and peer reviewer alike. However, while
much attention has been paid to the quality and nature of feedback
given to student writers by teachers and writing tutors, very little
exploration has been done of the impact of the feedback given by
students to their peers’ work
This paper represents a preliminary investigation of data gathered as part of a larger, cross-institutional study of peer review of writing assignments in undergraduate courses. Employing the web-based tool MyReviewers we collected collected student writing, commentary, and scores from one semester of a first-year writingin-the-disciplines program at the University of Pennsylvania. We used a simple “bag of words” approach to explore whether the type of comments students receive correlates with the score they give to the students’ assignment. Results suggest that students in the lower quartile receive significantly different commentary--more prescriptive and negative--from that given to students in the upper quartile. This has considerable implications, for it suggests that good writers receive the kind of positive reinforcement from peers that many in the field of writing studies consider most effective for advancing writing skills, while underperforming writers receive the sort of commentary from peers that are generally regarded as the sort that hinders development of writing skills.
Writing feedback is typically divided into two categories: direct
and indirect instruction. Direct instruction includes telling,
suggesting, explaining, and exemplifying (Mackiewicz &
Thompson) and is often contrasted with open problem solving or
discovery learning
In the context of writing center tutoring, Mackiewicz and
Thompson (2015) suggest that motivation plays a key role in the
amount of time and effort that students devote to writing
tasks. They explain the importance of motivational scaffolding
strategies in encouraging students through “praise, assurances of
caring, and statements reinforcing student writers’ ownership of
their work”
Though students may like to receive positive feedback, including
praise, research highlights the complexity of feedback in terms of
circumstance and effect. Straub (1997) surveyed 142 first-year
college writing students to investigate their perceptions about
teacher comments on their writing and found that students preferred
praise, even when it was merely in the form of the word “good”
adjacent to criticism. However, students most preferred praise that
was accompanied by reasons for the positive evaluation
Negative feedback, which is equally as complicated as positive
feedback, may either hinder or bolster learning, depending on the
student and context. High self-efficacy learners view their
performance optimistically, and therefore, may seek negative
feedback to outperform on tasks
The Critical Writing Program has over the past decade developed and refined a genre-, discourse- and discipline-based shared curriculum for introducing students to what is called "authentic" writing situations that involve real genres, audiences, motives, and subject knowledge, as well as introduce and provide students with practice in using a shared vocabulary and concepts about writing, from knowledge domains such as genre and process to more specific aspects of rhetorical analysis and production. The curriculum emphasizes peer review and reflection throughout. The data set consists of the work of 1,183 undergraduates, mostly first-year, who completed a writing seminar at the University of Pennsylvania in Spring 2016. This data set includes up to five drafts of a literature review as well as the peer reviews each draft received. Peer reviews consist of rubric-guided comments and numeric scores. Most drafts are accompanied by the student writer’s pre- or post-outline that provides a rhetorical analysis as well as line of reasoning for the draft. All drafts receive from one to six peer reviews (comments and scores) as well as comments and scores from their instructor. In addition, most students will provide a revision plan that responds to the feedback they have received from their peers and instructor.
All undergraduate students at Penn across the four undergraduate schools--College of Arts & Sciences, Wharton, Engineering, and Nursing--are required to complete a writing seminar. Penn’s writing seminars are administered by the Critical Writing Program; most seminars within the program are situated within a specific discipline, bounded by a particular, discipline-based inquiry, and taught by a Ph.D. in that discipline who frequently is engaged in the line of inquiry focused upon in the seminar. Thus the topics and disciplines vary and along these lines while all students are assigned the genre of the literature review, we recognize that literature reviews written in the bench sciences are often substantially different form those written in, say, the humanities. Some instructors teach two sections of the same topic. In addition to discipline-based seminars, we have two sets of seminars that share a single topic across disciplines: Craft of Prose (14 sections representing 161 students) and Upper Division seminars (8 sections representing 104 students). Students, sometimes with the help of their advisors, choose the type of seminar that best suits their needs, including their self-assessment of their competence and confidence as writers. Craft of Prose seminars are designed for students who may have less preparation in writing or have confidence issues or other concerns about writing. Upper Division seminars are designed for upperclassmen and transfer students who were uncomfortable in seminars designed mainly for freshmen. We also have single topic Global English classes for international students who are concerned with learning how to write for an American academic audience. For more information on our directed self-placement criteria, visit: http://writing.upenn.edu/critical/seminars/choosing_the_right_sem inar.php The specific data in this set includes peer review scores and comments produced by 1,183 undergraduates enrolled in 90 writing seminars during the spring 2016 semester. The Excel file housing the data is organized into 19 columns and 14,010 rows. The column headings include:
Draft 1 of the literature review was a one-onone review. Draft 2 was a multiple-reciprocal peer review with 1-6 peers. Draft three of the literature review was a multiple reciprocal review. Drafts 4 and 5 were optional. The first draft of the public argument peer review was one-on-one, and the second draft was multiple-reciprocal. The students also completed a multiple-reciprocal peer review for their final portfolio drafts.
student writer's ability to produce voice, vocabulary, syntax, sentence structure, punctuation, and tone appropriate to the genre and audience.
writer's ability to produce voice, vocabulary, syntax, sentence structure, punctuation, tone, source handling, etc., appropriate to the genre and audience.
assessments of Comment on Cognitive and Heuristic Processes, Comments on Invention, Comments on Reasoning, Comments on Presentation, and General Comments into one field.
Students are given a detailed rubric, the same one used by instructors in our program to assess individual students in their classes as well as to assess mid-term and final portfolios. The rubric acts as a guide for formative as well as, at semester’s end, summative assessment.
Cognition/Metacognition: Knowledge of Writing • Recognizes the purpose of the assignment • Conceives of a procedure for fulfilling it • Perceives the problem(s) to be solved in the assignment • Follows directions through all stages of the assignment • Able to detect flaws in reasoning in one’s own or other’s reasoning (outlines and peer reviews) • Able to identify and evaluate (in plan, outlines, peer reviews, cover letters, other artifacts): o Rhetorical Strategies o Audience o Purpose o Genre o Plan/Arrangement o Complex Synthesis o Presentation Invention: Idea/Audience (test of novelty, creativity, persuasion) • Selection of an appropriate and engaging subject within the topic • Ability to select and work successful within a genre • Selection of an appropriate proposition and reasons to support it, attuned to the audience and purpose • • • • •
Selection of the appropriate amount and type of evidence and materials to support the proposition, attuned to the audience and purpose Arrangement and style attuned to the audience, purpose, and genre, including ability to evaluate the strength of reasons and evidence Identification of shared premises to enable an effective introduction and conclusion Ability to grasp feedback or detect problems with invention and revise accordingly Ability to vary voice and style to accommodate different audiences and genres Reasoning: Development/Coherence (test of reasonableness) • Creation or selection of an appropriately justificatory or explanatory proposition • Creation or selection of reasons that directly support the proposition • Selection of evidence that confirms, illuminates or otherwise develops the reasons • Ability to test argument through strategies of counterargument • Demonstration of logical coherence: all reasons support the proposition, all evidence supports the reasons, and to the extent possible, reasons do not contradict each other • Demonstration of semantic coherence: sentences and paragraphs stick together Presentation • Ability to produce a voice and style appropriate to the genre and audience • Control of vocabulary, syntax, sentence structure, punctuation, tone • Ability to integrate rhetorical strategies and sources so that they create a consistency of style appropriate to the genre and audience • Demonstrated ability to proofread and polish work for an outside reader • Creation and use of grammar checklist to identify context and patterns of error in mechanics and usage, as well as to correct them • Appropriate formatting, citation, documentation of sources
We exclude peer reviews where the numerical Rubric Score is listed as 0, since many of these appear to be cases where the score is simply missing, although the reviewer evaluated the draft positively. We also exclude reviews where the score was greater than 0 but less than 2 out of 4 -- these represent evaluations with failing grades, and in such cases the reviewers often didn’t bother to supply detailed comments. Among the remaining examples, the bottom quartile includes 3046 reviews with scores between 2 and 3.3 out of 4, and the top quartile includes 3054 reviews with scores above 3.78. The combined comments in the bottom-quartile reviews comprise 1,022,709 words, and the combined comments in the top-quartile reviews comprise 759,637 words.
In order to evaluate the degree of association between individual words* [FN: here “word” simply means a string of alphabetic letters set off by space, punctuation, or other symbols] and score quartiles, we used the “algorithm from section 3.5.1 of Monroe et al. 2008.
This method, originally developed for a study of political writing, starts with a simple ratio of estimated word frequencies in two collections of text. The problem is that when the overall frequency of a word is low, so is our confidence that the ratio is not a statistical accident -- and so is the value of that word as a predictive marker of the distinction under study. Thus in this collection, the word judged occurs five times in the bottom-quartile reviews, and only once in the top quartile; given the different overall word counts in the two groups, the maximum-likelihood estimate is that judged is about 4 times as common in bottom-quartile reviews as in topquartile reviews. But we can’t be very confident that in the next batch of reviews, this ration might not be quite different, or even reversed. And in any case, judged doesn’t occur often enough to be a very strong indicator of a reviewer’s sentiment.
In contrast, the word should occurs 3,780 times in the bottomquartile comments, and 1,914 times in the top-quartile comments. Allowing for the groups’ overall word counts, this tells us that the frequency of should is about 1.5 times greater in the bottom-quartile comments than in the top-quartile comments. But in this case, the overall frequency is high enough that we can be fairly confident that should will also be about 50% more frequent in the low-quartile comments in next semester’s sample -- and should is common enough to be a useful indicator of overall review sentiment. In order to deal with these issues, the cited method shrinks the odds ratio for each word based on a factor derived from a simple statistical estimate of the process generating the counts, along with an estimate of that word’s overall frequency in a relevant more general source. The result is a number, the “weighted log-odds ratio”, that we can use to rank words according to their apparent affinity for one text sample or the other.
The bottom quartile has more words per (combined) comment than the top quartile: 336 vs. 249.
The words most reliably associated with the bottom quartile include: There's obviously some mixture of content with commentary here, but overall it makes sense -- most if not all of the content admixture could be removed by limiting the list to words that occur at least at a rate of 100 per million in both sets as well as removing words that are related to the specific content of a particular seminar, such as “Picasso.” "But" is the 6th most negative word:
but 5123 (5009.25) 2742 (3609.62) 5.949 The evaluations in the top quartile by rubric score are more heavily weighted with evaluative words of all types. The evaluations in the lower quartile are more heavily weighted with prescriptive terms such as “should,” “try,” “more,” “needs, “unnecessary.” One symptom: on the negative side, there's just one evaluative word that's both reasonably common overall (frequency > 15/100k) and more than twice as common in the lower quartile of evaluations: "unclear". The rest of that end of the list of the lower quartile includes:
At the upper end of the quartile, more than 20 positive terms are employed:
unclear incorrect unnecessary needs clearer WORD easy great very nice flows logically organized job well supported
Our next steps will include identifying cases where the commentary and grade are asymmetrical. We will also analyze instructor commentary and scores to see if similar patterns obtain. Of particular interest will be to explore correlations between the types and length of commentary as well as scores from peers in relation to improvement in drafts and final scores. We anticipate a corpus of several thousand papers from our own program as well as from the other institutions participating in a broader NSF-funded study of peer review: USF, MIT, UCNS, and Dartmouth, and will engage in a range of corpus-based approaches to text analysis. We hope to contribute to research on the effectiveness of different kinds of feedback, particularly as concerns struggling writers.
This research is supported by the National Science Foundation under Award #1544239, “Collaborative Research: The Role of Instructor and Peer Feedback in Improving the Cognitive, Interpersonal, and Interpersonal Competencies of Student Writers in STEM courses. Our thanks for the vital contributions of Joe Moxley, Principal Investigator, Alex Rudniy and the USF team Rajeev Reddy Rachamalla, Dat Ba Le, Brooke Downey, and Natalie Kass; Norbert Elliot, Professor Emeritus of English at New Jersey Institute of Technology and our NSF grant consultant; fellow investigators Chris Anson, Suzanne Lane, and Christiane Donahue; and Brighid Kelly, who solves all problems, big and small.