Revision is a core writing skill that presents challenges to both novice and expert writers. Within the context of peer review, peer feedback has the potential to provide rich guidance for reviewing, especially when making content-level changes. However, authors must review and evaluate each piece of feedback for meaningful critiques that can be applied to further drafts. In this work, we analyzed the impact of revision planning and comment feedback on subsequent essay performance. We found that the amount of feedback implemented, as well as learning gained from review are predictive of improved performance on future writing.
Revision has long been seen as one of the cornerstones of effective
writing [
There are, however, many factors that may influence how many
surface- or content-level changes are made during revision.
Students’ revision is often of higher quality when given feedback
[
Peer-review has been shown to have beneficial effects for revision.
Students were able to employ more strategic revision strategies
given peer feedback [
Yet, once feedback is received, it is not always implemented in
future drafts [
Web-based, computer-supported peer review has been shown to be an effective tool for improving students’ writing skills. Students learn as they read and review each other’s papers based on instructor provided-criteria. Students still need support, however, in organizing the reviews they receive and planning how to revise their own papers. This paper describes a revision environment that helps students to cluster and prioritize reviewers’ suggestions, develop a plan for revision their papers, and make note of lessons learned about writing for future use. We report here about students’ experiences in using the tool in a high school Advanced Placement course.
Scaffolded Writing and Rewriting in the Disciplines (SWoRD) is a web-based reciprocal peer review system developed at the University of Pittsburgh. It was originally developed especially for large undergraduate courses in academic disciplines where class size would otherwise discourage instructors from employing many writing assignments. Over the past 12 years, it has been used by over thirty-five thousand students across grade levels and across a variety of academic disciplines. Research shows that students learn as much, if not more, from giving good feedback as they do from receiving it [14]. Therefore, the SWoRD system is designed to support both the acts of giving and receiving feedback. The peer review process within SWoRD takes place in three phases: An Authoring phase, a Review phase, and a BackEvaluation phase. Figure 1 shows the peer-review process from the students’ perspective. In the first phase, students are provided with writing prompt to which they will respond. The instructor provides the prompt and set deadlines. Students may either enter text into the web interface, or upload a pre-existing document in order to submit their assignments. Once documents have been uploaded, students can begin requesting documents to review.
The Review phase takes place once the submission deadline has passed. The instructor provides a grading rubric that contains several rating dimensions and prompts for written comments. The instructor then selects how many documents each student is asked to review, and the system assigns the reviews from the existing pool of documents. In order to ensure that each submission receives an adequate number of reviews, students may elect to do additional reviewing for extra credit. When reviewing a document, students are presented with the grading rubric and comment prompts the instructor has provided along with the submitted document. The student reads the document and provides written feedback for each evaluative dimension, as well as numerical scores on a seven-point rating scale (1: Disastrous to 7: Excellent). Once students have submitted their scores, the system calculates a numerical accuracy score, taking into account consistency of each student’s scores with the mean of the other raters for the same documents. This is provided to the student as reviewer feedback once all the scores have been submitted.
In the final phase, students receive the feedback and scores generated by their peers. They can then evaluate the helpfulness of the feedback received, and provide extra feedback to the reviewers for future reviewing. As with reviewing evaluation dimensions, helpfulness is rated on a seven-point scale. The whole peer-review process can then be repeated for multiple drafts of the same paper.
During the course of peer review, students have the opportunity to learn from both giving and receiving feedback. During the review process, students are asked to critically evaluate a peer's submission on the same criterion with which their own writing will be judged. While reviewing, students may notice aspects of their peers' submissions that they can incorporate into their own work. Prior work [14] has shown connections between changes made in a document and the documents reviewed. The strongest connections came when the student both recognized it in a peers' work, as well as received feedback on the same topic from their peer reviewers. To support this process, the Revision Planning system has two components. The first component, Ideas for Revision, is available to the student during the reviewing process. It encourages them to make observations on the papers they are reading, and make note of changes that could be applied to their documents. They are able to identify the observation as a good idea that they’d like to consider for their own work, or a problem with the peer’s paper that they would like to avoid in their next draft. Figure 2 shows the Ideas for Revision page.
The second component, the Revision Planner, allows students to consider how they would address each comment they receive from their peers. For each comment, they can elect to ignore it or fix it. If they choose to fix it, they can then assign a priority and make notes on what the fix will be. If they choose to ignore it, they can select a reason from a drop-down menu, or add more text to explain why it is being ignored. Both the Revision Planner and the Ideas for Revision are visible during revision in their full capability. In addition, the system can generate checklist that the students can use to focus on the aspects of their document they have decided to improve. Figure 3 shows the Revision Planner page.
10 AP English Language and Composition teachers (n=941 students) fully participated in the study. 40% of the schools were from Title 1 schools, 40% had at least 30% or more students of color, and 50% had a high proportion of economically disadvantaged students.
All included teachers taught at least two sections of AP
English Language and Composition such that sections could be
randomly assigned to one of two conditions: all tools on vs. all tools
off. All classes implemented an initial peer review essay
assignment with a required revision and then a transfer essay
assignment. Across teachers, they could offer additional
intermediate essay assignments, but they had to keep this constant
across their sections. In the Tools-On condition (n=483 students),
students experienced all four tools turned on in all the assignments
prior to the transfer essay: thesis prompt, instant feedback [
The writing assignment consisted of essays writing assignments drawn from practice AP English Language and Composition exams, requiring students to analyze the rhetorical strategies employed in a given text. For the initial essay, students were required to write two drafts. Both drafts were subject to peerreview. The grading rubric asked students to evaluate the accuracy and quality of rhetorical devices identified, evaluate the textual evidence used in the analysis, and comment on the use of academic English grammar and style. For the transfer essay, only one draft was required. Some teachers elected to include peer review and a second draft of the transfer essay, but it was not required.
For each teacher (except for one who had not finished the transfer essay by the time expert coders completed their tasks), we randomly sampled 15 students from each of the conditions (i.e., 30 students per teacher), and then collected all the first draft essays from the initial and transfer assignments (i.e., two essays per sampled student). Writing experts (instructors of First Year Writing at a local university) were recruited and trained in using the same evaluation rubric to grade all of these sampled essays; they were blind to condition and whether the essay was the first or transfer essay. Each essay was scored by two experts. A third expert was also used if there was a discrepancy of grades of more than 2 points. For analysis, we used the mean score across raters.
From these essays, we further subsampled 10 students per teacher condition by selecting the highest 5 and lowest 5 quality essays as judged by the experts (n=200). For this subset of essays, undergraduate work-study students coded the comments (n=6483) for whether the comment was implemented in the second draft of the first essay, or not (Cohen’s Kappa = 0.779).
Considering all students for whom both essays were scored, and comments were annotated for implementation, there was an effect from the percentage of implemented critique comments on transfer essay score (Beta=0.359, p<0.05), controlling for performance on the initial essay score (i.e., controlling for initial writing ability). Figure 4 shows the scatter plot.
Using the revision planning tool appeared to influence the amount of revising that students did, with students being more likely to implement comments in the Tools On condition (p<.005). This effect was larger for lower ability authors (i.e., students with a first draft score lower than the class mean), as shown in Figure 5. The condition x writer ability interaction was statistically significant (p<.001).
Considering all students for whom essays were scored, there was no overall effect of condition on the transfer essay, in a univariate analysis on transfer essay overall score [F(2,167)<1], using condition as a fixed factor and essay 1 score as a covariate. However, if the analysis is restricted to only the 5 teachers for which there was a higher rate of use of the Ideas for Revision and Revision Planning tools, then the same analysis was statistically significant [F(1,147)=2.67, p=.03]; see Figure 6. Note the same result held whether or not Essay 1 score was included as a covariate. In addition, another follow-up regression analysis (including all the students with scored essays), we predicted transfer essay score using Essay 1, # of Ideas for Revision, and # of Revision Plans as predictors. # of Revision Plans was not a significant predictor (Beta=.19, t=1.5, p=.14), but # of Ideas for Revision was statistically significant (Beta=.25, t=2.0, p=.05). In other words, we have additional evidence that these tools were useful for improving writing, that considering amount of tool use as a useful filter on which teachers to consider, and it appears that amount of insights gathered from reviewing in particular was most helpful.
In this work we have presented a revision planning intervention which allows students to respond to feedback received, and note observations made during the reviewing process. We deployed the intervention in several AP English Language and Composition classrooms and evaluated the impact of the intervention on both second draft revisions and future writing. Students who used the revision planning intervention addressed more comments in the second drafts of their essays. When the analysis focused on classrooms in which revision planning was a more common activity, students who had access to the intervention also wrote higher quality subsequent essays, as judged by subject matter experts.
There are several promising directions for future work. While we investigated the amount of feedback to which the students attempted to respond, further investigation is required to understand both the quality of the feedback and the quality of the revision. Students may be addressing many comments which are incidental or even detrimental to the quality of the writing, or of future writing. Secondly, the feedback received within a single review is often focused on short-term improvements for the writing at hand, rather than long term development. The Ideas for Revision intervention, however, allows students to record more general observations that may be transferred to future writing. Extracting these themes from either the feedback received or Ideas for Revision observations may enable students to reify long term goals for improvement in writing.
Revision is a core writing skill that presents challenges to both novice and expert writers. Within the context of peer review, peer feedback has the potential to provide rich guidance for reviewing, especially when making content-level changes. We have shown that authors who implement more of the feedback received and who note observations made while reviewing peer work are able to produce higher quality writing in future assignments.
We thank participating teachers and students. This work funded by the Institute of Education Sciences, under grant R305A120370.