Impact of Revision Planning on Peer-Reviewed Writing Alok Baikadi Christian Schunn Kevin Ashley University of Pittsburgh University of Pittsburgh University of Pittsburgh 3939 O’Hara St 3939 O’Hara St 3939 O’Hara St Pittsburgh, PA 15213 Pittsburgh, PA 15213 Pittsburgh, PA 15213 baikadi@pitt.edu schunn@pitt.edu ashley@pitt.edu ABSTRACT with the process than by observing an expert performing the same Revision is a core writing skill that presents challenges to both task [19]. novice and expert writers. Within the context of peer review, peer Yet, once feedback is received, it is not always implemented in feedback has the potential to provide rich guidance for reviewing, future drafts [6]. Sometimes students indicate an intention to especially when making content-level changes. However, authors implement meaningful changes but do not follow through with the must review and evaluate each piece of feedback for meaningful intent [5]. One approach is to use checklists [17] to guide the critiques that can be applied to further drafts. In this work, we students in focusing on important aspects of the writing. Another is analyzed the impact of revision planning and comment feedback on to allow students to create revision memos, encouraging students subsequent essay performance. We found that the amount of to focus on intended revisions that they had identified [1]. feedback implemented, as well as learning gained from review are predictive of improved performance on future writing. While students do not always implement the feedback they receive, it has not been clear whether this was because they had forgotten Keywords about the feedback, or whether they had chosen to ignore it. In this work, we employed a revision planning application designed to peer review, revision, writing instruction scaffold the process of implementing feedback received in the peer- review process in several high school classrooms. We then 1. INTRODUCTION analyzed the impact of tool usage and implementation of feedback Revision has long been seen as one of the cornerstones of effective on future writing. The results show that the degree to which writing [7]. Practicing revision has been shown to not only improve students implemented the feedback they received is predictive of the produced writing, but also help on first drafts of future writings future writing success. In addition, students with access to the [9]. One of the discriminators between expert and novice writers is revision planning intervention were able to show gains in both the how they approach revision. While both groups often make many amount of feedback addressed, and in quality of a future writing surface-level edits, such as spelling, grammar, and stylistic assignment. revisions [2, 5, 15, 18], expert writers often make a higher proportion of content-level edits than do novices [4]. 2. SWoRD PEER-REVIEW SYSTEM There are, however, many factors that may influence how many Web-based, computer-supported peer review has been shown to be surface- or content-level changes are made during revision. an effective tool for improving students’ writing skills. Students Students’ revision is often of higher quality when given feedback learn as they read and review each other’s papers based on [4], especially if the feedback is substantial [9]. However, students instructor provided-criteria. Students still need support, however, will make more surface-level edits when given surface-level in organizing the reviews they receive and planning how to revise feedback [15]. They are also responsive to the grading rubrics that their own papers. This paper describes a revision environment that are presented to them by the teacher, often considering the teacher helps students to cluster and prioritize reviewers’ suggestions, to be the final judge of the intended audience [18]. Further, students develop a plan for revision their papers, and make note of lessons are more likely to engage in content-level revisions when provided learned about writing for future use. We report here about students’ with sufficient domain knowledge, and training in reading for experiences in using the tool in a high school Advanced Placement meaning [12]. course. Peer-review has been shown to have beneficial effects for revision. Scaffolded Writing and Rewriting in the Disciplines (SWoRD) is a Students were able to employ more strategic revision strategies web-based reciprocal peer review system developed at the given peer feedback [11], made fewer surface-level changes [16], University of Pittsburgh. It was originally developed especially for and add more details in their writing [13], especially when peers large undergraduate courses in academic disciplines where class provide justification for their feedback [8]. Students are also more size would otherwise discourage instructors from employing many likely to learn critical revision skills by observing a peer coping writing assignments. Over the past 12 years, it has been used by over thirty-five thousand students across grade levels and across a variety of academic disciplines. Research shows that students learn Figure 1 SWoRD Peer Review Timeline as much, if not more, from giving good feedback as they do from submissions that they can incorporate into their own work. Prior receiving it [14]. Therefore, the SWoRD system is designed to work [14] has shown connections between changes made in a support both the acts of giving and receiving feedback. document and the documents reviewed. The strongest connections came when the student both recognized it in a peers' work, as well The peer review process within SWoRD takes place in three as received feedback on the same topic from their peer reviewers. phases: An Authoring phase, a Review phase, and a Back- Evaluation phase. Figure 1 shows the peer-review process from the To support this process, the Revision Planning system has two students’ perspective. In the first phase, students are provided with components. The first component, Ideas for Revision, is available writing prompt to which they will respond. The instructor provides to the student during the reviewing process. It encourages them to the prompt and set deadlines. Students may either enter text into the make observations on the papers they are reading, and make note web interface, or upload a pre-existing document in order to submit of changes that could be applied to their documents. They are able their assignments. Once documents have been uploaded, students to identify the observation as a good idea that they’d like to can begin requesting documents to review. consider for their own work, or a problem with the peer’s paper that they would like to avoid in their next draft. Figure 2 shows the Ideas The Review phase takes place once the submission deadline has for Revision page. passed. The instructor provides a grading rubric that contains several rating dimensions and prompts for written comments. The The second component, the Revision Planner, allows students to instructor then selects how many documents each student is asked consider how they would address each comment they receive from to review, and the system assigns the reviews from the existing pool their peers. For each comment, they can elect to ignore it or fix it. of documents. In order to ensure that each submission receives an If they choose to fix it, they can then assign a priority and make adequate number of reviews, students may elect to do additional notes on what the fix will be. If they choose to ignore it, they can reviewing for extra credit. When reviewing a document, students select a reason from a drop-down menu, or add more text to explain are presented with the grading rubric and comment prompts the why it is being ignored. Both the Revision Planner and the Ideas instructor has provided along with the submitted document. The for Revision are visible during revision in their full capability. In student reads the document and provides written feedback for each addition, the system can generate checklist that the students can use evaluative dimension, as well as numerical scores on a seven-point to focus on the aspects of their document they have decided to rating scale (1: Disastrous to 7: Excellent). Once students have improve. Figure 3 shows the Revision Planner page. submitted their scores, the system calculates a numerical accuracy score, taking into account consistency of each student’s scores with 3. PEER REVIEW CORPUS the mean of the other raters for the same documents. This is 10 AP English Language and Composition teachers (n=941 provided to the student as reviewer feedback once all the scores students) fully participated in the study. 40% of the schools were have been submitted. from Title 1 schools, 40% had at least 30% or more students of color, and 50% had a high proportion of economically In the final phase, students receive the feedback and scores disadvantaged students. generated by their peers. They can then evaluate the helpfulness of the feedback received, and provide extra feedback to the reviewers All included teachers taught at least two sections of AP for future reviewing. As with reviewing evaluation dimensions, English Language and Composition such that sections could be helpfulness is rated on a seven-point scale. The whole peer-review randomly assigned to one of two conditions: all tools on vs. all tools process can then be repeated for multiple drafts of the same paper. off. All classes implemented an initial peer review essay assignment with a required revision and then a transfer essay 2.1 Revision Planning assignment. Across teachers, they could offer additional During the course of peer review, students have the opportunity to intermediate essay assignments, but they had to keep this constant learn from both giving and receiving feedback. During the review across their sections. In the Tools-On condition (n=483 students), process, students are asked to critically evaluate a peer's submission students experienced all four tools turned on in all the assignments on the same criterion with which their own writing will be judged. prior to the transfer essay: thesis prompt, instant feedback [3,10], While reviewing, students may notice aspects of their peers' Figure 2 Ideas for Revision Page Figure 3 Revision Planning interface ideas to consider, and revision planning. In the Tools-Off condition, the baseline version of SWoRD was used instead. The writing assignment consisted of essays writing assignments drawn from practice AP English Language and Composition exams, requiring students to analyze the rhetorical strategies employed in a given text. For the initial essay, students were required to write two drafts. Both drafts were subject to peer- review. The grading rubric asked students to evaluate the accuracy and quality of rhetorical devices identified, evaluate the textual evidence used in the analysis, and comment on the use of academic English grammar and style. For the transfer essay, only one draft was required. Some teachers elected to include peer review and a second draft of the transfer essay, but it was not required. 3.1 Data Annotation For each teacher (except for one who had not finished the transfer essay by the time expert coders completed their tasks), we Figure 4 Scatter plot of transfer essay scores as a function of randomly sampled 15 students from each of the conditions (i.e., 30 percentage of critique comments implemented. students per teacher), and then collected all the first draft essays from the initial and transfer assignments (i.e., two essays per Using the revision planning tool appeared to influence the amount sampled student). Writing experts (instructors of First Year Writing of revising that students did, with students being more likely to at a local university) were recruited and trained in using the same implement comments in the Tools On condition (p<.005). This evaluation rubric to grade all of these sampled essays; they were effect was larger for lower ability authors (i.e., students with a first blind to condition and whether the essay was the first or transfer draft score lower than the class mean), as shown in Figure 5. The essay. Each essay was scored by two experts. A third expert was condition x writer ability interaction was statistically significant also used if there was a discrepancy of grades of more than 2 points. (p<.001). For analysis, we used the mean score across raters. From these essays, we further subsampled 10 students per teacher condition by selecting the highest 5 and lowest 5 quality essays as judged by the experts (n=200). For this subset of essays, undergraduate work-study students coded the comments (n=6483) for whether the comment was implemented in the second draft of the first essay, or not (Cohen’s Kappa = 0.779). 4. RESULTS Considering all students for whom both essays were scored, and comments were annotated for implementation, there was an effect from the percentage of implemented critique comments on transfer essay score (Beta=0.359, p<0.05), controlling for performance on the initial essay score (i.e., controlling for initial writing ability). Figure 4 shows the scatter plot. Figure 5 Box plots of the proportion of comments implemented as a function of condition and writer ability. Considering all students for whom essays were scored, there was no overall effect of condition on the transfer essay, in a univariate analysis on transfer essay overall score [F(2,167)<1], using intervention, however, allows students to record more general condition as a fixed factor and essay 1 score as a covariate. observations that may be transferred to future writing. Extracting these themes from either the feedback received or Ideas for However, if the analysis is restricted to only the 5 teachers for Revision observations may enable students to reify long term goals which there was a higher rate of use of the Ideas for Revision and for improvement in writing. Revision Planning tools, then the same analysis was statistically significant [F(1,147)=2.67, p=.03]; see Figure 6. Note the same Revision is a core writing skill that presents challenges to both result held whether or not Essay 1 score was included as a covariate. novice and expert writers. Within the context of peer review, peer In addition, another follow-up regression analysis (including all the feedback has the potential to provide rich guidance for reviewing, students with scored essays), we predicted transfer essay score especially when making content-level changes. We have shown using Essay 1, # of Ideas for Revision, and # of Revision Plans as that authors who implement more of the feedback received and who predictors. # of Revision Plans was not a significant predictor note observations made while reviewing peer work are able to (Beta=.19, t=1.5, p=.14), but # of Ideas for Revision was produce higher quality writing in future assignments. statistically significant (Beta=.25, t=2.0, p=.05). In other words, we have additional evidence that these tools were useful for improving 6. ACKNOWLEDGMENTS writing, that considering amount of tool use as a useful filter on We thank participating teachers and students. This work funded by which teachers to consider, and it appears that amount of insights the Institute of Education Sciences, under grant R305A120370. gathered from reviewing in particular was most helpful. 7. REFERENCES [1] Bardine, B.A. and Fulton, A. 2008. Analyzing the Benefits of Revision Memos during the Writing and Revision Process. The Clearing House. 81 (4). 149–154. [2] Bridwell, L.S. 1980. Revising Strategies in Twelfth Grade Studentns’ Transactional Writing. Research in the Teaching of English. 14 (3). 197–222. [3] Falakmasir M. H., Ashley K. D., Schunn C., and Litman D. 2014. Identifying Thesis and Conclusion Statements in Student Essays to Scaffold Peer Review. In Proceedings, 12th International Conference on Intelligent Tutoring Systems, Lecture Notes in Computer Science, v. 8474. pp. 254-259. [4] Fitzgerald, J.1987. Research on Revision in Writing. Review of Educational Research. 57 (4). 481–506. [5] Fitzgerald, J. and Markham, L.R. 1987. Teaching children about revision in writing. Cognition and Instruction. 4 (1). 3–24. [6] Fitzgerald, J. and Stamm, C. 1990. Effects of Group Conferences on First Graders’ Revision in Writing. Figure 6 Mean transfer essay score (with SE bars) as a function Written Communication. 7 (1). 96–135. of condition for the 5 teachers whose students made more use [7] Flower, L. and Hayes, J. 1981. A cognitive process of the Ideas for Revision and Revision Planning tools. theory of writing. College Composition and Communication. 32 (4). 365–387. 5. CONCLUSIONS AND FUTURE WORK [8] Gielen, S., Peeters, E., Dochy, F., Onghena, P., and In this work we have presented a revision planning intervention Struyven, K. 2010. Improving the effectiveness of peer which allows students to respond to feedback received, and note feedback for learning. Learning and Instruction. 20 (4). observations made during the reviewing process. We deployed the 304–315. intervention in several AP English Language and Composition [9] Hillocks, G.J. 1982. The Interaction of Instruction, classrooms and evaluated the impact of the intervention on both Teacher Comment, and Revision in Teaching the second draft revisions and future writing. Students who used the Composing Process. Research in the Teaching of revision planning intervention addressed more comments in the English. 16 (3). 261–278. second drafts of their essays. When the analysis focused on [10] Nguyen, H., Xiong, W., and Litman, D. 2016. Instant classrooms in which revision planning was a more common Feedback for Increasing the Presence of Solutions in Peer activity, students who had access to the intervention also wrote Reviews, Proceedings Conference of the North American higher quality subsequent essays, as judged by subject matter Chapter of the Association for Computational experts. Linguistics: Demonstrations (NAACL-HLT), San Diego, There are several promising directions for future work. While we CA. investigated the amount of feedback to which the students [11] Keen, J. 2010. Strategic revisions in the writing of Year 7 attempted to respond, further investigation is required to students in the UK. The Curriculum Journal. 21 (3). understand both the quality of the feedback and the quality of the 255–280. revision. Students may be addressing many comments which are [12] McCutchen, D., Francis, M., and Kerr, S. 1997. Revising incidental or even detrimental to the quality of the writing, or of for Meaning : Effects of Knowledge and Strategy. future writing. Secondly, the feedback received within a single Journal of Educational Psychology. 89 (4). 667–676. review is often focused on short-term improvements for the writing [13] Morris Kindzierski, C.M. 2009. “I Like It the Way It at hand, rather than long term development. The Ideas for Revision Is!”: Peer-Revision Writing Strategies for Students With Emotional and Behavioral Disorders. Preventing School [17] Smede, S.D. 2000. Interior Design: Revision as Focus. Failure: Alternative Education for Children and Youth. The English Journal. 90 (1). 117–121. 54 (1). 51–59. [18] Yagelski, R.P. 1995. The role of classroom context in the [14] Patchen, M. Peer Review of Writing: Learning From revision strategies of student writers. Research in the Revision Using Peer Feedback and Reviewing Peers' Teaching of English. 29 (2). 216–238. Text. Doctoral Dissertation, University of Pittsburgh, [19] Zimmerman, B.J. and Kitsantas, A. 2002. Acquiring 2011. writing revision and self-regulatory skill through [15] Patthey-chavez, G.G., Matsumara, L., and Valdés, R. observation and emulation. Journal of Educational 2014. Investigating the process approach middle to Psychology. 94 (4). 660–668. writing instruction in urban schools. Journal of Adolescent & Adult Literacy. 47 (6). 462–476. [16] Peterson, S. 2003. Peer Response and Students’ Revisions of Their Narrative Writing. Educational Studies in language and Literature. 3 239–272.