=Paper=
{{Paper
|id=Vol-1633/ws1-paper7
|storemode=property
|title=Impact of Revision Planning on Peer-Reviewed Writing
|pdfUrl=https://ceur-ws.org/Vol-1633/ws1-paper7.pdf
|volume=Vol-1633
|authors=Alok Baikadi,Christian Schunn,Kevin Ashley
|dblpUrl=https://dblp.org/rec/conf/edm/BaikadiSA16
}}
==Impact of Revision Planning on Peer-Reviewed Writing==
https://ceur-ws.org/Vol-1633/ws1-paper7.pdf
Impact of Revision Planning on Peer-Reviewed Writing
Alok Baikadi Christian Schunn Kevin Ashley
University of Pittsburgh University of Pittsburgh University of Pittsburgh
3939 O’Hara St 3939 O’Hara St 3939 O’Hara St
Pittsburgh, PA 15213 Pittsburgh, PA 15213 Pittsburgh, PA 15213
baikadi@pitt.edu schunn@pitt.edu ashley@pitt.edu
ABSTRACT with the process than by observing an expert performing the same
Revision is a core writing skill that presents challenges to both task [19].
novice and expert writers. Within the context of peer review, peer Yet, once feedback is received, it is not always implemented in
feedback has the potential to provide rich guidance for reviewing, future drafts [6]. Sometimes students indicate an intention to
especially when making content-level changes. However, authors implement meaningful changes but do not follow through with the
must review and evaluate each piece of feedback for meaningful intent [5]. One approach is to use checklists [17] to guide the
critiques that can be applied to further drafts. In this work, we students in focusing on important aspects of the writing. Another is
analyzed the impact of revision planning and comment feedback on to allow students to create revision memos, encouraging students
subsequent essay performance. We found that the amount of to focus on intended revisions that they had identified [1].
feedback implemented, as well as learning gained from review are
predictive of improved performance on future writing. While students do not always implement the feedback they receive,
it has not been clear whether this was because they had forgotten
Keywords about the feedback, or whether they had chosen to ignore it. In this
work, we employed a revision planning application designed to
peer review, revision, writing instruction scaffold the process of implementing feedback received in the peer-
review process in several high school classrooms. We then
1. INTRODUCTION analyzed the impact of tool usage and implementation of feedback
Revision has long been seen as one of the cornerstones of effective on future writing. The results show that the degree to which
writing [7]. Practicing revision has been shown to not only improve students implemented the feedback they received is predictive of
the produced writing, but also help on first drafts of future writings future writing success. In addition, students with access to the
[9]. One of the discriminators between expert and novice writers is
revision planning intervention were able to show gains in both the
how they approach revision. While both groups often make many amount of feedback addressed, and in quality of a future writing
surface-level edits, such as spelling, grammar, and stylistic assignment.
revisions [2, 5, 15, 18], expert writers often make a higher
proportion of content-level edits than do novices [4]. 2. SWoRD PEER-REVIEW SYSTEM
There are, however, many factors that may influence how many Web-based, computer-supported peer review has been shown to be
surface- or content-level changes are made during revision. an effective tool for improving students’ writing skills. Students
Students’ revision is often of higher quality when given feedback learn as they read and review each other’s papers based on
[4], especially if the feedback is substantial [9]. However, students instructor provided-criteria. Students still need support, however,
will make more surface-level edits when given surface-level in organizing the reviews they receive and planning how to revise
feedback [15]. They are also responsive to the grading rubrics that their own papers. This paper describes a revision environment that
are presented to them by the teacher, often considering the teacher helps students to cluster and prioritize reviewers’ suggestions,
to be the final judge of the intended audience [18]. Further, students develop a plan for revision their papers, and make note of lessons
are more likely to engage in content-level revisions when provided learned about writing for future use. We report here about students’
with sufficient domain knowledge, and training in reading for experiences in using the tool in a high school Advanced Placement
meaning [12]. course.
Peer-review has been shown to have beneficial effects for revision. Scaffolded Writing and Rewriting in the Disciplines (SWoRD) is a
Students were able to employ more strategic revision strategies web-based reciprocal peer review system developed at the
given peer feedback [11], made fewer surface-level changes [16], University of Pittsburgh. It was originally developed especially for
and add more details in their writing [13], especially when peers large undergraduate courses in academic disciplines where class
provide justification for their feedback [8]. Students are also more size would otherwise discourage instructors from employing many
likely to learn critical revision skills by observing a peer coping writing assignments. Over the past 12 years, it has been used by
over thirty-five thousand students across grade levels and across a
variety of academic disciplines. Research shows that students learn
Figure 1 SWoRD Peer Review Timeline
as much, if not more, from giving good feedback as they do from submissions that they can incorporate into their own work. Prior
receiving it [14]. Therefore, the SWoRD system is designed to work [14] has shown connections between changes made in a
support both the acts of giving and receiving feedback. document and the documents reviewed. The strongest connections
came when the student both recognized it in a peers' work, as well
The peer review process within SWoRD takes place in three as received feedback on the same topic from their peer reviewers.
phases: An Authoring phase, a Review phase, and a Back-
Evaluation phase. Figure 1 shows the peer-review process from the To support this process, the Revision Planning system has two
students’ perspective. In the first phase, students are provided with components. The first component, Ideas for Revision, is available
writing prompt to which they will respond. The instructor provides to the student during the reviewing process. It encourages them to
the prompt and set deadlines. Students may either enter text into the make observations on the papers they are reading, and make note
web interface, or upload a pre-existing document in order to submit of changes that could be applied to their documents. They are able
their assignments. Once documents have been uploaded, students to identify the observation as a good idea that they’d like to
can begin requesting documents to review. consider for their own work, or a problem with the peer’s paper that
they would like to avoid in their next draft. Figure 2 shows the Ideas
The Review phase takes place once the submission deadline has for Revision page.
passed. The instructor provides a grading rubric that contains
several rating dimensions and prompts for written comments. The The second component, the Revision Planner, allows students to
instructor then selects how many documents each student is asked consider how they would address each comment they receive from
to review, and the system assigns the reviews from the existing pool their peers. For each comment, they can elect to ignore it or fix it.
of documents. In order to ensure that each submission receives an If they choose to fix it, they can then assign a priority and make
adequate number of reviews, students may elect to do additional notes on what the fix will be. If they choose to ignore it, they can
reviewing for extra credit. When reviewing a document, students select a reason from a drop-down menu, or add more text to explain
are presented with the grading rubric and comment prompts the why it is being ignored. Both the Revision Planner and the Ideas
instructor has provided along with the submitted document. The for Revision are visible during revision in their full capability. In
student reads the document and provides written feedback for each addition, the system can generate checklist that the students can use
evaluative dimension, as well as numerical scores on a seven-point to focus on the aspects of their document they have decided to
rating scale (1: Disastrous to 7: Excellent). Once students have improve. Figure 3 shows the Revision Planner page.
submitted their scores, the system calculates a numerical accuracy
score, taking into account consistency of each student’s scores with 3. PEER REVIEW CORPUS
the mean of the other raters for the same documents. This is 10 AP English Language and Composition teachers (n=941
provided to the student as reviewer feedback once all the scores students) fully participated in the study. 40% of the schools were
have been submitted. from Title 1 schools, 40% had at least 30% or more students of
color, and 50% had a high proportion of economically
In the final phase, students receive the feedback and scores disadvantaged students.
generated by their peers. They can then evaluate the helpfulness of
the feedback received, and provide extra feedback to the reviewers All included teachers taught at least two sections of AP
for future reviewing. As with reviewing evaluation dimensions, English Language and Composition such that sections could be
helpfulness is rated on a seven-point scale. The whole peer-review randomly assigned to one of two conditions: all tools on vs. all tools
process can then be repeated for multiple drafts of the same paper. off. All classes implemented an initial peer review essay
assignment with a required revision and then a transfer essay
2.1 Revision Planning assignment. Across teachers, they could offer additional
During the course of peer review, students have the opportunity to intermediate essay assignments, but they had to keep this constant
learn from both giving and receiving feedback. During the review across their sections. In the Tools-On condition (n=483 students),
process, students are asked to critically evaluate a peer's submission students experienced all four tools turned on in all the assignments
on the same criterion with which their own writing will be judged. prior to the transfer essay: thesis prompt, instant feedback [3,10],
While reviewing, students may notice aspects of their peers'
Figure 2 Ideas for Revision Page
Figure 3 Revision Planning interface
ideas to consider, and revision planning. In the Tools-Off condition,
the baseline version of SWoRD was used instead.
The writing assignment consisted of essays writing assignments
drawn from practice AP English Language and Composition
exams, requiring students to analyze the rhetorical strategies
employed in a given text. For the initial essay, students were
required to write two drafts. Both drafts were subject to peer-
review. The grading rubric asked students to evaluate the accuracy
and quality of rhetorical devices identified, evaluate the textual
evidence used in the analysis, and comment on the use of academic
English grammar and style. For the transfer essay, only one draft
was required. Some teachers elected to include peer review and a
second draft of the transfer essay, but it was not required.
3.1 Data Annotation
For each teacher (except for one who had not finished the transfer
essay by the time expert coders completed their tasks), we Figure 4 Scatter plot of transfer essay scores as a function of
randomly sampled 15 students from each of the conditions (i.e., 30 percentage of critique comments implemented.
students per teacher), and then collected all the first draft essays
from the initial and transfer assignments (i.e., two essays per Using the revision planning tool appeared to influence the amount
sampled student). Writing experts (instructors of First Year Writing of revising that students did, with students being more likely to
at a local university) were recruited and trained in using the same implement comments in the Tools On condition (p<.005). This
evaluation rubric to grade all of these sampled essays; they were effect was larger for lower ability authors (i.e., students with a first
blind to condition and whether the essay was the first or transfer draft score lower than the class mean), as shown in Figure 5. The
essay. Each essay was scored by two experts. A third expert was condition x writer ability interaction was statistically significant
also used if there was a discrepancy of grades of more than 2 points. (p<.001).
For analysis, we used the mean score across raters.
From these essays, we further subsampled 10 students per teacher
condition by selecting the highest 5 and lowest 5 quality essays as
judged by the experts (n=200). For this subset of essays,
undergraduate work-study students coded the comments (n=6483)
for whether the comment was implemented in the second draft of
the first essay, or not (Cohen’s Kappa = 0.779).
4. RESULTS
Considering all students for whom both essays were scored, and
comments were annotated for implementation, there was an effect
from the percentage of implemented critique comments on transfer
essay score (Beta=0.359, p<0.05), controlling for performance on
the initial essay score (i.e., controlling for initial writing ability).
Figure 4 shows the scatter plot.
Figure 5 Box plots of the proportion of comments implemented
as a function of condition and writer ability.
Considering all students for whom essays were scored, there was
no overall effect of condition on the transfer essay, in a univariate
analysis on transfer essay overall score [F(2,167)<1], using intervention, however, allows students to record more general
condition as a fixed factor and essay 1 score as a covariate. observations that may be transferred to future writing. Extracting
these themes from either the feedback received or Ideas for
However, if the analysis is restricted to only the 5 teachers for Revision observations may enable students to reify long term goals
which there was a higher rate of use of the Ideas for Revision and for improvement in writing.
Revision Planning tools, then the same analysis was statistically
significant [F(1,147)=2.67, p=.03]; see Figure 6. Note the same Revision is a core writing skill that presents challenges to both
result held whether or not Essay 1 score was included as a covariate. novice and expert writers. Within the context of peer review, peer
In addition, another follow-up regression analysis (including all the feedback has the potential to provide rich guidance for reviewing,
students with scored essays), we predicted transfer essay score especially when making content-level changes. We have shown
using Essay 1, # of Ideas for Revision, and # of Revision Plans as that authors who implement more of the feedback received and who
predictors. # of Revision Plans was not a significant predictor note observations made while reviewing peer work are able to
(Beta=.19, t=1.5, p=.14), but # of Ideas for Revision was produce higher quality writing in future assignments.
statistically significant (Beta=.25, t=2.0, p=.05). In other words, we
have additional evidence that these tools were useful for improving 6. ACKNOWLEDGMENTS
writing, that considering amount of tool use as a useful filter on We thank participating teachers and students. This work funded by
which teachers to consider, and it appears that amount of insights the Institute of Education Sciences, under grant R305A120370.
gathered from reviewing in particular was most helpful.
7. REFERENCES
[1] Bardine, B.A. and Fulton, A. 2008. Analyzing the
Benefits of Revision Memos during the Writing and
Revision Process. The Clearing House. 81 (4). 149–154.
[2] Bridwell, L.S. 1980. Revising Strategies in Twelfth
Grade Studentns’ Transactional Writing. Research in the
Teaching of English. 14 (3). 197–222.
[3] Falakmasir M. H., Ashley K. D., Schunn C., and Litman
D. 2014. Identifying Thesis and Conclusion Statements
in Student Essays to Scaffold Peer Review. In
Proceedings, 12th International Conference on
Intelligent Tutoring Systems, Lecture Notes in Computer
Science, v. 8474. pp. 254-259.
[4] Fitzgerald, J.1987. Research on Revision in Writing.
Review of Educational Research. 57 (4). 481–506.
[5] Fitzgerald, J. and Markham, L.R. 1987. Teaching
children about revision in writing. Cognition and
Instruction. 4 (1). 3–24.
[6] Fitzgerald, J. and Stamm, C. 1990. Effects of Group
Conferences on First Graders’ Revision in Writing.
Figure 6 Mean transfer essay score (with SE bars) as a function Written Communication. 7 (1). 96–135.
of condition for the 5 teachers whose students made more use [7] Flower, L. and Hayes, J. 1981. A cognitive process
of the Ideas for Revision and Revision Planning tools. theory of writing. College Composition and
Communication. 32 (4). 365–387.
5. CONCLUSIONS AND FUTURE WORK [8] Gielen, S., Peeters, E., Dochy, F., Onghena, P., and
In this work we have presented a revision planning intervention Struyven, K. 2010. Improving the effectiveness of peer
which allows students to respond to feedback received, and note feedback for learning. Learning and Instruction. 20 (4).
observations made during the reviewing process. We deployed the 304–315.
intervention in several AP English Language and Composition [9] Hillocks, G.J. 1982. The Interaction of Instruction,
classrooms and evaluated the impact of the intervention on both Teacher Comment, and Revision in Teaching the
second draft revisions and future writing. Students who used the Composing Process. Research in the Teaching of
revision planning intervention addressed more comments in the English. 16 (3). 261–278.
second drafts of their essays. When the analysis focused on [10] Nguyen, H., Xiong, W., and Litman, D. 2016. Instant
classrooms in which revision planning was a more common Feedback for Increasing the Presence of Solutions in Peer
activity, students who had access to the intervention also wrote Reviews, Proceedings Conference of the North American
higher quality subsequent essays, as judged by subject matter Chapter of the Association for Computational
experts. Linguistics: Demonstrations (NAACL-HLT), San Diego,
There are several promising directions for future work. While we CA.
investigated the amount of feedback to which the students [11] Keen, J. 2010. Strategic revisions in the writing of Year 7
attempted to respond, further investigation is required to students in the UK. The Curriculum Journal. 21 (3).
understand both the quality of the feedback and the quality of the 255–280.
revision. Students may be addressing many comments which are [12] McCutchen, D., Francis, M., and Kerr, S. 1997. Revising
incidental or even detrimental to the quality of the writing, or of for Meaning : Effects of Knowledge and Strategy.
future writing. Secondly, the feedback received within a single Journal of Educational Psychology. 89 (4). 667–676.
review is often focused on short-term improvements for the writing [13] Morris Kindzierski, C.M. 2009. “I Like It the Way It
at hand, rather than long term development. The Ideas for Revision Is!”: Peer-Revision Writing Strategies for Students With
Emotional and Behavioral Disorders. Preventing School [17] Smede, S.D. 2000. Interior Design: Revision as Focus.
Failure: Alternative Education for Children and Youth. The English Journal. 90 (1). 117–121.
54 (1). 51–59. [18] Yagelski, R.P. 1995. The role of classroom context in the
[14] Patchen, M. Peer Review of Writing: Learning From revision strategies of student writers. Research in the
Revision Using Peer Feedback and Reviewing Peers' Teaching of English. 29 (2). 216–238.
Text. Doctoral Dissertation, University of Pittsburgh, [19] Zimmerman, B.J. and Kitsantas, A. 2002. Acquiring
2011. writing revision and self-regulatory skill through
[15] Patthey-chavez, G.G., Matsumara, L., and Valdés, R. observation and emulation. Journal of Educational
2014. Investigating the process approach middle to Psychology. 94 (4). 660–668.
writing instruction in urban schools. Journal of
Adolescent & Adult Literacy. 47 (6). 462–476.
[16] Peterson, S. 2003. Peer Response and Students’
Revisions of Their Narrative Writing. Educational
Studies in language and Literature. 3 239–272.