=Paper=
{{Paper
|id=Vol-1633/ws1-paper4
|storemode=property
|title=Teaching Students to Give and To Receive: Improving Disciplinary Writing Through Peer Review
|pdfUrl=https://ceur-ws.org/Vol-1633/ws1-paper4.pdf
|volume=Vol-1633
|authors=Julia Morris,Jennifer Kidd
|dblpUrl=https://dblp.org/rec/conf/edm/MorrisK16
}}
==Teaching Students to Give and To Receive: Improving Disciplinary Writing Through Peer Review==
https://ceur-ws.org/Vol-1633/ws1-paper4.pdf
Teaching Students To Give And To Receive: Improving
Interdisciplinary Writing Through Peer Review
Julia Morris Jennifer Kidd
Old Dominion University Old Dominion University
5115 Hampton Boulevard 5115 Hampton Boulevard
Norfolk, Virginia 23529 Norfolk, Virginia 23529
1 (704) 490 2068 1 (757) 683 3248
Jmorr005@odu.edu Jkidd@odu.edu
ABSTRACT collaboration of six faculty members. These faculty, representing
The context for this study is a multi-disciplinary collaboration of four disciplines, implemented peer review into their classrooms in
six faculty members using peer review in their respective order to improve undergraduate student writing. Students
disciplines with the goal of improved student writing. Faculty engaged in formative peer review and reflected on their
members developed their own assignments and methods for experience in an online survey. The results of the initial fall 2015
implementing peer review, but each followed the same guidelines. study showed variation between classes, indicating the effect of
Students submitted drafts to peers who made comments and used the instructor and assignment for the peer review process.
a rubric to provide formative feedback. The instructors used a The results from the spring 2016 student data show a more
variety of tools to support peer review, including Google Drive, complete picture of how peer review can be implemented to
Blackboard, and Expertiza, a dedicated peer-review system. promote student investment in this autonomous learning process.
Students reflected on the peer review process in an online survey Further, results show that students’ can, and do, recognize that
after each round of peer review. The survey results varied peer review should promote substantive changes in their writing.
considerably between the classes, suggesting the importance of This study’s hypothesis stated that feedback from the fall 2015
the instructor, assignment, and peer review process. There were students would facilitate increased satisfaction with and utility of
also common themes that emerged across courses, such as the the peer review process in spring 2016 students. This paper
common value of giving reviews. This paper examines one describes the progression of responses from education students in
participating faculty member’s fall 2015 and spring 2016 one class between the fall and spring semesters.
education course and how students’ perceptions of peer review
evolved positively across the two semesters.
2. REVIEW OF LITERATURE
Keywords Once considered the “neglected variable in education,” peer
review and student interaction have the potential to transform
Peer review, feedback, student writing, multi-disciplinary writing practices in higher education [1]. Comer et al. discuss how peer-
to-peer interactions in two MOOCs enhance learners’
1. INTRODUCTION understanding of course learning objectives and highlight the
Technology enhanced peer review enables students the value of online learning environments. This is due to the fact that
opportunity to work collaboratively with each other, more than interactions occur almost entirely in written form. Comer et al.’s
ever before. Online peer review systems have increased students’ findings indicate that peer review fosters a networked learning
opportunities to provide and be given expeditious feedback. experience as online interactions require the primary form of
Writers are able to benefit from multiple perspectives more communication to occur through writing, thus improving both
immediately through the various modalities of technology, further course specific and composition skills.
expanding upon present classroom peer review processes. Peer Lui and Sadler [2] also demonstrate the success of technology-
review promotes learning autonomy for students during the infused peer review for higher education students by comparing
writing process. The positive effects of learning for both face-to-face peer review with online peer review. The authors find
reviewers and reviewees, facilitated by advances in classroom that technology greatly enhanced the number of comments, the
technology, have allowed students to improve their writing number of revision-oriented comments, and the number of
through peer review. revisions made by students after engaging the technology-
This study describes the data from one faculty member who enhanced peer review process. However, the incorporation of
participated in this yearlong exploration into student perceptions technology builds upon but does not eradicate pre-existing issues
of peer review. This study describes the second phase of an prevailing in face-to-face peer review. One of the most prevalent
exploration into these perceptions through the multi-disciplinary points discussed is the concept of anonymity. The literature is
varied in its stance on anonymity. Lee [3] discusses how
anonymity is peer review’s most significant inhibitor as it does
not encourage self-regulated learning. Similarly, Huahui et al. [4]
found that non-anonymous peer review partners encourage a
social presence, born of an optimal level of participation and
interaction, which promoted a “more supportive learning
environment” [4, p.812]. However, in contrast, Raes et al. [5]
concludes that increased anonymity can decrease peer pressure
and increase comfortability with peer review. Similarly, Lu and (pre-writing) assignments or they could be complete papers in an
Bol [6] found students participating in anonymous peer review initial state. Reviewers provided formative feedback via
outperformed students in identifiable peer review, and provided comments and a rubric. Authors made changes as desired and
more critical feedback to their peers. With these antithetical submitted a final product. This was graded by either the
conclusions, the best approach to anonymity in peer review is still instructor, or by students via a summative round of peer
undetermined. evaluations.
The instructors used a variety of tools to support peer
The skills required and refined by peer review prove advantageous
review including Google Docs, Blackboard, and Expertiza, a
across classrooms. It is understood that peer review has ventured
peer-review system developed at North Carolina State University.
beyond English and education courses; it is now a regularly
Students reflected on their peer review experience in an
utilized tool in computer programming, engineering,
anonymous online survey with scaled and open-ended items after
environmental science, and business and entrepreneurship courses
each round of peer review. To assess the impact of peer review on
[7,8]. However, as peer review transcends disciplines, it remains
student writing, instructors graded a selection of student
a cooperative effort between students and faculty. Peer review is
submissions before and after the peer reviews using a common
no longer an educational arrangement devised to benefit students
interdisciplinary writing rubric. This enabled them to judge the
and alleviate staff workload, but instead, a “rather complex
quality of student writing and to assess the extent to which
undertaking,” that merges students’ ability to assess and students’
students revised (and hopefully, improved) their writing after
knowledge of course content [7, p. 181]. With the unavoidability
participating in peer reviews. Faculty also completed mid and end
of variability amongst students in any given classroom, faculty
of semester surveys to gauge their perceptions of the costs and
must mediate reviews in order to assure students of reliability.
benefits of peer review. As the data collection from faculty is still
The nature of student perceptions of peer review is limitedly underway, this paper presents the results of the student reflection
studied. Studies at an Australian university found that 90% of survey, focusing on the data from one instructor.
surveyed students expect peer review to be helpful [7]. Half of The instructor discussed in this study has taught
these students expressed anxieties about peer review due to an foundational educational assessment to pre-service teachers for
uncertainty of how to be constructive without seeming too harsh. 10(+) semesters. After implementing peer review during the first
Conversely, some students were worried that reviews they semester of this study and reflecting on the corresponding fall
received would be too nice and thus provide little substance. Post data, changes were made in order to respond to students’ concerns
peer review, the number of students who considered peer review and suggestions for the spring semester. Several changes were
to be helpful dropped to 70%, indicating what Mulder et al. refer implemented to bring about change. Specifically, instead of a
to as the “modest downward shift” in positive perceptions of peer variety of tools being used (i.e. Google Docs, Expertiza, face-to-
review. No study has yet simultaneously considered student face), students conducted all peer review in one consistent tool.
perceptions in conjunction with what feedback is incorporated The assignments were all submitted, peer reviewed, received, and
affects learning outcomes [8]. In a recent study, students’ most meta-reviewed through Expertiza. The instructor also added a
critical impressions found formative peer review to be plagued calibration assignment where students were given the opportunity
with inconsistencies in quality and reliability [6]. As a result, less to simulate peer review, comparing their attempts at review with
than one-third of surveyed first-year students felt they had helpful the instructor’s expert review. Students were also guided through
feedback via peer review, complaining of “unreliable” and the peer review process gradually. Peer review stages were time
“inconsistent credibility.” More than half of students reported restricted meaning students could not work ahead of the current
that they were disillusioned with partners who lacked expertise. state of review (i.e. submission, peer review, and/or meta-review).
These students instead preferred an “expert review” from faculty This is a semester-long course and therefore methods described
or staff. These impressions, according to the authors, suggest that from the fall 2015 semester were amended for the new students
peer review can be effective, as students are looking for ways to enrolled in the spring 2016 semester. The results discussed reflect
improve their artifacts, but further research must be done that the trend in data from fall to the spring semester.
explores more effective implementation. When this study is
complete it will provide new insight into students’ attitudes and
behaviors, helping to delineating what type of feedback students 4. RESULTS
find most helpful and to uncover the processes and procedures
that prompt students to revise their work. This preliminary report 4.1 Peer Review Design
of student attitudes begins this work. This excerpt of the study into student perceptions of peer reviews
discusses results from an education course geared towards pre-
service teachers. Students were asked to participate in multiple
3. METHODOLOGY rounds of formative peer review, culminating in a final round of
Six faculty members agreed to incorporate peer review into their summative assessment where peers graded each other’s developed
undergraduate courses with the intention of improving student lesson. Students were asked to create a digital lesson to teach
writing. Although two of the participants had significant peers about a topic in education. Students completed multiple
experience peer review, the intention was to create a learning stages for this scaffolded assignment, each followed by a round of
community with the faculty learning from and with one another as peer reviews: a 3-paragraph research essay on their topic,
they designed and later revised writing prompts, rubrics, and peer followed by formative peer reviews; an initial draft of their lesson,
review protocols. Each faculty member developed their own followed by formative peer reviews; and, their final lesson,
assignments and peer review process, but followed the same submitted for summative peer review during which students
general guidelines. Students were required to submit a draft(s) on assigned grades to one another. All peer review was conducted
a major writing assignment (instructors could institute a single or through Expertiza and was designed to be anonymous.
multiple rounds of peer review). The drafts could be preliminary
Figure 1: Peer Review Procedures by Course
o Round 1 52% (N=91)
Subject/Course Number Technology Purpose & (N=86) o Round 2 44%
of Process of o Round 2 43% (N=74)
Responses Peer Review (N=71) o Round 3
o Round 3 0% (N= 84)
(N/A) Anonymity
Education 331 Expertiza Three
Anonymity o I knew who
Rounds:
o I knew who reviewed my work
1 & 2 were
reviewed my work Yes 2%
formative; 3
Yes 37% (N=4)
was
(N=58) No 98%
summative
No 63% (N=163)
(N=102) o I knew whose
-Round 1 = 3-
o I knew whose work I reviewed
paragraph
work I reviewed Yes 14%
essay
Yes 55% (N=24)
(N=88) No 86%
-Round 2 =
No 45% (N=143)
complete draft
(N=72)
-Round 3 =
final draft 4.3 Quantitative Survey Results
All reviews The quantitative survey items revealed that education students’
were impressions varied between semesters; however, the overall
anonymous impressions were positive (see table below). Across both
semesters, students found being reviewed and reviewing to be
4.2 Participants beneficial and reported comfortability in both roles. During the
fall semester, the lowest scores were reported when students were
In the fall 2015 semester and spring 2016 semester, students in an
asked whether they would like to see a similar peer review process
education course agreed to participate in the research (see table
implemented by more instructors. In variation, the lowest scores
below). The students were asked to complete an anonymous
in the spring semester were reported when students were asked
online survey after engaging in each round of peer review.
whether they received new insight into their work. Highest scores
Students completed multiple rounds of peer review, and thus
were reported in the fall semester when students were asked if
completed the survey multiple times during the semester.
they intended to change, or had already changed, their work based
Accordingly, the numbers reported below indicate survey
on the peer review process. The spring semester’s students
responses rather than numbers of students.
reported the highest scores when asked about the peer review
Figure 2: Demographics/Logistics of Participants system’s ease of use. The overall rating of the peer review
experience improved between the two semesters.
Fall 2015 Spring 2016
Gender Gender Figure 3: Mean Scores from the Quantitative Survey Items
o Female 88% o Female 82%
Fall Sprin
(N=142) (N=137)
Formative Assessment Survey Questions 2015 g
o Male 11% o Male 18%
(N=17) (N=30) 2016
o Prefer not to
answer <1% (N=1) Student Status N=160 N=16
Student Status o Full-time 92% 7
o Full-time 88% (N=153)
(N=141) o Part-time 8%
o Part-time 12% (N=14) 1. The reviews I received addressed the 3.90 3.77
(N=19) Age questions/ concerns I had about my work.
Age o 18-22 56%
o 18-22 66% (N=93) 2. The reviews I received gave me new 3.86 3.76
(N=104) o 23 or older 44% insight into my work.
o 23 or older 34% (N=74)
(N=56) Delivery Modality
Delivery Modality o Face to Face 3. The reviews I received helped me 3.97 3.81
o Face to Face 73% (N=122) understand what I needed to change about
70% (N=110) o Online 27% my work.
o Online 30% (N=45)
(N=50) Round of Peer Review 4. I trust the feedback I received. 3.87 3.86
Round of Peer Review o Round 1 54%
between the semesters support current literature that students
* 5. I plan to change (or already changed) 4.11 4.20
benefit more meaningfully while acting as the reviewer. The
my work based on the review process.
increase in Question 11 indicates that spring students were more
satisfied with the peer review experience than their fall
* 6. I felt comfortable giving feedback to 3.86 4.11 counterparts. The decrease in Questions 1, 2, 3, 4, and 9 indicate
my peers. that students did not receive the type of helpful feedback they
were anticipating. These results are further enforced by the open-
* 7. I felt comfortable receiving feedback 3.97 4.10 ended responses. Similarly, the increase in Questions 5, 6, 7, 8,
from my peers. 10, and 11 indicate that spring students were more comfortable
with the peer review process than fall students.
* 8. The peer review system was easy to 3.82 4.29
use. 4.4 Summary of Qualitative Data
The survey included five open-ended questions. Students were
asked to consider both what they liked and what they found
9. The reviews I received were beneficial 3.90 3.80 challenging about the peer review process, what kind of feedback
to me. they valued most, and what suggestions they had for peer review.
They also had an opportunity to add additional information not
* 10. The process of reviewing other 3.89 4.17 specifically requested. In both the fall and spring semesters,
students’ work was beneficial to me. education students saw peer review in a positive light. One fall
student observed, “When I am reviewing the work of someone
else, I find my self [sic] noticing things that I need to work on in
* 11. I wish more of my instructors would 3.34 3.81 my own work; I end up with a whole sheet of paper of revisions I
use this type of peer review in their classes. need to make on my own work.” Similarly, a spring semester
student stated, “I was able to understand how to review as well as
when I reviewed what I should improve.”
Students valued the peer review feedback as a form of copy
Overall Evaluation of Peer Review Process editing much more in the fall semester than those students in the
(Composite of 1-11) 42.5 43.7 spring. While students from both groups mentioned the benefits
of input on formatting, grammar, and sentence structures, spring
students appreciated, and were looking for, more substantive
changes. One spring student stated in response to the item that
asked what kind of feedback was most beneficial: “The feedback
* = Increase occurred
about what was good, the feedback about what I could change,
> All means are from a 1-5 scale AND [sic] advice on how to do so.” Another stated, “One reader
stated what she learned from my paper and I think having
someone reflect what they see in your lesson is about is helpful,
The greatest difference between the semesters occurred when making sure what your readers are getting and what you wanted to
students were asked if they would like to see peer review of this communicate are lining up.” When giving feedback, however,
type implemented in other courses (3.34 vs. 3.81). In contrast, the students were uncomfortable giving critical feedback and
largest decrease occurred when students were asked if the reviews questioned their ability to give good feedback because they felt
they received helped them identify necessary changes (3.97 vs. they lacked expertise. Assigning grades during the summative
3.81). Overall, the increased positivity between results in the fall round of review (round 3) was felt to be especially difficult for
and spring semester was greater than the rate of decrease, and the this reason. Students felt it was hard to think of what to say when
comprehensive results increased from the fall to the spring (42.5 they saw the work as being quality work or to not repeat what
vs. 43.7). others had said when such feedback was visible to them.
The consensus impression was positive, with every question Technology concerns were less prevalent in the spring responses.
indicating more strongly agree/agree responses across both Where fall students struggled while experimenting with multiple
semesters. Fall students responded most positively to the items technology platforms (i.e. Google Docs, Expertiza), spring
related to recognizing and making changes in their artifacts (i.e. students found the technology to be an asset. The instructor
“The reviews I received helped me understand what I needed to streamlined the spring peer review assignment to exist entirely in
change…” and “I plan to change my work based on the review Expertiza. Students responded very positively to Expertiza, with
process”). In contrast, spring students rated the utility and one student reporting: “I really enjoy using Expertiza for this type
logistics of the peer review process most positively (i.e. “The of assignment.” Where technology issues were mentioned in
process of reviewing other students’ work was beneficial to me,” more than 60% of open-ended responses from the fall responses,
and “The peer review system was easy to use”). While the technology was only mentioned 6 times in the (4%) 167 responses
responses were predominately positive, a small number of made in the spring data. Instead, the predominant theme of the
students responded very negatively to certain items. Fall students spring data found students to be critical of their peers’ level of
responded most negatively to the idea of using peer review in investment in the peer review process. Students were
other classes. In the spring, students responded most negatively disillusioned about mismatched feedback, where “the chosen
to the helpfulness of the reviews they received (i.e. “The reviews I [ratings] did not always match up with the comments.” Valid,
received gave me new insight into my work”). The differences reliable, constructive, and thorough feedback emerged as
students’ greatest desires, and subsequent disappointments. an average of 4.11 to 4.2 (see Figure 3). Students across the study
Students were neutral/positive about receiving summative scores questioned peer review’s credibility and reliability, which reflects
from one another but were dissatisfied/negative about reviewing previous scholarship [4]. While in the survey information does
unexplained or mismatched feedback that accompanied that grade. present trends towards overall improvement in students’
One comment read, “I didn’t have any problem with being graded perceptions of peer review, it should be acknowledged that not
by other students. I just didn’t like how I never received feedback every question showed improvement. For instance, questions 5
on why they gave me the grade [they] did.” Another student and 9 show contradictory changes. While question 5 showed that
recounted, “I received a good grade but it was not a perfect score more students in the spring 2016 semester made changes to their
and I wish I knew what was lacking…I heard a couple of my work based on the peer review process, question 9 showed that
peers stating the same desire.” fewer students in the spring 2016 found the reviews they received
during peer review to be beneficial. The instructor attributed this
Positive responses from students discussed how students enjoyed
difference to students’ participation in the calibration training
seeing others’ work as this helped “clarify” the assignment.
prior to the first round of peer review. Having practiced giving
Students from both semesters valued constructive criticism more
critical feedback and having seen expert feedback from the
than complimentary “vague” commentary, as well as differing
instructor, spring 2016 students were more critical of their peers’
perspectives on their work. Students from the spring semester felt
feedback and therefore found fewer reviews to be beneficial when
a deeper practical connection to the peer review process after it
compared with fall 2015 students. However, due to this training,
was compared to grading. A practice, or training exercise, was
students found the peer review process to be more helpful in the
added to the spring semester. These students completed a
spring 2016 semester as they were more engaged in the peer
calibration assignment in Expertiza that allowed them to assess
review process. As supported by current literature, students learn
two artifacts against an expert assessment. Students were asked to
more from reviewing their peers than being reviewed. Further
consider what was effective and ineffective in two example
research and analysis should investigate how to foster reliability
lessons plans. These lesson plans were created by students from a
in reviews, or more precisely, how to help students trust the
past semester, and each were representative of noteworthy
feedback they receive from their reviewers. More significant
positive and negative aspects. Students compared their rankings
stakes should be placed on open-ended responses as students so
to an “expert” review completed by the instructor. By evaluating
highly value commentary from their reviewers.
these two lessons, spring students had an advantage over fall
As this snapshot is part of a larger study, the data from
students; they were provided with a model to guide their own
this course will be combined with the arcs from other instructors’
submissions and peer review responses. As pre-service teachers,
courses in order to provide a more thorough understanding of how
spring students were instructed during the assignment
the structure of peer review can effectively promote student
introduction that peer review is a dry run for future students.
investment and learning. This research is ongoing and part of a
While this point was mentioned in the fall semester, peer review
larger investigation into student perceptions of peer review. The
was presented as a more practical skill for these teachers during
effects of these peer review processes on course instructors will
the peer review training process.
also be discussed. Student perceptions of peer review have shown
While the data responses from both semesters were mostly a positive trend as the data from this study is further probed;
positive, the negative commentary evolved from the fall to the students are feeling more and more positively about using peer
spring to show a progressive direction for peer review in this review in their classes. More importantly, students are becoming
instructor’s classroom. Far fewer students were disillusioned with more critical of using the peer review process in order to
peer review during the spring semester. Negative-toned maximize their outcomes of making changes to their writing.
commentary was centered almost entirely on students’ 6. ACKNOLWEDGEMENTS
dissatisfaction with feedback (or, a lack thereof). Students wanted The Peerlogic project is funded by the National Science
“slacker” peers to be held accountable for their failure to provide Foundation under grants 1432347, 1431856, 1432580, 1432690,
substantive and “constructive” feedback. Of the twenty open- and 1431975.
ended responses coded as “negative” (indicating unhappiness,
dissatisfaction, et cetera), 16 mentioned a dissatisfaction with 7. REFERENCES
incomplete, mismatched, or unreliable feedback. A student [1] Comer, D. K., Clark, C. R., & Canelas, D. A. (2014). Writing
responded by stating, “I think that for the reviews, students should to Learn and Learning to Write across the Disciplines: Peer-to-
be graded on completing the comment section. It was really Peer Writing in Introductory-Level MOOCs. International Review
frustrating receiving grades below a [perfect] score and not have Of Research In Open And Distance Learning, 15(5), 26-82.
[sic] an explanation as to why. It made me feel as though my
peers were not actually taking time to assess my work.” [2] Liu, J., & Sadler, R. W. (2003). The effect and affect of peer
review in electronic versus traditional modes on L2 writing.
5. CONCLUSIONS Journal Of English For Academic Purposes, 2193-227.
While there are changes in students’ impressions of peer review [3] Lee, Chun-Yi. (2015). The Effects of Online Peer
between the fall and spring semesters, this study’s hope to find Assessment and Family Entrepreneurship Experience on Students’
students making changes to their writing after engaging in peer Business Planning Performance. Turkish Online Journal Of
review consistently occurs between both semester. Students Educational Technology, 14(1), 123-132.
consistently rated that they planned to make changes to their work [4] Huahui Zhao1, Z., Sullivan, K. H., & Mellenius, I. (2014).
after engaging in the peer review process. The improvements Participation, interaction and social presence: An exploratory
implemented by the course instructor, including training and study of collaboration in online peer review groups. British
streamlined technology, positively affected the students in the Journal Of Educational Technology, 45(5), 807-819.
spring semester, increasing their agreeableness with this goal from
[5] Raes, A. T. (2015). Increasing anonymity in peer assessment [8] Mulder, R. A., Pearce, J. M., & Baik, C. (2014). Peer Review
by using classroom response technology within face-to-face in Higher Education: Student Perceptions before and after
higher education. Studies In Higher Education, 40(1), 178-193. Participation. Active Learning In Higher Education, 15(2), 157-
[6] Lu, R., & Bol, L. (2007). A Comparison of Anonymous 171.
Versus Identifiable e-Peer Review on College Student Writing [9] Lundstrom, K., & Baker, W. (2009). To give is better than to
Performance and the Extent of Critical Feedback. Journal Of receive: The benefits of peer review to the reviewer's own writing.
Interactive Online Learning, 6(2), 100-115. Journal of Second Language Writing, 18(1), 30-43.
[7] Wang, Y., Ai, W., Liang, Y., & Liu, Y. (2015). Toward
Motivating Participants to Assess Peers’ Work More Fairly:
Taking Programing Language Learning as an Example. Journal
Of Educational Computing Research, 52(2), 180-198.