=Paper=
{{Paper
|id=Vol-1633/ws2-paper4
|storemode=property
|title=Weighted Log-odds-ratio, Informative Dirichlet Prior Method to Enhance Peer Review Feedback for Low- and High-scoring College Students in a Required First-year Writing Program
|pdfUrl=https://ceur-ws.org/Vol-1633/ws2-paper4.pdf
|volume=Vol-1633
|authors=Valerie Ross,Mark Liberman,Lan Ngo,Rodger LeGrand
|dblpUrl=https://dblp.org/rec/conf/edm/RossLNL16
}}
==Weighted Log-odds-ratio, Informative Dirichlet Prior Method to Enhance Peer Review Feedback for Low- and High-scoring College Students in a Required First-year Writing Program==
https://ceur-ws.org/Vol-1633/ws2-paper4.pdf
Weighted log-odds-ratio, informative Dirichlet prior
method to compare peer review feedback for top and
bottom quartile college students in a first-year writing
program
Valerie Ross Mark Liberman Lan Ngo Rodger LeGrand
University of Pennsylvania University of Pennsylvania University of Pennsylvania University of Pennsylvania
3808 Walnut Street Williams Hall, Room 619 3808 Walnut Street 3808 Walnut Street
Philadelphia, PA 19104 Philadelphia, PA 19104 Philadelphia, PA 19104 Philadelphia, PA 19104
215-573-2729 215-898-0363 215-573-2729 215-573-2729
vross@writing.upenn.edu myl@cis.upenn.edu lngo@writing.upenn.edu legrand@writing.upenn.ed
u
of comments students receive correlates with the score they give to
ABSTRACT the students’ assignment. Results suggest that students in the lower
quartile receive significantly different commentary--more
The purpose of this paper is to use a weighted log-odds-ratio,
prescriptive and negative--from that given to students in the upper
informative Dirichlet prior method (“bag of words” approach) to
quartile. This has considerable implications, for it suggests that
analyze student comments and scores posed to MyReviewers, a
good writers receive the kind of positive reinforcement from peers
web-based tool designed for collecting student writing as well as
that many in the field of writing studies consider most effective for
their peers’ comments and scores of their colleagues’ drafts. Our
advancing writing skills, while underperforming writers receive the
preliminary findings suggest that students who receive lower scores
sort of commentary from peers that are generally regarded as the
may also be receiving significantly different kinds of feedback that
sort that hinders development of writing skills.
some in the field of writing studies have suggested may have a
negative impact on student learning and motivation. Findings point Writing feedback is typically divided into two categories: direct
to the possibility of identifying the the effectiveness of different and indirect instruction. Direct instruction includes telling,
kinds of feedback on lower and higher performing student writers; suggesting, explaining, and exemplifying (Mackiewicz &
evaluating the impact of feedback on student revision and grading Thompson) and is often contrasted with open problem solving or
practices; and identifying and analyzing symmetries and discovery learning (see, for example, Kirschner, Sweller, & Clark,
asymmetries in teacher and student feedback commentary and 2006). In discussing their analysis of discourse between writing
scores. center tutors and writing students, Mackiewicz and Thompson
(2015) describe directiveness as worrisome but admittedly
1. INTRODUCTION necessary. Though directiveness provides students with essential
Asking students to give writing feedback to their peers is a common knowledge, it may curtail opportunities for learners to generate
practice and may be introduced to students as early as kindergarten solutions on their own and may not foster motivation and curiosity
and continue into graduate school based on the understanding that (Glogger-Frey, Fleischer, Gruny, Kappich, & Renkl,
evaluating and sharing feedback with peers may accelerate the 2015). However, Mackiewicz and Riley’s (2003) analysis of a
learning of the writer and peer reviewer alike. However, while technical editor’s role in providing feedback to writers shows that
much attention has been paid to the quality and nature of feedback indirect strategies are less clear in communicating the hearer’s
given to student writers by teachers and writing tutors, very little obligation to implement the implied directive, thereby potentially
exploration has been done of the impact of the feedback given by creating ambiguity. Research on directiveness in various K-12
students to their peers’ work (Kelly 2016, Poe 2016, Bouzidi, L. & settings has also highlighted the important role of direct instruction
Jaillet A 2009, Stanley 1992). in student learning. Glogger-Frey et al. (2015) studied the effect of
an open problem solving approach (i.e. indirect approach) to
This paper represents a preliminary investigation of data gathered teaching physics to eighth grade students. Contrary to their
as part of a larger, cross-institutional study of peer review of writing expectation that open problem solving would cultivate knowledge
assignments in undergraduate courses. Employing the web-based transfer, they found that the students were less prepared for learning
tool MyReviewers we collected collected student writing, and transfer in physics than students who had received direct
commentary, and scores from one semester of a first-year writing- instruction. Their findings echo Kirschner et al.’s observation that
in-the-disciplines program at the University of Pennsylvania. We there is little research to support the effectiveness of solely using
used a simple “bag of words” approach to explore whether the type minimal guidance when teaching. As such, Mackiewicz and
Thompson (2015) suggest a balance between directness and
indirectness in providing writing support to students.
In the context of writing center tutoring, Mackiewicz and
Thompson (2015) suggest that motivation plays a key role in the
amount of time and effort that students devote to writing
tasks. They explain the importance of motivational scaffolding
strategies in encouraging students through “praise, assurances of
caring, and statements reinforcing student writers’ ownership of 2. DATA SET
their work” (Mackiewicz and Thompson, 2015, p. 5). Drawing on
studies by Harris (1992) and Lunsford (1991), Mackiewicz and The Critical Writing Program has over the past decade developed
Thompson (2015) particularly emphasize the effectiveness of and refined a genre-, discourse- and discipline-based shared
helping students to maintain control of their own writing, and their curriculum for introducing students to what is called "authentic"
research underscores tutors’ use of motivational scaffolding in writing situations that involve real genres, audiences, motives, and
fostering students’ ability to monitor their own learning. However, subject knowledge, as well as introduce and provide students with
the effect of motivational scaffolding may depend on various practice in using a shared vocabulary and concepts about writing,
factors, including a student’s self-efficacy level. For example, from knowledge domains such as genre and process to more
Boyer, Phillips, and Wallis (2008) examined tutorial dialogue in the specific aspects of rhetorical analysis and production. The
context of computer science learning and found that direct curriculum emphasizes peer review and reflection throughout.
encouragement appeared to aid students with low self-efficacy,
The data set consists of the work of 1,183 undergraduates, mostly
though it may not have been helpful for high self-efficacy
first-year, who completed a writing seminar at the University of
learners. As Boyer et al. (2008) suggest, balancing motivational
Pennsylvania in Spring 2016. This data set includes up to five
scaffolding and cognitive scaffolding--which encourages students
drafts of a literature review as well as the peer reviews each draft
to reflect on their own thinking and reasoning (Mackiewicz &
received. Peer reviews consist of rubric-guided comments and
Thompson, 2015)--remains an issue to be studied.
numeric scores. Most drafts are accompanied by the student
Though students may like to receive positive feedback, including writer’s pre- or post-outline that provides a rhetorical analysis as
praise, research highlights the complexity of feedback in terms of well as line of reasoning for the draft. All drafts receive from one
circumstance and effect. Straub (1997) surveyed 142 first-year to six peer reviews (comments and scores) as well as comments and
college writing students to investigate their perceptions about scores from their instructor. In addition, most students will provide
teacher comments on their writing and found that students preferred a revision plan that responds to the feedback they have received
praise, even when it was merely in the form of the word “good” from their peers and instructor.
adjacent to criticism. However, students most preferred praise that
was accompanied by reasons for the positive evaluation (Straub All undergraduate students at Penn across the four undergraduate
1997). Hattie and Timperley (2007) reviewed research, including schools--College of Arts & Sciences, Wharton, Engineering, and
meta-analyses, on feedback and consider it to be one of the Nursing--are required to complete a writing seminar. Penn’s
strongest influences on learning and achievement. According to writing seminars are administered by the Critical Writing Program;
their review, positive feedback may increase a student’s persistence most seminars within the program are situated within a specific
and, for high self-efficacy students, positive feedback builds their discipline, bounded by a particular, discipline-based inquiry, and
ability to cope with future negative feedback (Deci, Koestner, & taught by a Ph.D. in that discipline who frequently is engaged in
Ryan,1999; Hattie & Timperley, 2007; Swann, Pelham, & the line of inquiry focused upon in the seminar. Thus the topics and
Chidester; 1988). On the other hand, students with a low level of disciplines vary and along these lines while all students are
self-efficacy may react to positive feedback by avoiding tasks to assigned the genre of the literature review, we recognize that
limit the risk of receiving future negative feedback (Hattie & literature reviews written in the bench sciences are often
Timperley, 2007). Research on second language learning indicates substantially different form those written in, say, the
that low-performing students may continue to underperform if they humanities. Some instructors teach two sections of the same
are consistently given positive feedback rather than information on topic. In addition to discipline-based seminars, we have two sets
how to move forward (Hiver, 2014). of seminars that share a single topic across disciplines: Craft of
Prose (14 sections representing 161 students) and Upper Division
Negative feedback, which is equally as complicated as positive seminars (8 sections representing 104 students). Students,
feedback, may either hinder or bolster learning, depending on the sometimes with the help of their advisors, choose the type of
student and context. High self-efficacy learners view their seminar that best suits their needs, including their self-assessment
performance optimistically, and therefore, may seek negative of their competence and confidence as writers. Craft of Prose
feedback to outperform on tasks (Hattie & Timperley, 2007). For seminars are designed for students who may have less preparation
low self-efficacy students, disconfirmation of their performance in writing or have confidence issues or other concerns about
may adversely impact their motivation and future performance writing. Upper Division seminars are designed for upperclassmen
(Brockner, Derr, & Laing, 1987; Hattie & Timperley, 2007; and transfer students who were uncomfortable in seminars designed
Moreland & Sweeney, 1984). Negative feedback from either a mainly for freshmen. We also have single topic Global English
teacher or peer may hurt a student’s confidence. After surveying classes for international students who are concerned with learning
200 students of English as a foreign language, Kaivanpanah, Alavi, how to write for an American academic audience. For more
and Sepehrinia (2015) note that negative feedback from classmates information on our directed self-placement criteria, visit:
may be confusing and harmful to a student’s confidence. However, http://writing.upenn.edu/critical/seminars/choosing_the_right_sem
disconfirmation of performance may be welcomed if presented in inar.php
terms of guidance (Straub, 1997). A study by Muis, Ranellucci,
Trevors, and Duffy (2015) emphasizes the complex nature of The specific data in this set includes peer review scores and
negative feedback and its impact. In examining the attitudes, comments produced by 1,183 undergraduates enrolled in 90 writing
emotions, engagement, and learning outcomes of kindergarten seminars during the spring 2016 semester. The Excel file housing
students who received immediate feedback from a literacy learning the data is organized into 19 columns and 14,010 rows. The column
app on iPad, the researchers had expected negative evaluative headings include:
feedback to decrease enjoyment and increase boredom (Pekrun,
2006). Ultimately, results from their study demonstrated that the
impact of negative feedback on the students was mixed.
Column Definition Comment on Peer reviewer's written assessment of student
Heading Cognitive and writer's knowledge of writing and rhetorical
Heuristic awareness.
Class Code Signifies the discipline for the writing Processes
seminar topic and the course title. The data
set includes writing seminars in 21 Comment on Peer reviewer's written assessment of student
disciplines. Invention writer's creativity, novelty, and inventiveness
in what they select to persuade their target
Section Unique numerical identifier for each writing audience.
Number seminar.
Comment on Peer reviewer's written assessment of student
Instructor Full name of the course instructor. Reasoning writer's reasonableness and logical
coherence.
Date Date on which a peer review was completed.
Comment on Peer reviewer's written assessment of student
Project The writing assignment and draft number. Presentation writer's ability to produce voice, vocabulary,
Draft 1 of the literature review was a one-on- syntax, sentence structure, punctuation, tone,
one review. Draft 2 was a multiple-reciprocal source handling, etc., appropriate to the
peer review with 1-6 peers. Draft three of the genre and audience.
literature review was a multiple reciprocal
review. Drafts 4 and 5 were optional. The General Peer reviewer's final comments, insights, and
first draft of the public argument peer review Comments observations of student writer's writing.
was one-on-one, and the second draft was
multiple-reciprocal. The students also Combined Compiles the peer reviewer’s written
completed a multiple-reciprocal peer review assessments of Comment on Cognitive and
for their final portfolio drafts. Heuristic Processes, Comments on Invention,
Comments on Reasoning, Comments on
Student Name Full name of the student writer (anonymized) Presentation, and General Comments into
one field.
Grader Name Full name of the student peer reviewer
(anonymized)
2.1 Rubric
Rubric Score Represents the total score for across the 4 Students are given a detailed rubric, the same one used by
scoring categories identified in our rubric of instructors in our program to assess individual students in their
Cognitive and Heuristic Processes, Invention, classes as well as to assess mid-term and final portfolios. The
Reasoning, and Presentation (see below). rubric acts as a guide for formative as well as, at semester’s end,
summative assessment.
Final Grade The letter grade conversion of the cumulative Cognition/Metacognition: Knowledge of Writing
rubric score. • Recognizes the purpose of the assignment
• Conceives of a procedure for fulfilling it
Grade Peer reviewer's numerical assessment of • Perceives the problem(s) to be solved in the assignment
Cognitive and student writer's knowledge of writing and • Follows directions through all stages of the assignment
Heuristic rhetorical awareness. • Able to detect flaws in reasoning in one’s own or other’s
Processes reasoning (outlines and peer reviews)
• Able to identify and evaluate (in plan, outlines, peer
Grade Peer reviewer's numerical assessment of reviews, cover letters, other artifacts):
Invention student writer's novelty and persuasiveness o Rhetorical Strategies
for a targeted audience. o Audience
o Purpose
Grade Peer reviewer's numerical assessment of o Genre
Reasoning student writer's reasonableness and logical o Plan/Arrangement
coherence. o Complex Synthesis
o Presentation
Grade Peer reviewer's numerical assessment of
Presentation student writer's ability to produce voice,
Invention: Idea/Audience (test of novelty, creativity,
vocabulary, syntax, sentence structure,
persuasion)
punctuation, and tone appropriate to the
• Selection of an appropriate and engaging subject within
genre and audience.
the topic
• Ability to select and work successful within a genre
• Selection of an appropriate proposition and reasons to
support it, attuned to the audience and purpose
• Selection of the appropriate amount and type of evidence quartiles, we used the “algorithm from section 3.5.1 of Monroe et
and materials to support the proposition, attuned to the al. 2008.
audience and purpose
This method, originally developed for a study of political writing,
• Arrangement and style attuned to the audience, purpose,
starts with a simple ratio of estimated word frequencies in two
and genre, including ability to evaluate the strength of
collections of text. The problem is that when the overall frequency
reasons and evidence
of a word is low, so is our confidence that the ratio is not a statistical
• Identification of shared premises to enable an effective accident -- and so is the value of that word as a predictive marker
introduction and conclusion of the distinction under study. Thus in this collection, the word
• Ability to grasp feedback or detect problems with judged occurs five times in the bottom-quartile reviews, and only
invention and revise accordingly once in the top quartile; given the different overall word counts in
• Ability to vary voice and style to accommodate different the two groups, the maximum-likelihood estimate is that judged is
audiences and genres about 4 times as common in bottom-quartile reviews as in top-
quartile reviews. But we can’t be very confident that in the next
Reasoning: Development/Coherence (test of reasonableness) batch of reviews, this ration might not be quite different, or even
• Creation or selection of an appropriately justificatory or reversed. And in any case, judged doesn’t occur often enough to be
explanatory proposition a very strong indicator of a reviewer’s sentiment.
• Creation or selection of reasons that directly support the In contrast, the word should occurs 3,780 times in the bottom-
proposition quartile comments, and 1,914 times in the top-quartile comments.
• Selection of evidence that confirms, illuminates or Allowing for the groups’ overall word counts, this tells us that the
otherwise develops the reasons frequency of should is about 1.5 times greater in the bottom-quartile
• Ability to test argument through strategies of comments than in the top-quartile comments. But in this case, the
counterargument overall frequency is high enough that we can be fairly confident
• Demonstration of logical coherence: all reasons support that should will also be about 50% more frequent in the low-quartile
the proposition, all evidence supports the reasons, and to comments in next semester’s sample -- and should is common
the extent possible, reasons do not contradict each other enough to be a useful indicator of overall review sentiment.
• Demonstration of semantic coherence: sentences and
paragraphs stick together In order to deal with these issues, the cited method shrinks the odds
ratio for each word based on a factor derived from a simple
statistical estimate of the process generating the counts, along with
Presentation an estimate of that word’s overall frequency in a relevant more
• Ability to produce a voice and style appropriate to the general source. The result is a number, the “weighted log-odds
genre and audience ratio”, that we can use to rank words according to their apparent
• Control of vocabulary, syntax, sentence structure, affinity for one text sample or the other.
punctuation, tone
• Ability to integrate rhetorical strategies and sources so 4. PRELIMINARY RESULTS
that they create a consistency of style appropriate to the
genre and audience The bottom quartile has more words per (combined) comment than
• Demonstrated ability to proofread and polish work for an the top quartile: 336 vs. 249.
outside reader
• Creation and use of grammar checklist to identify context
and patterns of error in mechanics and usage, as well as
to correct them
• Appropriate formatting, citation, documentation of
sources
3. METHOD
We exclude peer reviews where the numerical Rubric Score is
listed as 0, since many of these appear to be cases where the score
is simply missing, although the reviewer evaluated the draft
positively. We also exclude reviews where the score was greater
than 0 but less than 2 out of 4 -- these represent evaluations with
failing grades, and in such cases the reviewers often didn’t bother
to supply detailed comments. Among the remaining examples, the
bottom quartile includes 3046 reviews with scores between 2 and
3.3 out of 4, and the top quartile includes 3054 reviews with scores
above 3.78. The combined comments in the bottom-quartile
reviews comprise 1,022,709 words, and the combined comments in
the top-quartile reviews comprise 759,637 words.
In order to evaluate the degree of association between individual
words* [FN: here “word” simply means a string of alphabetic
letters set off by space, punctuation, or other symbols] and score
The words most reliably associated with the bottom quartile and 26778 26183.4 23808 31341.3 -8.526
include:
is 20103 19656.6 16680 21957.9 -4.52
Word Low Low Q High High Q Weighted very 3391 3315.7 5269 6936.21 -14.319
Q Freq Q Freq Log
Count Per Count Per Odds well 3474 3396.86 4763 6270.1 -11.643
Million Million Ratio
was 3185 3114.28 3842 5057.68 -8.508
be 10219 9992.09 5710 7516.75 7.261 good 3222 3150.46 3169 4171.73 -4.742
sentence 9278 9071.98 5092 6703.2 7.221 topic 2738 2677.2 2751 3621.47 -4.619
more 7664 7493.82 3505 4614.05 10.11 piece 2647 2588.22 2591 3410.84 -4.236
paragraph 7001 6845.54 3443 4532.43 8.445 clear 2206 2157.02 2211 2910.6 -4.13
not 6424 6281.36 3516 4628.53 6.309 all 1773 1733.63 2083 2682.86 -5.637
but 5123 5009.25 2742 3609.62 5.949 job 1440 1408.03 1984 2611.77 -7.544
should 3780 3696.07 1914 2519.62 5.687 great 1149 1123.49 1811 2384.03 -8.504
some 2984 2917.74 1529 2012.8 4.925 really 1330 1300.47 1682 2214.22 -6.091
however 2701 2641.02 1255 1652.1 5.617 interesting 1447 1414.87 1653 2176.04 -4.985
than 1938 1894.97 945 1244.02 4.536 easy 676 660.99 1096 1442.79 -6.835
seems 1719 1680.83 720 947.821 5.626 strong 824 805.703 1091 1436.21 -5.281
sure 1268 1239.84 549 722.714 4.78 read 906 885.882 1059 1394.09 -4.177
rather 1052 1028.64 425 559.478 4.708 written 593 579.833 770 1013.64 -4.303
try 888 868.282 316 415.988 4.962 liked 130 127.113 324 426.52 -5.147
needs 793 775.392 253 333.054 5.439 enjoyed 99 96.8017 257 338.319 -4.64
media 731 714.768 208 273.815 5.235 picasso 27 26.4005 112 147.439 -4.294
pass 300 293.339 35 46.0746 5.514 twins 1 0.97779 111 146.122 -5.658
5
chaplin 205 200.448 19 25.0119 4.765
identical 5 4.88898 76 100.048 -4.47
mid 168 164.27 8 10.5313 4.835
poincare 0 0 63 82.9343 -4.81
The words most reliably associated with the top quartile
include: There's obviously some mixture of content with commentary
here, but overall it makes sense -- most if not all of the content
Word Low Low Q High High Weighted admixture could be removed by limiting the list to words that
Q Freq Q Q Log Odds occur at least at a rate of 100 per million in both sets as well as
Count Per Count Freq Ratio removing words that are related to the specific content of a
Million Per particular seminar, such as “Picasso.”
Million
"But" is the 6th most negative word:
but 5123 (5009.25) 2742 (3609.62) 5.949
the 71418 69832.2 56903 74908.1 -5.51
The evaluations in the top quartile by rubric score are more
heavily weighted with evaluative words of all types. The
evaluations in the lower quartile are more heavily weighted with
prescriptive terms such as “should,” “try,” “more,” “needs, fits 2.419
“unnecessary.” One symptom: on the negative side, there's just
one evaluative word that's both reasonably common overall
(frequency > 15/100k) and more than twice as common in the strong 2.400
lower quartile of evaluations: "unclear". The rest of that end of
the list of the lower quartile includes:
really 2.292
WORD RATIO Q4/Q1
nicely 2.251
unclear 2.004
convincing 2.211
incorrect 1.969
presentation 2.155
unnecessary 1.825
persuasive 2.122
needs 1.729
coherent 2.118
clearer 1.688
At the upper end of the quartile, more than 20 positive terms are engaging 2.111
employed:
interesting 2.071
WORD RATIO Q4/Q1
easy 2.939 consistent 1.983
great 2.857 supports 1.949
very 2.816 clearly 1.932
helps 1.927
nice 2.716
flows 2.553 appropriate 1.925
logically 2.547 This may be because the Q4 (lower-scoring) evaluations are full of
specific complaints/suggestions such that even the negative
evaluative words are diluted in frequency.
organized 2.500
4. FUTURE DIRECTIONS
job 2.497 Our next steps will include identifying cases where the commentary
and grade are asymmetrical. We will also analyze instructor
commentary and scores to see if similar patterns obtain. Of
well 2.485 particular interest will be to explore correlations between the types
and length of commentary as well as scores from peers in relation
to improvement in drafts and final scores. We anticipate a corpus
supported 2.456 of several thousand papers from our own program as well as from
the other institutions participating in a broader NSF-funded study
of peer review: USF, MIT, UCNS, and Dartmouth, and will engage
in a range of corpus-based approaches to text analysis. We hope to
contribute to research on the effectiveness of different kinds of
feedback, particularly as concerns struggling writers.
5. ACKNOWLEDGMENTS [10] Kelly, L. Effectivenss of Guided Peer Review of Student
Essays in a Large Undergraduate Biology
This research is supported by the National Science Foundation Course. International Journal of Teaching and Learning
under Award #1544239, “Collaborative Research: The Role of in Higher Education 2015, Volume 27, Number 1, 56-68
Instructor and Peer Feedback in Improving the Cognitive, DOI= http://www.isetl.org/ijtlhe/ ISSN 1812-9129
Interpersonal, and Interpersonal Competencies of Student Writers
in STEM courses. Our thanks for the vital contributions of Joe
[11] Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why
Moxley, Principal Investigator, Alex Rudniy and the USF team
minimal guidance during instruction does not work: an
Rajeev Reddy Rachamalla, Dat Ba Le, Brooke Downey, and
analysis of the failure of constructivist, discovery,
Natalie Kass; Norbert Elliot, Professor Emeritus of English at New
problem-based experiential, and inquiry-based
Jersey Institute of Technology and our NSF grant consultant;
teaching. Educational Psychologist, 41(2), 75–86.
fellow investigators Chris Anson, Suzanne Lane, and Christiane
http://dx.doi.org/10.1207/s15326985ep4102_1
Donahue; and Brighid Kelly, who solves all problems, big and
small.
[12] Lunsford, A. (1991). Collaboration, control and the idea
6. REFERENCES of a writing center. The Writing Center Journal, 12(1), 3-
10. https://www.ebscohost.com/academic/education-source
[1] Boyer, K. E., Phillips, R., Wallis, M., Vouk, M., &
Lester, J. (2008). Balancing cognitive and motivational [13] Mackiewicz, J., & Thompson, I. K. (2015). Talk about
scaffolding in tutorial dialogue. Intelligent Tutoring writing: The tutoring strategies of experienced writing
Systems, 5091, 239-249. DOI= https://link.springer.com center tutors. New York, NY and London, UK:
Routledge.
[2] Bouzidi, L., & Jaillet, A. (2009). Can Online Peer
Assessment be Trusted?.Educational Technology & Society [14] Moreland, R. L., & Sweeney, P. D. (1984). Self-
12 (4), 257–268. expectancies and reactions to evalua-tions of personal
performance. Journal of Personality, 52(2), 156-176.
[3] Brockner, J., Derr, W. R., & Laing, W. N. (1987). Self- DOI=10.1111/j.1467-6494.1984.tb00350.x
esteem and reactions to negative feedback: Towards
greater generalizability. Journal of Research in [15] Muis, K. R., Ranellucci, J., Trevors, G., Duffy, M. C.
Personality, 21(3), 318-334. DOI= 10.1016/0092- (2015). The effects of technology-mediated immediate
6566(87)90014-6 feedback on kindergarten students’ attitudes, emotions,
engagement and learning outcomes during literacy skills
[4] Deci, E. L., Koestner, R., & Ryan, M. R. (1999). A meta- development. Learning and Instruction, 38, 1-13.
analytic review of experiments examining the effects of DOI=http://dx.doi.org/10.1016/j.learninstruc.2015.02.001
extrinsic rewards on intrinsic motivation. Psychological
Bulletin ,125(6), 627-668. DOI= 10.1037/0033- [16] Pekrun, R. (2006). The control-value theory of achievement
2909.125.6.627 emotions: Assumptions, corollaries, and implications for
educational research and practice. Educational Psychology
[5] Glogger-Frey, Fleischer, C, Gruny, L., Kappich, J., & Renkl, Review, 18(4), 315-341. DOI= 10.1007/s10648-006-9029-9
A. (2015). Inventing a solution and studying a worked
solution prepare differently for learning from direct [17] Poe, S. and Emily O. Gravett (2016) Acknowledging
instruction. Learning and Instruction, 39, 72-87. Students' Collaborations through Peer Review: A
DOI=10.1016/j.learninstruc.2015.05.001 Footnoting Practice, College Teaching, 64:2, 73-83, DOI=
10.1080/87567555.2015.1094441.http://dx.doi.org/10.1080/
[6] Harris, M. (1992). Collaboration is not collaboration is not 87567555.2015.1094441 Published online: 08 Mar 2016.
collaboration: Writing center tutorials vs. peer-response
groups. College Composition and Communication, 43(3), [18] Stanley, Coaching Student Writers to Be Effective Peer
369-383. DOI= https://www.jstor.org Evaluators, JOURNAL OF SECOND LANGUAGE
WRITING, 1(3), 217-233 (1992)
[7] Hattie, J., & Timperley, H. (2007). The power of
feedback. Review of Educational Research, 77(1), 81-112.
[19] Straub, R. (1997). Students’ reactions to teacher co9ments:
DOI= 10.3102/003465430298487
An exploratory study. Research in the Teaching of
English, 31(1), 91-119. DOI=
[8] Hiver, P. (2014). Attractor states. In Z. Dornyei, A.
http://www.jstor.org/stable/40171265
Henry, & P.D. MacIntyre (Eds.), Motivational dynamics
in language learning (pp. 20-28). Multilingual Matters:
Bristol, UK. [20] Swann, W. B., Pelham, B. W., & Chidester, T. (1988).
Change through paradox: Using self-verification to alter
[9] Kaivanpanah, S., Alavi, S. M., & Sepehrinia, S. (2015). beliefs. Journal of Personality and Social Psychology,
The Language Learning Journal, 43(1), 74-93. DOI= 54(2), 268-273. DOI= 10.1037/0022-3514.54.2.268
10.1080/09571736.2012.705571