=Paper=
{{Paper
|id=Vol-2181/paper-06
|storemode=property
|title=Recognition and Reward System for Peer-Reviewers
|pdfUrl=https://ceur-ws.org/Vol-2181/paper-06.pdf
|volume=Vol-2181
|authors=Zeeshan Jan
|dblpUrl=https://dblp.org/rec/conf/semweb/Jan18
}}
==Recognition and Reward System for Peer-Reviewers==
https://ceur-ws.org/Vol-2181/paper-06.pdf
Recognition and Reward System for
Peer-Reviewers
Zeeshan Jan
KMI - Knowledge Media Institute,
The Open University, UK.
zeeshan.jan@open.ac.uk
Abstract. Peer reviewing plays an important role in academic publish-
ing process that scrutinizes and provides feedback of the scientific work
prior to publication. Peer-reviewers put their efforts in reviewing oth-
ers research work voluntarily, without any expectations of incentives or
rewards. The peer-review process has been criticized for its defects like
slowness, bias and abuse of the process. In this paper, we present a model
to address these issues by using the approach of recording peer-review
data on the blockchain. By using semantic web and linked data technolo-
gies, this system would be able to expose its data, and interact with other
systems. This system will be used to quantify, recognize and incentivize
the peer-reviewing efforts by researchers.
Keywords: Peer-review · Incentive · Linked-data · Blockchain.
1 Problem Statement
Peer-reviewing is the activity when researchers assess the scientific work of other
researchers before publishing. The process of peer-reviewing is questioned and
criticized for the identified defects in it [12]. The slowness, abuse, bias and in-
consistency in the process raise questions about the validity and ’working’ of the
peer-review process.
Researchers spend their valuable time in peer-reviewing that somehow, is less
recognized and quantified by the community. Such efforts of researchers should
be recognized, quantified and incentivized at some public platform.
There is a need of defining the actual and potential requirements of the peer-
review process, along with a model to address the defects of the process as well
as to incentivize peer-reviewers.
2 Relevancy
The modern form of peer-review process used by journals, conferences and pub-
lishers is influenced by the review process adopted by the Royal Society of Ed-
inburgh in 1731, when publications were inspected by a selected group of knowl-
edgeable members [13]. Peer-review is at the heart of the processes of science
[12]. The process of peer-reviewing is an important part of scholarly communica-
tion process and it has been discussed and criticized for its methods, approaches
and defects but yet stands as the backbone of the scientific publication process.
Published research work is piled up when new researchers develop their under-
standing by reading previous studies, identifying research questions and gaps,
discovering new research areas of study and contributing their research work
[14].
Researchers should get incentives for their peer-reviewing efforts, and these
efforts should be quantified at some platform to acknowledge researchers’ peer-
reviewing contributions. Such platform should ensure transparency and fairness
in the process. By bringing improvements in the mechanism of peer-reviewing
and making it transparent, fair and attractive for researchers, it will have a con-
structive impact on research and science. Researchers would be more motivated
to do peer-reviewing actively. This linked data of peer-reviews, comments and
incentives, can be exploited and used by other scholarly applications.
3 Related Work
In this section peer-review patterns, their problems and incentive models for
peer-reviewers are discussed.
3.1 Peer Review Patterns
Peer reviewing practice follows a number of patterns, for instance, in [8], case
studies of six peer-review patterns were presented. However, we focus on Open
Peer review process for the pilot study.
Closed Review: This process is categorized in two ways, single blind and double
blind. Single blind process ensures that identities of reviewers remain unknown
to the authors, while in Double blind process, in addition to the withholding of
peer-reviewers names from authors, the names of the authors are not revealed
to the reviewers. However, peer-reviewers can identify the articles after they are
published.
Open Review: Open peer-review process consists of publishing the manuscripts
along with the reviews they get, to ensure the transparency in peer-reviewing.
In open peer-review, names of the authors and reviewers are mutually revealed,
to avoid the potential bias.
3.2 Defects of Peer-review
Smith [12] mentions about a systematic review of all the available evidence on
peer-review, that concluded ’the practice of peer review is based on faith in its
effects, rather than on facts’ [6]. Stephen Lock [7] as an editor of the BMJ, con-
ducted a study in which only he decided about the publication of a consecutive
series of papers submitted to the journal, he would publish. Lock, then sent the
papers for usual process of peer-review and there was a little difference between
the papers he selected for publication and those selected after peer-review pro-
cess [7]. Smith [12] enlists a number of defects in peer-review process including
being poor at detecting gross defects, almost useful for detecting fraud, slow, ex-
pensive, profligate of academic time, highly subjective, a kind of lottery, prone
to bias and easily abused [12].
Slowness in the process: There are many journals take even more than a year
to review, accept and then publish a paper [12]. That makes peer-review process
as the frustrating phase of publishing scientific work.
Prone to bias: Traditionally, peer-review scores are not made public [15] there-
fore, there are high chances of occurrence of bias. Wenneras finds strong evidence
of gender bias against women in the process of peer-review while awarding fi-
nancial grants [15]. According to the study, peer-reviewers deemed women to be
deficient in scientific competence [15].
Abuse of the system: Peer-review process can be abused at many different
levels. From stealing research ideas to producing harsh review to block the re-
search progress of competitor, many researchers have been victim of the abuse
[12]. Another possible (potential) abuse of the system can be the misuse of the
access over peer-review database, as such systems are centralized. Someone can
easily alter the data and change the decision if they want to abuse the access
over system.
Fake peer-reviews: Haug, explains about the 64 retracted articles from 10 dif-
ferent journals by the publisher Springer. Haug further mentions about a South
Korean researcher Hyung-in Moon, who admitted to have email addresses so
that he could provide peer-reviews of his own articles using those email and fake
identities. Through such editorial checks, more than 250 articles were retracted
because of the fake-reviews, about 15% of the total number of retractions [3].
Publons1 is a platform for peer-reviewers where researchers can add their re-
view records to their profiles by forwarding their review receipt email to Publons.
It also allows journal integrations through a formal partnership of journals with
Publons. This platform is exposed to the potential abuses listed above.
3.3 Incentive
There have been several approaches to incentivising peer-reviewing, some of
which are listed in table 1 below. To improve the peer-review process, punishing
the peer-reviewers is also brought under discussion.
Gropp [2] raised a question that if peer-review is properly incentivized, and
if peer-reviewers are being asked to evaluate the right things? Gropp proposed
an incentive model to filter out the good peer-reviewers who produce thorough
reviews, submit them timely and are responsive to any query sent about the
review. Such peer-reviewers’ name are put into a research funds lottery, to give
them incentive for peer-reviewing.
1
https://publons.com/home/
TheWinnower is a platform that publishes post-publication peer-reviews and
is exploring incentivizing peer-reviewers to highlight their work by elevating the
review report to the level of an original research publication [10].
ReviewerCredits 2 came up with an idea to give credit to peer-reviewers. Peer-
reviewers would have to contact ReviewerCredits which would contact the journal
concerned for verification. After receiving verification, a peer-reviewer’s profile
is credited with ReviewerCredits.
Punishing the peer-reviewer submitting after deadline, is suggested by Hauser
[4]. Hauser opines, if habitual late-reviewers stick to their habit of being late
in submitting the peer-review, another week delay (for their own publication)
should be added as a top-up.
Table 1. Comparison of incentive models
Approach Punishment Incentive Model
Model
Hauser [4] Reviewer’s Reviewers’ articles in priority queue
article in ed- for publication, if accepted.
itorial limbo
for a cer-
tain period.
(even if peer-
reviewers
refuse to
review)
Gropp [2] No Filter good peer-reviewers and put
them in to a lottery fund
ReviewerCredit3 No Credit awards for peer-reviewers’
profiles as reputation indicators.
ScienceMiles No Digital Currency, to be spent on
other platforms, as well as mea-
sured as reputation.
4 Research Questions
Given the problems discussed in section 1, we intend to answer these research
questions:
Q1 : What are the actual and potential requirements of peer-review pro-
cesses? (e.g., open, closed, blind, double-blind, conferences, journals etc.)
Q2 : How are currently implemented processes addressing and failing to ad-
dress these requirements?
Q3 : How does recording data and incentivizing peer-reviewers address and
fail to address these requirements?
Q4 : How can currently implemented processes be compared with the blockchain-
based processes in terms of addressing the requirements answered by Q1 ?
2
https://www.reviewercredits.com/
Q5 : By knowing that the peer-review data would be recorded permanently,
does it affect the researchers’ behavior in terms of:
i Papers they accept
ii Papers they reject
iii Papers they rate
5 Hypotheses
By having above research questions, we derive hypotheses as:
H1 : Blockchain-based data-recording model for peer-reviewing process keeps
the data tamper-free, traceable and immutable.
H2 : Blockchain-based incentivizing model for peer-reviewing process ensures
fair and transparent distribution of rewards.
H3 : The features of immutability and transparency ensure tamper-free and
traceable data. Therefore, researchers would be more careful in terms of accept-
ing, rejecting and rating papers.
6 Preliminary results
We are in a process to annotate the open-review data from ESWC. We have also
deployed Ethereum4 based blockchain in our private network of the institute. We
have written and implemented the SmartContracts for Authors, Reviews and
Submissions from the ESWC-20185 . Our next step is to implement customized
ERC20 Token 6 SmartContracts for researchers.
7 Approach
We plan to implement Ethereum [1] based blockchain that would record peer-
review linked-data. This would allow us to implement an incentive system for
researchers, as blockchains are the only generally-accepted trusted way to have
exchangeable tokens without a central authority.
Annotation (Linked-data)
The approach would be to annotate the peer-review data by reusing Confer-
ence [9] and ScholOnto [11] ontologies. Annotating the data using linked-data
technologies would ensure the integration of third-party applications with the
system.
4
https://www.ethereum.org/
5
https://2018.eswc-conferences.org/program/submissions/
6
https://theethereum.wiki/w/index.php/ERC20_Token_Standard
Blockchain
SmartContract is a piece of code embedded and stored in blockchain network,
that is responsible for managing the transactions and is protected from any
amendments, deletion and tampering [5]. We plan to implement SmartCon-
tract(s) enabling storage of linked-data on the blockchain.
Fig. 1. ScienceMiles Model
8 Evaluation plan
Based on the research questions posed in section 4, the evaluation plan for each
question is:
E1 : Are actual and potential requirements of peer-review process addressed
by currently implemented peer-reviewing processes?
E2 : Does blockchain-based data-recording model fulfill these requirements
and add the promising features (i.e., transparency, immutability, decentraliza-
tion) to the peer-review process?
E3 : Does blockchain-based incentive model for peer-review process ensure
the transparent and fair distribution of awards amongst researchers (peer-reviewers)?
E4 : Comparison between currently implemented peer-review processes and
blockchain-based peer-review process in terms of the number of:
i Papers accepted by a journal/conference.
ii Papers rejected by a journal/conference.
iii Papers rated by peer-reviewers.
9 Reflections
Existing approaches to incentivize peer-reviewers are based on centralized and
manual systems requiring trust in third parties. Apart from the idea of a lottery
fund for researchers, these models offer no real-world value incentives. While
we mentioned about the defects in peer-review process and identified that we
can address these issues by developing a system based on the linked-data and
blockchains technologies that ensure data integration, re-usability of data, secu-
rity and transparency.
Benefits of peer-review process on Blockchain
– Transparency: Blockchain offers transparency in recording transactions in
the network to make them accountable and audit-able. In peer-review sys-
tem, the impact of transparency would strengthen the importance, reliability
and trust in peer-reviews. With distributed database technology, each mem-
ber in the network would be able to lookup, trace and validate any record
of peer-review submissions and incentives given to peer-reviewers. The novel
concept of immutability of data in blockchain would ensure the sanctity of
records and prevent any fabrication or tampering of data.
– Decentralized data: Blockchain follows peer-to-peer network infrastruc-
ture where the network peers(nodes) validate, store and reconcile data. Con-
sensus is a concept of reviewing such transactions and taking decisions about
their validity. By implementing peer-review system on blockchain, each node
of the network would have the same copy of data which makes it persistent,
incorruptible, secure and immutable, in the presence of Consensus. That
safeguards the data from hacking attacks and attempts to manipulate it.
– Trust in peer-review process: Features of blockchain ensure immutabil-
ity, persistence, security and publicly availability of data. These features
would enhance the trust in peer-review process.
– Recognition: With the implementation of peer-review process on the blockchain,
the contributions by peer-reviewers can be quantified as the recognition.
Peer-reviewers would get recognition based on their reviews and rewards
available publicly. Journals and conferences can also announce the best re-
viewers of the year title awards to the researchers.
– Preventing possible abuse: Currently, peer-review systems are imple-
mented by some centralized third-party entities that share the access to data
with conference/journal program committee. The centralized-stored data can
be abused potentially. Implementing peer-review system on blockchain can
help in preventing any possible abuse of authority over a reviews database.
Researchers can be verified through their official (institutional) emails, hence
it would be very difficult to create multiple identity. In case they switch their
institutes (they can add or amend their email addresses) system can detect
the duplicity of a record based on name and date of birth of a researcher.
– Less bias: Immutable reviews and comments are publicly available, so re-
viewers would want to come up with a valid and clear stance about the
article, hence it would help in reducing the bias in the reviewing process if
any.
– Improved quality of the scientific output: Encouraging timely and well-
thought unbiased reviews will improve the quality of the scientific output.
Fair and transparent incentive model can help in increasing motivation for
peer-reviewing.
– Rewards: By participating in the public platform, authors, reviewers and
commentators would be entitled to be awarded by ScienceMiles. These Sci-
enceMiles in the form of a digital currency, are the acknowledgement by the
research community, for the research community.
– Large pool of peer-reviewers: This system would gather a large pool of
well-reputed (verified) peer-reviewers, that would be helpful for journals to
find the suitable peer-reviewers for their publications.
Acknowledgements
I would like to thank Prof. John Domingue and Dr. Allan Third for supervising
and guiding me through my PhD research.
References
1. V. Buterin et al. Ethereum white paper. GitHub repository, 2013.
2. R. E. Gropp, S. Glisson, S. Gallo, and L. Thompson. Peer review: A system under
stress. BioScience, 67(5):407–410, 2017.
3. C. J. Haug. Peer-review fraudhacking the scientific publication process. New
England Journal of Medicine, 373(25):2393–2395, 2015.
4. M. Hauser and E. Fehr. An incentive solution to the peer review problem. PLoS
Biology, 5(4):e107, 2007.
5. M. Iansiti and K. R. Lakhani. The truth about blockchain. Harvard Business
Review, 95(1):118–127, 2017.
6. T. Jefferson, P. Alderson, E. Wager, and F. Davidoff. Effects of editorial peer
review: a systematic review. Jama, 287(21):2784–2786, 2002.
7. S. Lock et al. A difficult balance: editorial peer review in medicinecontinued.
Nuffield Provincial Hospitals Trust, 1985.
8. D. Millard, P. Sinclair, and D. Newman. Peerpigeon: A web application to support
generalised peer review. 2008.
9. A. G. Nuzzolese, A. L. Gentile, V. Presutti, and A. Gangemi. Semantic web
conference ontology - a refactoring solution. In The Semantic Web: ESWC 2016
Satellite Events, volume 9989 of Lecture Notes in Computer Science, pages 84–87.
Springer, 2016.
10. B. Schmidt, A. Deppe, J. Bordier, and T. Ross-Hellauer. Peer review on the move
from closed to open. In ELPUB, pages 91–98, 2016.
11. S. B. Shum, E. Motta, and J. Domingue. Scholonto: an ontology-based digital
library server for research documents and discourse. International Journal on
Digital Libraries, 3(3):237–248, 2000.
12. R. Smith. Peer review: a flawed process at the heart of science and journals.
Journal of the royal society of medicine, 99(4):178–182, 2006.
13. R. Spier. The history of the peer-review process. TRENDS in Biotechnology,
20(8):357–358, 2002.
14. M. Ware. Peer review: An introduction and guide. Bristol: Publishing Research
Consortium, 2013.
15. C. Wenneras and A. Wold. Nepotism and sexism in peer-review. Women, sience
and technology: A reader in feminist science studies, pages 46–52, 2001.