This paper overviews eleven plagiarism detectors that have been developed and evaluated within PAN'11. We survey the detection approaches developed for the two sub-tasks “external plagiarism detection” and “intrinsic plagiarism detection,” and we report on their detailed evaluation based on the third revised edition of the PAN plagiarism corpus PAN-PC-11.
Copying another author’s text and claiming its authorship is called plagiarism. While research on automatic plagiarism detection has been conducted for decades, the standardized evaluation of plagiarism detection algorithms has a short history [13]. In this regard we have organized three competitions on plagiarism detection, the latest one held in conjunction with the 2011 CLEF conference. This paper overviews the submitted detectors and evaluates their performances. 1.1
Let s = hsplg; dplg; ssrc; dsrci denote a plagiarism case where splg is a passage of document dplg and a plagiarized version of some source passage ssrc in dsrc. Given dplg, the task of a plagiarism detector is to detect s by reporting a corresponding plagiarism detection r = hrplg; dplg; rsrc; d0srci. We say that r detects s iff splg \ rplg 6= ;, ssrc \ rsrc 6= ;, and dsrc = d0src. This task can be tackled with external plagiarism detection as well as with intrinsic plagiarism detection.
Algorithms for external plagiarism detection attempt to detect s by retrieving dsrc from a document collection D (e.g., the web) and by extracting ssrc and splg from dsrc and dplg based on a detailed comparison of the two documents. Algorithms for intrinsic plagiarism detection attempt to detect s by analyzing the writing style of dplg, whereas significant style variations from one passage to another may indicate that splg has been written by a different author than the rest of dplg.
Document Purpose source documents suspicious documents – with plagiarism – without plagiarism 50% To evaluate plagiarism detectors we have developed an evaluation framework consisting of the PAN plagiarism corpus 2011 (PAN-PC-11) and detection performance measures [13]. The framework was already employed in the 1st and 2nd competition on plagiarism detection, and the corpus has been revised for this year. Table 1 gives an overview of important corpus parameters. Notable changes compared to previous versions of the corpus include the significantly larger portion of plagiarism that is obfuscated by paraphrasing or translation, and the addition of manually translated plagiarism. These changes are based on insights gained from last year’s competition, namely, that verbatim plagiarism is detected without problems and that automatically translated plagiarism is detected too easily.
Let S denote the set of plagiarism cases in the corpus, and let R denote the set of detections reported by a plagiarism detector for the suspicious documents. To simplify the notation, a plagiarism case s = hsplg; dplg; ssrc; dsrci, s 2 S, is represented as a set s of references to the characters of dplg and dsrc, forming the passages splg and ssrc. Likewise, a plagiarism detection r 2 R is represented as r. Based on this notation, precision and recall of R under S can be measured as follows [13]: prec(S; R) = 1 jRj r2R
S X j s2S(s u r)j ; jrj r ec(S; R) = 1 jSj s2S
S X j r2R(s u r)j ; jsj where s u r = s \; r iofthredrwetiescet.s s,
Observe that neither precision nor recall account for the fact that plagiarism detectors sometimes report overlapping or multiple detections for a single plagiarism case. This is undesirable, and to address this deficit also a detector’s granularity is quantified g ran(S; R) = where SR S are cases detected by detections in R, and Rs R are detections of s. I.e., SR = fs j s 2 S ^ 9r 2 R : r detects sg and Rs = fr j r 2 R ^ r detects sg. The above measures have been computed for every plagiarism detector that took part in PAN’11; however, they do not allow for an absolute ranking among them. Therefore, the three measures are combined into a single overall score as follows: plagdet(S; R) =
F1 log2(1 + g ran(S; R)) ; where F1 is the equally weighted harmonic mean of precision and recall. 2
This section surveys the intrinsic plagiarism detectors and evaluates their performances. 2.1
Intrinsic plagiarism detection has attracted renewed interest in PAN’11. An analysis of
the submitted notebooks reveals a generic set of building blocks all of which employ
a chunking strategy, a writing style retrieval model, and an outlier detection algorithm;
however, the specifics differ significantly. In all cases, the mentioned building blocks
are arranged within a retrieval process similar to that described by the authors of [
Chunking All of the submitted detectors employ sliding window chunking with chunk
sizes ranging from 200 to 1000 words. The slide stepping of the window ranges from
40 to 500 words. The best performing detectors use chunk sizes of 400 words [9] and
1000 words [
Retrieval Model Retrieval models for intrinsic plagiarism detection are comprised of
a model function that maps texts onto feature representations along with a similarity
measure to compare representations. The submitted detectors use either word-based
features or character-based features: Oberreuter et al. [9] use a word vector including
stop words with tf -weighting, Akiva [
Post-processing With regard to post-processing most detectors merge overlapping and consecutive chunks that have been identified as outliers in order to decrease detection granularity. 2.2
Table 2 shows the detection performances of the aforementioned detectors
according to the detection performance measures plagdet, precision, recall, and granularity.
The overall best performing detector stems from Oberreuter et al. [9]; it outperforms
all other detectors on all except one performance measure. The detector of
Kestemont et al. [
While the baseline detector performs better on medium and long plagiarism cases, the submitted detectors perform better on short and medium length cases. Automatically translated plagiarism is detected better than manually corrected translation plagiarism. Regarding the ratio of plagiarism per document, the picture is not clear, with the first detector performing similar on documents with a hardly and medium ratio of plagiarized text, the second performing better on the latter, and the third performing better on the former. Finally, with regard to document length, a medium length seems to be best for intrinsic plagiarism detection. 2.3
The outstanding performance of the detector of Oberreuter et al. [9] looks very encouraging, but it should be taken with a grain of salt. The detector’s retrieval model quantifies the uniqueness of a word with regard to the whole suspicious document. However, during construction of the PAN-PC-11, plagiarism cases have been inserted into the suspicious documents from randomly chosen source documents, so that no topic overlap between a suspicious document and its sources can be expected. I.e., with a high probability, words have been inserted into the suspicious documents that did not occur beforehand. A retrieval model which builds on computing word uniqueness hence benefits from this construction principle. Moreover, it is surprising that a retrieval model which builds on words instead of writing style features which have been shown in the past to outperform word-based style quantification should perform that well on intrinsic plagiarism detection. Presumably, the PAN’11 performance may not be achieved in different settings.
These results are nonetheless important as they pinpoint a problem with
constructing a corpus for intrinsic plagiarism detection. Randomly inserting text into a document
may preserve writing style, but it obviously doesn’t represent plagiarist behavior, and
it hence opens the door to detection approaches which may not be applicable in
practice. Though, at the time of writing, no better way of constructing a corpus for intrinsic
plagiarism detection evaluation is at hand, this will be an important subject for future
research. Also, the second best performing detector of Kestemont et al. [
This section surveys the external plagiarism detectors and evaluates their performances. 3.1
External plagiarism detection continues to be an important part of the PAN plagiarism detection competition. Several of the plagiarism detectors that have been evaluated this year are enhanced versions of detectors that have been evaluated in previous years. An analysis of the submitted notebooks reveals that the generic retrieval process for external plagiarism detection described at length in the overview papers of PAN’09 and PAN’10 did not change much [11, 12]: For a given suspicious document and a collection of potential source documents, (1) all documents are pre-processed using an indexing pipeline that normalizes the word tokens by removing stop words, stems the remainder, and replaces words with one of their synonyms. Moreover, non-English documents are typically translated to English using Google Translate or other translation services. (2) A set of candidate source documents is retrieved from the collection of source documents, (3) each candidate document is compared in detail to the suspicious document in order to extract similar passages of text, and finally, (4) the extracted passages are post-processed to filter false positive detections, while the remainder is returned to the user as potential plagiarism detections.
While the specifics of the algorithms applied in each of the aforementioned steps
have not changed much, it can be observed that some participants focus on certain
aspects such as runtime performance, cross-language detection, and obfuscation. The
detector of Cooke et al. [
Table 3 shows the detection performances of the detectors that took part in PAN’10;
the table reports the detection performance measures plagdet, precision, recall, and
granularity. The best-performing detector of Grman and Ravas [
The precision performance of the top five detectors is very high, while the recall performances varies from poor to medium, depending on the corpus parameter. In fact, only verbatim plagiarism is detected with high recall. The granularity of the top five detectors is close to 1.0 in most cases, while the remaining detectors show comparably unstable performance characteristics. With regard to the corpus parameters it can be seen that manual obfuscation in terms of paraphrasing and translation is much more difficult to be detected than automatic obfuscation. While the length of a case has a certain influence on detection performance (short cases are less well detected), the document length as well as the ratio of plagiarism per document have no effect on detection performance.
Compared to the performances reported in PAN’09 and PAN’10, a drop in the plagdet performance can be observed in PAN’11. This fact does not indicate a worse detection performance compared to previous years, but should be attributed to an increased detection difficulty: during the construction of the PAN-PC-11 corpus we have lowered the ratio of plagiarism cases that are not obfuscated, while we have increased the number of cases that are manually or automatically obfuscated. Within the previous as well as this competition, it became clear that verbatim plagiarism poses no challenge to detection anymore, so that the high absolute performance values reported may lead to the false conclusion that plagiarism detection in general is close to being solved. By changing the respective corpus parameters, we address this issue; however it forecloses a direct comparison between performances of this year and those of earlier years. 3.3
The external plagiarism detection sub-task of the PAN’11 competition on plagiarism detection has matured in the past three years: in the first year the size of the test corpus posed the biggest difficulty, in the second year the revised corpus introduced new challenges such as manual obfuscation, whereas in the third year the difficulty of the task was increased. Furthermore, many of the plagiarism detectors have been evaluated more than once. Unfortunately, however, some participants chose not to share their approaches, so that some of the achieved performances cannot be replicated or verified. 4
The results of the 3rd international competition on plagiarism detection, PAN’11, can be summarized as follows: 11 plagiarism detectors have been developed from which 7 detectors tackle external plagiarism detection, 2 detectors tackle intrinsic plagiarism detection, and 2 detectors handle both. Five of the detectors have been evaluated the second time, and one for the third time. One of the new detectors dominates all other detectors in terms of detection performance on the third revised version of the PAN plagiarism corpus PAN-PC-11. The corpus features plagiarism cases with an increased level of detection difficulty compared to previous corpus versions.
The lessons learned from the competition include that the portion of our corpus which is dedicated to intrinsic plagiarism detection may be biased: the evaluation may not favor realistic detection approaches over less realistic ones. Moreover, the increased detection difficulty draws a clearer picture of the detection performances of today’s plagiarism detectors.
We would like to thank Yahoo Research for sponsoring the PAN competition for the third time now. We further thank the participants of PAN for their dedicated work without which this event would not be possible. This work was partly funded by the European Commission as part of the WIQEI IRSES project (grant no. 269180) within the FP7 Marie Curie People Framework, by MICINN as part of the TextEnterprise 2.0 project (TIN2009-13391-C04-03) within the Plan I+D+i, and as part of the VLC/CAMPUS Microcluster on Multimodal Interaction in Intelligent Systems. [8] Rao Muhammad Adeel Nawab, Mark Stevenson, and Paul Clough. External Plagiarism Detection using Information Retrieval and Sequence Alignment: Notebook for PAN at CLEF 2011. In Notebook Papers of CLEF 2011 LABs and Workshops, 19-22 September, Amsterdam, The Netherlands, September 2011. [9] Gabriel Oberreuter, Gaston L’Huillier, Sebastián A. Ríos, and Juan D. Velásquez.
Approaches for Intrinsic and External Plagiarism Detection: Notebook for PAN at CLEF 2011. In Notebook Papers of CLEF 2011 LABs and Workshops, 19-22 September, Amsterdam, The Netherlands, September 2011. [10] Yurii Palkovskii, Alexei Belov, and Iryna Muzyka. Using WordNet-based Semantic Similarity Measurement in External Plagiarism Detection: Notebook for PAN at CLEF 2011. In Notebook Papers of CLEF 2011 LABs and Workshops, 19-22 September, Amsterdam, The Netherlands, September 2011. [11] Martin Potthast, Benno Stein, Andreas Eiselt, Alberto Barrón-Cedeño, and Paolo Rosso. Overview of the 1st International Competition on Plagiarism Detection. In Benno Stein, Paolo Rosso, Efstathios Stamatatos, Moshe Koppel, and Eneko Agirre, editors, SEPLN 09 Workshop on Uncovering Plagiarism, Authorship, and Social Software Misuse (PAN 09), pages 1–9. CEUR-WS.org, September 2009.
URL http://ceur-ws.org/Vol-502. [12] Martin Potthast, Alberto Barrón-Cedeño, Andreas Eiselt, Benno Stein, and Paolo Rosso. Overview of the 2nd International Competition on Plagiarism Detection. In Martin Braschler and Donna Harman, editors, Notebook Papers of CLEF 10 LABs and Workshops, September 2010. ISBN 978-88-904810-0-0. [13] Martin Potthast, Benno Stein, Alberto Barrón-Cedeño, and Paolo Rosso. An Evaluation Framework for Plagiarism Detection. In Chu-Ren Huang and Dan Jurafsky, editors, 23rd International Conference on Computational Linguistics (COLING 10), pages 997–1005, Stroudsburg, PA, USA, August 2010.
Association for Computational Linguistics. [14] Sameer Rao, Parth Gupta, Khushboo Singhal, and Prasenjit Majumder. External & Intrinsic Plagiarism Detection: VSM & Discourse Markers based Approach: Notebook for PAN at CLEF 2011. In Notebook Papers of CLEF 2011 LABs and Workshops, 19-22 September, Amsterdam, The Netherlands, September 2011. [15] Diego Antonio Rodríguez Torrejón and José Manuel Martín Ramos.
Crosslingual CoReMo System: Notebook for PAN at CLEF 2011. In Notebook Papers of CLEF 2011 LABs and Workshops, 19-22 September, Amsterdam, The Netherlands, September 2011. [16] Efstathios Stamatatos. Intrinsic Plagiarism Detection Using Character n-gram Profiles. In Benno Stein, Paolo Rosso, Efstathios Stamatatos, Moshe Koppel, and Eneko Agirre, editors, SEPLN 2009 Workshop on Uncovering Plagiarism, Authorship, and Social Software Misuse (PAN 09), pages 38–46. Universidad Politécnica de Valencia and CEUR-WS.org, September 2009. URL http://ceur-ws.org/Vol-502.