This paper overviews 18 plagiarism detectors that have been developed and evaluated within PAN'10. We start with a unified retrieval process that summarizes the best practices employed this year. Then, the detectors' performances are evaluated in detail, highlighting several important aspects of plagiarism detection, such as obfuscation, intrinsic vs. external plagiarism, and plagiarism case length. Finally, all results are compared to those of last year's competition.
Research and development on automatic plagiarism detection is the prominent topic in the broad field of text reuse studies. The evaluation of plagiarism detectors, however, is still in its infancy: recently the first standardized evaluation framework for plagiarism detection has been published [17]. Using this framework the 2nd PAN competition on plagiarism detection was held in conjunction with the 2010 CLEF conference. Altogether 18 groups from all over the world developed plagiarism detectors for PAN, which is 5 more than in last year’s competition [16]; 5 groups attended for the second time. In this paper we overview the participants’ detection approaches in a comparative manner, and we report on the evaluation of their detection performances. 1.1
We define a plagiarism case s = hsplg, dplg, ssrc, dsrci as a 4-tuple which consists of a passage splg in a document dplg that is the plagiarized version of some source passage ssrc in dsrc. When given dplg, the task of a plagiarism detector is to detect s, say, by reporting a plagiarism detection r = hrplg, dplg, rsrc, d′srci which consists of an allegedly plagiarized passage rplg in dplg and its source rsrc in d′src, and which approximates s as closely as possible. We say that r detects s iff splg ∩ rplg 6= ∅, ssrc ∩ rsrc 6= ∅, and dsrc = d′src. To accomplish this task, the plagiarism detector can resort to two strategies: external plagiarism detection and intrinsic plagiarism detection.
In external plagiarism detection, it is assumed that the source document dsrc for a given plagiarized document dplg can be found in a document collection D, such as the Web. Typically, plagiarism detection then divides into three steps [21]: first, a set of candidate source documents Dsrc is retrieved from D, where ideally |Dsrc| ≪ |D| to speed up subsequent computations. Second, each candidate dsrc ∈ Dsrc is compared in detail with dplg, and a plagiarism detection r is reported if two highly similar passages rplg and rsrc are identified between dplg and dsrc. Third, the set R of reported detections is post-processed to filter out false positives.
In intrinsic plagiarism detection, the plagiarism detector attempts to detect
plagiarized passages solely based on information extracted from dplg [
To evaluate a plagiarism detector we employ our recently published evaluation framework for plagiarism detection, which comprises the PAN plagiarism corpus 2010, PANPC-10, and three plagiarism detection performance measures [17]. The plagiarism detector is asked to detect all plagiarism cases in the corpus, and then the accuracy of the detections is measured.
During construction of the PAN-PC-10, a number of different parameters have been varied in order to create a high diversity of plagiarism cases. Table 1 gives an overview: the corpus is divided into documents suspicious of plagiarism, and potential source documents. Note that only a subset of the suspicious documents actually contains plagiarism cases, and that for some cases the sources are unavailable. Also, the fraction of plagiarism per document and the document lengths have been varied. As for the plagiarism cases, one of their most salient properties is whether and how they have been obfuscated; i.e., real plagiarists rewrite their source passages in order to make detecting them more difficult. A variety of obfuscation strategies have been employed in our corpus, including artificial (automatic) obfuscation, simulated (manual) obfuscation, and automatic translation from German and Spanish to English. Besides the obfuscation type also the length of the plagiarism cases has been varied, and the fact whether or not the topic of a plagiarized document matches that of the source document.
The performance of a plagiarism detector is quantified by the well-known measures precision and recall, supplemented by a third measure called granularity. Let S denote the set of plagiarism cases in the suspicious documents of the corpus, and let R denote the set of plagiarism detections the detector reports for these documents. To simplify the notation, a plagiarism case s = hsplg, dplg, ssrc, dsrci, s ∈ S, is represented as a set s of references to the characters of dplg and dsrc that form the passages splg and ssrc. Likewise, a plagiarism detection r ∈ R is represented as r. Based on these representations, the precision and the recall of R under S can be measured micro-averaged (mic) and macro-averaged (mac): precmic(S, R) = r ecmic(S, R) = | S(s,r)∈(S×R)(s ⊓ r)|
| Sr∈R r| | S(s,r)∈(S×R)(s ⊓ r)| | Ss∈S s| ; ; where s ⊓ r = precmac(S, R) =
r ecmac(S, R) = s ∩ r if r detects s, ∅ otherwise.
1 X | Ss∈S(s ⊓ r)| |R| r∈R |r| 1 X | Sr∈R(s ⊓ r)| |S| s∈S |s| ; ;
Precision and recall do not account for the fact that plagiarism detectors sometimes report overlapping or multiple detections for a single plagiarism case. This is clearly undesirable, and to address this deficit also a detector’s granularity is quantified as follows: gran(S, R) = 1
X |Rs|, |SR| s∈SR where SR ⊆ S are cases detected by detections in R, and Rs ⊆ R are detections of s. I.e., SR = {s | s ∈ S ∧ ∃r ∈ R : r detects s} and Rs = {r | r ∈ R ∧ r detects s}.
The above measures are computed for each plagiarism detector; however, they do not allow for an absolute ranking among detectors. Therefore, the three measures are combined into a single, overall score as follows: plagdet(S, R) =
F1 log2(1 + gran(S, R)) where F1 is the equally-weighted harmonic mean of precision and recall. 2
This section surveys the plagiarism detectors developed for PAN. We summarize the best practices that have been employed this year for external plagiarism detection, discuss their suitability for practical use, and compare them with those of last year. Finally, Section 2.3 reports on this year’s approaches to intrinsic plagiarism detection. 2.1
All except one participant submitted lab reports describing their plagiarism detectors. After analyzing all 17 reports, certain algorithmic patterns became apparent to which many participants followed independently. We have unified and organized the different approaches in the form of a “reference retrieval process for external plagiarism detection”: given a suspicious document d and a document collection D, the task is to detect all plagiarism cases s in d. The process follows the aforementioned three steps, i.e., candidate retrieval, detailed analysis, and post-processing.
Candidate Retrieval. In order to simplify the detection of cross-language plagiarism, non-English documents in D are translated to English using machine translation (services). Then, to speed up subsequent computations, a subset Dsrc of D is retrieved that comprises candidates for plagiarism in d. Basically, this is done by comparing d with every document in D using a fingerprint retrieval model: d is represented as a fingerprint d of hash values of sorted word n-grams extracted from d. Note that sorting the n-grams brings them into a canonical form which cancels out plagiarism obfuscation locally. Beforehand, d is normalized by removing stop words, by replacing every word with a particular word from its synonym set (if possible), and by stemming the remainder. Again, these steps cancel out some obfuscation.
Since many suspicious documents are to be analyzed against D, the entire set D is represented as fingerprint collection, D, which is stored in an inverted index. Then, the postlists for the values in d are retrieved, and all documents that occur in at least k postlists are considered as candidate source documents Dsrc. Note that the value of k increases as n decreases. This approach is equivalent to an exhaustive comparison of d with every fingerprint in D using the Jaccard coefficient, but optimal in terms of runtime efficiency when repeating the task with different suspicious documents. Detailed Analysis. The suspicious document d is compared in-depth with each candidate source document dsrc ∈ Dsrc. This is done by means of heuristic sequence alignment algorithms, that, inspired by their counterparts in bioinformatics, work as follows: first, the sorted word n-grams that match exactly between d and dsrc are extracted as seeds. Second, the seeds are merged stepwise into aligned passages by applying merge rules. A merge rule decides whether two seeds or aligned passages can be merged, e.g. by checking whether they fulfill a certain condition with regard to their relative positions in the two documents. Typically, a number of merge rules are organized in a precedence hierarchy: a superordinate rule is applied until no two seeds can be merged anymore by that rule, then the next subordinate rule is applied on the resulting aligned passages, and so on until all rules have been processed. Third, the obtained pairs rplg and rsrc of aligned passages are returned as plagiarism detections r = hrplg, d, rsrc, dsrci. Post-Processing. Before being returned to the user, the set R of plagiarism detections from the previous step is filtered in order to reduce false positive detections. In this regard a set of “semantic” rules is applied that, for instance, require detections to have at least a certain length or that discard detections whose passages rplg and rsrc do not exceed a similarity threshold under some retrieval model. Moreover, ambiguous detections that report different sources for approximately the same plagiarized passage in d are dealt with, e.g., by discarding the less probable alternative. 2.2
Compared to last year, the detectors have matured and specialized to the problem domain. With regard to their practical use in a real-world setting, however, some developments must be criticized. In general, many of this year’s developments pertain only to plagiarism detection in local document collections, but cannot be applied easily for plagiarism detection against the Web.
A novelty this year is that many participants approach cross-language plagiarism cases straightforwardly by automatically translating all non-English documents to English. In cross-language information retrieval, this solution is often mentioned as an alternative to others, but it is hardly ever applied. Reasons for this include the fact that machine translation technologies are difficult to set up in the first place. All of this is of course alleviated to some extent by online translation services like Google Translate.1 With regard to plagiarism detection, however, this solution can only be applied locally, and not on the Web.
Most of the retrieval models for candidate retrieval employ “brute force”
fingerprinting, say, instead of selecting few n-grams from a document, as is custom with
near-duplicate detection algorithms like shingling and winnowing [
With regard to detailed analysis, one way or another, all participants employ sequence alignment heuristics, but few notice the connections to bioinformatics and image processing. Hence, due to the lack of a formal framework, participants come up with rather ad hoc rules. Finally, in order to minimize the granularity, some participants discard overlapping detections with ambiguous sources partly or altogether. It may or may not make sense to do so in a competition, but in a real-world setting this cannot hold. 1 http://translate.google.com 2 http://lucene.apache.org and http://terrier.org 2.3
Intrinsic plagiarism detection has received less attention this year as it turns out that
developing algorithms to detect both kinds of plagiarism cases and combining them
into a single detector is still too difficult a task to be accomplished within a few months
time. Moreover, intrinsic plagiarism detection is still in its infancy compared to external
plagiarism detection, and so is research on combining the two. The winning participant
reports to have successfully reimplemented last year’s best approach, however, the
developments were dropped in favor of external plagiarism detection [
In this section we report on the detection performances of the plagiarism detectors that took part in PAN. Their overall performance is analyzed, which determines this year’s winners, and then, each detector’s performance is analyzed with regard to the aforementioned parameters of our evaluation corpus. Again, we discuss the results and compare them with those of last year.
t e d g a l p a
P O 5 . 0 5 . 0 5 . 1 O
O
O Our evaluation corpus comprises a number of parameters that have been varied in order to create a high diversity of plagiarism cases; see Table 1 for an overview. In the following a detailed analysis of the detectors’ performances in terms of these parameters is given.
5
.
0
5
.
0
5
.
0
5
.
0
5
.
1
5
.
1
Detection Task. Table 3 summarizes the detection performances with regard to portions
of the evaluation corpus that are intended for external plagiarism detection and
intrinsic plagiarism detection. Since most of the participants focused on external plagiarism
detection, the trends that appear on the entire corpus can be observed here as well. The
only difference is that the recall values are between 20% and 30% higher than on the
entire corpus, which is due to the fact that about 30% of all plagiarism cases in the corpus
are intrinsic plagiarism cases. Only Muhr et al. [12] and Suárez et al. [22] made serious
attempts to detect intrinsic plagiarism; their recall is well above 0, but their precision is
poor. Nevertheless, combining intrinsic and external detection pays off overall for Muhr
et al., and the intrinsic-only detection of Suárez et al. even detects some of the external
plagiarism cases. Grozea and Popescu [
Precision rkza ouZ rhu zae tre jnóo iraeeP iilsvkko abohS ttrnoo licoMj-ssàu aabwtpauGianaV rzàue irnaha tIfene
u p Mro re rre r s a a S lz 5 . 0 5 . 0 5 . 0 5 . 0 5 . 0 5 . 0 5 . 0 5 . 0 5 . 0 5 . 0 p o t
Greb T tections if much plagiarism is found in a document. Altogether, the general behavior of the plagiarism detectors with regard to these corpus parameters is similar.
5 . 1 5 . 1 5 . 1 0 0 1
The detection task, the obfuscation strategies, and the length of a plagiarism case are the key parameters of the corpus that determine detection difficulty: the most difficult cases to be detected are those without source, and short ones with simulated obfuscation. The difficulty to detect simulated plagiarism may be in part due to the fact that is was created with specific intructions to create high obfuscation.
Since this year’s evaluation corpus has been re-developed from scratch, a
comparison to last year’s detection results is not straightforward. In this respect, the
secondtime participation of last year’s winner forms an important connection: Grozea and
Popescu [
A number of lessons learned can be derived from this year’s results: research and development on external plagiarism detection focuses too much on retrieval from local document collections instead of Web retrieval, while the less developed intrinsic plagiarism detection does not get much attention. Besides Web retrieval, another challenge 5 . 0 5 . 0 5 . 0 5 . 0 5 . 0 5 . 0 5 . 0 5 .
0 0 0 1 1 1 2 r r i l k u a n e a i a n
h e l r o a o e t jó i i k h o c s a rz Z u z u e sa re sv b ttr i s
Mo e r o r r r B e o
r o k S G P l
u
Mj a o
G b O a P à b a a n s
t i i wu a i
r i z d
n e o b e r e n lme n p n in t g r a u ra o ll e b ta a à wg h lc A ft
I GV r a Hu a la a a
e a N t s o C
S h m c a
t h
S T performances of PAN 2009 detectors are shaded dark, those of PAN 2010 detectors light. 5 . 1 5 . 1 5 . 1 5 . 1 in external plagiarism detection is obfuscation: while artificial obfuscation appears to be detectable relatively easy if a plagiarism case is long, short plagiarism cases as well as simulated obfuscation is not. Regarding translated plagiarism, again, automatically generated cases pose no big challenge in a local document collection, while we hypothesize that simulated cross-language plagiarism will. Future competitions will have to address these shortcomings.
We would like to thank our co-organizers Efstathios Stamatatos and Moshe Koppel for
their help along the way. Our special thanks go to the participants of the competition
for their devoted work. Last not least we thank Yahoo! Research for their sponsorship.
This work is partially funded by CONACYT-Mexico and the MICINN project
TEXTENTERPRISE 2.0 TIN2009-13391-C04-03 (Plan I+D+i).
[11] Daniel Micol, Óscar Ferrández, and Rafael Muñoz. A Textual-Based Similarity Approach
for Efficient and Scalable External Plagiarism Analysis: Lab Report for PAN at CLEF
2010. In Braschler et al. [
Lab Report for PAN at CLEF 2010. In Braschler et al. [
FASTDOCODE: Finding Approximated Segments of N-Grams for Document Copy
Detection: Lab Report for PAN at CLEF 2010. In Braschler et al. [
ISBN 978-88-904810-0-0. [16] Martin Potthast, Benno Stein, Andreas Eiselt, Alberto Barrón-Cedeño, and Paolo Rosso.
Overview of the 1st International Competition on Plagiarism Detection. In Benno Stein,
Paolo Rosso, Efstathios Stamatatos, Moshe Koppel, and Eneko Agirre, editors, SEPLN
2009 Workshop on Uncovering Plagiarism, Authorship, and Social Software Misuse (PAN
09), pages 1–9. CEUR-WS.org, September 2009. URL http://ceur-ws.org/Vol-502.
[17] Martin Potthast, Benno Stein, Alberot Barrón-Cedeño, and Paolo Rosso. An Evaluation
Framework for Plagiarism Detection. In Chu-Ren Huang and Dan Jurafsky, editors,
Proceedings of the 23rd International Conference on Computational Linguistics (COLING
2010), pages 997–1005, Beijing, China, August 2010. Association for Computational
Linguistics.
[18] Viviane P. Moreira Rafael C. Pereira and Renata Galante. UFRGSPAN2010: Detecting
External Plagiarism: Lab Report for Pan at CLEF 2010. In Braschler et al. [
Lab Report for PAN at CLEF 2010. In Braschler et al. [