This paper overviews the five source retrieval approaches that have been submitted to the seventh international competition on plagiarism detection at PAN 2015. We compare the performances of these five approaches to the 14 methods submitted in the two previous years (eight from PAN 2013 and six from PAN 2014). For the third year in a row, we invited software submissions instead of run submissions, such that cross-year evaluations are possible. This year's stand-alone source retrieval overview can thus to some extent also be used as a reference to the different ideas presented in the last three years-the text alignment subtask will be depicted in another individual overview.
The retrieval and extraction of text reuse from large document collections is central to
applications such as plagiarism detection, copyright protection, and information flow
analysis. Appropriate algorithms have to be able to deal with all kinds of text reuse
ranging from verbatim copies and quotations to paraphrases and translations to
summaries [
The source retrieval subtask has been running for four years in a row now and we can observe the standard multi-year life cycle of repeated shared tasks. Basically, there are three phases: an innovation phase, a consolidation phase, and a production phase. In the innovation phase, new evaluation resources are being developed and introduced for the first time, such as new corpora, new performance measures, and new technologies. The introduction of such new resources typically stirs up a lot of dust and is prone to errors and inconsistencies that may spoil evaluation results to some extent. This cannot be avoided, since only the use of new evaluation resources by many different parties will 1 Some of the concepts found in this paper have been described earlier, so that, because of the inherently incremental nature of shared tasks, and in order for this paper to be self-contained, we reuse text from previous overview papers.
Suspicious document
Source retrieval Document collection
Candidate documents
Text alignment
Knowledge-based post-processing Plagiarism Detection
Suspicious passages reveal their shortcomings. Therefore, the evaluation resources are released only sparingly so they last for the remainder of a cycle. This phase spanned the first and also to some extent the second year of the source retrieval subtask. In the consolidation phase, based on the feedback and results obtained from the first phase, the new evaluation resources are developed to maturity by making adjustments and fixing errors. This phase spanned the second and to some extent the third year of the source retrieval subtask. In the production phase, the task is repeated with little changes to allow participants to build upon and to optimize against what has been accomplished, and, to make the most of the prior investment in developing the new evaluation resources. Meanwhile, new ideas are being developed to introduce further innovation.
This third production phase in part could be observed in last year’s third edition of the source retrieval subtask and was the motivation behind organizing the subtask for a fourth time. However, as will be described in more detail later, no real progress in the probably most important directions for a source retrieval method (i.e., improving the recall of reused sources and minimizing the effort until the first evidence for reuse is detected) can be observed in this year’s submission. New querying strategies might be the key but no participant developed new ideas in that direction. Before further elaborating the different submitted approaches of this year’s edition and the respective evaluation results, we describe the task setting and test environment in a self-contained way (note again that a lot of the respective passages have already been contained in the previous years’ overview papers). 2
Terminology. Figure 1 shows a generic retrieval process to detect plagiarism in a given suspicious document dplg, when also given a (very large) document collection D of potential source documents. This process is also referred to as external plagiarism detection since plagiarism in dplg is detected by searching for text passages in D that are highly similar to text passages in dplg.2 The process is divided into three basic steps, 2 Another approach to detect plagiarism is called intrinsic plagiarism detection, where detectors are given only one suspicious document and are supposed to identify text passages in it which deviate in their style from the remainder of the document. which are typically implemented in most plagiarism detectors. First, source retrieval, which identifies a feasible set of candidate source documents Dsrc D that are likely sources for plagiarism regarding dplg—this is the problem tackled in the source retrieval subtask. Second, text alignment, where each candidate source document dsrc 2 Dsrc is compared to dplg, extracting all passages of text that are highly similar—this is the problem tackled in the text alignment subtask described in another overview paper. Third, knowledge-based post-processing, where the extracted passage pairs are cleaned, filtered, and possibly visualized for later inspection—this is not really reflected in the PAN plagiarism subtasks so far.
Shared Tasks on Plagiarism Detection. We have organized shared tasks on
plagiarism detection annually since 2009. In the innovation phase of our shared task at
PAN 2009 [21], we developed the first standardized evaluation framework for
plagiarism detection [20]. This framework was consolidated in the second and third task at
PAN 2010 and 2011 [
Contributions. Since the source retrieval subtask probably now is in the production phase of its life cycle, we refrain from changing the existing evaluation resources too much, whereas we continue to maintain them. Therefore, our contributions this year consist of (1) a survey of this year’s submitted approaches, which reveals that there are hardly any new trends among participants in the source retrieval subtask, and (2) an analysis of this years participants’ retrieval performances in direct comparison to participants from previous years, which reveals that no real progress in the important direction of increased recall can be observed.
In this connection, our goal with both shared tasks is to further automate them.
Hence, we continue to develop the TIRA evaluation platform [
In source retrieval, given a suspicious document and a web search engine, the task is to retrieve all source documents from which text has been reused whilst minimizing retrieval costs. The cost-effectiveness of plagiarism detectors in this task is important since using existing search engines is perhaps the only feasible way for researchers as well as small and medium-sized businesses to implement plagiarism detection against the web, whereas search companies charge considerable fees for automatic querying their APIs.
In what follows, we briefly describe the building blocks of our evaluation setup, provide brief details about the evaluation corpus, and discuss the performance measures (see the task overview from 2013 for more details on these three points [16]). We then survey the submitted softwares in Section 4, and finally in Section 5, report on their achieved results in this year’s setup. 3.1
For the evaluation of source retrieval from the web, we consider the real-world scenario of an author who uses a web search engine to retrieve documents in order to reuse text from them in their to-be-written text. A plagiarism detector typically uses a search engine, too, to find reused sources of a given document. Over the past years, we assembled the necessary building blocks to allow for a meaningful evaluation of source retrieval algorithms; Figure 2 shows how they are connected. The setup was described in much more detail in the task overview of 2013 [16].
Two main components are the TIRA experimentation platform and the ClueWeb09
with two associated search engines. TIRA [
The ClueWeb corpus 2009 (ClueWeb09)3 is one of the most widely adopted web crawls which is regularly used for large-scale web search-related evaluations. It consists of about one billion web pages, half of which are English ones. Although an updated version of the corpus has been released,4 our evaluation is still based on the 2009 version since our corpus of suspicious documents was built on top of ClueWeb09. Indri5 and
Submitted plagiarism detector
Virtual machines TIRA experimentation platform
Search proxy API ChatNoir Cluster
Indri
Performance measures and source oracle ClueWeb09 (0.5 billion
English web pages) n liuao true t c u vea tr
s ic ra t f ta in S
Web search
Web
ChatNoir [17] are currently the only publicly available search engines that index the ClueWeb09 corpus; their retrieval models are based on language modeling and BM25F, respectively. For developer convenience, we also provide a proxy server which unifies the APIs of the search engines. At the same time, the proxy server logs all accesses to the search engines for later performance analyses. 3.2
The evaluation corpus employed for source retrieval is based on the Webis text reuse corpus 2012 (Webis-TRC-2012) [19, 18]. The corpus consists of 297 documents that have been written by 27 writers who worked with our setup as shown in the first row of Figure 2: given a topic, a writer used ChatNoir to search for source material on that topic while preparing a document of 5700 words length on average, reusing text from the found sources.
As in the last year, we use the same 98 documents from the Webis-TRC-2012 as training documents and another 99 documents are sampled as test documents—also the same as in 2014. The remainder of the corpus will be used within future source retrieval evaluations. 3.3
Given a suspicious document dplg that contains passages of text that have been reused from a set of source documents Dsrc, we measure the retrieval performance of a source retrieval algorithm in terms of precision and recall of the retrieved documents Dret taking into account the effect of near-duplicate web documents as follows (cf. the 2013 task overview [16] for more details).
For any dret 2 Dret, we employ a near-duplicate detector to judge whether it is a true positive detection; i.e., whether there is a dsrc 2 Dsrc of dplg that is a nearduplicate of dret. We say that dret is a true positive detection for a given pair of dsrc and dplg iff (1) dret = dsrc (equality), or (2) the Jaccard similarity of the word n-grams in dret and dsrc is above 0.8 for n = 3, above 0.5 for n = 5, and above 0 for n = 8 (similarity), or (3) the passages in dplg known to be reused from dsrc are contained in dret (containment). Here, containment is measured as asymmetrical set overlap of the passages’ set of word n-grams regarding that of dret, so that the overlap is above 0.8 for n = 3, above 0.5 for n = 5, and above 0 for n = 8. This three-way approach of determining true positive detections inherently entails inaccuracies. While there is no straightforward way to solve this problem, this error source affects all detectors, still allowing for relative comparisons.
Let ddup denote a near-duplicate of a given dsrc that would be considered a true positive detection according to the above conditions. Note that every dsrc may have more than one such near-duplicate and every ddup may be a near-duplicate of more than one source document. Further, let Ds0rc denote the set of all near-duplicates of a given set of source documents Dsrc of dplg and let Dr0et denote the subset of Dsrc that have at least one corresponding true positive detection in Dret: Ds0rc = fddup j ddup 2 D and 9dsrc 2 Dsrc : ddup is a true positive detection of dsrcg; Dr0et = fdsrc j dsrc 2 Dsrc and 9dret 2 Dret : dret is a true positive detection of dsrcg: Based on these sets, we define precision and recall of Dret regarding Dsrc and dplg as follows: prec = jDret \ Ds0rcj ; jDretj r ec = jDr0et \ Dsrcj : jDsrcj Rationale for this definition is that retrieving more than one near-duplicate of a source document does not decrease precision, but it does not increase recall, either, since no additional source is obtained. A further graphical explanation of how we take nearduplicates into account for precision and recall is given in Figure 3. Note that Dret as defined above does not actually contain all duplicates of the retrieved documents, but only those that are already part of Dsrc.
Finally, to measure the cost-effectiveness of a source retrieval algorithm in retrieving Dret, we count the numbers of queries and downloads made and compute the workload in terms of queries and downloads until the first true positive detection is made (i.e., until the first real evidence for text reuse is found). The last measure highlights the probable end user needs that some evidence should be found fast in order to quickly flag such a suspicious document for a further detailed analysis.
Duplicate Hull D'ret Retrieved Dret
Documents D Duplicate Hull D'src
Sources Dsrc Retrieved Dret
Duplicate Hull D'ret
Documents D Duplicate Hull D'src
Sources Dsrc Dret Ç D'src D'ret Ç Dsrc
The Source Oracle To allow for participation in the source retrieval task without the need of having a text alignment component at hand, we provide a source oracle that automatically enriches a downloaded document with information about whether or not it is considered a true positive source for the given suspicious document. Note that the oracle employs the aforementioned conditions to determine whether a document is a true positive detection. However, the oracle does not, yet, tell for which part of a suspicious document a downloaded document is a true positive detection. Hence, applying a custom text alignment strategy can still be beneficial to derive such a mapping and potentially to adjust the query strategy accordingly. 4
Five teams submitted softwares for the source retrieval task, all of whom also
submitted a notebook describing their approach—the approaches of Han [
Chunking Given a suspicious document, it is divided into (possibly overlapping) passages of text. Each chunk of text is then processed individually. Rationale for chunking the suspicious document is to evenly distribute “attention” over a suspicious document so that algorithms employed in subsequent steps are less susceptible to unexpected characteristics of the suspicious document.
The chunking strategies employed by this year’s participants are no chunking (i.e.,
the whole document as one chunk) [26], 500-word chunks [22] (overlap size not
detailed), paragraphs as chunks [23] (not detailed how paragraphs are split),
paragraphs split at empty lines as chunks [26], or individual sentences and headings as
chunks [
Note that chunks typically seem to be implemented as non-overlapping by the participating approaches. The potentially interesting question of whether overlapping chunks might help was not really tackled by any approach so far—except that Suchomel and Brandejs [26] use different types of chunks in combination. A problem with nonoverlapping longer chunks might be that typical plagiarism cases have no fixed length and overlapping chunks might reduce the risk of, for instance, having more than one source in one chunk of 500 words. Furthermore, relying on the given document structure (e.g., chunking by lines or paragraphs) bears the risk of failing for some unseen documents that are not as well-formatted as the ones in our evaluation corpus (e.g., all the text in a single line). Maybe mixed chunking strategies as seen in Suchomel and Brandejs [26]’ approach is the most promising direction. Notably, their document level queries seem to also guarantee an early recall measured in queries (cf. Section 5) and mixed chunking probably at least will not decrease total recall.
Keyphrase Extraction Given a chunk, “keyphrases” are extracted from it in order to formulate queries with them. Rationale for keyphrase extraction is to select only those phrases (or words) which maximize the chance of retrieving source documents matching the suspicious document. Keyphrase extraction may also serve as a means to limit the amount of queries formulated, thus reducing the overall costs of using a search engine. This step is perhaps the most important one of a source retrieval algorithm since the decisions made here directly affect the overall performance: the fewer keywords are extracted, the better the choice must be or recall is irrevocably lost.
Some participants use single words while others extract whole phrases. Most of the
participants preprocessed the suspicious document by removing stop words before the
actual keyphrase extraction. Phrasal search was provided by the Indri search engine.
All participants did use Indri when submitting phrasal queries; some of which also
combine phrases with non-phrasal ChatNoir queries, the search engine that the original
essay authors had used. In particular, Han [
Altogether, the participants’ approaches to “keyphrase extraction” are more or less simplistic and do not really rely on established keyphrase extraction techniques from the NLP community. Giving several such established methods a try at least on document level might be an interesting direction for some more general keyphrases. Queries from such phrases in combination with queries from “extracted keyphrases” or just single words from shorter chunks might be a promising direction. Some first steps in this direction are shown in the method of Suchomel and Brandejs [26] but might still be enhanced. Not relying on just one strategy alone but combining different keyphrase extraction ideas might ensure a higher recall. This way, just as with chunking, the risk of algorithm error is further diminished and it becomes possible to exploit potentially different sources of information that complement each other.
Query Formulation Interestingly, most of the participants hardly put effort in
finding good keyphrase combinations as queries. Instead, typically the top-k tf idf -ranked
terms form the first query, then the next k terms, etc. This way, mostly non-overlapping
queries are generated for the individual chunks. This non-overlap-approach is in line
with many query-by-document strategies [
In particular, Han [
Given a set of queries, the search controller schedules their submission to the search
engine and directs the download of search results. Rationale for this is to
dynamically adjust the search based on the results of each query, which may include
dropping queries, reformulating existing ones, or formulating new ones based on the
relevance feedback obtained from the search results. Han [
Note that still (just as in the last years) none of the teams did try to reformulate existing queries or formulating new ones based on the available number of search results, the search snippets, or the downloaded documents, which probably leaves room for substantial improvement. Another interesting aspect might be the scheduling of the queries themselves. The experimental results (cf. Section 5) seem to suggest that some document-level queries in the first submission positions guarantee an early recall with respect to the number of submitted queries (e.g., Suchomel and Brandejs [26]). Simply scheduling queries in the order of chunks in the documents instead, might run into problems with early recall as maybe there is not that much reused text at the beginning of a document. This might also be an interesting point for future research that none of the approaches has investigated so far.
Download Filtering Given a set of search engine results, a download filter removes all documents that are probably not worthwhile being compared in detail with the suspicious document. Rationale for this is to further reduce the set of candidates and to save invocations of the subsequent detailed comparison step. Many of the participants in the last years and also this year seem to focus a little too much on that part of a source retrieval system. This can be seen in the not improved overall recall and also in the number of submitted queries till the first evidence of text reuse is found compared to the approaches from 2013. Another evidence is that some approaches hardly download ten documents per suspicious document. In this case, the download filtering needs to be almost optimal which puts a higher burden on the search strategy. As downloads probably are not the most important bottleneck and probably queries are more costly in a general environment, reducing downloads too much does not seem to be the most promising strategy. A text alignment system is probably easily able to compare a suspicious document against even several thousands of potential sources in a couple of minutes. Maybe downloading more documents and putting more effort on the formulation of good queries is a promising future area.
In this year, Han [
Some participants download very few documents based on their filtering while
others download more documents per query than most participants did in the last years. As
described above, the second option seems to be more promising with respect to recall
since downloads typically are not that costly. An further interesting option for future
research might be based on the User-over-Ranking hypothesis [
Table 1 shows the performances of the five plagiarism detectors that took part in this year’s source retrieval subtask as well as those of the last years’ participants whose approaches were also evaluated on this year’s test corpus using the TIRA experimentation platform. Since there is currently no single formula to organize retrieval performance and cost-effectiveness into just one absolute score or order, the detectors are ordered alphabetically, whereas the best performance value for each metric is highlighted—note that these individual “best” performances should be compared to the other metrics as for instance the fastest approach also has the highest number of no detections etc. As can be seen, there is no single detector that performs best on all accounts. Rather, different detectors have different characteristics.
Arguably, highest possible recall at a reasonable workload (queries and downloads) is the goal of source retrieval while, at the same time, it would be nice to quickly detect some first evidence for text reuse if there is one. Since downloads in most environments would be much cheaper than query submissions (that would accompanied by costs at most major web search engine’s APIs) the most interesting metrics probably are: recall and number of no detections, number of submitted queries, and number of queries and downloads until the first detection.
Focusing on recall first, interestingly the top-6 approaches are from previous years
with the best one by far still being the approach of Kong et al. [
The ensemble of all submitted approaches of the last three years would achieve an average recall of 0.85 retrieving all sources for 48 topics. Only for 14 topics the recall is below 0.6 (which is the best individual average recall). These numbers did not change from the ensemble of the 2013 and 2014 approaches also indicating that this year’s methods do not contain real innovations with respect to recall-oriented source retrieval.
A per-participant analysis also reveals some interesting observations when
comparing the approaches from different years. For instance, after doubling the recall in the
last year, Suchomel and Brandejs [26] were only able to slightly increase their recall
this year with way more query and downloading effort. Rafiei et al. [22] and Ravi N
and Gupta [23] manage to enter the competition with medium recalls while Kong et al.
[
As some further not-just-recall-oriented observations, for no metric, any of this
year’s participants achieved the best performance. This is not too surprising given the
fact that no real “innovations” are contained in this year’s submissions. They are rather
very similar to methods that participated in the last year such that no “radical” change
could be expected. Still, some notable achievements can be observed for the five
participants of this year’s competition. Rafiei et al. [22] in their first year manage to find
sources for all but one of the suspicious documents. Also Ravi N and Gupta [23] enter
the competition with a very good result: their number of downloads till the first
detection is almost the top-performing one and is a little better than the also very good one
of Han [
Altogether, the current strategies might still be a little too focused on saving downloads (and queries) compared to for instance increase the recall. Also runtime should probably not be the key metric to optimize (e.g., using threads instead of sequential processing does not decrease the actual costs for using the search engines). A reasonable assumption probably is that recall is most important to the end user of a source retrieval system. Investing a couple of queries and a couple of downloads (maybe even hundreds to thousands) to achieve a recall above 0.8 might be a very important research direction since still none of the participating approaches can reach such levels. In the end, whatever source the source retrieval step misses, cannot be found by a later text alignment step. This probably is a key argument for a recall-oriented source retrieval strategy that also takes into account basic considerations on total workload of query submissions and downloads. It would be interesting to see efforts in that direction of substantially improved recall at a moderate cost increase in future approaches.
Another interesting direction from a practical standpoint is to find at least one source for a document that contains text reuse and to report a first detection as early as possible. This way the real end user of a detection system could focus on the really important suspicious documents very quickly and could even deepen the search depth or increase the allowed number of queries without the fear of missing text reuse in too many suspicious documents. However, this probably should be future work when better overall recall levels are reached.
Altogether, even though new participants entered the source retrieval subtask this year, the ideas underlying the approaches did not contain “real” innovations compared to the approaches from the previous years since mostly the same ideas are just reused. Even though the source retrieval subtask now is expected to be in the production phase of its shared task life cycle—it is well-defined and all evaluation resources are set up and provide for a challenging testbed—most of the approaches still struggle to retrieve at least 50% of the text reuse sources with no real progress on the recall side this year. The combined ensemble of all approaches would result in a really good recall but at a rather high cost in the number of queries and downloads. None of this year’s approaches really tried to work towards that recall at reduced costs resulting in the described “lack” of innovation.
Without really new ideas, the task does not seem to be interesting enough for
another edition. Instead, we are planning to continue to pursue automating source retrieval
evaluations without the requirement of an organized shared task. The key is the
development of the TIRA experimentation platform [
We thank the participating teams of the editions of the source retrieval subtask for their shared ideas and for their devoted work towards making their softwares run on TIRA.
International Competition on Plagiarism Detection. In: Forner, P., Karlgren, J.,
Womser-Hacker, C. (eds.) Working Notes Papers of the CLEF 2012 Evaluation Labs (Sep
2012), http://www.clef-initiative.eu/publication/working-notes
16. Potthast, M., Gollub, T., Hagen, M., Tippmann, M., Kiesel, J., Rosso, P., Stamatatos, E.,
Stein, B.: Overview of the 5th International Competition on Plagiarism Detection. In:
Forner, P., Navigli, R., Tufis, D. (eds.) Working Notes Papers of the CLEF 2013 Evaluation
Labs (Sep 2013), http://www.clef-initiative.eu/publication/working-notes
17. Potthast, M., Hagen, M., Stein, B., Graßegger, J., Michel, M., Tippmann, M., Welsch, C.:
ChatNoir: A Search Engine for the ClueWeb09 Corpus. In: Hersh, B., Callan, J., Maarek,
Y., Sanderson, M. (eds.) 35th International ACM Conference on Research and Development
in Information Retrieval (SIGIR 12). p. 1004. ACM (Aug 2012)
18. Potthast, M., Hagen, M., Völske, M., Stein, B.: Crowdsourcing Interaction Logs to
Understand Text Reuse from the Web. In: Fung, P., Poesio, M. (eds.) Proceedings of the
51st Annual Meeting of the Association for Computational Linguistics (ACL 13). pp.
1212–1221. Association for Computational Linguistics (Aug 2013),
http://www.aclweb.org/anthology/P13-1119
19. Potthast, M., Hagen, M., Völske, M., Stein, B.: Exploratory Search Missions for TREC
Topics. In: Wilson, M.L., Russell-Rose, T., Larsen, B., Hansen, P., Norling, K. (eds.) 3rd
European Workshop on Human-Computer Interaction and Information Retrieval
(EuroHCIR 2013). pp. 11–14. CEUR-WS.org (Aug 2013),
http://www.cs.nott.ac.uk/ mlw/euroHCIR2013/proceedings/paper3.pdf
20. Potthast, M., Stein, B., Barrón-Cedeño, A., Rosso, P.: An Evaluation Framework for
Plagiarism Detection. In: Huang, C.R., Jurafsky, D. (eds.) 23rd International Conference on
Computational Linguistics (COLING 10). pp. 997–1005. Association for Computational
Linguistics, Stroudsburg, Pennsylvania (Aug 2010)
21. Potthast, M., Stein, B., Eiselt, A., Barrón-Cedeño, A., Rosso, P.: Overview of the 1st
International Competition on Plagiarism Detection. In: Stein, B., Rosso, P., Stamatatos, E.,
Koppel, M., Agirre, E. (eds.) SEPLN 09 Workshop on Uncovering Plagiarism, Authorship,
and Social Software Misuse (PAN 09). pp. 1–9. CEUR-WS.org (Sep 2009),
http://ceur-ws.org/Vol-502
22. Rafiei, J., Mohtaj, S., Zarrabi, V., Asghari, H.: Source Retrieval Plagiarism Detection based
on Noun Phrase and Keyword Phrase Extraction—Notebook for PAN at CLEF 2015. In: [
Plagiarism Source Retrieval—Notebook for PAN at CLEF 2015. In: [
Retrieval—Notebook for PAN at CLEF 2015. In: [