This paper overviews 18 plagiarism detectors that have been evaluated within the fifth international competition on plagiarism detection at PAN 2013. We report on their performances for the two tasks source retrieval and text alignment of external plagiarism detection. Furthermore, we continue last year's initiative to invite software submissions instead of run submissions, and, re-evaluate this year's submissions on last year's evaluation corpora and vice versa, thus demonstrating the benefits of software submissions in terms of reproducibility.
Text is reused in many ways, such as quotations, translations, paraphrases, summaries, and boilerplate text. Under the right circumstances, all of these kinds of text reuse may also be considered plagiarism [25]. A frequent research topic concerning text reuse and plagiarism is algorithms that detect them. Particularly the detection of plagiarism has received considerable attention in terms of publications over the past two decades. Our focus is on the evaluation of such algorithms with respect to their retrieval performance; since 2009 we have been organizing four annual competitions on plagiarism detection [26, 27, 29, 30] and this paper reports on the results of the fifth edition.1
During the first three editions of our lab we developed the first standardized evaluation framework for plagiarism detection [28]. A total of 32 teams of researchers took part in these evaluations, nine of whom more than once, and the framework has been adopted by the research community since. While evaluation frameworks should accurately emulate the real world around a given computational task in a controlled laboratory environment, most frameworks do so only to some extent, since they typically rest on design choices that affect their generalizability. This is also true for our framework, which has been shown to exert a number of shortcomings due to its semiautomatic construction that render it less realistic and sometimes lead to impractical algorithm design. As of last year, we started developments on a new, more realistic evaluation framework 1 Some of the concepts found in this paper have been described earlier, so that, because of the inherently incremental nature of evaluations, and in order for this paper to be self-contained, we reuse text from these sources.
Suspicious document
Source retrieval Document collection
Candidate documents
Text alignment
Knowledge-based post-processing Plagiarism Detection
Suspicious passages that consists of entirely manually generated text reuse and plagiarism [32]. This year, we employ the new framework for the second time to evaluate a total of 18 plagiarism detectors, revising it along the way. 1.1
Since last year, we evaluate plagiarism detectors step-wise instead of as a whole. Our focus is on the source retrieval task and the text alignment task, and we research and develop new evaluation frameworks for each of them. In the following two sections, we detail the evaluations of both tasks. Our contributions are as follows: 1. Software Submissions. For the second time, we asked participants to submit their software instead of outputs of software runs. The submitted softwares were run and 2 Another approach to detect plagiarism is called intrinsic plagiarism detection, where detectors are given only a suspicious document and are supposed to identify text passages in them which deviate in their style from the remainder of the document. This year, we focus on external plagiarism detection.
evaluated at our site using the TIRA experimentation platform [9]. This improves the sustainability of our evaluations because submitted softwares are maintained in executable state so that they can be run against new corpora later on. 2. Text Alignment Evaluation Across Years. Since software submissions were introduced last year for the text alignment task, this puts us in the position to re-evaluate last year’s submissions against this year’s evaluation corpora and vice versa. We report on the results of this cross-year evaluation and present a combined ranking, demonstrating the benefits of software submissions in terms or reproducibility. 3. Survey of Retrieval Approaches. We survey the 18 submitted softwares as described by their authors, analyze how they work, and organize them into generic retrieval processes for both of the two tasks source retrieval and text alignment. 4. Performance Measures for Source Retrieval. Regarding the source retrieval task, we shed light onto measuring the performance of a source retrieval algorithm. In particular, we show that near-duplicate retrieval results should be discounted when measuring source retrieval performance. 2
In source retrieval, given a suspicious document and a web search engine, the task is to retrieve all source documents from which text has been reused whilst minimizing retrieval costs. The cost-effectiveness of plagiarism detectors in this task is important since using existing search engines is perhaps the only feasible way for researchers as well as small and medium-sized businesses to implement plagiarism detection against the web, whereas search companies charge considerable fees for automatic usage.
In what follows, we describe the building blocks of our evaluation setup, provide details about the evaluation corpus and how it was constructed, discuss performance measures, survey the submitted softwares, and finally, report on the evaluation of these softwares. 2.1
For the evaluation of source retrieval from the web, we consider the real-world scenario of an author who uses a web search engine to retrieve documents in order to reuse text from them in a document. A plagiarism detector typically uses a search engine, too, to find reused sources of a given document. Therefore, to evaluate a plagiarism detector, an evaluator must collect realistic samples of documents that contain reused text and feed them into the detector while keeping the web environment under full control. To do so in a reproducible manner, the web environment must be representative, yet static, so that evaluations of different detectors yield comparable results when done asynchronously. Meeting both constraints at the same time poses a significant engineering challenge. Over the past years, we assembled the necessary building blocks to allow for a meaningful evaluation of source retrieval algorithms; Figure 2 shows how they are connected: – A large-scale web corpus (in our case, the ClueWeb09) that can be readily served and browsed as if it were the real web (i.e., links of delivered web pages are rewritten so that they point to the servers hosting the web corpus instead of the real web).
S ubmitted plagiarism detector
Virtual machines TIR A experimentation platform
S earch proxy AP I C hatNoir C luster
Indri
Performance measures and source oracle C lueWeb09 (0.5 billion
E nglish web pages) Participant – Web search engines (Indri and ChatNoir) that index the web corpus using two different retrieval models and that offer both a user interface as well as an API for automatic usage. – An evaluation corpus (Webis-TRC-12) that consists of long, manually written documents from authors who searched for sources in the web corpus using one of the search engines and then reused text from sources retrieved. – An experimentation platform (TIRA) that allows participants to submit their plagiarism detector for evaluation and for conservation in an executable state for future evaluations. – A search proxy API that, for later analysis, monitors and logs a participant’s use of the search engines as well as web page downloads from the corpus. – Performance measures that measure the retrieval quality and cost-effectiveness of a plagiarism detector. – A source oracle that provides feedback to a detector about whether or not a web page the detector decided to download is a true positive detection (i.e., a source document, or a duplicate thereof, that was actually used).
This setup has been used for both corpus construction as well as for evaluation runs, both of which are detailed in the following sections. Beforehand, we briefly describe the underlying infrastructure which must be kept operational for both purposes and maintained for future evaluations.
The TIRA experimentation platform The TIRA experimentation platform is developed alongside our evaluation lab at our site [9]. It facilitates information retrieval experiments by providing a unified execution environment that allows for local instantiation of experiment software, web dissemination of evaluation results, platform independent software development, and result retrieval and visualization. TIRA itself consists of a number of building blocks the full description of which is beyond the scope of this report. Suffice it to say that one of them, depicted in Figure 2 bottom left, facilitates both platform independent software development and software submissions at the same time by its capability to create and remote control virtual machines on which our lab’s participants deploy their plagiarism detectors.
The ClueWeb, Indri, and ChatNoir The currently most widely adopted web crawl that is regularly used for large-scale web search-related evaluations is the ClueWeb corpus 2009 (ClueWeb09).3 The corpus consists of about one billion web pages, half of which are English ones. It has been successfully employed to evaluate retrieval models and search engines within the annual TREC evaluation conference4 and has therefore been widely adopted. As of this year, an updated version of the corpus has been released,5 however, our evaluation is still based on the 2009 version.
Indri6 and ChatNoir [31] are currently the only publicly available search engines that index the ClueWeb09 corpus. Indri’s retrieval model combines language modeling and inference networks [20], whereas ChatNoir implements the classic BM25F model [33] and incorporates PageRank and SpamRank scores. This way, two of the most widespread retrieval models are available within our evaluation setup. For developer convenience, we also provide a proxy server which unifies the APIs of the search engines. At the same time, the proxy server logs all accesses to the search engines for later performance analysis. 2.2
The evaluation corpus employed for source retrieval is based on the Webis text reuse corpus 2012 (Webis-TRC-2012) [32]. The corpus consists of 297 documents that have been written by 27 writers who worked with our setup as shown in the first row of Figure 2: given a topic, a writer used ChatNoir to search for source material on that topic while preparing a document of 5700 words length on average, reusing text from the found sources. The writers were instructed to modify their text reuse as much as they deemed necessary to avoid automatic detection. To do so, writers applied two general strategies, namely paraphrasing of a reused passage and interleaving of two or more reused passages from different source documents. Some writers made only superficial 3 http://lemurproject.org/clueweb09 4 http://trec.nist.gov 5 http://lemurproject.org/clueweb12 6 http://lemurproject.org/clueweb09/index.php#Services modifications while others made many, so that a spectrum of cases ranging from no paraphrasing and no interleaving to much paraphrasing and much interleaving can be observed.
Last year, we sampled 40 documents from the Webis-TRC-2012 as training and test documents. This year, these documents were provided for training, and another 58 documents were sampled as test documents. The remainder of the corpus will be used within future labs on this task. 2.3
Given a suspicious document dplg that contains passages of text that have been reused from a set of source documents Dsrc, we measure the retrieval performance of a source retrieval algorithm in retrieving Dsrc in terms of precision and recall. Let Dret denote the set of documents that are retrieved by a source retrieval algorithm when given dplg, then it may seem straightforward to define precision as prec = jDret \ Dsrcj=jDretj and recall as r ec = jDret \ Dsrcj=jDsrcj. However, this definition turns out to be overly simplistic since it disregards an important side-effect of using a large-scale web corpus for evaluation, namely near-duplicate web documents.
The web is rife with documents that are near-duplicates of each other [4]. A source retrieval algorithm may therefore retrieve a document dret 2 Dret that is almost the same as a source document dsrc 2 Dsrc in terms of its main contents, but not exactly the same. Because of this, dret would be counted falsely as a negative detection despite the fact that a human assessor would consider it a true positive one. To relax this constraint, we employ a near-duplicate detector to judge whether a retrieved document dret is a true positive detection; i.e., whether a document dsrc exists in dplg’s set of source documents Dsrc that is a near-duplicate of dret. If one of the following conditions holds, we say that dret is a true positive detection for a given pair of dsrc and dplg: 1. Equality. The document dret is a true positive detection of dsrc if dret = dsrc. 2. Similarity. The document dret is a true positive detection of dsrc if their n-gram Jaccard similarity is above 0.8 for n = 3, above 0.5 for n = 5, and above 0 for n = 8. The thresholds have been determined experimentally. 3. Containment. The document dret is a true positive detection of dsrc if the passages in dplg known to be reused from dsrc are contained in dret. Containment is measured as asymmetrical set overlap of the passages’ set of n-grams regarding that of dret, so that the overlap is above 0.8 for n = 3, above 0.5 for n = 5, and above 0 for n = 8.
This way of determining whether a retrieved document dret is a true positive detection inherently entails inaccuracies, so that not all near-duplicates of a source document dsrc will be considered true positive detections. While there is no straightforward way to solve this problem, this error source affect all detectors at the same time, still allowing for relative comparisons.
Let ddup denote a near-duplicate of a given dsrc that would be considered a true positive detection according to the above conditions if it was retrieved by a source retrieval algorithm. Note that every dsrc may have more than one near-duplicate and every ddup may be a near-duplicate of more than one source document. Further, let Ddup denote the set of all near-duplicates of a given set of source documents Dsrc of dplg and let Dr0et denote the subset of Dsrc that have at least one corresponding true positive detection in Dret: Ddup = fddup j 9dsrc 2 Dsrc : ddup is a true positive detection of dsrcg; Dr0et = fdsrc j dsrc 2 Dsrc and 9dret 2 Dret : dret is a true positive detection of dsrcg: Based on these sets, we define precision and recall of Dret regarding Dsrc and dplg as follows: Rationale for this definition is the fact that, retrieving more than one near-duplicate of a source document does not decrease precision, but it does not increase recall, either, since no additional information is obtained.
Finally, to measure the cost-effectiveness of a source retrieval algorithm in retrieving Dret, we count the numbers of queries and downloads made and compute the workload in terms of queries and downloads until the first true positive detection is made. The Source Oracle As described at the outset, the task subsequent to source retrieval is text alignment, where the candidate sources found are compared in closer detail with a given suspicious document (see Figure 1). Because of this connection, one problem of implementing and evaluating a source retrieval algorithm is that a working text alignment algorithm is required a priori. Since such an algorithm is not at hand, we decouple the source retrieval task from the text alignment task by means of a source oracle so as to allow for participation without the need to develop a text alignment algorithm. The oracle automatically enriches a downloaded document with information about whether or not it is considered a true positive source for the given suspicious document. Note that the oracle employs the aforementioned conditions to determine whether a document is a true positive detection. However, the oracle does not, yet, tell for which part of a suspicious document a downloaded document is a true positive detection detection. Hence, applying a custom text alignment strategy can still be beneficial. 2.4
Nine of the 14 participants submitted runs for the source retrieval task, eight of whom
also submitted a notebook describing their approach. An analysis of these descriptions
reveals the same building blocks that were commonly used in last year’s source retrieval
algorithms: (
The chunking strategies employed by the participants are no chunking (i.e., the whole document as one chunk) [3, 18, 42], 50-line chunks [3], TextTiling [14] to identify the topically related passage and therein 4-sentence chunking [13], paragraph chunking [19, 42], anomaly sections based on intrinsic plagiarism detection [21], 100word chunks [43], 5-sentence chunks [44], and combinations thereof. Note that chunks typically are stated as non-overlapping. An interesting question could be to identify the potential of overlapping chunks (except maybe in the cases of the whole-document and TextTiling chunks). Typical plagiarism cases have no fixed length and overlapping chunks would reduce the risk of, for instance, having more than one source in one chunk of 50 lines or 100 words, etc. Furthermore, relying on the given document structure (e.g., chunking by lines or paragraphs) bears the risk of failing for some unseen documents that are not as well-formatted as the ones in our evaluation corpus. Keyphrase Extraction Given a chunk, keyphrases are extracted from it in order to formulate queries with them. Rationale for keyphrase extraction is to select only those phrases (or words) which maximize the chance of retrieving source documents matching the suspicious document. Keyphrase extraction may also serve as a means to limit the amount of queries formulated, thus reducing the overall costs of using a search engine. This step is perhaps the most important one of a source retrieval algorithm since the decisions made here directly affect the overall performance: the fewer keywords are extracted, the better the choice must be or recall is irrevocably lost.
Phrasal search was provided by the Indri search engine. However, only Lee et al. [19] made use of Indri. Some participants use single keywords while others extract whole phrases. Most of the participants preprocessed the suspicious document by removing stop words before the actual keyphrase extraction. In particular, Elizalde [3] applies three different strategies. The first approach generates one query per 50-lines chunk containing the top-10 words scored by tf idf values; a word’s document frequency is obtained from the external Brown corpus. The second approach uses ten queries, each of which is formed by one of the ten longest named entities extracted from the whole document using the Python Natural Language Toolkit NLTK. The third approach uses 15 queries, each of which is formed by one of the top-15 noun phrases extracted via Barker and Cornacchia’s head noun phrase extractor [1]. Haggag and ElBeltagy [13] use KPMiner [2] to extract one keyphrase per topically related passage and combine this phrase with the rarest word (frequency on document level) of each 4-sentence chunk within the passage until the query contains ten keywords. Kong et al. [18] apply two different strategies. First, they use the top-20 words scored by tf idf values from the whole document; a word’s document frequency is obtained from the external Wall Street Journal corpus. Second, they use a Pat Tree [10] to extract one 2-gram, one 3-gram, two 4-grams, and forty 5-grams with highest tf idf scores that contain at least one of the top-10 tf idf terms. Lee et al. [19] use the most unique 8-gram from each paragraph (uniqueness determined via the Google Books n-grams) that starts with a word that is not contained in any keyphrase obtained for a previous paragraph. Nourian [21] use the first ten keywords of a chunk as one phrase (stopping or keyword extraction method not specified any further). Suchomel et al. [42] apply three different strategies. First, they use the top-5 words scored by tf idf values from the whole document; a word’s document frequency is obtained from an external web crawl. These top-5 keywords are then also combined with their most frequent two or three term collocations. Second they use a “representative” sentence of at least 6 words length from passages in the document that an intrinsic plagiarism detector identified as differing according to writing style; however, what makes a sentence representative is not explained. Third, for a paragraph they extract the longest sentence. Veselý et al. [43] use all the longest non-overlapping n-grams (n 5, n is odd) that return less than 300 but more than 0 results. To determine these keyphrases they basically employ the open end query formulation [36] with a query-level version of the User-over-Ranking hypothesis [39, 11]. Williams et al. [44] use a very similar and simplistic keyphrase extraction strategy: the first three disjunct sequential 10-grams of each “reduced” chunk (only nouns, adjectives and verbs) form the keyphrases. Note that this is very similar to the winning approach from 2012, where Jayapal [15] used the first such 10-gram per chunk only (also allowing pronouns).
Altogether, the participants’ approaches to keyphrase extraction can basically be
divided into four different categories. (
Search Control Given a set of queries, the search controller schedules their submission to the search engine and directs the download of search results. Rationale for this is to dynamically adjust the search based on the results of each query, which may include dropping queries, reformulating existing ones, or formulating new ones based on the relevance feedback obtained from search results. Some teams did not implement a search controller and simply submit all formulated queries. The ones who implemented search contral applied the following ideas.
Haggag and El-Beltagy [13] drop a query when more than 60% of its terms are contained in a previous downloaded document. Lee et al. [19] stop submitting queries when most of the suspicious document is found as plagiarized or the number of queries exceeds the number of paragraphs in the suspicious document. However, it remains unclear what “most of a document” means and why paragraphs are a good upper bound on the query budget. In practice, not all documents are well-formatted. A document without paragraph breaks would then allow for a single query only. Suchomel et al. [42] schedule queries dependent on the keyphrase extractor which extracted the words: the order of precedence corresponds to the order in which they have been explained above. Whenever later queries were formulated for portions of the suspicious document that were already mapped to a source, these queries are not submitted and discarded from the list of open queries.
Note that none of the teams did try to reformulate existing queries or formulating new ones based on the available number of search results, the search snippets, or the downloaded documents, which leaves significant room for improvement. Download Filtering Given a set of downloaded documents, a download filter removes all documents that are probably not worthwhile being compared in detail with the suspicious document. Rationale for this is to further reduce the set of candidates and to save invocations of the subsequent detailed comparison step.
In particular, Elizalde [3] focuses on the top-10 results of a query and downloads a result document when at least 90% of the words in a 160-character snippet are contained in the suspicious document. Haggag and El-Beltagy [13] only consider the top-ranked result and download it when at least 50% of the query terms are contained in a 500 character snippet. It remains unclear whether checking not single words but phrases from the snippets could be beneficial. Kong et al. [18] download a document when the cosine similarity of the snippet (presumably 500 characters long) and the suspicious document exceeds some threshold (not specified in the paper). Taking into account that ChatNoir’s snippets are centered around the passage of a search result that contains most of the query’s terms, the cosine similarity should typically be rather high for all search results (the query terms in the snippet are also contained in the suspicious document). Lee et al. [19] download the top-k results of a query (k is not specified) and documents that appear frequently in the results (frequency threshold not specified). Suchomel et al. [42] download documents when more than 20% of the word 2-grams in the 500 character snippet also appear in the suspicious document. Veselý et al. [43] first compute queries without submitting them and submit all queries computed when 20% of the words in the suspicious document are contained in the queries without computing any new query afterwards. The retrieval scores of ChatNoir are used to compute the sum of these scores for a document over all queries and in the end the 15 documents with highest sum are downloaded (however, this is not consistent with the overall workload we measured, which is around 31 downloads per suspicious document). Williams et al. [44] download the top-3 documents whose snippets share at least five word 5-grams with the suspicious document. Table 1 shows the performances of the nine plagiarism detectors that implemented source retrieval. Since there is currently no formula to organize retrieval performance and cost-effectiveness into an absolute order, the detectors are ordered alphabetically, whereas the best performance value for each metric is highlighted. As can be seen, there is no single detector that performs best on all accounts. Rather, different detectors have different characteristics. The detector of Williams et al. [44] achieves the best trade-off between precision and recall and therefore the best F1 value. This detector is followed closely by that of Haggag and El-Beltagy [13], which achieves best precision but mediocre recall, whereas the detector of Kong et al. [18] achieves best recall at the cost of poor precision. It is not easy to decide which of these detectors solves the task best, since each of them may have their justification in practice. For example, the detector of Haggag and El-Beltagy downloads only about six documents on average per suspicious document and minimizes the time to first detection. Despite the excellent trade-off of Williams et al.’s detector, it incurs the second-highest costs in terms of queries on average, which is more than thrice as much as the other mentioned detectors. Kong et al.’s detector has highest download costs, but one may argue that downloads are much cheaper than queries, and that in this task recall is more important than precision.
Interestingly, the ensemble of all submitted approaches would achieve an average recall of 0.82 retrieving all sources for 23 topics. Only for eight topics the recall is below 0.65 (which is the best individual average recall). For just three topics none of the detectors detects a source (one of which actually has none). 3
In text alignment, given a pair of documents, the task is to identify all contiguous passages of reused text between them. The challenge with this task is to identify passages of text that have been obfuscated, sometimes to the extent that, apart from stop words, little lexical similarity remains between an original passage and its plagiarized counterpart. Consequently, for evaluators, the challenge is to provide a representative corpus of documents that emulate this situation. To study this task, we employ a corpus construction methodology similar to that which has been used in previous evaluations of this task, while fixing some of its deficiencies. We evaluate the performance of plagiarism detectors based on the traditionally employed measures. Finally, we exploit the benefits of software submissions for the first time and evaluate the detectors that have been submitted last year on this year’s evaluation corpus and vice versa. 3.1 The evaluation corpus for text alignment is also based on the aforementioned WebisTRC-13. But instead of employing the documents of that corpus directly, pairs of documents that comprise reused passages have been constructed automatically, as was done in previous years [28]. One frequent point of criticism about automatically generating plagiarism is that it is difficult to ensure that documents between which text is plagiarized are about the same topic, so that plagiarism could be simply detected by analyzing topic drift [32]. Using the documents that have been retrieved manually as sources for the documents of the Webis-TRC-13 as a basis for constructing plagiarism cases, however, allows us to mitigate this problem.
Corpus Construction The corpus is constructed within eight steps: 1. Documents. The documents used for our corpus are web documents obtained from the ClueWeb 2009 corpus. We have compiled a set of documents on 145 topics which have been manually searched, browsed, and found relevant to a given topic. For each topic, we collected a set Dtopic that contains between 1 and 270 documents for a total of 10 630 documents. The documents have been judged by the writers who wrote documents on these topics for the aforementioned Webis-TRC-2012. 2. Pre-Processing. The HTML documents were converted to plain text, extracting their main content using the BoilerPipe library. Passages of text with eight words or less were discarded as well as documents with less than 100 words. In total, 6500 documents remained after pre-processing, divided into 144 remaining topics with at least two and up to 170 documents. 3. Withheld Documents. For each topic, one document is withheld in order not to be used for plagiarism. 4. Source Set Formation. Each of the suspicious documents that are generated in a later step has a set of source documents Dsrc. In this step, these sets are formed by randomly choosing a topic from which to draw documents, a targeted size jDsrcj between 5 and 75 documents, and documents from Dtopic until the targeted size is reached or no further documents are available: in each iteration, a document is chosen at random from Dtopic and added to Dsrc, unless a duplicate of the chosen document is already present in Dsrc. In this connection, we consider two documents duplicates if their n-gram cosine similarity is above 0.6 for n = 1, above 0.25 for n = 3, and above 0 for n = 8. These thresholds were experimentally determined with respect to the documents used.7 A total of 520 source sets were created this way. 5. Withheld Source Sets. In addition to the above source sets, for each topic one source set was created similarly but ensuring that no sentence of a source document has a duplicate sentence in the withheld document of that topic chosen in Step 3. Two sentences are considered duplicates if their 1-gram cosine similarity is above 0.9. 6. Passage Extraction. Assuming a log-normal distribution of document lengths, we estimate its parameters based on the documents withheld in Step 3. For a given source set Dsrc created in Step 4, we extract a set of passages from its documents, so that each passage is at least 50 words long, and every source document contributes at least one passage. Given these constraints, we sample the number of passages to be extracted per source document from a Poisson distribution with = 3, favoring lesser but larger passages. Further, passages are drawn so they are no duplicates of previously drawn passages (1-gram cosine similarity above 0.9), so they are not adjacent to previously drawn passages, and so that at least one passage is drawn per document. The resulting 520 passage sets contained between 5 and 134 passages. 7. Obfuscation. Every passage from the passage sets extracted in Step 6 is obfuscated in order to emulate plagiarist behavior based on the following four strategies: no obfuscation, random obfuscation, and, for the first time, cyclic translation obfuscation and summary obfuscation. With the exception of summary obfuscation which was constructed independently of the rest of the corpus, the strategies are applied uniformly distributed ensuring that only one strategy is applied in a suspicious document. Details about these strategies can be found below. 8. Suspicious Document Generation. In this step, a suspicious document dplg is generated by randomly concatenating an (obfuscated) passage set. Special care is taken with regard to their formatting so that paragraph breaks are no easy predictor of boundaries of reused passages.
Finally, the resulting suspicious documents are paired with their respective source documents, and the withheld documents are paired with the withheld source sets from Step 5 to form examples of document pairs that do not contain plagiarism. The corpus contains in total 3653 suspicious documents and 4774 source documents, which are grouped into 7 Note that the duplicate detector differs from that applied in source retrieval. In source retrieval, we adjusted the duplicate detector to maximize precision and therefore false positive detections, whereas in text alignment, we maximize recall for the same reason (i.e., to make sure that no unintended duplication is found between pairs of documents that may lead text alignment algorithms astray). 10000 pairs, so that there are 6000 pairs containing plagiarism (i.e., 2000 for each of the mentioned obfuscation strategies), 2000 containing unobfuscated plagiarism, and 2000 without plagiarism. Half of this corpus has been released as training corpus, and the other half was used as test corpus.
Random Obfuscation Random obfuscation is a naïve approach to obfuscation in that the resulting passages are not human-readable and bear no semantics. The purpose of this type of obfuscation is to test whether text alignment algorithms are capable of identifying reused passages from a bag-of-words model point of view. The obfuscation strategy itself is a sequence of random text operations such as shuffling, adding, deleting, and replacing words or short phrases at random. Replacing words is done based on a synonym database such as WordNet, and phrases are shuffled while maintaining the original part-of-speech sequence. Moreover, some sentences may be shuffled randomly. The longer the sequence of random operations, the more an obfuscated passage differs from its original, and presumably the more difficult it is to identify them automatically. This kind of obfuscation has been used in all previous evaluation corpora that have been employed to evaluate this task.
Cyclic Translation Obfuscation A new kind of obfuscation strategy we introduce this year is cyclic translation obfuscation. Here, a plagiarized passage of text is run through a sequence of translations, so that the output of one translation forms the input of the next one while the last language of the sequence is the same as the passage’s original language. Rationale of this strategy is to exploit the fact that translating a text inherently involves paraphrasing it, so that translating a text back and forth between languages is a way ob obtaining alternative versions of a text without changing its semantics.
We employ the APIs of three different translation web services for this task, namely Google Translate,8 Microsoft Translator,9 and MyMemory.10 In every cyclic translation sequence, all three services are employed, since preliminary experiments revealed that employing only one of the services yield little to no difference of the obfuscated text to its unobfuscated counterpart. An explanation for this may be found in the fact that the language models used for translation appear to deterministically favor one alternative translation over others, even across many languages, while different translation services are based on independently trained models which introduce more variation in a cyclic translation sequence.
A cyclic translation sequence is constructed randomly. The intermediate languages of a sequence—start and end are always English—are drawn at random from two sets of languages, namely the Indo-European languages French, German, Italian, Spanish, and Swedish, and a mixture of different language families such as Arabic, Chinese, Hebrew, Hindi, and Japanese. First, one of the two sets is chosen and then up to three intermediate languages are employed as intermediate languages.
Summary Obfuscation Another new kind of obfuscation strategy we introduce this year is summary obfuscation. Its rationale is that including an unattributed summary 8 http://translate.google.com 9 http://www.microsoft.com/translator 10 http://mymemory.translated.net of another’s document in one’s own text can be considered a case of plagiarism since the main ideas of the document are maintained in condensed form. Presuming that the summary is not based on a simple concatenation of some sentences from the original document, the lexical and syntactic similarities between summary and original document may be very restricted. Actually, summary obfuscation can be viewed as a form of plagiarism of ideas rather than the simple case of reusing exact phrases or sentences.
To build a corpus of plagiarism cases based on this idea, we used existing resources from the research field of automatic text summarization. In more detail, the set of original documents was taken from the Document Understanding Conference (DUC) 2001 corpus for text summarization.11 These are newswire stories and newspaper articles originally published in Wall Street Journal, Associated Press, San Jose Mercury News, Financial times, LA Times, and FBIS. For each such document there are two summaries of approximately 100 words created by human assessors. To produce plagiarism cases, these summaries were planted into documents of the DUC 2006 text summarization corpus.12 This corpus also comprises newswire stories and newspaper articles originally published in Associated Press, New York Times, and Xinhua News Agency. Hence, for each original document, we produced two plagiarism cases based on summary obfuscation. In addition, for each original document, eight more documents from DUC 2006 corpus were used as suspicious documents.
Given the genre of the DUC 2001 and DUC 2006 corpora, the similarity between an original document and its summary may be easily identified by using named-entity occurrences. That is, both the original document and its summary will talk about the same persons, locations, organizations, etc. To weaken this kind of similarity, we introduced some noise in the DUC 2006 documents by replacing some of their named-entities with the named-entities of the original document or the named-entities of the summaries. In particular, we randomly selected some parts of the suspicious documents of similar length to the summaries (e.g., 100 words) and transformed them to noisy areas by replacing their named-entities with those of the original text. That way, a suspicious document appears similar to the original since it refers to the same proper names, locations, organizations, etc. The popular Stanford Named Entity Recognizer13 was used to detect time, location, organization, person, money, percent, and date entities in original and suspicious documents. In total, there are 237 original documents and 2607 suspicious ones in this part of our corpus. 474 of the suspicious documents contain plagiarized summaries, and 1896 are noisy documents. 3.2
To assess the performance of the submitted detailed comparison approaches, we employ the performance measures used in previous evaluations. For this paper to be selfcontained, we summarize the definition found in [28]: let S denote the set of plagiarism cases in the corpus, and let R denote the set of detections reported by a plagiarism detector for the suspicious documents. To simplify notation, a plagiarism case 11 http://www-nlpir.nist.gov/projects/duc/data/2001_data.html 12 http://www-nlpir.nist.gov/projects/duc/data/2006_data.html 13 http://nlp.stanford.edu/software/CRF-NER.shtml prec(S; R) = 1 jRj r2R
S X j s2S(s u r)j ;
jrj where s u r = r ec(S; R) = 1 jSj s2S
S X j r2R(s u r)j ;
jsj s \ r if r detects s, otherwise. s = hsplg; dplg; ssrc; dsrci, s 2 S, is represented as a set s of references to the characters of dplg and dsrc, specifying the passages splg and ssrc. Likewise, a plagiarism detection r 2 R is represented as r. Based on this notation, precision and recall of R under S can be measured as follows: ; Observe that neither precision nor recall account for the fact that plagiarism detectors sometimes report overlapping or multiple detections for a single plagiarism case. This is undesirable, and to address this deficit also a detector’s granularity is quantified as follows: g ran(S; R) = 1
X jSRj s2SR jRsj; where SR S are cases detected by detections in R, and Rs R are detections of s; i.e., SR = fs j s 2 S ^ 9r 2 R : r detects sg and Rs = fr j r 2 R ^ r detects sg. Note further that the above three measures alone do not allow for a unique ranking among detection approaches. Therefore, the measures are combined into a single overall score as follows: plagdet(S; R) =
F1
log2(1 + g ran(S; R))
;
where F1 is the equally weighted harmonic mean of precision and recall.
Nine of the 18 submitted detectors implement text alignment, and for six of them also a
notebook describing their approach has been submitted. An analysis of these notebooks
reveals a number of building blocks that are commonly used to build text alignment
algorithms: (
Seeding Given a suspicious document and a source document, matches (so-called „seeds”) between the two documents are identified using some seed heuristic. Seed heuristics either identify exact matches or create matches by changing the underlying texts in a domain-specific or linguistically motivated way. Rationale for this is to pinpoint substrings that altogether make up for the perceived similarity between suspicious and source document. By coming up with as many reasonable seeds as possible, the subsequent step of extending them into aligned passages of text becomes a lot easier.
A number of seed heuristics have been applied by this year’s participants: Rodríguez Torrejón and Martín Ramos [34] use sorted word 3-grams and two kinds of sorted word 1-skip-3-grams. Kong et al. [18] use sentence pairs as seeds which exceed a given similarity threshold. Suchomel et al. [42] use sorted word 4-grams and unsorted stop word 8-grams (the latter having been introduced in [38]). Shrestha and Solorio [37] also use stop word 8-grams in addition to named entity 5-grams (i.e., word 5-grams that contain at least one named entity) as well as all other word 5-grams. The latter, however, are processed separately from the former, since otherwise they would subsume the named entitiy n-grams. Also, at the expense of runtime, they introduce inexact n-gram matching (i.e., the n-grams need not overlap entirely but their Jaccard similarity must be above a given threshold). Palkovskii and Belov [24] use sorted word 5-grams. Before computing seeds, some participants choose to collapse whitespace, reduce cases, remove stop words, and stem the remaining words, if applicable to their respective seed heuristics. Extension Given seed matches identified between a suspicious document and a source document, they are extended into aligned text passages between the two documents of maximal length, which are then reported as plagiarism detections. Rationale for merging seed matches is to determine whether a document contains plagiarized passages at all rather than just seeds matching by chance, and to identify a plagiarized passage as a whole rather than only its fragments.
Most of the participants’ extension heuristics are rule-based, merging seeds into aligned passages if they are adjacent in both suspicious and source document and the size of the gap between them is below some threshold. The exact rule depends on the seeds used, and instead of using just one rule, many participants develop sets of constraints that have to be fulfilled by aligned passages in order to be reported as plagiarism detections. Since the rules are usually highly involved with their respective setup, we exemplify only one rule set here in order to give an idea of what they may look like: Suchomel et al. [42] employ a 2-step merge heuristic, where in the first step, adjacent seed matches that are no more than 4000 chars apart are merged. The resulting passages from the first step are then merged again, considering pairs of adjacent passages in turn, and checking if the gap between them contains at least four seeds so that there is at least one seed per 10 000 chars of gap length between them. To be merged, adjacent passages further have to fulfill the constraints that their gap is smaller than 30 000 chars, that their combined size is bigger than twice the gap size, and that the ratio of seeds per chars of the adjacent passages does not drop by a factor of more than three in the potentially merged passage. The only participants who go beyond rule-based merging are Palkovskii and Belov [24], who employ clustering for unsupervised merging. Filtering Given a set of aligned passages, a passage filter removes all aligned passages that do not meet certain criteria. Rationale for this is mainly to deal with overlapping passages and to discard extremely short passages.
Kong et al. [17] discard passages whose word overlap under a modified Jaccard coefficient is below a threshold. Suchomel et al. [41] discard overlapping passages that are shorter than 300 chars, and keep only the passages longer than 300 chars. Palkovskii and Belov [23] discard passages shorter than 190 chars. Gillam et al. [8] discard passages shorter than 50 words that have less than 0.75 cosine similarity under a vector space model. Other participants do not apply passage filtering. Remarks Since six of this year’s participants took part in previous years as well, many of them simply reuse their earlier solutions. While there is no problem with doing so, innovative ideas become less frequent compared to parameter tuning and small adjustments to an algorithm. However, this year’s best performing approach submitted by Rodríguez Torrejón and Martín Ramos [34], has been evaluated for the fourth time in a row, showing that persistent development may eventually yield good results. A number of new ideas could be observed: – Palkovskii and Belov [24] continue their development of obfuscation-specific detection approaches by targeting summary obfuscation in particular. – Suchomel et al. [42], Rodríguez Torrejón and Martín Ramos [34], and Shrestha and Solorio [37] employ more than one seed heuristic at the same time. In particular, the seed heuristic of Stamatatos [38] based on stop word n-grams is used more often. – Shrestha and Solorio [37] employ inexact seed matching (i.e., in order for a pair of seeds to be linked across documents, they need not be exactly equal but only approximately equal according to some similarity measure). This approach may lead to more relaxed seeding heuristics, but also introduces runtime overhead. 3.4
In this section, we report on the evaluation of this year’s submissions on the aforementioned evaluation corpus. Moreover, we conduct the first cross-year evaluation of all softwares submitted last year and this year on the evaluation corpora of both years. We further differentiate performance with regard to obfuscation strategies to provide insights into how the softwares deal with different strengths of obfuscation. In addition to that, we reveal how the changes made to softwares that have been submitted in different versions in both years affect performance, and whether or not they improved. We also shed light on the question of corpus difficulty and find that last year’s evaluation corpus was more difficult than this year’s corpus.
Overall Results of 2013 Table 2 shows the overall performances of the nine plagiarism detectors that implement text alignment and were submitted this year. The overall best performing approach is that of Rodríguez Torrejón and Martín Ramos [34], closely followed by that of Kong et al. [18]. The former detector has unbalanced precision and recall, while the latter does, but with worse granularity. The three new approaches submitted this year from Saremi and Yaghmaee [35], Shrestha and Solorio [37] and Nourian [21] achieve mid-range performances. Two detectors’ performances do not exceed the baseline. In terms of precision and granularity, almost all detectors perform well, whereas recall sets them apart. In terms of runtime, all detectors are in the range of minutes, one requiring only 1.2 minutes, while two others lag far behind because they employ resource-intensive named entity recognition algorithms.
Cross-Year Evaluation of 2012 and 2013 Tables 3 to 6 show the performances of all 18 plagiarism detectors submitted last year and this year that implement text alignment on both years’ respective evaluation corpora. The overall performance of the detectors with regard to the plagdet score can be found in Table 3. As can be seen, the best performing detectors across both years are those of Oberreuter et al. [22] and Kong et al. [17], both of which have been first evaluated in 2012. This year’s best performing detectors from Rodríguez Torrejón and Martín Ramos [34] comes close to them, however, only when evaluated on the 2013 evaluation corpus. On the 2012 corpus it is far off, which suggest this detector may be overfitted to the 2013 corpus.
Regarding different obfuscation strategies, it appears the detectors’ performances on the 2012 corpus correlate mostly with their overall performance, but on the 2013 corpus this is not the case. Especially for summary obfuscation, the two best performing detectors are from Suchomel et al. [41, 42] which otherwise achieve mid-range performance only. The detector of Oberreuter et al. [22] fails on summarized plagiarism, while that of Kong et al. [17] achieves third-best performance on this kind of obfuscation. It is unfortunate that the detector of Palkovskii and Belov [24], which implements a detection approach that targets summary obfuscation, does not compete with the others. Regarding unobfuscated plagiarism (column “None” in the tables), it is interesting to observe that many detectors detect this kind of plagiarism with scores above 0.9 on the 2012 corpus but less so in the 2013 corpus where the scores are mostly below that number. It is unclear why this is the case, since unobfuscated plagiarism is not changed when inserted into a suspicious document with the exception of text formatting.
Table 4 shows the detectors’ performances with regard to precision. In general, achieving a high precision appears to be less of a problem compared to achieving a high recall. This is underpinned by the fact that our basic baseline approach outperforms almost all detectors in precision. However, the detectors that perform best in precision typically have deficiencies in terms of recall, but not the other way around: the aforementioned overall best performing detectors achieve mid-range precision. The only obfuscation strategy that poses a comparably higher challenge in terms of precision is random high obfuscation, which has been adjusted to emulate extreme obfuscation. Table 5 shows the detectors’ performances with regard to recall. The best performing detectors are the two versions submitted by Kong et al. [17, 18]. They dominate all others in terms of recall, but not each other; the 2013 version performs best on the 2012 corpus and vice versa. By contrast, this year’s best performing detector by Rodríguez Torrejón and Martín Ramos [34] achieves only mid-range recall. The best performing approaches with regard to precision from Gillam et al. [8, 7] performs poor with regard to recall, suggesting that the implemented approach is too conservative for this Software Submission Team Year Nourian 2013 Jayapal 2012 Baseline R. Torrejón Oberreuter Gillam Gillam Jayapal Shrestha Kueppers Saremi Kong Suchomel Kong R. Torrejón Palkovskii Palkovskii Suchomel Sánchez-Vega Baseline Gillam Gillam Jayapal task; the authors reveal their primary concerns up to now has been near-duplicate texts instead of paraphrases. In general, a visible correlation of recall performance on the entire 2012 corpus with the performances for each obfuscation strategy can be observed, and to a lesser extent on the 2013 corpus. Table 6 shows the detectors’ performances with regard to granularity. Many detectors achieve perfect granularity on almost all obfuscation strategies, which may indicate that the problem of fragmented detections of a contiguous plagiarism case is under control, however, especially obfuscated plagiarism naturally poses a higher challenge with regard to this performance measure, which can be seen looking at random obfuscation as well as summary obfuscation. The only approaches that apparently do not do anything about granularity appear to be the ones of Jayapal [15, 16]. In general, however, these numbers must be taken with a grain of salt, since participants often resort to post-retrieval filtering in order to optimize granularity only for the sake of achieving a good ranking instead, while some admit that they would not do this in practice.
Comparing Detector Versions between 2012 and 2013 Since six of nine teams submitted versions of their detectors in both 2012 and 2013, this allows for an analysis of performance changes across versions of the same detector, and whether the adjustments made pay off in terms of improved performance. Such analyses are a novelty for evaluation labs such as ours and they yield insights into task design. Figure 3 shows the performance differences of each of the six detector pairs when subtracting their respective 2012 performance values from their 2013 ones for each of our four performance measures and both years’ evaluation corpora. Regarding plagdet, half of the participants achieve a performance improvement and the other half decreased the performance of their detectors. The highest performance gain of about 0.08 plagdet performance was achieved by Jayapal and Goswami [16], but only on the 2013 corpus, while Palkovskii and Belov [24] suffer a significant performance loss of at least 0.2 plagdet on both corpora. Clearly, the modifications made on the latter detector should be carefully reviewed or even reverted.
Precision and recall are related measures in that one can typically be traded for the other. Regarding them, it can be seen that all but one detector decrease in precision performance, but only Jayapal and Goswami [16] and Suchomel et al. [42] materialize a return in terms of increased recall. The modifications made by Gillam [7] and Kong et al. [18] result in a slight decrease of recall, and those of Palkovskii and Belov [24] in a big loss of recall. The detector of Rodríguez Torrejón and Martín Ramos [34] improves a lot in terms of precision but only slightly in terms of recall. Finally, all but one detector suffer losses in terms of granularity. Most of these losses are rather small, except for that of Jayapal and Goswami [16] which amounts to more than 10 granularity loss. Comparing Corpus Versions between 2012 and 2013 Software submissions not only allow for more sustainable evaluations and assessing software versions in terms of performance changes, but also for measuring the difficulty of different evaluation corpora. By evaluating every detector on the evaluation corpora of both 2012 and 2013, a distribution of comparable performance values is obtained that sheds light on how difficult it is to identify cases of plagiarism in the two corpora relative to each other. Figure 4 shows the performances of all detectors on both corpora for each of our four performance mea
PAN'12 corpus PAN'13 corpus Gillam Jayapal Kong Palkovskii S uchomel Torrejón Gillam Jayapal Kong Palkovskii S uchomel Torrejón Precis ion Difference
R ecall Difference PAN'12 corpus PAN'13 corpus PAN'12 corpus
PAN'13 corpus sures, ordered from high to low performance. In terms of plagdet, the 2013 evaluation corpus is consistently easier than the 2012 corpus. While the recall curve difference is comparable to that of the plagdet curves, the precision curves are closer to each other, which indicates that precision difficulty is similar across both corpora. In terms of granularity, more than half of the detectors achieve almost equal performance. The remainder perform better on the 2013 corpus because their granularity values are smaller. As a result, our revised corpus construction process outlined above yields plagiarism cases that are more easily detected in general.
Besides these differences, the obfuscation-specific performances shown in Tables 3 to 6 show that the random obfuscation strategy employed in 2013 compares to that of random low obfuscation of 2012; despite other intentions, we did not accomplish to hit the middle ground between random low and random high obfuscation, which contributes to the 2013 corpus being less difficult. The cyclic translation obfuscation appears to be on a level of difficulty similar to that of random (low) obfuscation, since most detectors achieve similar performances on them. The most difficult portions of the 2012 corpus is random high obfuscation, and that of the 2013 corpus is summary obfuscation, which can be seen particularly when considering recall performance.
P lagdet
PAN'12 corpus PAN'13 corpus
Granularity
PAN'13 corpus 8 4 2 1 1 1 The differences in performance across the resulting curves indicate corpus difficulty relative to each other. The bigger—smaller, in case of granularity—the area under a curve, the easier the corresponding corpus.
With this fifth international competition on plagiarism detection at PAN 2013 we
introduced a number of improvements to the evaluation methodology for plagiarism
detectors. (
This year, we collected a total of 18 plagiarism detectors from 14 teams, half of which implement source retrieval and the other half text alignment. For the task of text alignment, this is the second year in which we ask for software submissions instead of run submissions, which gives us the opportunity to conduct a cross-year evaluation of all detectors submitted in both years on all evaluation corpora available. Our crossyear evaluation reveals that this year’s best performing detector cannot keep up with the best performing detectors of last year. Moreover, considering the six detectors that have been submitted in both years, we analyzed whether their retrieval performance has been improved. In fact, for three of the detectors this is not the case; i.e., the adjustments should be carefully reviewed or even reverted. Finally, we used the submitted softwares to compare the difficulty of our corpora, and we found out that this year’s evaluation corpus is significantly easier than the last year’s corpus in terms of detecting the contained plagiarism cases.
Altogether, we see a lot of room for further improvement with respect to both the methodology of plagiarism detection evaluation and the organization paradigm of software submissions.
We thank the participating teams of this task for their devoted work. This paper was partially supported by the WIQ-EI IRSES project (Grant No. 269180) within the FP7 Marie Curie action. [6] Pamela Forner, Roberto Navigli, and Dan Tufis, editors. CLEF 2013 Evaluation Labs and Workshop – Working Notes Papers, 23-26 September, Valencia, Spain, 2013. URL http://www.clef-initiative.eu/publication/working-notes. [7] Lee Gillam. Guess Again and See if They Line Up: Surrey’s Runs at Plagiarism
Detection—Notebook for PAN at CLEF 2013. In Forner et al. [6].
[8] Lee Gillam, Neil Newbold, and Neil Cooke. Educated Guesses and Equality
Judgements: Using Search Engines and Pairwise Match for External Plagiarism
Detection—Notebook for PAN at CLEF 2012. In Forner et al. [5]. ISBN
978-88-904810-3-1. URL
http://www.clef-initiative.eu/publication/working-notes.
[9] Tim Gollub, Benno Stein, and Steven Burrows. Ousting Ivory Tower Research:
Towards a Web Framework for Providing Experiments as a Service. In Bill
Hersh, Jamie Callan, Yoelle Maarek, and Mark Sanderson, editors, 35th
International ACM Conference on Research and Development in Information
Retrieval (SIGIR 12), pages 1125–1126. ACM, August 2012. ISBN
978-1-4503-1472-5. doi: http://dx.doi.org/10.1145/2348283.2348501.
[10] Gaston H. Gonnet, Ricardo A. Baeza-Yates, and Tim Snider. New indices for
text: Pat trees and pat arrays. In Information Retrieval: Data Structures &
Algorithms, pages 66–82. 1992.
[11] Matthias Hagen and Benno Stein. Applying the User-over-Ranking Hypothesis
to Query Formulation. In Advances in Information Retrieval Theory. 3rd
International Conference on the Theory of Information Retrieval (ICTIR 11),
volume 6931 of Lecture Notes in Computer Science, pages 225–237, Berlin
Heidelberg New York, 2011. Springer. doi:
http://dx.doi.org/10.1007/978-3-642-23318-0_21.
[12] Matthias Hagen and Benno Stein. Candidate Document Retrieval for Web-Scale
Text Reuse Detection. In 18th International Symposium on String Processing
and Information Retrieval (SPIRE 11), volume 7024 of Lecture Notes in
Computer Science, pages 356–367, Berlin Heidelberg New York, 2011. Springer.
doi: http://dx.doi.org/10.1007/978-3-642-24583-1_35.
[13] Osama Haggag and Smhaa El-Beltagy. Plagiarism Candidate Retrieval Using
Selective Query Formulation and Discriminative Query Scoring—Notebook for
PAN at CLEF 2013. In Forner et al. [6].
[14] Marti A. Hearst. TextTiling: Segmenting text into multi-paragraph subtopic
passages. Computational Linguistics, 23(
Association for Computational Linguistics. [29] Martin Potthast, Andreas Eiselt, Alberto Barrón-Cedeño, Benno Stein, and Paolo Rosso. Overview of the 3rd International Competition on Plagiarism Detection. In Vivien Petras, Pamela Forner, and Paul D. Clough, editors, Working Notes Papers of the CLEF 2011 Evaluation Labs, September 2011. ISBN 978-88-904810-1-7. URL http://www.clef-initiative.eu/publication/working-notes. [30] Martin Potthast, Tim Gollub, Matthias Hagen, Jan Graßegger, Johannes Kiesel, Maximilian Michel, Arnd Oberländer, Martin Tippmann, Alberto Barrón-Cedeño, Parth Gupta, Paolo Rosso, and Benno Stein. Overview of the 4th International Competition on Plagiarism Detection. In Pamela Forner, Jussi Karlgren, and Christa Womser-Hacker, editors, Working Notes Papers of the CLEF 2012 Evaluation Labs, September 2012. ISBN 978-88-904810-3-1. URL http://www.clef-initiative.eu/publication/working-notes. [31] Martin Potthast, Matthias Hagen, Benno Stein, Jan Graßegger, Maximilian Michel, Martin Tippmann, and Clement Welsch. ChatNoir: A Search Engine for the ClueWeb09 Corpus. In Bill Hersh, Jamie Callan, Yoelle Maarek, and Mark Sanderson, editors, 35th International ACM Conference on Research and Development in Information Retrieval (SIGIR 12), page 1004. ACM, August 2012. ISBN 978-1-4503-1472-5. doi: http://dx.doi.org/10.1145/2348283.2348429. [32] Martin Potthast, Matthias Hagen, Michael Völske, and Benno Stein.
Crowdsourcing Interaction Logs to Understand Text Reuse from the Web. In 51st Annual Meeting of the Association of Computational Linguistics (ACL 13) (to appear). ACM, August 2013. doi: http://dx.doi.org/. [33] Stephen E. Robertson, Hugo Zaragoza, and Michael J. Taylor. Simple BM25 extension to multiple weighted fields. In Proceedings of the 2004 ACM CIKM International Conference on Information and Knowledge Management, Washington, DC, USA, November 8-13, 2004, pages 42–49, 2004. [34] Diego A. Rodríguez Torrejón and José Manuel Martín Ramos. Text Alignment Module in CoReMo 2.1 Plagiarism Detector—Notebook for PAN at CLEF 2013.
In Forner et al. [6]. [35] Mehrin Saremi and Farzin Yaghmaee. Submission to the 5th International Competition on Plagiarism Detection. http://www.webis.de/research/events/pan-13, 2013. URL http://www.clef-initiative.eu/publication/working-notes. From Semnan University, Iran. [36] Jacob Shapiro and Isak Taksa. Constructing web search queries from the user’s information need expressed in a natural language. In Proceedings of the 2003 ACM Symposium on Applied Computing (SAC), March 9-12, 2003, Melbourne, FL, USA, pages 1157–1162. ACM, 2003. [37] Prasha Shrestha and Thamar Solorio. Using a Variety of n-Grams for the Detection of Different Kinds of Plagiarism—Notebook for PAN at CLEF 2013.
In Forner et al. [6]. [38] Efstathios Stamatatos. Plagiarism Detection Using Stopword n-Grams. JASIST, 62(12):2512–2527, 2011. doi: http://dx.doi.org/10.1002/asi.21630. [39] Benno Stein and Matthias Hagen. Introducing the User-over-Ranking Hypothesis. In Advances in Information Retrieval. 33rd European Conference on IR Resarch (ECIR 11), volume 6611 of Lecture Notes in Computer Science, pages 503–509, Berlin Heidelberg New York, April 2011. Springer. doi: http://dx.doi.org/10.1007/978-3-642-20161-5_50. [40] Benno Stein, Sven Meyer zu Eißen, and Martin Potthast. Strategies for Retrieving Plagiarized Documents. In Charles Clarke, Norbert Fuhr, Noriko Kando, Wessel Kraaij, and Arjen P. de Vries, editors, 30th International ACM Conference on Research and Development in Information Retrieval (SIGIR 07), pages 825–826, New York, July 2007. ACM. ISBN 987-1-59593-597-7. doi: http://dx.doi.org/10.1145/1277741.1277928. [41] Šimon Suchomel, Jan Kasprzak, and Michal Brandejs. Three Way Search Engine Queries with Multi-feature Document Comparison for Plagiarism Detection—Notebook for PAN at CLEF 2012. In Forner et al. [5]. ISBN 978-88-904810-3-1. URL http://www.clef-initiative.eu/publication/working-notes. [42] Šimon Suchomel, Jan Kasprzak, and Michal Brandejs. Diverse Queries and Feature Type Selection for Plagiarism Discovery—Notebook for PAN at CLEF 2013. In Forner et al. [6]. [43] Ondrˇej Veselý, Tomáš Foltýnek, and Jirˇí Rybicˇka. Source Retrieval via Naïve Approach and Passage Selection Heuristics—Notebook for PAN at CLEF 2013.
In Forner et al. [6]. [44] Kyle Williams, Hung-Hsuan Chen, Sagnik Ray Chowdhury, and C. Lee Giles.
Unsupervised Ranking for Plagiarism Source Retrieval—Notebook for PAN at CLEF 2013. In Forner et al. [6].