This paper describes an approach submitted to the 2013 PAN competiton for the source retrieval sub-task. Three different methods for extracting queries were used, which employed tf-idf, noun phrases and named entities, in order to submit very different queries and maximize recall. To plagiarize is to take someone else's work or ideas and pose it as your own. It has become a major problem for Universities and other academic institutions since Internet has become widespread. Current plagiarism detection methods should check the whole Web to find possible matches. For this reason, since last year, the Plagiarism detection track in PAN1 has been divided in two sub tasks: source retrieval and detailed comparison. This notebook reports an approach presented to the PAN 2013 plagiarism competition for the first sub-task.
Introduction
Source retrieval - sometimes called candidate document retrieval - is the first step of the
plagiarism detection process. It consists in finding a set of documents which are likely
to contain plagiarism, analyzing the document from a global perspective, either by using
an index or querying a search engine. After this stage a second step is performed:
detailed comparison, in which the previously retrieved documents are compared
exhaustively against the suspicious document. Source retrieval is a recall oriented problem,
since in the second step it is possible to increase the precision of the overall system,
while sometimes lowering the recall[
This year the corpus utilized was ClueWeb09[7] and two different search engines were available to search it: ChatNoir[6] and Indri[8]. The ChatNoir engine only supports keyword search, while Indri has quite a complex query language grammar. Both engines were used in this work: ChatNoir for keyword queries and Indri where an exact match to a phrase was needed.
The approach used to solve this task consisted in three different strategies to find
plagiarized texts, which will be discussed in the following subsections. It was developed
using Python and the Natural Language Toolkit [
This first strategy consisted in keyword based queries, submitted to the ChatNoir
engine. The text was divided in 50 line chunks, non alphabetical characters and stopwords
were removed. Lemmatization was applied using the WordNet lemmatizer[
For this approach, NLTK was used to identify Named Entities which were ranked according to the amount of words included. The top 10 entities were submitted to Indri to search for an exact match. This yields at most 10 queries per document.
The rationale behind this is that even when there is some paraphrasing, the Named Entities (places, people, etc) will remain unchanged. Also, the longest NEs will be less common and hence appear in less documents. 2.3
Finally, an existing keyphrase extractor was adapted to the task of plagiarism detection.
Barker and Cornacchia[
In this work, the default NLTK POS tagger was used, and the noun phrases were found by using fixed patterns. With m = 20 and n = 15, this strategy generated at most 15 queries per document.
A slight modification to the algorithm was introduced: all the nouns present were used in the ranking, not just the head nouns. For example, in the phrase “the Church of Ireland”, the phrase would count both towards “Church” and “Ireland”.
The queries were posed to the Indri search engine. 2.4
In all cases, only the top 10 results of every query were analyzed. For each result, a 160 character snippet was requested. The words were POS-tagged and only verbs, adjectives and nouns were considered. If more than 90% of those words (or their stemmed form) were present in the suspicious text, the document was regarded as promising and downloaded. Submission elizalde13 foltynek13 gillam13 haggag13 kong13 lee13 nourian13 suchomel13 williams13 The goal behind using three different approaches of query extraction, with different chunk lengths is to generate different sets of queries, thus maximizing recall. This sacrifices precision. The reasoning behind this is that the second phase in plagiarism detection - detailed comparison - will improve performance, while recall won’t be improved, but rather lowered. The results obtained in the competition clearly are a consequence of these decisions.
Since in some contexts queries are charged while downloads aren’t, another decision made was to minimize the number of queries. For that reason, very large chuncks (50 lines) were used for the first strategy, while for the other strategies a fixed lower bound on the amount of queries was set (10 and 15 queries per document, respectively). However, a large number of documents (10) were downloaded for each query, to ensure recall was high.
When looking at the results, we can see that the average queries per document are 44:5, while the average downloads are 107:22. This yields approximately 2:4 downloads per query, which is far lower than 10. There are two reasons that can explain this: on one side, two of the strategies employ exact match searches, which typically result in fewer documents. On the other side, this could mean that filtering downloads using the text snippets lowers the number of downloaded documents dramatically. 6. Potthast, M., Hagen, M., Stein, B., Graßegger, J., Michel, M., Tippmann, M., Welsch, C.: ChatNoir: A Search Engine for the ClueWeb09 Corpus. In: Hersh, B., Callan, J., Maarek, Y., Sanderson, M. (eds.) 35th International ACM Conference on Research and Development in Information Retrieval (SIGIR 12). p. 1004. ACM (Aug 2012) 7. Potthast, M., Hagen, M., Völske, M., Stein, B.: In: 51st Annual Meeting of the Association of Computational Linguistics (ACL 13) 8. Strohman, T., Metzler, D., Turtle, H., Croft, W.B.: Indri: a language-model based search engine for complex queries. Tech. rep., in Proceedings of the International Conference on Intelligent Analysis (2005)