This paper describes an architecture of the source retrieval system used at PAN 2014 lab on uncovering plagiarism, authorship, and social software misuse. The system is based on the systems used in past years at PAN 13 [6] and PAN 12 [5]. The majority of features were adapted with some improvements described in this paper. The source retrieval subsystem forms an integral part of a modern system for plagiarism discovery.
Systems which compute similarities between documents can significantly help with plagiarism detection. They automate the tedious work such as locating possible sources of plagiarism and finding similar text passages. If the suspicious passages are highlighted by the system, the supervisor only checks whether a passage is a plagiarism case or not. The state of the art anti-plagiarism systems evaluate document similarities in order to select suspicious passages of a source document. This evaluation is refered in PAN as the document alignment. Documents are algorithmically aligned to a corpus of known documents. However, if the corpus does not contain the original document the similarity between the original and the suspicious document can not be detected. Therefore a potential source documents should be retrieved from all documents prior to text alignment calculations. The corpus of all documents is usually very large, for example the entire web, and a search engine is utilized as a retrieval tool. It is ideally utilized automatically in the same manner as plagiarizing users would do manually. The global view of the system for unoriginal text detection is depicted in figure 1.
With the usage of a given search engine, the problem of source retrieval is then reduced to the problem of combining proper queries and passing them to the search engine. Selecting and downloading search engine results also influence total performance of the source retrieval system. The queries pose the most expensive piece of operation, whereas the downloads are quite cheap. In the real-world scenario, we are often limited by a number of queries executed in a given time period or by a total number of queries per document. During the operation the system should maximize the recall and precision of retrieved results and also minimize the total number of executed queries as much as possible.
The following sections describe the key parts of the system for source retrieval used
at PAN 2014. More information about the task and the competition can be found in the
task overview written by the lab organizers [
The Web
Plagiarism discovery Candidate Document
Retrieval
Candidate Documents
Detailed Document Comparison Documents
Indices Additional Sources
Knowledge-based post-processing
Similar Passages
For obtaining the search results two search engines have been utilized: The
Chatnoir [
Several types of queries were prepared.This year we combined keywords-based queries, paragraph-based queries and headers-based queries together. Some of the prepared queries could be discarded from the execution, no query refinement was applied according to the results, but some of the top scored keywords could appear in the different combination in more than one query. 2.1 From the whole suspicious document, there were extracted keywords using TF-IDF scoring of lemmas created via Python NLTK lemmatizer and omitting english stopwords. Firstly we created so called pilot query from top scored six terms. This query was passed to both Chatnoir and Indri search engines with Indri setting for combine belief operator of the query.
Based on the three top-scored single-term keywords, their collocations of two more words were extracted. Those served as phrasal search queries, which were passed to Indri in order to lookup a contextual occurrence of the selected keywords. All other subsequent keywords-based queries were passed to Chatnoir only and they were created by combining collocations of top scored keywords together with the rest of the extracted keywords up to 6 tokens long. From the rest of the keywords, if any, the remaining six-term long queries were created. The total number of used keywords-based queries depended on each document characteristics. From some of the documents, there were identified fewer keywords than from others. Usually there were prepared around 10 keywords-based queries per document. 2.2
From each paragraph of the suspicious document, a single paragraph-based query was created. The longest sentence from the specific paragraph was used to build the query. From the selected sentence six subsequent terms were selected from a random position within the sentence. The query was created from those six terms, only the punctuation was removed. The resulting query was passed to the Indri search engine with proximity term number of 1 denoting the phrasal search. 2.3
The headers-based queries were used in the form in which they appeared in the text
limited to six words in length. They were passed to Indri as a phrasal query as well. For
discovering the headers in the text the approach adopted from Suchomel et al. 2012 [
Three types of text chunking were applied: sentence and word chunking for keywords extraction; headers detection; and paragraph chunking. For each type of query the corresponding chunking method was always applied on the whole document from the beginning. 3
All prepared queries were processed according to their priority. Starting with the keywords-based, then the paragraph-based and last the header-based queries. Only all keywords-based queries – the pilot query, collocation queries and remaining keywordscombined queries – were executed for each suspicious document. After each query all its results were processed and positions of discovered similarities were stored. To each subsequent (not keywords-based) query its position was also attached, if that position collided with any of already found similarities, the query was omitted from the queue of prepared queries. 4
For each search engine result a snippet based on the given query can be obtained prior to the full document download. The snippet is obtained via ChatNoir API, it is considered as no operation and it provides rough information about the document content. It contains a portion of the document up to 500 characters around a given textual string. We generated the snippet for all documents from results based on specific query for each term in that query. The all snippets from one document were concatenated together and its tuples of two terms were compared with tuples from the suspicious document. Concordance of the tuples of 20 % or more was the threshold for decision about the document download.
The downloaded results were textually aligned to the suspicious document using
feature type selection for computing similarities described in Suchomel, Kasprzak et
al. 2013 [
This paper described the key aspects and changes from our erstwhile systems for candidate document retrieval used at PAN 14 lab on uncovering plagiarism. The architecture stems from PAN 12 and PAN 13 labs and the real-world anti-plagiarism system which is in use at Masaryk University. The results of the PAN show that this approach is one of the best for a real-life adoption, since it achieved a decent recall with just a fraction of used queries. Such approach is applicable for detection of suspicious texts, which may contain plagiarism, that can then be selected for further investigation.