This paper describes our approach to the Detailed Analysis subtask of the PAN 2012 competition. Our experiments deal with monolingual plagiarism cases, only. We use a simple set-based algorithm, that employs Dice's coefficient as a similarity measure. Furthermore we employ basic strategies from Information Retrieval and Natural Language Processing for stop word removal and language detection. We achieved the 7th place with an overall PlagDet score of 0.35.
2.1
Our detection algorithm is simple and works as follows: in a first step we use a language
detection algorithm to determine the natural language the document is written in. This
language detector is based on the classification algorithm from [
The next processing step deals with the tokenization of the raw text, whereby we store the original positions of the token within the document. We perform only a simple string normalization by collapsing whitespace.
Afterwards the token sequence (for both the suspicious and the source document) has to be divided into smaller sequences. This processing step is called chunking. The selection of a suitable chunk size is crucial for the overall performance of the plagiarism detector: a small chunk size leads usually to more accurate results, but the computation effort rises. A longer chunk size tends to reduce the computation complexity, but the overall precision decreases, because longer chunks contain non-plagiarized text with a higher probability than smaller ones. We use a chunking algorithm with a semi-fixed window size of 250 characters. To prevent tokens from being split, the algorithm expands the window by 1 character at a time until a separator character (usually a single space) is hit. We experimented with another chunking algorithm, which produces chunks on a sentence level (1 natural sentence = 1 chunk), but we drop this approach due to a below average implementation of the sentence detection. So we decided to ”emulate” a sentence detection algorithm by defining the rough length of a sentence with approximately 250 characters.
At this point we chunked the suspicious document into n and the source document into m chunks. We conduct an exhaustive search by comparing all n suspicious chunks to the m source chunks, therefore we have to perform n m comparisons by computing the pair-wise similarity of the chunks. Before we compute the similarity of two chunks, we use language-dependent pre-compiled stop lists to eliminate stop words (words with a high frequency are usually considered useless or even harmful for comparison tasks). By removing these words, we prevent our algorithm from detecting similarities between chunks, which are solely based on shared articles, pronouns, etc.
The adjusted chunks are then treated as a simple set of words, so we can use the Dice coefficient to compute the similarity of the sets. Dice’s coefficient is defined as follows: 2
jA T Bj jAj + jBj Obviously the function’s range is the interval [0; 1], whereby 0 means no similarity between the sets and 1 indicates total equality. We consider a chunk (or set) as plagiarized, if the Dice similarity between the sets is higher (or equal) 0:4. The Dice similarity is relatively robust against simple obfuscation techniques like re-ordering or word substitution (to a certain degree). Due to the simple approach we are not able to detect cross-language plagiarisms adequately. 2.2
The described detection algorithm produces a series of detected plagiarism cases, which are analyzed further in a post-processing step. We used non-overlapping chunks (cf. section 2.1), so there is never a plagiarism case contained within another, but adjacent cases are most probable to appear. Therefore two plagiarism cases are merged, if their distance to each other is less than 500 characters. 2.3
The algorithm described in section 2.1 is adjustable by two parameters: the chunk size (in characters) and the similarity threshold. We performed a series of experiments to determine the optimal parameter pair for a high PlagDet score. Due to runtime issues (which remained unsolved at this point), we were unfortunately unable to perform the experiments on the whole PAN 2012 training corpus. Therefore we generated a smaller subcorpus with roughly 10% of the size of the original corpus and performed our experiments on the smaller one. The experimental results are shown in table 1. A chunk size of approximately 250 characters with a similarity threshold of 0:4 seemed to be a good choice for our algorithm. We performed no further experiments with even smaller chunk sizes. It is noticeable, that regardless of the chunk size all experiments with a similarity threshold of 0:4 achieved the highest PlagDet score within their respective series. 3
Despite our simple approach we achieved 7th place (as a first-time contestant) in the
Detailed Comparison sub-task of the Plagiarism Detection task of the PAN 2012 workshop.
Table 2 shows the final results for our contribution. We achieved an overall PlagDet
score of 0:35, that was mainly biased by a low recall of 28%. The low recall can be
explained by numerous reasons. The most obvious explanation is the simple nature of our
detector, because words from the analyzed documents are not subject to further
processing like stemming (e.g. [
Our algorithm required nearly 28 seconds to analyze a document pair. This runtime issues can easily be explained by a poor implementation of our algorithm. Actually the current version of our detector is much faster and requires only about 10% of the time to analyse a document pair, but naturally the memory requirements increased. We achieved a good precision ( 78%) and granularity score (1.27), so there is little requirement on enhancing our merging algorithm at this time.
The proposed algorithm is at least suitable to detect monolingual plagiarism cases with
low obfuscation, but most probably fails on highly obfuscated plagiarism. Therefore
our future work includes the employment of language-dependent stemming techniques
(e.g. [