Plagiarism detection has been considered as a classification problem which can be approximated with intrinsic strategies, considering self-based information from a given document, and external strategies, considering comparison techniques between a suspicious document and different sources. In this work, both intrinsic and external approaches for plagiarism detection are presented. First, the main contribution for intrinsic plagiarism detection is associated to the outlier detection approach for detecting changes in the author's style. Then, the main contribution for the proposed external plagiarism detection is the space reduction technique to reduce the complexity of this plagiarism detection task. Results shows that our approach is highly competitive with respect to the leading research teams in plagiarism detection.
Plagiarism in academia is rising and multiple authors have worked to describe this
phenomena [
Two main strategies for plagiarism detection have been considered by researches
[
The main contribution of this work is the usage of outlier detection techniques on text-based data to enhance two plagiarism detection strategies, one for intrinsic plagiarism detection using deviation parameters with respect of the writing style of a given document, and another one to reduce the search space for external plagiarism detection based on the generation of segments of n-gram for approximated plagiarism decision where unrelated documents are discarded efficiently.
This paper is structured as follows: First, in Section 2, a short summary on plagiarism detection is introduced. In Section 3 the proposed external plagiarism detection method is described. Afterwards, in Section 4, the proposed intrinsic plagiarism detection method is described. In Section 5 results are presented. Finally, in Section 6 conclusions are discussed. 2
According to Schleimer et al. [
When comparing texts against a reference set of possible sources, comes the
complication of choosing the right set of documents to compare to. And now more than ever,
with the possibilities that Internet bring to plagiarists, this task becomes more
complicated to achieve. For this, the writing style can be analyzed within the document and an
examination for incongruities can be done. The complexity and style of each text can be
analyzed based on certain parameters such as text statistics, syntactic features,
part-ofspeech features, closed-class word sets, and structural features, as stated by Meyer zu
Eissen [
Stamatatos [
Other approaches have been proposed, such as the one presented by Seaward &
Matwin [
In terms of external plagiarism detection algorithms, the use of n-grams have shown
to give some flexibility to the detection task, as reworded text fragments could still be
detected [
Kasprzak & Brandejs [
The proposed algorithm is based on two phases; First, it executes a plagiarism search
space reduction method, and then executes an exhaustive search to find plagiarized
passages. The search space reduction method aims at quickly identify those pair of
documents that potentially have some text in common, possibly one of them having
plagiarized from the other. For this, the method’s general tactics are to remove
stopwords, and consider word 4-grams. If two documents have at least two word 4-grams
coincidences close enough as to be in the same paragraph, the documents are given to
the next phase. Otherwise the pair is discarded. For more details, please refer to [
For the exhaustive search, word tri-grams are used (compared to use both word
bi-grams and word tri-grams in [
The system does not consider plagiarism detection between different languages.
The overall mechanism for finding the plagiarized passages is described in Oberreuter
et al. [
For intrinsic plagiarism detection, we first considered some ideas other authors had
investigated. To characterize the writing style of an author, different details can be
considered. As studied by Stein et al. [
In the following, some of the core ideas developed in this research are presented: – To be able to distinguish different authors within the same document, one must characterize the writing style present on the text. 4
– The use of “n-gram profiles” compares segments of the document against the whole document. This approach works based on the assumption that the document has a main author, who wrote the majority, if not all, the text. Therefore, it is logical that the comparison between the style of a particular segment with the whole document style could lead to detections of important variations, meaning that other authors are involved. – Based on reading and contemplation, one of the characteristic that showed to be of interest, is the author’s use of words. Different authors tend to use different words to write their ideas, be them on the same topic or not.
These ideas lead to the following intuition for the development of the algorithm: If some of the words used on the document are author-specific, one can think that those words could be concentrated on the paragraphs (or more general, on the segments) that the mentioned author wrote.
First, the document is preprocessed removing numbers and all other Characters that don’t belong to the a–z group. All Characters are considered lowercase. Second, the method uses word uni-grams and considers all non-numerical words; stopwords are not removed. Next, a frequency-based algorithm to test self-similarity of document is proposed. A hard (not normalized) frequency vector v is built for all words on the given document. Then, the complete document is clustered creating groups C. As a first approach, these groups or segments c 2 C are created using a sliding window of length m over the complete document. Afterwards, for each segment c 2 C, a new frequency vector vc is computed, which is used in further steps to compare whether a segment is deviated with respect to the footprint of the complete document. This is performed by using the Algorithm 4.
Algorithm 1 Intrinsic plagiarism evaluation
As presented in Algorithm 1, the general footprint or style of the document is represented by the average of all differences computed for each segment and the complete document. Note that every segment is compared against the whole document only in terms of the words present in the segment. Also, this algorithm takes into account the intuition; if certain words are only used on a certain segment, the comparison of that segment against the whole document would lead to a low value, because the frequency of those words would be the same in both the whole document and in the segment. Finally, all segments are classified according to its distance with respect to the document’s style. As an example, in Figure 2, a graphical representation of this evaluation is presented.
In this case, the average value of the comparison of all segment with the whole document represents the document “main” style. This value is roughly computed by the difference on the frequency of words between vectors v and vc; 8c 2 C. If the variation is significant, the style function will be lower than the average value minus (the threshold), then the segment is classified as suspicious. In this example, real plagiarized annotations are presented along with the style function value of each segment. Five cases of plagiarism could be discovered; the value of the style function on those cases is lower than the threshold. The evaluation results will now be presented. Three experiments were conducted: First, we participated with our external plagiarism detector applied to the external corpus provided by the competition. Second, our intrinsic plagiarism detector was applied to the intrinsic corpus. Last, we applied the intrinsic plagiarism detector on the external plagiarism corpus. Parameters of the algorithms were tuned considering the PAN@2010 corpus in the case of the external approach, and PAN@2009 corpus in the case of the intrinsic one.
In the PAN@2010 competition, as shown in Table 1, the best results were achieved
by Kasprzak & Brandejs approach [
Our proposed model, applied to the PAN@2010 corpus, achieves better results. The new method is more precise (0.94), thus reducing false-positive detections. The recall also improved, getting a score of 0.6. This value is acceptable considering that at the moment we do not consider detecting plagiarism between different languages, presented in the corpus. Also the corpus considers intrinsic plagiarism, which is not considered in this particular case.
In Table 2 the results from PAN@2011 competition are shown. The revised method achieves third place, obtaining high precision (0.91) but a low recall score (0.22). This could be explained as we do not consider translated plagiarism, and possibly because our plagiarism space reduction technique could be filtering lots of used source documents. The best team, Grman & Ravas, get’s slightly better precision, and a recall score of 0.39.
A sliding window of 400 words was used, and a threshold parameter = 0:075. These were iteratively adjusted depending on text length. Sensibility analysis for all parameters was intentionally excluded by authors due to lack of space.
The results for the intrinsic task at PAN@2009 are shown in Table 3 and for PAN@2011 in Table 4. The results are based on the quality of the detection, which only considers the information on each document itself.
The winner was Stamatatos approach [
The proposed method gets an overall score of 0.3457, greater than any other approach, with a positive difference of 0.0995 with the winner’s approach. Our model gets the best result at F-measure, precision and granularity.
These results are confirmed with similar results on the PAN@2011 competition presented in Table 4; the proposed model gets roughly the same overall score, 0.3254, with comparable precision (0.34) and worse but not significantly different recall (0.31). We get the best results in the competition, followed by Luyckx et al. with an overall score of 0.17, almost doubling their score.
We used the same intrinsic plagiarism detection algorithm with the same parameters on the external plagiarism corpus. The results are presented in Table 5. The recall score is the third lowest; this can be explained as the intrinsic detector provides no information on the source of the copied passages, which reduces considerably the metric itself. Also, the algorithm achieves a precision of 0.36, comparable to the precision obtained when applied to the intrinsic corpus (0.34). Overall, the intrinsic detector would have ranked 8 if participated on the external competition, out of 10 teams. In this lab report two approaches for plagiarism detection were described. The first method compares suspicious documents against a collection of possible sources, while the second one compares the writing style within a particular document to determine if the text was written by one or more authors.
The third place at the external plagiarism detection competition PAN@2011 was obtained, out of 9 participant teams. The precision of the proposed method, of particular importance at plagiarism detection, is close to perfect, with a score of 0.94. Future work in this task would be to integrate an automatic translator to the system, thus providing a way to detect plagiarism for cross-language tasks. Also, to investigate new ways to improve the total number detections, or recall.
The proposed intrinsic algorithm, which introduces a new variant to compute writing style differences, achieves remarkable results, obtaining the first place at the PAN@2011 competition, almost doubling the score of the second team. The method does not utilize language-dependent features such as verbs or stopwords, thus providing a starting point to experiment with other languages. Nevertheless, it is important to note that in this task, of significant difficulty, much work is still needed. The best score so far has been 0.325, indicating that in the field of writing style modeling new approaches need to be developed.
Last, the proposed intrinsic model was applied to the external corpus. This provided results for a real case scenario were one has no prior information on the suspect documents. The results indicate that the precision is still low in this case, and that a significant part of plagiarized passages are left undetected. Nevertheless, it proves that it still can be usefully as it can be the only way to get plagiarism detection done when no reference collection is available.
Authors would like to thank continuous support of “Instituto Sistemas Complejos de Ingeniería” (ICM: P-05-004- F, CONICYT: FBO16; www.isci.cl); and FONDEF project (DO8I-1015) entitled, DOCODE: Document Copy Detection (www.docode.cl). Gabriel Oberreuter is currently “Becario CONICYT”. Finally, authors would like to thank PAN Competition Organizers for constructing such a great workshop and motivate the development of plagiarism detection techniques.