In this paper, we describe an approach to create monolingual English plagiarism detection corpus for the task of text alignment corpus construction in PAN 2015 competition. We propose two different obfuscation methods to fragment obfuscation for creating the cases of plagiarism. The first method is an artificial obfuscation which consists of variety of obfuscation strategies such as synonym substitution, random change of order, POS preserving change of order and addition/deletion. The second obfuscation method is a simulated obfuscation, in which the SemEval dataset is used for creating the cases of plagiarism by using pairs of sentences with their similarity scores.
Plagiarism is defined as re-use of another person’s ideas, processes, results, or words
without explicitly acknowledging the original source [
The plagiarism detection task has been running for seven years in PAN
competition and each year, it provides a corpus for evaluating of submitted systems. The
evaluation corpora in PAN are used for text alignment and source retrieval task for
plagiarism detection [
In this lab report, we have described our approach to generate a monolingual English corpus for the task of text alignment corpus construction. We employ two obfuscation strategies, artificial and simulated ones. Our main contribution is using the SemEval dataset for constructing simulated plagiarism cases. The similarity score of paired sentences in SemEval dataset have been used for establishing the degree of obfuscation for plagiarism cases.
In the following, in section 2 we describe our approach for corpus construction. Then in section 3 we will discuss the statistics of the resulted corpus which is based on Wikipedia articles. Finally, we will conclude and discuss about some future works in section 4. 2
In this section, an overview of our approach for constructing a monolingual English plagiarism detection corpus is presented. Our approach includes four main steps: document clustering, fragment extraction, fragment obfuscation and inserting plagiarism cases into the source and suspicious documents. The process of each step is described in the following sections. 2.1
The documents which are used in the corpus are derived from the Wikipedia Internet
encyclopedia project. In this step, the collection of Wikipedia documents is clustered
into different topically related categories. Since pages on similar subjects are intended
to be grouped together via categories, a bipartite graph of documents-categories has
been created to cluster the documents based on their topics. To detect communities of
the graph, the infomap community detection algorithm [
The documents used in the corpus are divided into two categories: 50% of the documents are considered as source and 50% are designated as suspicious documents. Note that only 25% of suspicious documents contain plagiarism cases.
We have used two different methods for fragment extraction. In the first method, the fragments are extracted from the source documents, while in the second method, the SemEval datasets is used for fragment extraction. The length of fragments is evenly distributed between 3 and 12 sentences. The distribution of fragments’ length is shown in Table 1. We have proposed two obfuscation strategies for obfuscation of fragments: Artificial obfuscation and simulated obfuscation. In the following, we described our obfuscation strategies.
Artificial Obfuscation. For the purpose of generating artificial plagiarism, obfuscation strategies were applied to fragments extracted from source documents. We have used five obfuscation strategies as follows: None (No Obfuscation) Random Change of Order Source fragment without any change considered as the obfuscation fragment. In other words, the obfuscation fragment is an exact copy of source fragment.
Given source fragment, the obfuscation fragment is created by shuffling words at random. POS-preserving Change of Order In order to accomplish this obfuscation strategy, the sequence of parts of speech (POS) tags in source fragment is determined. Then, words are shuffling randomly, while retaining the original POS sequence. Synonym Substitution Addition / Deletion The plagiarized fragment is created in such a way to replace some words by one of their synonyms.
The obfuscated fragment is created by inserting or removing words at random.
Simulated Obfuscation. The pairs of sentences from the dataset of semantic textual
similarity task in SemEval are used for constructing the simulated plagiarism cases.
The dataset includes pairs of semantically similar sentences with their corresponding
similarity score. The similarity score can range from exact semantic equivalence to
complete unrelatedness, corresponding to quantified values between five and zero [
In this strategy, both source and plagiarized fragments are constructed by SemEval dataset sentences. Source fragments constructed by original sentences and corresponding plagiarized fragments are created by corresponding sentences of original ones in the dataset.
To consider the degree of obfuscation in plagiarized fragments, a combination of sentences with a variety of similarity scores is used in a fragment. The number of sentences and their similarity scores specifies the degree of obfuscation for each plagiarized fragment. More precisely, using sentences with higher degree of similarity (e.g. 5) could lead to plagiarized fragments with lower degree of obfuscation and vice versa. The distribution of different sentences for creating different degrees of obfuscation (namely “Low”, “Medium”, and “High” obfuscation) is shown in Table 2. In this step, one or more plagiarism cases according to the suspicious document’s length, within the same cluster have been selected. Then, each of them inserted at random positions in suspicious documents. For simulated plagiarism cases, the corresponding source fragments also inserted at random positions in source documents.
The fraction of plagiarism in each document is not fixed. The percentage of plagiarism in each suspicious document is distributed between 5% and 60% of its length. The ratio of plagiarism per suspicious documents is shown in Table 3.
Finally, for each pair of source and suspicious documents, a Metadata file is created which contains meta information about the plagiarism cases. The tags in the file include: this_length: The length of plagiarism case in the suspicious document. this_offset: Start offset of the plagiarism case in the suspicious document. source_reference: Name of source document. source_length: The length of source fragment in the source document. source_offset: Start offset of the source fragment in the source document. 3
In this section, the results and statistics of monolingual English corpus for the PAN 2015 competition is presented. This corpus is based on Wikipedia documents. The results of corpus construction are shown in Table 4.
Document Purpose The number of source documents: The number of suspicious documents: Hardly (5% - 20%) Medium (20% - 40%) Much (40% - 60%)
Plagiarism per Document
Plagiarism cases The number of plagiarism cases:
- No obfuscation cases: - With obfuscation cases: - Random obfuscation: - Simulated obfuscation: Short (3 – 5 sentences): Medium (6 – 8 sentences): Long (9 – 12 sentences):
Case Length The established English mono-lingual plagiarism detection corpus is available at the website1 of “Research Institute for Information and Communication Technology” for research purposes. 4
In this lab report, we described our approach for constructing a monolingual plagiarism detection corpus. We have used two obfuscation strategies to create our corpus. The first is artificial obfuscation strategy in which the plagiarized fragments are automatically created. In the second strategy, named simulated obfuscation, either source or plagiarized fragments were created by SemEval dataset. The degree of obfuscation in simulated plagiarism cases is based on similarity scores of paired sentences. This corpus is intended to be used for testing the performance of plagiarism detection systems for English language. Although this corpus is in English text, the obfuscation strategy can also be exploited in other languages. In our future work, we plan to improve our corpus by implementing other obfuscation techniques.
This work has been accomplished in ICT research Institute, ACECR, under the support of Vice Presidency for Science and Technology of Iran - grant No. 1164331. The authors gratefully acknowledge the support of aforementioned organizations. Special thanks go to the members of ITBM research group for their valuable collaboration. 1 http://www.ictrc.ir/plaglab/corpora/MonoLingual_English_Corpus(mohtaj15).zip