Plagiarism; an un-attributed reuse of text, is very significant problem specifically for higher education institutions. Consequently, a number of automated plagiarism detection system have been developed to cater this problem. The comparison of these automated plagiarism detection systems is difficult sue to problem in collecting real cases of plagiarism by students / scholars. This paper describes development of corpus containing simulated cases of plagiarism by the people having different level of writing skills. This corpus will be a very valuable addition in the set of evaluation resources presently available for comparison of plagiarism detection systems.
The un-acknowledged reuse of information is generally known as plagiarism [9]. Plagiarism is acknowledged as a significant & increasing problem in higher education [7] [11] [20] [12] [5]. Resultantly, plagiarism & its detection has recently received much attention [1] [8] [21] and higher education institutions are now using automated systems to detect plagiarism in students’ / scholars’ work. Numerous approaches for plagiarism detection are available [2] [19]. However, one of the barriers preventing a comparison among techniques is the lack of a standardised evaluation resource.
This corpus will be a valuable addition to the set of existing corpora for the
plagiarism detection task. This corpus, (
The aim of this corpus collection is to investigate how text is reused by students / scholars while writing an article, and to determine whether algorithms can be discovered to detect and quantify such reuse automatically. It is hoped that results will generalise beyond the text reuse & plagiarism in academia and provide broader insights into the nature of text derivation and paraphrase; but the selected scenario provides an ideal initial case study, and one with considerable potential practical application.
There can be three types of plagiarism in a benchmark corpus; (
A number of corpora have been constructed for evaluation of state of the art
techniques for plagiarism detection. An outstanding effort for developing plagiarism
corpora is the PAN International Competitions on Plagiarism Detection1. A series of
evaluation labs have been held on plagiarism detection as part of the CLEF conferences2.
A number of benchmark corpora generated as an outcome of this series of
competitions [18] [17] [14] [16] [15]. Both mono-lingual and cross-lingual examples of
plagiarism are present in these corpora, 90% of these are mono-lingual, and remaining
10% are cross-lingual. The distribution of plagiarised and non-plagiarised examples is
uniform i.e 50% documents in each corpora are plagiarised, and remaining 50% are
non-plagiarised. The plagiarised documents are created using different techniques: (
The Short Answer Corpus [3] contains examples of simulated plagiarism. The Short Answer Corpus was created by asking participants to answer the five questions on different topics from Computer Science domain. In order to create non-plagiarised and plagiarised documents, each participant answered each of the five questions only once. The each answer consists of 200-300 words. All the documents (answers to the questions) in this corpus were manually (simulated) created. This corpus contains total 100 documents, 95 of which are suspicious documents and 5 documents are source Wikipedia3 articles. Out of 95 suspicious documents, 57 documents are plagiarised with different levels of rewrite (near copy = 19, light revision = 19 and heavy revision = 19) and remaining 38 documents are non-plagiarised.
Plagiarism is not acceptable type of text reuse, but there are other forms of text
reuse that are acceptable, for example reuse of news agency text by newspapers. The
METER corpus4 [6] is another benchmark corpus, which was mainly built for the study
of text reuse in journalism. However, this corpus can also be used for the evaluation
of plagiarism detection systems. The METER corpus contains total 1,716 documents,
1 http://pan.webis.de/ Last visited: 02-06-2015
2 http://clef2015.clef-initiative.eu/CLEF2015/ Last visited: 02-06-2015
3 http://www.wikipedia.org/
4 http://nlp.shef.ac.uk/meter/ Last visited: 18-03-2015
771 documents are Press Association (PA) articles and the remaining 945 documents
are news stories published by nine different British newspapers. Each news story
(suspicious document) was manually examined to access level of text reuse, and based on
the amount of text reused from the PA article (potential source document) classified at
document level as: (
All of the above mentioned corpora contains documents with different levels of rewrite. They lack in categorisation of documents on the basis of writer (having certain level of writing skills) of document. To the best of our knowledge, no standard evaluation resource is available for study the variation in text rewritten by groups of people having different writing skills. 3 3.1
Previous studies have shown that detecting paraphrased plagiarism is a difficult task and an open challenge [10] [14]. The proposed corpus aims to collect paraphrased examples of plagiarism from participants i.e. collection contains simulated cases of plagiarism.
In previous studies, simulated examples are generated from university students [3] or by paying workers on Amazon Mechanical Turk [13]. However, none of these contain paraphrased examples of plagiarism generated by different groups of people. This study aims to collect paraphrased examples of plagiarism from different groups. We selected following four groups: i. Undergrad in progress: The students of undergrad program, who have not written final year project report. ii. Undergrad: The people, who have completed undergrad, and they have written report for their final year project. This group also includes the students of Masters program, who have written report for their final year prject of undergrad program but they have not written their Master’s thesis. iii. Masters: The group of people, who have completed master degree, and have written masters thesis. This group also includes the students of PhD program, who have written their masters thesis but they have not written their PhD thesis. iv. PhD: The group of people, who have completed their PhD degree.
Another important point is that participants were asked to selected text of their own research area i.e. in which they have sound knowledge and experience. Because to efficiently paraphrase a text one mush have the domain knowledge. The participants were asked to generate paraphrased plagiarism examples with different amount of text because people may have variation in the amount of text reused for plagiarism. The three variants were: small, medium and large.
Documents were collected from domains including (
The proposed corpus contains total 1000 documents (500 source documents, and 500
suspicious documents). Out of these 500 document pairs, 250 pairs are plagiarised, and
remaining 250 document pairs are non-plagiarised. The 250 plagiarised pairs were
created by inserting text fragments into documents. Only one source-plagiarised fragment
pair was inserted into one source-suspicious document pair. The source-plagiarised
fragment pairs belonging to Technology domain were inserted into source-suspicious
documents belonging to same domain i.e Technology, and similarly source-plagiarised
fragment pairs from other domains were inserted into source-suspicious document pairs
which belonged to the same domain. Table 2 presents the domain vise statistics of
fragment pairs in the corpus.
To collect corpus while ensuring authenticity and least dependencies on the
contributors, three types of contributors were selected for our study: (
In this section we will review the other participant’s corpora. We are reviewing only those corpora which are completely in English i.e. both the source and suspicious documents are in English, as it is not possible for us to review the corpora in the other unfamiliar languages. We randomly observed some of the document pairs of the entire corpus and reported our observations. Alvi15’s corpus constitute of obfuscation strategies: ‘non plagiarized’, ‘human retelling’, ‘synonym replacement’ and ‘character substitution’. In ‘non plagiarized’ class, we couldn’t find any matching pairs at all, as it supposed to be. In ‘human retelling’ class, the inserted text in the source document has been paraphrased in the suspicious document. In ‘synonym replacement’ category, most of the words in the inserted text buffer have been replaced by their synonyms. In ‘character substitution’, the character we found substituted mostly is ‘the’ has been replaced by ‘thy’ at some points in the corpus. Mohtaj15’s corpus compromise of obfuscation strategies: ‘non plagiarized’, ‘no-obfuscation’, ‘random obfuscation’, and ‘simulated obfuscation’. In ‘non plagiarized’ class, we couldn’t find any matching strings. In ‘no-obfuscation’, string buffers has been inserted into the documents pairs with no obfuscation i.e. the inserted text is exactly same in both the source and suspicious documents. In ‘random obfuscation’, the text has been randomly obfuscated i.e. the words of the matched string has been reordered randomly and makes no sense grammatically and has no meaning. In ‘simulated obfuscation’, the text that has been inserted randomly in the documents pairs has been paraphrased. Palkovskii15’s corpus consist of obfuscation strategies: ‘non plagiarized’, ‘no-obfuscation’, ‘random obfuscation’, ‘translation obfuscation’ and ‘summary obfuscation’. In ‘non plagiarized’ class, again we couldn’t find any matching text in the observed pairs. Again in ‘no-obfuscation’, buffers has been inserted into the documents pairs with no obfuscation i.e. no change in text at all. Similarly in ‘random obfuscation’, the text has been randomly obfuscated i.e. the words of the matched string has been reordered randomly and makes no sense grammatically and has no meaning. In ‘summary obfuscation’, the source document is basically the short summary of the suspicious document. In "translation obfuscation", the inserted text has been paraphrased in the suspicious document. Overall we find all the three corpora error free and true in realism. 5
This paper explained the construction of a new corpus for text reuse & plagiarism detection research. This corpus contains examples of simulated plagiarism, and has been created manually. Our corpus is available to others for evaluation of techniques developed for plagiarism & text reuse detection. The corpus allows much more deeper analysis of different strategies used by people having different level of education.
In future, we plan to gather more document pairs to increase the size of the corpus. Also we will apply & evaluate different techniques using this corpus for text reuse & plagiarism detection.
We thank all the volunteers for their contribution in corpus construction. 18. Potthast, M., Stein, B., Eiselt, A., Barrón-Cedeño, A., Rosso, P.: Overview of the 1st International Competition on Plagiarism Detection. In: Stein, B., Rosso, P., Stamatatos, E., Koppel, M., Agirre, E. (eds.) SEPLN 09 Workshop on Uncovering Plagiarism, Authorship, and Social Software Misuse (PAN 09). pp. 1–9. CEUR-WS.org (Sep 2009), http://ceur-ws.org/Vol-502 19. White, D.R., Joy, M.S.: Sentence-based natural language plagiarism detection. Journal on
Educational Resources in Computing (JERIC) 4(
Australian Computer Society, Inc. (2004) 21. Zu Eissen, S.M., Stein, B.: Intrinsic plagiarism detection. In: Advances in Information Retrieval, pp. 565–569. Springer (2006)