In this work we describe the construction of a plagiarism detection/text reuse corpus submitted for the PAN-2015 Evaluation Lab. Our corpus consists of four different text reuse scenarios namely, (1) no-plagiarism, (2) story-retelling, (3) synonym-replacement and (4) character-substitution. Among these scenarios the most interesting one is story retelling - through it we find patterns of textual similarity between story retellings. We use Grimm brothers fairy tales as described in the Project Gutenberg as the source of our documents. The corpus consists of 200 pairs of documents, with 50 document pairs for each type of text reuse. Empirical observation shows interesting patterns of textual similarity within the corpus. Furthermore, plagiarism detection using various approaches shows the difficulty of detection of various groups within the corpus.
The PAN Lab Evaluation Series [
Text reuse detection has been a well researched area in the news domain [
The corpus consists of 200 document pairs with 50 document pairs each within the following four different groups namely, (1) no-plagiarism, (2) (human) story-retelling, (3) synonym-replacement and (4) character-substitution. The no-plagiarism group contains completely different short stories that may share some genre-specific terms leading to minor textual overlap. The story retelling group describes pairs of story fragments taken from two different retellings by human writers. The third group, synonym replacement, describes story fragment pairs with replacement of words and phrases with their synonyms. Finally, character substitution refers to technical disguise, where letters in words are replaced with their look-alike unicode equivalent characters. Empirical observation reveals interesting similarity patterns within the corpus.
The corpus consists of documents from the Grimms fairy tales as available on Project Gutenberg website. The corpus is small in comparison to other PAN corpora. This is because the number of tales available within the Grimm’s collection ranges from a maximum of 200 in some editions to less than 50 within other editions. In order to have a balanced collection of documents within each group, our corpus consists of 200 document pairs, with 50 pairs for each group. Some statistics related to the passage length within the corpus documents are shown in table 1.
Here, a passage is defined as a contiguous maximal-length sequence of characters
(or text) that consists of similar text between two versions. For corpus construction, we
selected passages from two versions of a tale that correspond to the same events, since
different versions of the same story may sometimes differ in details of events.
The no-plagiarism group consists of stories that are completely different but may have
minor textual overlap due the occurrence of some genre-specific words. For this group,
we found ferret [
In this work we do not address this question definitively i.e., “Does story retelling
involve text reuse and/or plagiarism?”. Therefore, we use the term ‘textual similarity’
when referring to similar passages of text within two story retellings. However, from
the literature we see that Clough et al [
In figure 1 we show fragments from two retellings of the story ‘King Thrushbeard’ (also called ‘King Grisly Beard’). Here we give a correspondence between sentence fragments found between the two retellings: 1. (Fragment 1) The wedding of the King’s eldest son was to be celebrated.
(Fragment 2) The king’s eldest son was passing by, going to be married. 2. (Fragment 1) She thought of her lot with a sad heart.
(Fragment 2) She bitterly grieved. 3. (Fragment 1) She put in her jars to take home.
(Fragment 2) She put into her basket to take home.
We see that in pair 1, fragment 2 corresponds to a change of voice from fragment 1
with some modification; likewise in pair 2, fragment 2 corresponds to a summarization
of fragment 1, while the two fragments in pair 3 are exactly the same with minor word
replacement. It may be relevant to mention here that the original Grimm’s tales are in
the German language, with these two retellings being two different translated versions
in English. Earlier, Barzilay et al [
It happened that the wedding of the King's eldest son was to be celebrated, so the poor woman went up and placed herself by the door of the hall to look on. When all the candles were lit, and people, each more beautiful than the other, entered, and all was full of pomp and splendour, she thought of her lot with a sad heart , and cursed the pride and haughtiness which had humbled her and brought her to so great poverty. The smell of the delicious dishes which were being taken in and out reached her, and now and then the servants threw her a few morsels of them: these she put in her jars to take home.
She had not been there long before she heard that the king's eldest
son was passing by, going to be married; and she went to one of the
windows and looked out. Everything was ready, and all the pomp and
brightness of the court was there. Then she bitterly grieved for the
pride and folly which had brought her so low. And the servants gave her
some of the richmeats, which she put into her basket to take home.
The third group in our corpus is synonym replacement. This refers to replacement of
words (and some phrases) with synonymous words and equivalents. For this purpose,
we initially used Wordnet [
Since the size of the corpus is not large and words belong to a particular domain, we created a customized list of synonyms for commonly occurring words and phrases in the documents. While this approach certainly produced meaningful texts, it is may not be scalable for large number of documents and/or corpora involving diverse topics. In addition to these, we also removed some articles (a, an, the), and replaced alternate occurences of some pronouns with proper nouns. Below we give an example of synonym replacement: 1. (Fragment 1) The King, who had a bad heart, and was angry... 2. (Fragment 2) the monarch, who had a worse heart, and was enraged... 2.4
Character Substitution or Technical Disguise
Substitution of characters with their unicode equivalents in order to exploit the
weakness of a plagiarism detection approach is known as technical disguise [
Most word n-gram based approaches might fail to detect this type of obfuscation since a unit of similarity in these approaches is a word. In the example shown below we replace an ‘e’ with an ‘e’ and an ‘a’ with an ‘a’ to emphasize the change that happens. 1. (Sentence 1) Now there lived in the country two brothers, sons of a poor man, who declared themselves willing to undertake the hazardous enterprise. 2. (Sentence 2) Now there lived in the country two brothers, sons of a poor man, who declared themselves willing to undertake the hazardous enterprise.
We tested the corpus using the simple PAN Baseline approach as well as our hashing
and merging based approach [
From figure 2, we see that the two approaches performed very well in detecting no-plagiarism and synonym replacement. However the overall score was low in case of story retelling and is close to zero in case of character substitution.
In this work we constructed a corpus in PAN format based on story retelling. We used various translations of Grimms’ fairy tales in the construction of this corpus. The question whether a story retelling involves text reuse has not been addressed in this work comprehensively. However, given the nature of story retelling, it can be expected that text reuse may happen in case of story retelling. Apart from story retelling our other strategies were synonym replacement and character substitution in the form of technical disguise.
In future, this corpus can be improved upon by enhancing synonym replacement with a comprehensive automatic paraphrasing strategy. Another possible area of exploration could be to extend the domain of story retelling to modern short stories. Some of the stories in our corpus contain archaic language or words, however this is to be expected since we used versions of fairy tales that, unlike modern short stories, are in the public domain. Detailed reviews of various corpora have been submitted earlier. Due to space limitations, here we give a summary of corpora review as shown in table 4. Regarding the errors, – Synonym errors refer to an error in replacement of word by its synonym. Such a replacement may make the resulting text somewhat incomprehensible. – Demarcation Errors refer to errors in the alignment of the source and the suspicious text. For example, the source text may be marked a few characters earlier or later than it should have been. However, this is a minor error and might have also been observed due to programming errors in creation of XML files, or in the viewer program used for reviewing the corpora.
The average errors per document figure is given as a range, since definition of synonym error is not precise – likewise demarcation errors might not be present at all, but may have been observed due to the character set used.
Due to lack of expertise in language and/or non-availability of language resources, review of non-English corpora could not be successfully carried out.
Documents Average Errors Observations Reviewed Per Document and Errors
- - Corpus Name alvi15 cheema
15 mohtaj
15 asghari
15 hanif
15 khoshnava-15 kong15
Mono/Bi Lingual
Mono (English)
Mono (English)
Mono (English)
Mono (English) Bi (EngPersian) Bi (EngUrdu)
Mono (Persian)
Mono (Chinese)
Method of Construction
Refer to Notebook Possibly Automatic + Manual < 10