This paper briefly describes the approach taken to Persian Plagiarism Detection based on modification to the approach used for PAN between 2011 and 2014 in order to adapt to Persian. This effort has offered us the opportunity to evaluate detection performance for the same approach with another language. A key part of the motivation remains that of undertaking plagiarism detection in such a way as to make it highly unlikely that the content being matched against could be determined based on the matches made, and hence to allow for privacy.
Detection of plagiarism has been shown to be beneficial to both
education and research, ensuring that students and researchers
alike are demonstrating their own understanding and findings. The
same techniques can be used against the document archives of an
organization to improve how such information is managed. And
we have successfully demonstrated how related techniques can be
used to protect by preventing the accidental propagation of
corporate information deemed to be of high value in an Innovate
UK project on Intellectual Property Protecting Cloud Services in
Supply Chains (IPCRESS) collaboratively with Jaguar Land
Rover and GeoLang Ltd [
The Persian Plagdet training data, dated 2016-07-17 comprises 1563 source documents and 1525 suspicious documents. There are some 2749 associated annotations, the breakdown of which is shown below in Table 1. 01-no-plagiarism 02-no-obfuscation 03-random-obfuscation 04-simulated-obfuscation
Total files It is striking that there is such a significant minority of unobfuscated cases, in contrast to 59% ‘random’. A highperforming system must therefore be able to address the nature of this randomness, and this was uncovered during training as it seemed difficult to push performance beyond a certain point.
Following re-implementation of our approach in Python for ease of inspection (and necessary upgrading to Python 3.4 so that wide characters were handled properly), and a switch to a Persian stoplist1, initial tests were conducted to ascertain performance against the training data.
Key parameters for the system are: 1. length used for match (windowSize, measured in words) 2. distance within which two non-overlapping matches
can be merged (merge_dist, measured in characters) 1
Obtained from: https://github.com/kharazi/persianstopwords/blob/master/persian, comprising some 778 words. An alternative of just 330 words was also considered but not used: http://www.ranks.nl/stopwords/persian
minimum proportion of shared words in matched segments (match_threshold), required due to the methodology used as verification of match minimum length of a final match (min_length), used later to filter out short matches.
Brute force efforts were used to obtain approximate values for parameters 1-3, with: 1. windowSize from 15 to 35, in increments of 5 2. merge_dist from 100 to 225 in increments of 25, and
then honed in increments of 5 3. match_threshold from 0.5 to 0.95 in increments of 5 The best initial combination was windowSize=30, merge_dist=210, match_threshold=0.75, producing the following: Following the investigation into the make-up of the training data, explained above, the performance against each subset was evaluated. Initially, we determine the detection capability with match_threshold=0 (second column), i.e. without content checking, then look at each subset with checking (subsequent columns), as shown in Table 3.
Such scores demonstrate acceptable performance for the approach used, which is geared towards copying with low levels of obfuscation, as even without subsequent checking of matches recall is very high for non-obfuscated text. The low recall for obfuscation therefore leads us to explore the nature of the kind of obfuscation in use, and here the use of so-called random obfuscation within the training data certainly merits discussion.
To begin with, if passages are constructed which would not be meaningful within the language, there would seem to be limited gain from its treatment here, as there is a question over how this reflects the reality of the problem being addressed. In addition, if the extent of change is high as would require significant human effort to reproduce, the likelihood of such highly edited passages in the wild would seem to be lessened unless approaches are partially automated and do not undergo post-editing. It also becomes difficult to address the difference between necessary inclusion within a focused discussion, and an act of deliberate copying. Consider the example passages below with matches between source and suspicious, with colour used to identify those passages shared. These passages are not fully matchable: lowermost in the table are two matched sub-passages where the maximal fragment length is 5, and the sub-matches are mostly smaller. Clearly, when the initial extents are short this will favour those approaches that address short n-grams. In addition, these passages include extents of text in that are quite different between the two, and this brings implications to the treatment of the gap between passages as well as to any verification step as addresses word overlap.
. ذًاُداد ٍذیصق لیکشت یعیش باتک تسیتسرداً یلصا ىاوتخاس ، تسا ٍاک ، صّزخ َوُ ّ اذخذک ، جٌگ ،ْخبآذِع ٍ رذًا ٍزفط َهاً َلیپِلیشیب يیلّا . ذسیٌْیه راک ّد ىازیا یهلاسا یرِْوج َب شباتک ٍذش یحازج ّ کذٌق غیت یًًْاقزیغ نسا ىدزک ضازتعا رد ىاتسلگ نیُازبا . لاسٌِک مزش یازب یشّزفرابّراْخ زگا ، یایتاراشتًا رّآهزش ، تسا _ٍذش_َتشًْ ٍذش اوغی ىآ رد باتک ای تسا زیذه َب باطخ ۳۱۳۱ زوق زیت ۱۳ خیرات َپت ٍاگتهاقا َب ذخاه ار َب لاغشا یتیاکش ذًاُدزک يیٌچ یًاسک یذٌه َقلاع یفاٌه سا َک َیّاس رابتیصعه نٌکیوً فلت سا ار نتقّ يه . يیا َهاً پاچ . اوش ماذقا يیا کش َٔلوع یازب خش رازقزب ذًاوب اًِدزک َلثه ّ اٌِتفر شک ذشاب يیا تسا زتِب … نساٌشیوً ، ذشاب نُ زگا ار عجزه ّ ماقه يیٌچ ٍاک مزبب Assuming that the test data would be formulated similarly to the training data, peak performance would be somewhat constrained, and with time available, further tests were performed using rather shorter initial word numbers (windowSize values) and lower values for other parameters.
merge_dist windowSize match_threshold min_length Plagdet Score
From Table 4, above, values used for the submission based on test data are in the final column. Results from all participants are shown below in Table 5. Although only achieving 8th place of 9, with all other participants are from Iranian institutions, we are satisfied that we have managed to maintain the core of our approach, which we were already aware was only robust to a certain extent of obfuscation and is not readily tuned to random effects as may not necessarily be readable unless via a codebook. 0.92204 0.90593 0.87103 0.83015 0.80083 0.77496 0.72662 0.39968 0.38994 1.00146 1 1 1.03968
1.0 1.22759
1 1.52803 3.53698
In this paper, we briefly described the approach taken to Persian
Plagiarism Detection, based on modification to the approach used
for PAN between 2011 and 2014. Detection performance for
Persian is appropriate with respect to the nature of application we
have in mind, and the large proportion of randomly obfuscated
data, allied to the manner in which obfuscation is conducted,
limits what our approach would achieve. Runtime performance is
also inadequate, however this is believed due in significant part to
using a Python implementation instead of our main C++ codebase
– and we are aware that some suggest Python 3 is significantly
slower than C++ for the majority of tasks [
It is worth noting that the first author has no familiarity with Persian languages, and only sought the co-author’s advice on reasons for the detection performance in respect to random obfuscation after it was apparent that an improved plagiarism detection score could not be achieved.
The authors gratefully recognize prior contributions of Neil
Cooke, Scott Notley, Peter Wrobel, Neil Newbold and Henry
Cooke in respect to the approach in previous competitions, and by
Neil Cooke and Peter Wrobel to the patents generated from these.
We recognize, also, prior support from the EPSRC and JISC
(EP/I034408/1) and by Innovate UK (TSB, 169201). The authors
are also grateful for the efforts of the Persian Plagdet organizers in
formulating proceedings [