Our submission this year is generated by the same method Encoplot that we have developed for the last year competition. There is a single improvement, we compare in addition each suspicious document with each other and flag the passages most probably in correspondence as intrinsic plagiarism.
Our method Encoplot[
In the second stage the most promising pairs are examined in detail by creating the
encoplot data for them and the passages in correspondence are extracted based on some
simple heuristics. The encoplot data is a subset of the dotplot. It consists of a subset of
the set of indexes on which the two documents have the same N -gram, thus: the first
occurrence of an N -gram in one document is paired with the first occurrence of the
same N -gram in the other document, the second occurrence with the second one and so
on. Computationally, it is worth mentioning that computing the encoplot data is done
in linear time, with the algorithm we have published in [
A critical choice is how to rank the similarity measures, how to prune the huge list of document pairs without loosing too much from performance. We had noticed in the previous competition that ranking all possible target documents for each fixed source leads to higher recall than ranking for each suspicious document (target) the sources. This can be seen in Figure 1(a), ranking the targets offers a consistent about 10% higher recall, therefore this is what we have used. Based on those graphs, we have decided for looking this time at the first 400 most likely targets for each source, up from the 50 we considered in the previous competition.
1 0.9 l 0.8 a c rem 0.7 u m ifxam 0.6 o n itaom 0.5 it s E 0.4 0.3 1 0.8 l a c e irum 0.6 m x a m f o iton 0.4 a m it s E 0.2
0.2 0 50 100 150 200 2R5a0nking3s0o0urces350 400 Rank
0 0 50 100 150 200 2R5a0nking3s0o0urces350 400
Rank (a) Pan’09 (b) Pan’10
We have kept the same method without any extra tuning - all hyperparameters but
this one were kept to the values given in the Encoplot paper [
It turns out that for this dataset, for low number of neighbors, it is much better to rank the possible sources for each target than to rank the possible targets for each source, as in Figure 1(b). The difference is as extreme as between 38% and 64% when the first 20 neighbors are used. For the number of neighbors we have chosen, 400, our choice looks on the first glance close to ideal – more about that in the Evaluation and Discussion section below. 3
Our position is that given the weaknesses of the intrinsic plagiarism detection and the
higher probability that the source is anyway available to the investigator, it is better to
attempt external plagiarism detection with an enlarged set of sources instead of risking
with the low-performing intrinsic plagiarism detection methods. It was unclear whether
this would be allowed by the rules – as a side note, in other data competitions such as
VOC[
Those passages have been reported as intrinsic plagiarism, as they have been identified as plagiarism by this document-set method but the source is not known. It may well be that in fact the two passages are copied from a third source. 4
Since our method was already proven in the last year competition (on which PAN-PC09 is based), we did not evaluate it again on this dataset.
Our results in the competition are given in the Table 1.
As far as the current competition is concerned, we have the confirmation that our decision to add cross-suspicious-documents detected passages led to an increase in recall that eventually gave a similar increase in the overall score. The precision is also slightly higher, as a result of the strong filtering of the cross-suspicious-documents detections. Hadn’t we filter those, we would have obtained an higher recall (2.5% higher) at the expense of a lower precision (2% lower). Overall this would have improved further (but only slightly – 1%) the overall score.
The most interesting question is why is our score lower this year? Could it be that we didn’t detect the non-artificial plagiarism as good as the artificial one? The score went down mostly as a result of the decrease in recall. We suggest as a possible explanation that the Figures 1(a) and 1(b) are over-optimistic for high number of neighbors. The actual recall we obtained in these two competitions corresponds to the ideal values for 10 to 20 neighbors. A likely explanation is that, as the kernel and the encoplot agree on their conclusions, encoplot will fail to notice common passages in the documents where the kernel failed to notice similarity. If this is true, then, according to Figure 1(b) we have used the lower performance ranking and better should have been to rank the sources for each target, that would have brought us a substantial performance boost without any change in the algorithms. 5
Although three groups have succeeded to outperform our method in the challenge, we have argued here that our results could have been improved through pruning by ranking the source documents for each suspicious one, instead of ranking the suspicious documents for each source document. Given the consistent performance and the versatility of Encoplot, we plan to apply it to new fields in the future.