In this paper, the basics of the three tuning approaches of the evolving CoReMo Plagiarism Detector are shown, focused for the Text Alignment task. In the last PAN edition, it was observed that the different corpora could condition the necessary tuning, and the results using an overfitted tuning from a different corpus could be far from the expected ones. This year's goal has been to find the way to get the system could be selftuned, looking for improving the performance of any fixed parameter tuning, and to be very closed to the overfitted performance for any corpus. All of these tuning approaches have a high Plagdet performance for any corpus, but it's intended to show the different advances effect on each corpus and for all the years corpora. They include new features for parameters selftuning, based on the size and the ratio of the compared documents. For the competition, our choice was based on the most constant detection quality tuning approach when any condition (called WideTuning).
To filter the chance matching, then the Reference Monotony is used, which rejects the matching ngram when it happens far from any minimum length matching group. The minimum group length and maximum distance are rule based since 2.1 version.
The detection seeds are finally bound when their distance is below a fixed threshold, but currently, in 2.3 version this threshold is rule based too.
In the Overview of the 5th International Competition of Plagiarism Detection [
For CoReMo 2.1 (PAN2013) compared to CoReMo 1.9 (PAN2012), although the result was better in the corpus 2013, it was not the same case for the 2012 corpus. Although the Overview’s authors believed an overfitting of the new system for the corpus of 2013 as the possible cause, the CoReMo authors considered to be the 1.9 version the really overfitted for the 2012 corpus, having an illogical but optimal granularity reduction binding distance of up to 80.000 characters, twenty times higher than the one used in version 2.1, being this last one large enough to achieve very similar but more accurate results thanks to the highest quality and number of its detection seeds.
However, the facts that the detection in the corpus PAN2012 could and have to be improved, and that its best fitting was far from the necessary one for the PAN2013 corpora, are leaving the door open to study the different features between the corpora, which may be the key to new selftuning methods to achieve acceptable results for all the corpora together, or even to improve them for each case.
To establish the way to set the automatic tuning of the granularity filter’s binding
distance (the greatest tuning parameter deviation), distinctive features were seek
between 201
By the analysis of the differences between the corpora used to train the system, possibly useful for tuning, it was essentially seen that the average size of the documents in PAN2013 was much lower than in PAN2012 and that the ratio between the sizes of suspicious documents and source also changed significantly for each corpus, so it was suspected its possible influence for the different optimal setting of CoReMo for each one.
For that purpose, the PAN12 and PAN13 competition and training corpora were analyzed by CoReMo 2.1, using different parameters for the chunk length and granularity filter binding distance to find the optimal settings for each one.
The CoReMo efficiency, ready for multicore systems, allowed to arrange hundreds of runs with different configurations on these corpora, as it only takes 4.5 seconds to analyze any of the PAN13 corpora, and about 45 seconds for the PAN12 ones.
The first and obvious possible improvement lies in tuning the chunk length proportionally to the size of the documents analyzed instead of a fixed length. This adjustment improved the result with the corpus of PAN2012, though curiously it had slightly worse outcome in PAN2013.
The second improvement was to propose different granularity filter binding
distances for the suspicious and source documents. This step actually achieved
improvements for all the corpora, although the optimum settings for the 201
The third improvement was to apply rules based on the ratio between suspicious and source document sizes, to decide when applying the above optimal settings, which improved the performance achieved for PAN2012 significantly (0.6905) at the expense of a slight decrease in the performance for PAN2013 to 0.8319.
Finally, noting that despite having very similar characteristics in terms of
documents and corpus size, the optimal chunk size settings were slightly different for
each PAN corpus, similar relationship rules between document sizes were arranged to
tune the chunk size, improving the results obtained for all the corpus (i.e. 0.6930 in
PAN201
The Table 2 describes the rules to define these selftuning. The authors think that these rules can still be more optimized by any kind of interpolation. 0,83 0,81 0,79 0,77 0,71 0,69 0,67 0,65 te 0,75 d lag 0,73 P Corpora
PAN 13Test PAN13Traininig PAN12Test
In Fig. 1, it can be seen the effect of each analyzed possible optimization on its own corpus and the other ones. The left hand side is for PAN2013 test corpus optimizations, the center is for the rule based tuning (focused to any corpora) and the right hand side is for PAN 2012 test corpus optimizations.
PAN2013 <---
Common ---->
PAN2012 fix 28% fixed 4000 8%/100%
Ruled Bind
WideTuning 30%/50%
fixed 80.000 fixed%40 Fig. 1. Optimal settings for each optimization applied to different corpora
Concluding the analysis of the tests, it can be stated that the auto setting will not get optimal performance for any corpus, but a very closed one to it and good enough to meet the expectations for all of them, despite their considerable differences. In its continuous evolution, CoReMo 2.3 includes the following new features since the 2.1 (PAN2013) version: • • • • •
Granularity Filter dumping distance tunable as fixed one or as proportional to document size.
Independent Granularity Filter settings for source and suspicious documents. Rules for autotuning Granularity Filter by the ratio suspicious/source sizes.
New autotuning chunk size rule based on the suspicious/source sizes ratio. Due to the new TIRA environment [5] features, for this issue, there different CoReMo tuning approaches have been provided: 1. CoReMo 2.1 Settings: the same as proposed last year, without the explained improvements, but without the existing bug when the 2013 competition, which left the last detection unregistered when it was near the end of the document. Though curiously it was the best performing of all the three at the EarlyBird evaluation, this fact doesn’t match with the training experiments by 2013 and 2012 corpora or by the final competition corpus. 2. Optimum Settings for PAN2013: it only includes the granularity filer distance proportional to the documents size, different for the source document (100% of its size) and the suspicious document (8% of its size). It achieved the best performance when training, but for PAN2012 corpora, which was almost a 5% below the WideTuning version. It also got the best PAN2014 performance of the three tuning approaches, as expected. 3. Wide Tuning Version: It exploits the four new features above explained, with self tuning for both granularity filter distances and chunk length, by rules based on the ratio between documents size. It Achieves the best Plagdet average for all the corpora, being significantly better than the others for the PAN2012 corpora, although slightly lower for 2013 ones.
Table 3 shows the Plagdet score achieved by the three tuning approaches for the different corpora used in training and the PAN2014 competition corpus, including the EarlyBird evaluation, where curiously, the previous year’s tuning approach (CoReMo 2.1) achieved the best results.
The tuning approach choosing to be submitted to compete for the ranking, based on which we would use for a production system, was the WideTuning version, although the EarlyBird evaluation forecasts and the features of the competition corpus get to feel that the best fit to win would be respectively the previous year tuning or the 2013 corpora optimized one.
During the last PAN edition, it was compared the combination of the different current and previous year’s corpora and approaches, and it was noted that the 2012 corpus was significantly the hardest to detect for any approach, having lower Plagdet scores and needing much larger runtimes. The WideTuning approach, allowed us to obtain for 2012 up to 5% more Plagdet score than the others, while on the 2013 corpora it falls less than 0.5% below the optimum one. It’s not the first time we prefer showing our research than our ranking progress, as we did in previous years using local translation systems instead of the best external Google Service.
The CoReMo 2.3 performance is still amazing with a 0.85 Plagdet score, precision 0.90, recall 0.80, and an optimal 1.00 granularity. Everything is very closed to the winning approaches, but with the fastest runtime of only 31 seconds: at least 4 times faster than the next one, and 100 times faster than the winning approach. Nevertheless, currently the TIRA system does not offer the ability to operate in the multicore mode, and CoReMo is optimized to use it, allowing to run the same job in just 4 seconds using an AMD FX8120 based machine.
The Table 4 shows that, as expected, the best tuning approach was the PAN2013 optimized (but it gets about 0.04 less Plagdet for PAN2012 test corpus than our choice), and the selected WideTuning version gets almost same best performance, having only 0,001 below score.
Both CoReMo 23 tuning versions get to overpass the last year's tuning performance, however, a possible overfitting effect may be present on the best PAN13 optimized due to its lowest PAN2012 test corpus performance (see Table 3). We recommend to check the PAN2012 test corpus performance for all the PAN14 participants approaches which could arrange it in a reasonable time, to look for any possible overfittings. Although these autosettings are in very good direction and achieving very good results at best runtime for any corpus, the authors think that the selftuning rules for granularity reduction could be significantly improved by another function different to simple thresholds to change fixed parameters values.
As we mentioned in the previous editions, these methods could be combined with others such as the use of thesaurus to minimize the effect of replacing words by other synonyms.
It would be wise to check the performance of all participant approaches, looking for any possible overfittings, on the PAN2012 competition corpus (or at least which could arrange it in a reasonable time), which has proved to be the most difficult ever to analyze, and, at least in our case, has demonstrated to require very different settings that for 2013 or 2014 corpora.