This paper describes our approach to the Author Identification task in the PAN 2013 evaluation lab. We use a profile-based approach and use the common n-grams (CNG) method that employs a normalized distance measure for short and unbalanced text introduced by Stamatatos[6]. We achieved the 9th place with an overall F1 score of 0:6.
Textual plagiarism can be described as “The unacknowledged use of an exact copy or a slightly modified version of the text written by another author.” Automated methods to detect plagiarism can approach this in two ways: either by looking at the textual content to detect passages that align within two documents, or by looking at the writing style of an author and detecting changes in the style when a plagiarized passage is encountered. The first approach is suitable for extrinsic analysis, when a large reference corpus is present from which plagiarized passages are suspected to originate. The latter is more suitable for intrinsic analysis when no reference corpus is available and plagiarism can be found by detecting writing style changes within a single document.
This paper describes our approach to the Author Identification subtask, that was sumbitted to the PAN 2013 evaluation lab, and is structured as follows. First in section 2 the submitted algorithm is explained in detail, then in section 3 the results of this algorithm for the Author Identification subtask are discussed, ending in section 4 with some final notes. The PAN 2013 Author Identification task focuses on determining whether an unknown document has the same author as a given set of known documents that are all written by a single author. For the task corpus the documents within one problem are matched for genre, register, theme and date of writing, making the problem closely resemble an intrinsic plagiarism analysis question: are there significant differences in writing style between two sections of the same document?
Our algorithm works as follows: first the texts are pre-processed, to increase the
accuracy of the author profiles. Then author profiles are created for the set of known
documents and for the unknown document, based on the Common N-Grams (CNG)
approach[
Each of these steps is discussed in more detail in the next subsections, including
the algorithm configuration for English and Greek. For Spanish it was decided to use
the same approach as for English, rather than evaluate choices on the small provided
training set. For English the training data was expanded using training instances
generated from books in Project Gutenberg[
Secondly because the limited amount of text for every test instance, being on average 1000 words for every document, we felt the need to reduce redundancy in character n-grams when words occur both capitalized at the beginning of a sentence and lowercase in the middle of a sentence, and common verb conjugations occurring both right before a comma and a space etc. We chose to remove all punctuation except spaces, and to convert each document to lowercase for this purpose. While this will discard stylistic information contained in capital and punctuation use, we expect this to strengthen the stylistic features concerning word preferences and conjugation usage.
With the same reasons for removing capitalization, we decided to remove diacritics
for English and Spanish. After experiments on the training corpus it was determined
that removing diacritics lowered accuracy for Greek, which can be explained by the
polytonic orthography used in Greek, where authors and publishers sometimes still use
diacritics heavily for indicating accents.
The CNG method was introduced by [
In [
Experiments done on the training corpus confirmed 3 as a good n-gram size for
Greek and suggested 4-grams for English. Experiments on varying the profile length
indicated 2300 and 1500 as a good profile length for 4-grams and 3-grams respectively.
It often happens that the ordered n-grams from L x to L+y have the same normalized
frequency, and in such cases we chose to also include all n-grams with the same size as
the Lth n-gram, making 1500 and 2300 the minimum profile lengths.
Having extracted an author profile P (A) for the unknown document and a profile P (B)
for the set of known author documents, the distance between these profiles is calculated
using the normalized distance function nd1 proposed by [
nd1(A; B) =
X g2P (A) 2(fA(g)
fB(g)) 2 fA(g) + fB(g) 4jP (A)j
To provide easy judgements on whether a specific distance dA;B indicates the same or a different author we make use of the corpus characteristics: within the corpus there are an equal amount of cases with the same author and different authors. Therefor within a single language the average distance d is calculated from all reported distances, and a distance dA;B < d is taken to indicate that documents A and B have the same author. 3
As a first-time contestant and despite using a simple approach we achieved an overall 9th place (out of 17) in the Author Identification subtask of the PAN 2013 evaluation labs. Our algorithm had a runtime of roughly 9 seconds, making it the 3rd fastest algorithm. Table 2 shows the detailed performance for our contribution.
The difference in performance for English and Spanish is remarkable, especially because the accuracy for English is much lower than that for Spanish, and the algorithm hasn’t been designed with Spanish in mind. A possible explanation for these results is in the pre-processing steps that are taken, which can reduce the available stylistic information in different amounts for different languages.
The very low performance for Greek, which is around the 50% that is attainable
by random guess, is unexplained as of yet. While the research in [
Language F1 Precision Recall English 0.600 0.600 0.600 Greek 0.467 0.467 0.467
Spanish 0.760 0.760 0.760
Table 2. The attained performance on the PAN 2013 test data. 4
The presented algorithm performs above the baseline for English and Spanish, but is failing for Greek. A good direction for further work would be an instance-based approach on the available training texts where more than one text is provided for an author, to get a better estimate of the average distance between documents of that author. This estimate could be used instead of the global average distance for all authors of a language.
Furthermore we plan to look at more limited profile lengths, based on the solution
proposed by [