=Paper=
{{Paper
|id=Vol-1743/paper10
|storemode=property
|title=An Empirical Method Exploring a Large Set of Features for Authorship Identification
|pdfUrl=https://ceur-ws.org/Vol-1743/paper10.pdf
|volume=Vol-1743
|authors=Seifeddine Mechti,Maher Jaoua,Rim Faiz,Lamia Hadrich Belguith
|dblpUrl=https://dblp.org/rec/conf/simbig/MechtiJFB16
}}
==An Empirical Method Exploring a Large Set of Features for Authorship Identification==
https://ceur-ws.org/Vol-1743/paper10.pdf
An Empirical Method Exploring a Large Set of Features for Authorship
Identification
Seifeddine Mechti Rim Faiz
LARODEC Laboratory LARODEC Laboratory
ISG of Tunis B.P.1088, 2000 Le Bardo, Tunisia IHEC Carthage, Tunisia
mechtiseif@gmail.com Rim.faiz@ihec.rnu.tn
Maher Jaoua Lamia Hadrich Belguith
MIRACL Laboratory MIRACL Laboratory
FSEGS, BP 1088, 3018 Sfax, Tunisia FSEGS, BP 1088, 3018 Sfax, Tunisia
maher.jaoua@fsegs.rnu.tn l.belguith@fsegs.rnu.tn
Abstract have been used to infer attributes discriminating
the authors styles. In this context, we propose a
In this paper, we deal with the author iden- hybrid method combining the stylistic and statis-
tification issues of the document whose tical attributes while relying on measure-ments of
origin is unknown. To overcome these inter-textual distances. In this paper, we present
problems, we propose a new hybrid ap- the results of our experiments, using several learn-
proach combining the statistical and stylis- ing techniques. The objective of the work pro-
tic analysis. Our introduced method is posed in (Stamatatos et al., 2014) is to determine
based on determining the lexical and syn- from a specific list the au-thor who wrote a given
tactic features of the written text in or- text. Thus, for this identification, we should focus
der to identify the author of the docu- on open-set or closed-set classification problems.
ment. These features are explored to In this context, we address a non-factoid question:
build a machine learning process. We was a particular text written by a well-defined au-
obtained promising results by relying on thor. This paper is organized as follows: In section
PAN@CLEF2014 English literature cor- 2, we depict the author identification approaches
pus. The experimental results are compa- proposed in literature. After that, we present our
rable to those obtained by the best state of hybrid method based on the statistical and stylistic
the art methods. analysis. In section 3, we describe the machine
learning process. The fourth section shows the
1 Introduction experiments carried out together with the sever-al
Recently, much more interest has been given to applied tests and algorithms. Then, we compare
a document authorship because of its application our simulation results with those obtained by us-
in many domains, such as e-commerce, forensic ing other methods. Finally, we end up this paper
linguistics, etc. For instance, in the latter, author by some concluding remarks, and we propose fu-
identification can make many investigations eas- ture research study.
ier. Addi-tionally, the author identification task
2 Related Work
is very useful in the plagiarism detection pro-
cess. Indeed, the probability of plagiarism in- In this section, we introduce author identifica-
creases where two parts of a document are not as- tion methods classified essentially into three cat-
signed to the same author. This task is planned egories. The first one is based on a stylistic anal-
in PAN@CLEF 2016.In addition, forensic anal- ysis. The second class contains techniques rely-
ysis or that of the documents paternity for legal ing on various statistical analyses. The third cat-
purposes can contribute to several investigations egory, which includes more recent methods, uses
focusing on various linguistic characteristics. In machine learning algorithms. The basic idea of
the literature, the automation of the author iden- the stylistic methods is the modeling of authors
tification task can draw on stylistic or statistical from a linguistic point of view. For instance, we
attributes. Currently, machine learning techniques can mention the works of Li et al.(2006), who
89
focused on topographic signs (Li et al., 2006), other authors for each problem. The training cor-
as well as the studies of Zheng et al. interested puses are represented in different forms. each text
in the co-occurrence of character n-grams (Zheng is considered as a vector in a space with several
et al., 2006). Other researchers were concerned variables. In addition, a variety of powerful al-
with the distribution of function words (Vartape- gorithms, including discriminating analysis (Sta-
tience et al., 2014) or the lexical features (Arga- matatos et al,2000), SVM (Lee et al., 2006), deci-
mon et al., 2007). In another work, Raghavan et sion trees (Zhao et Zobel, 2006), the neural net-
al.2006 exploited grammars excluding the prob- work (Argamon et al., 2007) and genetic algo-
abilistic context to model the grammar used by rithms (Moreau et al., 2014), can be used to con-
an author (Raghafan et al., 2010). Feng et al. struct a classification model. Finally, in a critical
dealt with the syntactic functions of words and study carried out by Baayen, the latter showed that
their relationships in order to discern entity coher- the stylistic methods revealed low performances
ence (Feng et al, 2013). Other surveys studied the for short texts (Baayen et al, 2008). He also proved
semantic dependency between the words of writ- that style can change over time or according to the
ten texts by means of taxonomies and thesaurus literary genre of the texts (poetry, novels, plays
(Maccarthy et al, 2006). Concerning statistical ...). Besides, despite their interesting results, the
methods, the first attempts emerged in (Mostler et statistical analysis ignores the writers style. In
Wallace., 1964). They compared the occurrence this case, neither the vocabulary nor the theme of
frequency of words, such as verbs, nouns, arti- the suspect document is taken into account. Ol-
cles, prepositions, conjunctions, and pronouns. In son criticized some studies which convert the lan-
the last few years, new methods, based on vari- guage into mathematical equations (Herchey el al.,
ous statistical tools, have been introduced in order 2007). We choose hybridization to take advantage
to discriminate between the potential authors of a of both the stylistic methods and statistics. On the
text. Among these methods, we can mention inter- one hand, we use the lexical and syntactic analysis
textual distance (Labbé,2014), the Delta method to address the problem of mathematical represen-
(Savoy, 2013), the LDA distribution (Blei et al, tation of a text (Section 3.1). On the other hand
2004) and the KL divergence distance (Herchey we apply the Delta rule to gather the writers who
et al., 2007). Indeed, (Labbé,2003) Labb demon- have almost the same style (section 3.2).
strated the effectiveness of intertextual distance in
quantifying the proximity between several texts 3 The Proposed Method
through a normalized index. Later, he revealed
the considerable Corneille contribution in plays The following section describes our hybrid extrin-
written by Moliere . In (Savoy, 2014), Buroows sic method for tauthor identification. First, we
proposed the Delta method in order to identify will extract the different types of stylistic features
the unknown documents author. He has sug- (syntactic, lexical and characters) and then the n-
gested selecting 40 to 150 most frequently used grams. In the second step of the authors selection,
words, especially the functional words, while ig- we will focus on the delta method. The third step
noring the punctuation signs. On the other hand, will be reserved for the application of the learning
in (Grieve, 2007), researchers demonstrated that model.
the Delta method could offer the best results. To
3.1 Feature Extraction
determine the document paternity, the authors in-
troduced a probabilistic model for author identifi- In order to extract features, also called style mark-
cation by addressing several topics (Savoy, 2012). ers, we use the tools of the Apache Open Library .
At this level, each corpus is modeled as a distri- These robust tools allow segmenting the texts and
bution of different themes; each theme represents analyzing the necessary syntax and semantics. For
a specific distribution of words. From a machine the lexical features, obtained by frequency calcu-
learning point of view (Stamatatos et al, 2014), au- lations, the text is regarded as a set of tokens. We
thor verification method can be either intrinsic or distinguish between the number of words that ap-
extrinsic. In fact, intrinsic methods use both the pear only once, the ratio V/N (V is the size of the
known and unknown texts of the problem , while hapaxes , and N is the length of the text), the av-
extrinsic methods utilize external documents of erage sentence length and the number of words
which appear twice. Then, we extract the lexi-
90
cal features, such as the number of nouns, verbs, by other authors represent the negative examples.
adjectives, adverbs and prepositions. In features This algorithm is determined after applying a test
extraction, we consider the text as a simple se- on multiple classifiers, such as: SVM, decision
quence of characters. We also take into account trees, Naive Bayes, decision table and KNN. We
the information concerning the frequencies of let- choose the algorithm that gives the best perfor-
ters, punctuation marks (number of colons, excla- mance.
mation marks, question marks and commas), up-
percase and lowercase characters as well as the nu- 4 Basic characteristichs of our Hybrid
merical and alphabetical characters. Finally, we method
resort to the n-grams classes. We make n vary
from 3 up to 7 characters. In fact, a small n=3 and Hybridization has always been considered as an
a large one are respectively used to capture the syl- interesting track because it overcomes the limita-
lables and the punctuation marks and to produce tions of the combined approaches. The following
the words. table 1 presents a comparison between the differ-
ent methods of author identfication: Verification
3.2 Authors Selection Model: The intrinsic models use the texts within
In this step, we select authors in order to prepare a verification problem (Zheng et al.,2006), (Feng
the machine learning process. We apply the Delta et al.,2013), (Mostler et Wallace.,1964). In other
method on the candidate document and all authors studies (Labbé, 2014), (Savoy, 2012) Labb and
existing documents. For each unknown author, Savoy consider other texts written by different au-
we select the three authors who have the lowest thors and attempt to transform the verification task
Delta measure with the candidate document. into a binary classification problem. However, Ac-
We note that different verification problems cording to PAN@CLEF 2014 and PAN@CLEF
(different folders) may share documents of the 2015, extrinsic models give better results than in-
same authors. For example, the known document trinsic ones (Stamatatos et al.,2014). Classifca-
of folder EN001 and that of folder EN002 may be tion: There are two methods of classification: ea-
written by the same author. Then, we calculate the ger methods, using a supervised learning (Zheng
distance based on the standardized frequencies et al.,2006), (Feng et al,2013), and lazy methods
(Z-score) between two documents Q and A using that do not apply any algorithm (Mostler et Wal-
the following equation: lace, 1964), (Labbé, 2003), (Savoy, 2012). In
1 Pm this paper, we resort to supervised learning using
D(Q, Aj ) = M i=1 )[Zscore(tiq )
Zscore(tij )] SVM. Attribution Paradigm: There are two attri-
bution paradigms (Stamatatos et al, 2000). In the
Where instance based representation each document is
tf r mean(i)
Zscore(tij ) = ij sd(i) represented separately (Feng et al., 2013), (Labb,
2003), (Savoy, 2012). However, the profile based
tf rij is the frequency of the term ti in the paradigm tries to construct an author profile us-
document Dj, mean represents the average, and ing all texts of the corresponding author. (Au-
sdi denotes the standard deviation. Finally, we use thor profile) (Zheng et al.,2006), (Mostler et wal-
the number of the most common terms between lace, 1964). Indeed, we choose the hybrid of the
100 and 400 words. two paradigms, a representation for each docu-
ment which are then combined in a single author
3.3 Application of a Classification Model profile. Text analysis: Most of the proposed stud-
We perform the machine learning process based ies used the part of speech POS tagging (Zheng
on the documents of the candidate author and et al., 2006), (Mostler et wallace, 1964) because
those of the three already selected authors. We use of the availability of taggers. Some other stud-
the Weka tool in order to represent the known au- ies resorted to intertextual distance (Labb, 2003),
thor and the other three authors by an ARFF file (Savoy, 2012). However, our method combines
with the already extracted features. In addition, statistical and stylistic features (sections 3.1, 3.2).
we apply a learning algorithm on this File in order The following section describes our hybrid extrin-
to get a prediction model where the known texts sic method for tauthor identification. First, we
are the positive examples, and documents written will extract the different types of stylistic features
91
Table 1: Author identifcation methods
Author (s) Verification Model Classification Attribution paradigms text analysis
(Zheng., 2006) extrinsic Eager Author profile POS taggig
(Feng, 2013) extrinsic Eager Instance based POS taggig
(Wallace et al., 2011) extrinsic Lazy Author profile POS tagging
(Labbé ,2014) intrinsic Lazy Instance based Intertextual
distance
(Savoy et al., 2013) intrinsic Lazy Instance based Delta method
Our Method extrinsic eager Hybrid Delta metod +
POS taggig
(syntactic, lexical and characters) and then the n- FP / False Positive: case was negative but pre-
grams. In the second step of the authors selection, dicted positive
we will focus on the delta method. The third step
will be reserved for the application of the learning
model. Recall= VP/(VP+FN)
5 Experiments and Evaluation C@1 score
The evaluation score C@1 has the advantage of
In this section, we show the experimental results considering the documents that the classifier is
of our method for authors identification. We first unable to assign to a category. For each problem,
describe the corpus and the evaluation measures. each score greater than 0.5 is considered as a
Then, we depict the performance of our system in positive response, while that below 0.5 is viewed
the identification of anonymous authors. as a negative response. Therefore, the test docu-
ment does not belong to this author. Nevertheless,
5.1 Corpus
all the scores equal to 0.5 correspond to the
The training corpus includes a set of folders from outstanding problems where the answer will be ”I
the PAN@CLEF 2014 computational conference. dont know ”. Then, c @ 1 is defined as follows:
Each folder contains up to five machine learning
documents and a test document in English. The
length of the documents varies from a few hundred c@1 = (1/n)*(nc+(nu*nc/n))
to a few thousand words. It is worth noting that (Penas et Rodrigo, 2011)
the experiments were carried with the 200 existing
problems in the corpus. where:
n = number of problems ;
5.2 Performance Measures nc = number of correct answers ;
To assess our results, we adopt the the C@1 mea- nu = number of unanswered problems
sure (Penas et Rodrigo., 2011) AUC and Recall
metrics. AUC score
Recall The AUC is a common evaluation metric for
In the context of classification tasks, the terms binary classification problems.
true positives, true negatives, false positives and the figure 1 present an exmample of AUC plot.
false negatives are used to compare the given Consider a plot of the true positive rate vs the false
classification of an item : positive rate as the threshold value for classifying
an item as 0 or is increased from 0 to 1: if the
TN / True Negative: case was negative and classifier is very good, the true positive rate will
predicted negative increase quickly and the area under the curve will
TP / True Positive: case was positive and pre- be close to 1. If the classifier is no better than ran-
dicted positive dom guessing, the true positive rate will increase
FN / False Negative: case was positive but linearly with the false positive rate and the area
predicted negative under the curve will be around 0.5.
92
sures reach high value with the choice of the most
frequent 250 words. Our system has proven its
effectiveness when the statistical and the stylistic
analysis were combined. Thus, we were able to
find the unknown author of a document in 59%
of the studied cases. In Table 2, we compare the
performance of our method with those of the win-
ner of PAN@CLEF 2014 competitive conference
for the English essays. From table 2, we notice
Table 2: Comparison between our performances
and Frery el. 2014
Baseline Our method Frery et al.(2014)
Figure 1: Example of AUC plot C@1 0.53 0.68 0.71
Recall 0.5 0.74 0.72
AUC 0.54 0.6 0.72
5.3 Result Analysis
The histograms below reveal the experiments con- that our method is useful in terms of recall. It no-
ducted to obtain the best possible documents pa- ticeably outperforms Frery et al.(2014), although
ternity: C@1 and AUC still need to be further improved.
Figure 2 (a) shows the accuracy reached with a Based on PAN@CLEF 2014 competitive confer-
test set of six well known classifiers in order to se- ence (Stamatatos et al, 2014), our classification
lect the best one. This accuracy is determined with results are so encouraging, which shows the effec-
all the stylistic features and the n-gram features tiveness of our method. Focusing on the step of
(variation of n between 3 and 7). The best accu- selecting the attributes, we are trying to improve
racy has been achieved by the use of the SVM al- our results in our future work.
gorithm with a slight advantage vis-a-vis the Nave
Bayes classifier. Figure1 (b) show that the char- 6 Conclusion
acter features are not very powerful in determin- In this paper, we have focused on author identifica-
ing the authors of documents whose origin is un- tion problem by applying a machine learning pro-
known. On the other hand, the syntactic features cess. Indeed, the introduced hybrid method is es-
give encouraging results. Combining these fea- sentially based on using both stylistic and statisti-
tures provides better performance than the use of cal characteristics. The experimental results reveal
each feature separately. Figure 1(c) depicts the the efficiency of the proposed technique in which
c@1 histogram of the n-grams method. It high- we use the Delta method prior to syntactic and lex-
lights that accuracy reaches a maximum for n= 3 ical features as well as n-grams and character fea-
and 4. Then, it decreases with the increase of n.. tures. We have also proven through the carried ex-
After that, we use the most frequent numbers of m periments how the heterogeneous models allowed
words (between 100 and 400). Figure 1(d) shows us to detect appropriately the document paternity.
that the best c@1 measure is given based on the In future research study, we will try to make our
SVM algorithm with 250 words. This measure de- technique more effective by utilizing text extrac-
creases with the increase of words number. tion tool. The main objective will be to show that
Figure 3 demonstrates that combining the syn- the authors style is clear in some specific parts of
tactic features, the lexical ones and the 3 grams the written text.
brings encouraging results in a machine learning We are also planning to apply our approach on
process. However, the use of the Delta method German, Spanish and Greek corpora to show the
to classify documents gives better results than the efficiency of our method in multilingual context.
stylistic method by which we obtain 0.54 c@1
score. In the hybrid evaluation step, this result
is somewhat improved by using the Delta method
during the step of authors selection. These mea-
93
Figure 2: author identification histograms
References
Stamatatos Efstathios, Daelemans Walter,
Verhoeven Ben , Potthast Martin, Stein Benno,
Juola Patrick, Miguel A. Sanchez-Perez, and
Barrn-Cedeo Alberto. 2014. Overview of the
Author Identification Task at CLEF. England.
Li Jiexun, Zheng Rong and Chen Hsinchun. 2006.
From fingerprint to writeprint. Communication
ACM 49(4), 7682.
Zheng Rong, Li Jiexun, Chen Hsinchun and
Huang Zan. 2006. A framework for authorship
identification of online messages: Writing style
features and classification techniques. Journal of
the American Society for Information Science
Figure 3: The C@1 Performance of different fea- and Technology, 57(3), 378-393.
tures according to words number Vartapetiance Anna and Gillam Lee. 2014. A
Trinity of Trials: Surreys 2014 Attempts at Author
Verification. Proceedings of PAN@CLEF2014.
Argamon Shlomo, Whitelaw Casey, Chase Paul,
Hota S. Raj, Garg Navendu and Levitan Shlomo.
2007. Stylistic text classication using functional
lexical features Journal of American society
of information science and technology 58(6),
802822.
Raghavan Sindhu, Kovashka Adriana and Mooney
Raymond. 2010. Authorship attribution using
probabilistic context free grammars. Proceedings
of ACL10, 3842.
94
Feng V. Wei and Hirst Graeme. 2013. Authorship tralian Computer Science Conference ACM Press,
verification with entity coherence and other rich 59-68,Australia.
linguistic features.Proceedings of CLEF13. Moreau Erwan, Jayapal Arun, and Vogel Carl.
Mccarthy M. Philip, Lewis A. Gwyneth, Dufty F. 2014. Author Verification: Exploring a Large
David and Mcnamara S. Danielle. 2006. Analyz- setof Parameters using a Genetic Algorithm
ing writing styles with coh-metrix. Proceedings Notebook for PAN at CLEF 2014. England.
of FLAIRS06, 764769. Peas Anselmo and Rodrigo lvaro. 2011. A Simple
Baayen R. Harald. 2008. Analyzing Linguistic Measure to Assess Nonresponse. In Proceedings
Data. A Practical Introduction to Statistics using Of the 49th Annual Meeting of the Association
R.Cambridge, Cambridge University Press, Cam- for Computational Linguistics, Vol.1, 1415-1424.
bridge. Frery Jordan, Largeron Christine, and Juganaru-
Mosteller Frederick and Wallace David. 1964. Mathieu Mihaela. 2014. UJM at CLEF in Author
Inference in an Authorship Problem,1964. In Identification. PAN@CLEF2014. England.
Journal of the American Statistical Association,
Volume 58, Issue 302, 275-309.
Labb Cyril. 2003. Intertextual Distance and
Authorship Attribution. Corneille and Molire, In:
Journal of Quantitative Linguistics, , pp. 213-231.
Burrows John. 2002. Delta: a Measure of Stylistic
Difference and a Guide to Likely Authorship, In
Journal Lit Linguist Computing.
Blei M. David, and Jordan I. Michael. 2004.
Variational methods for the Dirichlet process.
In Proceedings of the twenty first international
conference on Machine learning ACM.
Hershey R. John, Olsen A. Peder and Rennie
J. Steven. 2007. Variational Kullback Leibler
divergence for Hidden Markov models. IEEE
Workshop on Automatic Speech Recognition and
Under standing.
Grieve Jack. 2007. Quantitative authorship
attribution: An evaluation of techniques. Literary
and linguistic computing, 22(3),.251-270.
Savoy Jacques. 2012. Etude comparative de
stratgies de slection de prdicteurs pour lattribution
dauteur, COnfrence en Recherche dInformation et
Applications CORIA. 215-228, France.
Stamatatos Efstathios, Fakotakis Nikos and
Kokkinakis George. 2000. Automatic text
categorization in terms of genre and author,
Computational Linguistics, Volume 26,.471-495.
Lee C. Min, Mani Inderjeet, Verhagen Marc,
Wellner Ben, and Pustejovsky James. 2006.
Machine learning of temporal relations. In
Proceedings of the 21st International Conference
on Computational Linguistics and the 44th annual
meeting of the Association for Computational
Linguistics. 753-760.
Zhao Ying, and Zobel Justin. 2007. Searching
with style: Authorship attribution in classic
literature, In Proceedings of the Thirtieth Aus-
95