Electronic messages are extensively used to distribute information over such channels as e-mail, Internet newsgroups, Internet chat rooms, Internet forums and other user generated contents on the Web. These messages are quite different from other forms of writings particularly, because of their brevity. Unfortunately, unethical hands and criminals exploit the convenience of these media to carry out their obnoxious goals. Digital forensics require the use of scientifically derived and proven methods towards the preservation, collection, validation, identification, analysis, interpretation, documentation and presentation of digital evidence derived from digital sources for litigation purposes.
Authorship profiling is one of the major classes of authorship attribution problems. It seeks the demographic or psychological group of the author of an anonymous text. Its application in forensics and digital security has made it to be of growing importance in the present information age. Methodologies of profiling like any other authorship analysis consist majorly of feature extraction and application of analytical techniques. Choice of feature sets and analytical techniques may significantly affect the performance of authorship analysis [1]; thus, studies into optimization of authorship profiling of online messages can assist in improving the success of identifying sources of security threats perpetrated through web-based channels.
A number of previous studies ( [1]; [22]; [3]) have investigated some parameters that could affect the effectiveness of authorship attribution undertakings. These studies, however, focused on authorship identification problem and not on authorship profiling. Considering the potential of authorship profiling in investigating transnational digital breaches, the present study seeks to find through experiments the writing-style features, classification techniques as well as possible number of class options that can maximize the effectiveness of profiling the authorship of electronic messages. The following research questions were pursued in order to achieve the purpose the study:
Research Question 1: Which feature type set maximizes the effectiveness of profiling the country of affiliation of writers of online messages?
Research Question 2: Which classification scheme maximizes the effectiveness of profiling the country of affiliation of writers of online messages?
Research Question 3: Which class labelling option maximizes the effectiveness of profiling the country of affiliation of writers of online messages? Research Question 4: What is the performance of the resultant model in classifying electronic messages to writers' countries of affiliation?
Authorship attribution is a process of examining the characteristics of a piece of writing in order to draw conclusions about its author. Authorship attribution problems vary in complexity. They have been categorized into three major classes, namely, authorship identification, authorship profiling and authorship verification. The most straightforward version of these three is the identification problem which involves the determination of the actual author of a given text among a small set of candidate authors. Given a set of writings of a number of authors, the task in authorship identification is to assign a new piece of writing to one of them [4]. In authorship verification, there is no closed candidate set but there is one suspect and the challenge is to determine if the suspect is or is not the author. In this case, examples of the writing of a single author are given and the task is to verify that a given target text was or was not written by this author. Hence, verification can be thought of as a one-class classification problem and it is significantly more difficult than basic authorship identification problem [5].
In authorship profiling (also known as authorship characterization problem) there is no candidate set at all; the challenge is to provide as much demographic or psychological information as possible about the author. Unlike the identification problem, authorship profiling does not begin with a set of writing samples from known candidate authors. Instead, it exploits the sociolinguistic observation that different groups of people speaking or writing in a particular genre and in a particular language, use that language differently; that is, they vary in how often they use certain words or syntactic constructions in addition to variation in pronunciation or intonation [6]. Profiling problem is concerned with determining such characteristics as gender, educational and cultural backgrounds, language familiarity and so on of the author that produced a piece of work. This is a harder problem than the identification problem since it characterizes the writing style of a set of writers rather than the unique style of a single person [7].
Despite variations in the complexities of authorship problems, choices of appropriate linguistic features and analytical techniques are paramount.
One of the main components of authorship attribution methods is the extraction of linguistic features that represent the writing style of an author or author group. Language, like genetics, can be characterized by a very large set of potential features that may or may not show up in any specific sample, and that may or may not have obvious large-scale impact. By identifying the features characteristic of a group or individual of interest, and then finding those features in an anonymous document, one can support a finding that the document was written by that person or a member of that group [8]. The various feature sets, otherwise known as feature metrics in computational linguistics can be classified into four main classes, which are the lexical, syntactical, contentspecific and structural features [9]. Researchers vary in their choices of linguistic features; while some used feature(s) that belong to a single class (for example, [10]; [11]; [12]; and [9], others (such as [6]; [2]; [4]; [3]; [7]; [1]; [13]; [14]) used features across multiple feature classes.
The second component is the application of analytical techniques to feature sets for supervised or unsupervised learning. Different analytical techniques have been used in previous authorship attribution studies. These techniques can be classified into three, namely, the unitary invariant, multivariate and machine learning approaches [8]. Machine learning examines previous examples and their outcomes and learns how to reproduce these and make generalisations about new cases. Machine learning algorithms differ in terms of level of data and abilities to resolve data ambiguities such as noise or missing data. Machine learning techniques include rule based algorithms such as OneR, neural networks such as Multilayer Perceptron, statistical modelling algorithm such as Naive Bayes, decision trees such as J48, linear models such as linear regression and Support Vector Machine and instance-based learning algorithm such as Nearest Neighbour.
Unlike in the choice of feature sets, researchers are less varied in their choices of analytical techniques. While older studies tend to favour the use of Principal Component Analysis, the more recent ones tend towards the use of Support Vector Machine. Most previous studies reported the use of only a single analytical technique. Considering such statement as made by [15].
Experience shows that no single machine learning scheme is appropriate to all data mining problems. The universal learner is an idealistic fantasy. Real datasets vary and to obtain accurate models, the bias of the learning algorithm must match the structure of the domain. Data mining is an experimental science (pg 365).
Choice of machine learning scheme should be based on the result of a prior experiment that validates its suitability to the dataset.
A number of previous studies have shown relative performances of a number of feature types and analytical techniques in authorship analyses. [3] studied the results of authorship identification using many authors and limited data on learning. Their result showed that systematically increasing the amount of authors under investigation led to a significant decrease in performance. Their study also revealed that providing a more heterogeneous set of features improves the system significantly. [1] investigated the types of writing-style features and classification techniques that were effective for identifying the authorship of online messages. They reported that the accuracy kept increasing as more types of features were used and that Support Vector Machine (SVM) outperformed Neural Networks (NN), which in turn outperformed the C4.5 classifier. The best accuracy was achieved when SVM and all feature types were used but classifier performance reduced as the number of authors increased. [2] through experiment demonstrated that inclusion of stylistic idiosyncrasy features to letter n-grams, function words and to a combination of n-grams and function words consistently led to improved accuracy for identifying the native language of the author of a given English language text. The studies of [3] and [1] are situated within the identification domain of authorship attribution problems because they started with a close number of candidate authors, while that of [2] was a profiling problem. However, their focus was majorly to show the ability of idiosyncrasies in detecting writer's native language. It therefore, did not address some of the salient issues covered by [1] which are relative performances of analytical techniques and effect of increasing the number of candidate authors. Also, the corpus used by [2] was the International Corpus of Learner English (ICLE) which had between 579 and 846 words. These numbers were quite high for an online message, which are usually very short. The present study focuses on shorter texts which characterise online messages. Therefore, the present study seeks to find the writing-style (linguistic) features, classification techniques as well as possible number of class options that can maximize the effectiveness of profiling the native language of the author of an online message.
Given a number of online messages written in English language by nationals of selected African countries, namely, Cameroon, Ghana, Liberia, Nigeria and Sierra-Leone. The goal is to find the types of writing-style features, the classification technique as well as possible number of class options that can maximize the effectiveness of profiling the linguistic origin of anonymous electronic texts written by the nationals of any of the selected countries.
A multistage sampling technique was used to select a representative sample of electronic texts from the population of texts contained in the relevant country pages of the website www.topix.com. To get the texts that could be useful for a supervised learning approach of the study, each text was opened, read and assessed based on the number of words contained and a sense of affiliation to the respective country as depicted in the content. A comment was considered to be affiliated to (and labelled to be from) a particular country if it was found in that country's forum and if it contained such phrases as 'our country', 'our beloved country' and other related ones in its discourse. Initially the researcher targeted selecting texts with a hundred or more words; however, this was reduced to texts with twenty (20) or more words because of the scarcity of large texts on the discussion forums. The numbers of texts selected for the study in November 2011 and based on the assessment criteria are as shown in Table 1.
The corpora were subjected to pre-processing in order to put them in the format expected by the relevant software for text processing. The pre-processing tasks included deletion of e-mail headers, removal of control codes, text aggregation, and removal of non-ASCII characters. Text processing was achieved by extracting linguistic features from the sampled texts using computer codes written by the researcher in Python 2.6.4 programming language, based on the natural language toolkit (NLTK) version 2.0. Some of the specific issues handled in the course of text processing were tokenization, part of speech tagging and linguistic feature extraction.
Although there is no agreement on a best set of features for a wide range of application domains, selected feature metrics must be reliable characteristic of attribution domain [21]. Certain features were extracted in the present study, based on their relevance as determined from relevant literature on authorship attibution and Nigerian Englishes ( [16]; [17]). Extracted features were syntactic features comprising the twenty (20) most frequent function words in the topix.com corpus, Idiosyncratic features comprising frequency of occurrence of spelling errors, adverb-verb part of speech (POS) bigram distribution and article omission/inclusion distribution. Structural features comprising lexical diversity; and content specific features consisting of twenty (20) most frequent noun, adjective, verb and adverb unigrams in the topix.com corpus. The features extracted and their denotations are as shown in Table 2.
The decision to extract twenty most frequent features (function word, noun, adjective, verb and adverb unigrams) was as a result of a prior experiment which showed that the summation of the frequencies of occurrence of the twenty most frequent features accounted for at least 60% of the cumulative frequency of all features extracted in each case.
i. Class Labelling: According to the study of [3] learner's performance changes with number of candidate authors. To find out the effect of varying the number of classes on the classification performance in the present study, the dataset was copied into three different files having all parameters being the same except the class labels. The class labels were controlled as presented in Table 3. The texts in Dataset 1 bear their original class labels, that is, the actual countries of affiliation of the writers as determined from the forums and the texts. There are therefore five different class labels, representing the five country sources of the texts. Dataset 2 has three class labels; texts from Nigeria and Ghana bear their original country source labels while those from the other three countries were combined and labelled 'Non-Ghana-Nigeria'. This was informed by a previous study that showed varying degrees of similarity in the English language usage among the selected countries. Dataset 3 labelled texts from Nigeria as Nigeria while texts from the other four countries were combined under the label ' Non-Nigeria'. This was done to achieve a two-class dataset option. Each of the three datasets (Dataset 1, Dataset 2 and Dataset 3) with each of the feature set types (F1, F2, F3, F4) and all their possible combinations (F1+F2, F1+F2+F3, F1+F2+F3+F4, F1+F2+F4, F1+F3, F1+F4, F2+F3, F2+F3+F4, F2+F4, F3+F4, F3+F4+F1) were analysed using the four machine learning algorithms.
Ten fold cross validation was used to evaluate the models' performances based on percent correct (percentage of all datasets that are classified correctly) and Kappa statistic (measure of the agreement between predicted and observed categorization, while correcting for agreement that happens by chance.
Tables in Appendix 1 show the percent correct and kappa statistic values derived for each of the datasets in our experiment. The results are presented successively for Naive Bayes, SMO, J48 and multilayer perceptron. It could be observed from the tables that the percent correct values appear to be highest for Dataset 3 while Kappa statistics appear to be highest for Dataset 2. This observation cuts across virtually all features sets and classifiers. This implies that classifiers were better able to classify Dataset 3 correctly compared to other datasets while classifications achieved in Dataset 2 gave better agreement between predicted and observed categorization having corrected for agreement that happened by chance. Worthy to be noted is the result of SMO in Dataset 3, although the percent correct values were relatively high, Kappa statistics were all zero. Lack of coherence in the directions of the two performance measures led us to using the product of the two measures (percent correct and kappa statistic) as a basis for comparing models' performances. This decision to use the product was informed by the theory of Dimensional Analysis which is a problem-solving method that uses the fact that any number or expression can be multiplied by one without changing its value. One can only meaningfully add or subtract quantities of the same type but can multiply or divide quantities of different types. When two measurements are multiplied together the product is of a type depending on the types of the measurements. This analysis is routinely applied in physics and it is an engineering tool that is widely applied to numerous engineering problems for designing and testing all types of engineering and physical systems ( [18]; [19]). The result of the dot products of the two measures is as presented in Appendix 2. The table in Appendix 2 presents the performances of our models taking into consideration the two performance measures. We consider this table more representative of the models' performances because it combines the strengths and weaknesses of the two performance measures. Answers to research questions will, therefore, be based on the content of this table.
Research Question 1: Which feature set type maximizes the effectiveness of profiling the country of affiliation of writers of online messages?
Figure 1 is a derivative of the table in Appendix 2, it shows the product of percent correct and kappa statistic values derived for the feature set types in our experiment. The results are presented successively for Naive Bayes, SMO, J48 and Neural Network. Across all the three datasets, the feature set that combined all feature types (F1+F2+F3+F4) performed best. This is followed by (F2+F4), (F2+F3+F4) and (F1+F2+F3), while the performance of F1 was the least. Our result shows that inclusion of all features from all the four types (lexical, syntactic, idiosyncrasies and content specific) produced the most effective model. Again the result was consistent with those of [20] and [2] and [1] pg 365 who reported that combining feature types in their studies gave a better result. Using vocabulary richness only produced the poorest result probably because of the short length of online messages in the study.
Figure 2 shows the relative performances of the four classifiers across all feature types (F1+F2+F3+F4) and datasets. Neural Network (multilayer perceptron) performed best when compared to the other three classifiers. Its performance was particularly the highest on the feature set (F1+F2+F3+F4) contained in our two-class option dataset (Dataset 3). Most previous studies considered SVM most appropriate in authorship attribution (though most times without carrying out a prior experiment). [1] however, reported that there were no significant performance differences between SVM and neural networks. It could be observed that SVM implementation (SMO) outperformed the other three classifiers when the texts contained their natural class labels (Dataset 1) and performed most terribly on Dataset 3. This corroborates the submission of [15] that no single machine learning scheme is appropriate to all data mining problems because real datasets vary and to obtain accurate models, the bias of the learning algorithm must match the structure of the domain.
Meaning that the structure of our Dataset 3 is most amenable to neural network than any of the other machine learning schemes (Naive Bayes, SMO, J48) in our study. Worthy of note also is the usefulness of our application of the dimensional analysis principle which informed the multiplication of the two performance measures in our study. For example, if our comparison had been based on percent correct (in Appendix 1) only, we might have erroneously rated the performance of SMO relatively high on Dataset 3. The figure shows that the dataset having two class options (Dataset 3) performed best followed by the one having three class options (Dataset 2) and lastly the one having the instances labelled naturally, having five classes (Dataset 1). The result is consistent with those of [3] and [1] that reported that authorship attribution success improves with reduction in the number of authors or author classes. In the specific however, the present result shows that if we can reduce an authorship profiling problem to a two-class one, we can get an appreciable improvement in the effectiveness of authorship profiling task.
Using the TrainTestSplitMaker component of WEKA's knowledge flow interface to evaluate the performance of our model in classifying electronic messages to writers' countries of affiliation. Separate two-class label file was created for each country, resulting in a dataset for each country, where all attributes except the class attribute were the same. The class attribute for a particular country had instances labelled either as 'the country name' such as (Nigeria, Ghana, Cameroon) or as 'non country name' such as (Non-Nigeria, Non-Ghana, Non-Cameroon). Tables 4 shows the effectiveness of profiling authors' countries of affiliation by the resultant model. Application of our optimization method resulted in a remarkable improvement in the profiling of each country from the others. The study showed that we could achieve a percent correct ranging between 70.8% and 88.2% at Kappa statistics ranging between 0.04 and 0.34 compared to the highest possible percent correct value of 43.8% at kappa statistics of 0.26% if our method was not applied. This however is a trade-off on the efficiency of the profiling process because we needed to create separate labels for the class attribute. The extent of improvement in model performance however can be said to outweigh the additional effort. The detailed performance of the model is as shown in Table 5. The resultant model performed well when we consider the weighted averages of the performance measures of each dataset. It could however, be observed that the model was better at identifying texts that were not from the country as against those that were from the country in each case. It could also be observed that the performance of the model in predicting each country's texts vary directly with the number of each country's texts in the study corpus. The best performance was achieved in profiling Nigerian electronic texts from Non Nigeria texts, followed by that of Sierra Leone and then Ghana. Thus, it could be deduced that performance of our model could be much improved with bigger sub-corpora sizes.
The study through experiments sought the number of class options, feature set types and machine learning scheme that maximize the effectiveness of identifying the countries of affiliation of authors of online messages composed in English language. The online messages in our corpus were collected from online forums of five African countries with average length of 52 to 102 words. Using a product of percent correct and kappa statistics as our bases for model justification, the experiment showed that we achieved the most effective model when all feature set types, contained in a two-class dataset was analysed with the neural network (multilayer perceptron) machine learning scheme. Application of the parameters of the most effective model (derived from the experiment) to profiling the countries of affiliation of authors of the online messages resulted in about a hundred percent improvement in effectiveness.
The study achieved greater effectiveness but with a trade-off on efficiency. We look forward to having a model that can maximize both effectiveness and efficiency in profiling the authorship of online messages, and this constitutes a need for further studies. This approach in its present state can be very appropriate if a group is suspected and the purpose of authorship attribution is to affirm one's thought about the suspect's group of affiliation.