This paper presents the use of Bag-of-Words( BoW) and Psycho-Linguistic( P-L) approaches based upon the demographic trends in modeling multilingual( Roman-Urdu and English) SMS text( Short Message Service) for gender and age prediction. The data set1 was provided as a standard source to work for the multilingual author pro ling task in the contest FIRE'18-MAPonSMS2. The proposed approaches, as compared to the baseline results, adequately classify the test set to age and gender separately.
Authorpro ling is a task in automatic authorship identi cation that nds characteristics, particularly: demographic, of the author of a document. Having known the pro le of an author can help in resolving many issues, such as, crime investigation( e.g., by identifying the linguistic pro le of a suspected message), developing a recommendation system to recommend di erent products to di erent users( by nding the demographic features of authors through his reviews on a product) etc.
MAPonSMS task is about the prediction of gender and age of the authors based on the multilingual i.e. English and Roman-Urdu SMS data set of 350 documents.
In this paper, we present our proposed approaches and their contribution
towards the classi cation of gender and age for the contest. We have applied some
stylometric( lexical, syntactic, structural features) and content based( based
upon the textual content rather than the metadata) approaches to perform the
task. In stylometric approaches, features based upon Psycho-Linguistic have
proposed as taken from [
In the sections those follow, we present the related work in authorpro ling in section 2. In section 3 we describe the approaches we used to generate features of the given dataset. Section 3.4 provides an overview of the results for both prediction tasks separately. Section 4 concludes the paper and suggests potential improvements. 2
In the literature-world of automatic authorpro ling, a lot of work has performed
on datasets mainly collected from social media sites and blogs while SMS, as
data-source, remains neglected. Besides, the multilingual datasets are in the
languages which are spoken in developed countries while a little work( such as
given by Fatima et al. in [
In [
Fatima et al. in [
Cheng et al. propose Psycho-Linguistic and gender preferential cues along
with stylometric features for gender prediction in [
In third authorpro ling task at PAN 2015 [
Rangel et al. in the author pro ling task at PAN 2013 contest [
Researchers have been providing pieces of evidence for the last few decades
that a person's physical and mental health is strongly correlated with the words
he/she uses. Gottschalk et al. [
We conducted feature extraction on the dataset provided for age and gender prediction separately. The results show that both of the proposed approaches have proved better than the baseline approaches. 3.1
The given dataset was a training set to work for gender and age classi cation task for the contest FIRE18'-MAPonSMS. It is an SMS based multilingual corpus containing 350 total documents each from a di erent user. Each document contains multiple text messages and annotated with age groups and the gender such that the instances of all groups are balanced. For gender classi cation, it has 60% and 40% documents written by male and female authors respectively. For age classi cation, there are 31% documents categorized in age group 15-19, 50% in age group 20-25 while 19% in the group 25-xx. 3.2
We applied 1) Stylometry and 2) Content-based approaches for gender and age
prediction tasks among the renowned methods for authorpro ling i.e. stylometry,
content-based and topic-based [
task, we used 67 stylometric features in groups of three namely: character based( Table1), vocabulary richness( Table 2) and word based( Table3). Under the group of word based features are introduced some P-L as well. In content based( Table 4) both features are P-BoW.
For character based approaches, there are 43 features in total( as shown in
Table1) most of which have been employed by [
For vocabulary richness, we used total 9 features as given in Table2). These
vocabulary richness features have used by many researchers for age and gender
prediction problem such as in [
Feature Description Feature Description F1 Total number of all characters(C) F26 Count of underscore F2-F12 Punctuation marks(. , ? etc less m-dsh3) F27-34 Count of @, &, *, $, =, /, %, + sign F13 Percentage of punctuation marks to C F35 Count of all sorts of brackets F14-F15 Opening and closing curly braces F36 Count of white spaces F16-F17 Opening and closing square brackets F37 Percentage of of white spaces to C F18-F19 Opening and closing parenthesis F38 Percentage of letters to C F20-F21 Opening and closing angle brackets F39 Percentage of upper case letters to C F22 Count of white spaces F39 Percentage of upper case letters to C F23 Count of vertical lines F41 Percentage of white spaces to C F24 Count of uppercase letters F42 Percentage of digits to C F25 Count of digits F43 Percentage of tabs to C Intuition for proposing word ending with i/I ( F7 in Table3) and word ending with a/A( F6 in Table3) is the fact that in Roman-Urdu, many words are gender speci c( that discriminate a masculine noun4 from a feminine noun5). The ending of a word( things, abstract nouns, participles), if a vowel, usually helps in this gender classi cation. Words, ending with a are usually masculine whereas if a word ends with i or ii, it is usually a feminine6 word. For example, \Answer ne kr saki mein" is written in Roman-Urdu that means \I couldn't answer"( as written by a female author). The word \saki" means could that is a feminine version of participle while the same word is written as \saka" if referred by a male. There are many such words in Roman-Urdu those are used with the slight change of i and a letter, in the end, to refer to female and male author respectively. Other examples for such words are \khata-khati"( eat in English), \ata-ati"( come in English), \karta-karti"( do in English), \sota-soti"( sleep in English) and so on. Limitation of this approach is the fact that there are many neutral words( with no gender) that might have been counted as a masculine or a feminine. Additionally, a male author may refer to many feminine words and vice versa.
Two features( F9 and F10 in Table3) are related to the use of emojis and
smilies in the SMS document by each user. Emojies are combinations of di
erent characters to express emotions( emojis are also called emoticons, winks or
smileys). Chen et al. claim in [
Last Visited: 05, 08, 2018 7 http://cool-smileys.com/text-emoticons
Last Visited: 05, 08, 2018 count of emojis( F9 in 3) and the other is the average number of emojis per message( F10 in 3).
As the given multilingual dataset has been generated having collected from
di erent mobile users in Pakistan, another approach for proposing P-L features
is to see the tendency of authors to use English words in the multilingual dataset.
Urdu is Pakistan's o cial language yet English is used equally in o ces especially
for writing many o cial documents. Moreover, many text displays on di erent
banners, billboards, organization's name boards and many other activities use
English. Besides English is the medium of education in almost all of the
educational setups and institutes in Pakistan8. Studies show that some demographic
features a ect the language one uses [
Last Visited: 05,08,2018 9 one or more characters
Second group in P-BoW is the Negation Words, from [
As Roman-Urdu lacks standard lexicon, many spelling variations exist for a given word most of the times. For example, nahi, ni, nae, nii are all variations of a single word in Roman-Urdu that means no in English. So, it is important to mention here that any group of these P-BoW is not exhaustive because of the inherent inconsistency in the representation of the Roman-Urdu text.
total 75 features were generated out of which 70 are stylometric and 5 are content based.
In stylometric features, we introduced total 44 character based features as listed in Table 5 and 8 vocabulary richness features as given by Table 6.
Feature Description Feature Description
F1 Total number of all characters(C) F25 Percentage of digits
F2-F12 Punctuation marks(. , ? etc less m-dsh10) F26-36 Count of @, &, *, $, =, /, %, +,
F13 Percentage of punctuation marks to C F37 Count of all sorts of brackets
F14-F15 Opening and closing curly braces F38 Count of white spaces
F16-F17 Opening and closing square brackets F39 Percentage of letters to C
F18-F19 Opening and closing parenthesis F40 Count of upper case letters to C
F20-F21 Opening and closing angle brackets F41 Percentage of digits to C
F22 Percentage of white spaces F42 Percentage of tabs to C
F23 Count of vertical lines F43 Count of tabs
F24 Percentage of uppercase letters to C F44 Percentage of special characters to C
Percentage of certainty( F5) given in Table 8. As studies show that age strongly
a ects the use of language [
Table 8. Content based features for age classi cation .
Feature Description Feature Description F1 Percentage of assent(P-BoW) F4 Percentage of slang(P-BoW) F2 Percentage of negation(P-BoW) F5 Percentage of certainty(P-BoW) F3 Count of slang(P-BoW) 3.3
As the training dataset was annotated, the gender and age prediction tasks are supervised machine learning problems with Gender prediction a binary classi cation task( class attributes as male or female) whereas Age prediction a multiple classi cation task( class attributes as 15-19, 20-24, or 25-xx ). We used two classi ers: Random Forest( RF) and Meta Bagging( MB) from Meta class. Both of these algorithms are ensemble machine learning algorithms and are closely related. Note that Bagging was used with its default settings and REP Tree as its component classi er14.
We used 10-fold cross validation to evaluate the prediction models and reported accuracy as a measure to evaluate the performance because the dataset is balanced. Accuracy is the percentage ratio of correctly classi ed instances to incorrectly classi ed instances. 3.4
Gender Classi cation Table 5 shows the accuracy measure of di erent groups of features as reported by RF and MB. MB and RF gave accuracies of 60.8% and 52.85% for All P-BoW features respectively. All P-L and P-BoW features combined gave 66.85% accuracy for MB and 65.7% for RF. All word based features collectively gave an accuracy of 70.2% by MB and 71.4% by RF. All character based features combined resulted in 78.28% accuracy for MB and 77.14% for RF. Then the fth set of features i.e. combination of all features gave the highest accuracy of 80.29% by MB. 12 \Certainty is something that is certain or sure"
https://www.merriam-webster.com/dictionary/certainty Last Visited: 05, 08, 2018 13 P-BoW features of certainty and use of slangs proved more useful to discriminate age groups than the gender. 14 Although we also selected other Decision Tree algorithms as component classi ers for Bagging but REP Tree gave the best results.
It is evident that overall results were best reported by MB as 80.29% for combination of all features. The best group of features, if analyzed group-wise, was character based that gave an accuracy of 78.28% for MB. There is also an interesting fact about using P-L and P-BoW features combined. There were total 8 such features for gender classi cation which collectively gave an accuracy of 66.85% with MB. Of these 8 features, 4 were related to use of English or RomanUrdu words for which RF gave 65.7% accuracy. This shows that all P-BoW and P-L approaches contributed substantially to train gender classi er.
Table 9. Gender classi cation results using di erent groups of features. Feature Classi er Accuracy Feature All P-BoW MB 60.8% All character based
RF 52.85% All P-L and P-BoW MB 66.85%
RF 65.7% All word based 15 MB 70.2%
RF 71.4%
Classi er Accuracy MB 78.28%
RF 77.14% All features combined MB 80.29%
RF 78.57% Age Classi cation The accuracy scores for di erent sets of features with the names of classi ers are given in Table10 for Age classi cation. MB and RF gave accuracies of 49.14% and 44.85% for All P-BoW features respectively. All PBoW and P-L is the group of 10 psycholinguistic features. This combined group showed an accuracy of 55.7% for RF and 53.7% by MB. Then the combination of all word based features gave an accuracy result of 58% and 53.7% for RF and MB respectively. Group of all character based features when combined gave accuracy results of 55.14% by RF and 50.57% for MB. The combination of all stylometric and content based features gave the maximum accuracy of 60% for RF and 52% by MB
For age classi cation, results show that RF performed better than MB. All P-L and P-BoW features gave a combined accuracy of 55.7% for RF that is greater than 55.14% - the accuracy reported by RF for all character based( 44 in total) features combined. This shows a strong contribution of the P-L and P-BoW approaches( total 12 such features). We can infer from the results that best results( i.e., 60%) are reported by the set of all features combined using RF Decision Tree algorithm.
The accuracy values given by the classi ers for any set of features could not go beyond 60% for age classi cation, unfortunately. One reason of the proposed features for not being able to classify the age groups adequately can be due to the fact that the division of age groups is so closely related( 15-19, 20-24, 25-xx) in terms of many demographic traits such as education level, income background and even the type of educational institute( university) that the authors have many overlapping traits.
Table 10. Age classi cation results using di erent groups of features. Feature Classi er Accuracy Feature Classi er Accuracy All P-BoW RF 44.85% All character based RF 55.14%
MB 49.14% MB 50.57% All P-L and P-BoW RF 55.7% All word based16 RF 58%
MB 53.7 % MB 53.7% All features combined RF 60%
MB 52%
In this paper, we presented our approaches for gender and age prediction tasks
on the training data that is SMS based multilingual dataset containing
English and Roman-Urdu text of 350 documents each from a di erent user. We
used stylometric and content based approaches to extract the features and
reported 80.29% accuracy for gender and 60% accuracy for age prediction task
for MAPonSMS contest. The trained prediction models, when used to
predict the test set containing 150 multilingual documents, outperform the
baseline approaches. The improvement in accuracy for gender prediction goes from
0.60%( baseline) to 0.69% and for age prediction from 0.51%( baseline) to 0.53%.
The joint result accuracy improvement is from 0.32%( baseline) to 0.35%. 17
The task of authorpro ling for multilingual text displays a great cushion for
improvement, particularly for the gender classi cation task. To improve the
results some approaches, such as, preprocessing the dataset to normalize the
Roman-Urdu text( as discussed by [