-This paper describes the gender detection research done on Lithuanian texts using automatic machine learning methods. The main contribution of our work is investigations done namely on the very short (avg. ~ 39 tokens) non-normative texts. With this paper we analyze a fundamental problem: how to choose automatic methods (in particular, classifiers and feature types) that could achieve the highest accuracy in our solving author profiling task (when the short pure text itself is the only evidence used for determining the author's meta-information). The related research analysis helped us to select the methods which demonstrated encouraging results on the other languages and to apply them on the Lithuanian dataset. Out of a number of experimentally investigated classifiers with lexical or symbolic features the Naïve Bayes Multinomial method with character ngrams (of n = [1, 5]) feature type yielded the best performance reaching 83.6% of the accuracy.
INTRODUCTION
Due to the constant increase of electronic texts the various natural language processing works become especially relevant. However, lots of these texts are written anonymously or pseudonymously, therefore court linguistic analysts, administrators of Internet forums, supervisors of social networks more and more often face such problems as impersonation, bullying or harassment, discloser of confidential information, dissemination of disinformation, etc. Although to disclose an identity of a particular person sometimes is rather difficult, the meta-information (i.e., demographic characteristics: age, gender, etc.) also may provide some clues: e.g. a system detects that a 50 years old man is impersonating a 12 years old girl and encourages the police to dive more into details or even to take the decisive actions in finding a criminal.
Researchers confirm that the authors’ characteristics can
be determined during an analysis of the text style. It is
possible due to an existing phenomenon of a human stylome
(an analogue of a genome) which enables each person to
formulate sentences and to express their thoughts in special
unique ways [
characteristics (as age, gender or social status) or psychological state.
In general, the authorship identification has a long history
dating back to 1887 [
The ultimate goal of this research can be achieved by performing the following intermediate tasks: (1) a related work analysis (see Section II), (2) a construction of the representative corpus containing non-normative Lithuanian texts (see Section III), (3) an analytical selection of the most promising methods (see Section IV), (4) a precise experimental evaluation of selected methods (see Section V) (5) conclusions (recommendations) for the gender detection when using short non-normative Lithuanian texts and our further research plans (see Section VI).
II.
All existing author profiling approaches can be grouped
according to the following criteria: a percentage of training
instances in the dataset, an amount of information they
provide, (i.e., a recognition-training feedback) and the nature
of knowledge. According to these criteria the approaches are
[
The obsolete rule-based approaches use rules manually constructed by humans. The development process itself is very laborious and requires linguistic expertise. Moreover, created rules are tied to that specific solving problem, therefore are hardly transferable to new domains.
Unsupervised machine learning (or clustering methods) is selected when no meta-information is provided. The text samples are grouped according to the similarity between them. A main drawback of these methods is that their grouping does not necessarily correspond an imaginary grouping by a human. Mostly due to the very low accuracy these methods are not among the most popular choices in any author profiling tasks.
If texts are supplied with the necessary meta-information about the certain author characteristic (so-called class) the supervised machine learning is one of two best choices. The stylistic, lexical or symbolic text characteristics (extracted from the training instances) are provided as an input for a classifier. It generalizes all input information and produces a model as an output. This created model afterwards can be used for the author profiling of unseen texts. A main drawback of all supervised machine learning methods is that they require a comprehensive and representative dataset to create an exhaustive and robust model. An advantage is that the method can be flexibly adjusted to new tasks or domains: after adding new text samples the classifier can be easily retrained. The similarity-based approaches are very similar to the supervised machine learning by their nature. An only difference is that instead of creating the model they memorize and store all training instances and use similarity measures to determine to which of available classes some incoming instance is the most similar. An advantage of similarity-based methods is that they store the entire training set; therefore no information is lost during its generalization. Since both supervised machine learning and similarity-based approaches are the most accurate, they are the most popular for the various author profiling tasks. This important observation narrows down our research area to these approaches only.
The research done on various languages usually involves
the investigation of these popular approaches for supervised
machine learning (e.g., Naïve Bayes [
When investigating the Lithuanian non-normative texts we considered recommendations formulated for the other languages. However, a language factor itself is also very important, therefore must be taken into account as well. The Lithuanian language (that we are coping in this research) is rich in the vocabulary and morphology, has the rich word derivation system and the relatively free-word order in a sentence. Despite the Lithuanian language is rather complicated, some of previously mentioned language characteristics do not necessary complete the solving problem, i.e., it might occur that our investigated groups of individuals are bind to the very different sentence structures or vocabulary.
In fact the gender detection task for the Lithuanian
language is not absolutely new: it has been solved using the
supervised machine learning methods [
III.
A gender detection task was solved using the specifically prepared corpus of non-normative Lithuanian language texts. The corpus was composed of original posts (without any appearance of third party texts) harvested from the Facebook social network in October, 2016. It contains posts, comments and messages of 70 persons (for statistics see Figure 1) (32 and 38 texts belong to women and men, respectively (see Figure 2)). The youngest participant is 18 years old, the oldest – 77, the mean age of respondents is ~33.8. 43 and 27 people indicated that their level of education higher and secondary, respectively. 33 and 37 individuals claimed they are married and unmarried, respectively.
The corpus consists of 2.729 tokens in total1 (of which 1.433 are written by men and 1.296 by women) (see Figure 3)). The shortest text (without symbols and emoticons) is only 4 tokens length, the longest – 161, the average length of texts is ~39 tokens.
Fig. 2 A percentage of texts in our corpus written by men and women 1 It is important to notice, that instead of words we focus on tokens in this work. Besides regular words, tokens also include out-of-vocabulary words, numbers, and non-normative “words” with embedded digits or punctuation marks.
proper selection of the classifier and the proper selection of the feature type.
Supervised machine learning. A representative of this type is is the
Vector
Machine
(SVM)
method
(introduced by Cortes C. and Vapnik V. in 1995 [
Naïve
Bayes
Multinomial
(NBM)
(introduced by Lewis D. D. and Gale W. A. in 1994 [
Similarity-based. A representative of this type is the IBK method (introduced by Aha D. and Kibler D. in 1991
nearest neighbors’ classifier chooses the
appropriate k value, based on the k-time cross-check after
the distance evaluation (between a testing instance and all
samples in the training set).Another representative is
Kstar method (introduced by Cleary J. G. and Trigg L. E.
in 1995 [
These two lastmentioned methods store all available instances; therefore are prevented from the information loss during training.
The second direction involved the proper selection of the feature type. In our experiments we investigated: (sequences of 4 tokens in a window sliding one token at the time). For instance, from the phrase “gender detection from the Lithuanian texts” it would be generated 6 unigrams: “gender”, “detection”, “from”, “the”, “Lithuanian”, “texts” and 3 tetra-grams “gender detection from the”, “detection from the
Character features, in particular, character
ngrams similarly to token n-grams are sequences of
items,
but
instead
of
tokens
they
contain
characters. For instance, from the phrase “gender
detection” it would be generated the following
4grams: “gend”, “ende”, “nder”, “der_”, “er_d”,
“r_de”, etc. (where “_” denotes the whitespace). It
is important to mention that a value of n not
necessary has to be fixed: i.e., ranges are also
possible. With range, e.g., n = [
V.
EXPERIMENTS AND RESULTS
Our experiments were carried out on the corpus described in Section III using the methods and features described in
classifier.
We used the implementations of the methods incorporated
into the WEKA 3.8 machine learning toolkit2. WEKA [
In all our experiments we used 10 fold cross validation and evaluated
accuracy (1) and f-score (2). The results are considered acceptable and reasonable if the accuracy is above random (3) and
majority (4) baselines equal to 0.502 and 0.540, respectively.
_ = = here tp (true positives), tn (true negatives), fp(false positives), fn (false negatives) denote a number of correctly classified instances ci with ci and cj with any other cj, incorrectly classified instances ci with any other cj and any other cj with ci, respectively max( ) ∑ 2
Our preliminary experiments involved the selection of the
most accurate
classification technique
when
using token
unigrams (n=1), token tetra-grams (n=4) and character
tetragrams (n=4) (the results are presented in Figure 4). The best
results were achieved with SVM
and NBM
and character
tetra-grams3. These
methods also demonstrated the
best
2 Download from: http://www.cs.waikato.ac.nz/ml/weka/downloading.html
3 Since the f-score values demonstrate the same trend compared to the
accuracies, we do not present them in the following figures.
(1)
(2)
(3)
(4)
performance in gender detection tasks on the morphologically
complex Arabic language [
Later on we used only SVM and NBM (because they demonstrated the best performance in the preliminary experiments) by tuning a parameter n in the character n-grams (see obtained results in Figure 5).
The overall best results (reaching 0.836 of the accuracy)
on the short non-normative Lithuanian texts for the gender
detection task were achieved with the NBM and character
ngrams of n=[
By the way, the best of the only previously reported results
for the Lithuanian language in the gender detection task were
achieved with the SVM and lemma bi-grams as the feature
type [
The gender detection task is solved for a rather big group
of languages, reaching ~80% and ~56.53% of accuracy on the
normative English in [
VI.
CONCLUSION AND FUTURE WORKS
In this paper we report the first gender detection results using short (of only avg. ~39 tokens) Lithuanian nonnormative texts taken from the Facebook social network. During our research we investigated the most popular supervised machine learning (Naïve Bayes, Naïve Bayes Multinomial, Support Vector Machine) and similarity-based (IBK, kStart) techniques plus various lexical and character feature types.
The best results reaching 83.6% of accuracy were achieved
with the Naïve Bayes Multinomial method and character
ngrams (of n = [
Since the majority of the research done for the Lithuanian langue is mostly focused on the normative texts, in the future research we are planning to pay special attention to this problem by increasing the datasets and tackling the other author profiling tasks as age detection, social status detection, etc.