1 Introduction
Author profiling task considers different profile dimensions of the author of the text.
This year shared task [
12
][
11
] is focusing on gender and language variety. Previous
competitions explored properties like gender, age group [13] and personal traits [
8
].
This task is interesting from both industrial and scientific points of view. Applications
like accurate advertising targeting, security and forensic fields make this task highly
relevant for practice. Also, the task can be considered as a tool for filling missing
information about a person in some political or demographic research. Research
community also pays attention to the task special track of PAN [
7
] shared task is held since
2013. Each year contributed a new language or new profile dimension to classify. The
common part of all years was gender identification. The first task was on blog data in
Spanish and English [
10
]. Competition in 2014 concentrated on different sources like
reviews, tweets etc. [
9
]. The task of 2015 extended by additional languages and
realvalued personal traits [
8
]. The main characteristic of the most recent shared task was
cross-genre. The target was to develop a model such that it will be robust to the domain
of data [13]. Since gender identification was presented in all previous competitions,
there were many tested approaches. The main features were n-grams and various text
statistics [
4
].
Language variety task was first to appear at PAN 2017, but there were language
variety detection competitions like Discriminating between similar languages and national
language varieties (DSL) 2016 [
1
]. Winning approach of this contest used char n-grams
in wide range (1-7) with a linear classifier [
3
]. We used this method not only for
language variety task but also for gender classification. A new feature of the current shared
task is language variety. Each language has several variants. For instance, we have two
several Portuguese: Brazil variant and European one. The task is to distinguish one
from another. Languages and their varieties can be found in Table 1. Our approach tries
to automatically extract features for each of variant Portuguese, English, Spanish and
Arabic without any linguistic knowledge. We use char n-grams as features and logistic
regression as a classifier. Evaluation metric is accuracy for both subtasks.
Language Variety
Portuguese Portugal, Brazil
English Australia, Canada, Great Britain,
Ire
land, New Zealand, United States
Spanish Argentina, Chile, Colombia, Mexico,
Peru, Spain, Venezuela
Arabic Gulf, Levantine, Maghrebi, Egypt
Table 1. Languages and Varieties
2
Methodology
This section is about our approach to current PAN Author Profiling task. First, we
briefly discuss preprocessing steps. Then, we describe how we construct the feature
space. Finally, we explain our choice of logistic regression as our classifier.
2.1
Preprocessing
We did not perform any preprocessing like removing hashtags, HTML tags and urls,
because we considered it as potentially informative features.
2.2
Classification
Our main assumption was to consider all short texts written by a single author as an
object in machine learning task formulation. We formulated the problem as classification
task with two or more classes depending on language (Table 1). If language has more
than two varieties we used "one versus other" scheme.
Let dataset
D = f(xi; yi)g; i = 1; : : : ; m;
to be consisted of pairs "object-class", xi 2 Rn. Each object xi has one of Z class labels
yi 2 Y = f1; : : : ; Zg: We have to find mapping f^ 2 F : Rd ! Y, which minimizes
empirical risk on dataset D:
f^ = arg min
f2F
X [f (xi) 6= yi];
xi;yi2D
where F – family of models.
Feature space was constructed such that for each language corpus we performed
counting of character level n-gram in some range. This counts were used as features. The
number of authors and features for different tasks can be founded in Table 2. One can
see that the data is quite sparse. Density distribution of non-zero n-grams for Portuguese
is shown in Figure 1. We did not used higher-order n-grams because of RAM
restrictions, although [
3
] reported quality to increase up to 7 char n-gram level.
We performed classification by means of logistic regression model with regularization
parameter C = 1. Our choice was justified by the fact that logistic regression has high
bias and low variance.
Evaluation
In this section we describe our results during cross-validation and on the test set. Next
we present embedding of the data in low-dimensional space. Finally, we discuss about
feature importances of our classifier.
3.1
Results and Data Visualization
Evaluation metric this task is accuracy:
Accuracy =
T P + T N
T P + F P + T N + T F
We evaluated quality of gender and language variety subtasks separately by using
crossvalidation scheme with five folds. Results can be found in Table 3.
Example ROC-curve for language variety classification of Portuguese is shown at
Figure 2. FPR and TPR are false positive rate and true positive rate respectively with
various classification threshold. We evaluated test scores via TIRA. [
6
]
It was interesting to see how data is located in a feature space. To do so we exploited
modern dimensionality reduction and data visualization techniques. Our choice of
algorithm was t-SNE [
2
] since it reported to be fast when the number of objects is small
and tends to efficiently preserve local structure of the data. Also, Python scikit-learn
[
5
] implementation of the algorithm supports sparse matrices as an input. Example for
Portuguese authors is at Figure 3. Unfortunately, axes of this algorithm have no clear
interpretation.
We investigated absolute values of coefficients of our model for Portuguese language
variety. This values can be considered as feature importances (Figure 4). Axis x means
position in array of linear regression coefficients sorted in descending order. Axis y is
absolute value of the coefficient. One can see that on the one hand feature coefficients
have pretty low magnitude, but on the other hand there is group of features with
relatively high importance.
We explored a simple and robust method for gender and language variety classification
for PAN17 Author Profiling task. It turned out that high-order char n-grams are good
features that are easy to generate with no need of handcrafting or expert linguistics
knowledge. The main disadvantage of such features is that this is almost impossible to
perform error analysis. We trained logistic regression classifier for both subtasks and
evaluated accuracy measure. We will explore effects on quality measure due to adding
even more n-grams.
13. Rangel Pardo, F., Rosso, P., Verhoeven, B., Daelemans, W., Potthast, M., Stein, B.:
Overview of the 4th Author Profiling Task at PAN 2016: Cross-Genre Evaluations. In:
Working Notes Papers of the CLEF 2016 Evaluation Labs. CEUR Workshop Proceedings,
CLEF and CEUR-WS.org (Sep 2016), http://ceur-ws.org/Vol-1609/