The PAN 2016 author profiling task is a supervised classification problem on cross-genre documents (tweets, blog and social media posts). Our system makes use of concreteness, sentiment and syntactic information present in the documents. We train a random forest model to identify gender and age of a document's author. We report the evaluation results received by the shared task. The paper at hand presents a description of our approach to author profiling task at PAN 2016. The author profiling task includes two separate classification problems: gender classification and and age group classification. The latter is a multi-class (1824, 25-34, 35-49, 50-64, 65-xx) classification problem. The classification problem can be described as follows: An author profile in the context of the task is defined as an author's gender and age group. Given a set of documents with author profiles known, learn to identify the author's profile of documents of unknown authorship. The PAN 2016 author profiling task is cross-genre, meaning that the training documents will be on one genre and the evaluation will be on another genre. While this resembles real-world problems more closely, it also makes the task more challenging. The training corpus is a collection of tweets in English, Spanish and Dutch. However, our approach only focuses on documents in English language. This notebook paper is outlined as follows: in section 2 we describe our classification approach. In section 3 we present the results. Finally, we present the conclusion in section 4.
2.1
A number of features are based on the concreteness of words within tweets. The base
of this features is a dataset assembled by Brysbaert et al. with the help of Amazon
Mechanical Turk [
Concreteness has been studied in a variety of scenarios, with an emphasis on topics
like age-of-acquisition. There is some work on the link between the tendency to use
words with varying degree of concreteness and the age and gender of the person [
Our set of concreteness features consists of nine individual numeric features, based on three different scores being computed on a per word basis: 1. Mean concreteness: The score reflects the concreteness of the words within a tweet.
Concreteness thereby ranges from 5 to 1. 2. Standard deviation concreteness: This score encodes how strong the individual annotators agreed on the concreteness score. For words were all raters agreed, the score will be low. 3. Percent known: This score represent the percentage of all raters, who indicated that they know the word. This score ranges from 0:85 to 1.
In order to arrive at features at tweet level, all word based scores are aggregated. Therefore the minimum, the maximum and the arithmetic mean are computed for each of the three types of scores. 2.2
The motivation for this feature is to encode the main topics of a tweet in a concise way.
It is based on the publicly available WordNet Domains corpus 1 [
We developed an algorithm that creates a set of domains for a given short snippet of text. If available, the part-of-speech of the words can be utilised to narrow down the appropriate synset for each word. All domains of all words are combined while keeping track of a weight. The weight reflects how ambiguous the domain mappings are, thus words with many domains will yield lower weights.
Finally, the hierarchy of the domains is exploited, where each sub-domain distributed a share of its current weight to its parent. The ranked list of domains is finally
2 https://wordnet.princeton.edu/ pruned. All domains with a lower weight than half of the weight of the top ranked domain will be removed. On average a short snippet of text will yield a set of 1 to 5 domains.
In order to convert the set of domains into features we created a binary feature for each domain. If a tweet is associated with a certain domain, the corresponding feature will be set to true. 2.3
Sentiment in text in general, and more particular in tweets, might help to discriminate
different age groups as well as different genders. Based on this hypothesis, we
generate features that capture the polarity (whether positive or negative sentiment) of the
tokens in the tweet. For this task we use the well known sentiment library called
SentiWordNet [
In order to capture the polarity of the tweet and learn from its feature, we extract the score for each token. If the tokens aren’t defined in SentiWordNet we ignore them. We get the score of the most used context of the token. As a final step we model the polarity as four numeric features: we collect the tweet’s
These features represent the polarity distribution of a tweet seen as a bag of words. 2.4
Twitter provides some specific features like hashtags, retweets and replies. Especially hashtags are easy to use, but lack a direct equivalent in other blogs or message services. We expect users familiar with twitter to make use of these service specific features more often. We encode the usage of hashtags as three features:
2.5
The motivation behind the token length feature is somewhat similar to the hash tag usage. Users familiar with micro blogging or texting are used to the 140 character limit. As a consequence, we expect more frequent usage of abbreviations and acronyms. We encode the mean token length and the median token length. 2.6
In our approach we combine a number of different features into a single feature space. Therefore it is highly likely that the feature space itself will not be linearly separable. Depending on the actual classification algorithm this might be problem. Algorithms that have a low bias and a high variance will tend to cope easier with such scenarios. For example, a 1-NN algorithm does not impose the requirement of a linearly separable feature space. Apart from the implications of a high variance, there might be another culprit of such system.
In machine learning, a single object from which a model can be learned, or to which a model can be applied, is called an instance. In our case, an instance is a vector representation of a single document (i.e. a single tweet, blog or social media post).
It has been discovered that in many real-life datasets some instances behave
different to others. More precisely, certain instances have the tendency to be over-represented
in the neighbourhood of the remaining instances. These instance effectively behave like
hubs, hence the term hubness has been introduce to describe this phenomenon [
In order to deal with such hubs we introduce an optional step in the feature
engineering pipeline. This additional step is conduced after the feature space has been
created and before the actual classification. Instead of identifying individual hubs, we
try to detect regions, where multiple hubs are expected to be found. For this we
utilize a density based clustering algorithm, more precisely DBSCAN [
We make use of this property by filtering out all instances from high density regions. This is motivated by the intuition that instance, that are similar to each other will be less helpful for the learning algorithm than instances, that capture certain characteristics not present in the instances from the high density areas. The parameter of the DBSCAN algorithm can be effectively used to control the amount of instances being filtered out in this step. 3
We report the results as shown by the PAN 2016 evaluations done on TIRA [
As we ran into memory problems using the 4gb provided by the virtual machines on TIRA, we had to deactivate features in order to be able to successfully train a model. As the task at hand is cross-genre, we decided to deactivate the WordNet domains feature group. We expect the use of topics to be of minor help when dealing with tweets across different genres.
Table 1 shows the evaluation results obtained from TIRA. While the results (see table 1) on ’pan16-author-profiling-test-dataset2-english-201605-07’ are where we expected them to be, the results on the results on ’pan16-authorprofiling-test-dataset1-english-2016-03-08’ are extremely low. We cannot comment on this yet, as we have no further details on how the test sets look like.
An overview [
In this paper we presented our software developed for the PAN 2016 author profiling task. By extracting features like concreteness and sentiment, we trained a RandomForest to identify the gender and age class of an unknown tweet author. This is an initial approach towards authorship profiling. While our system achieved results in the region we expected on one of the test sets, it greatly underperformed on the other. Lacking the details on the test sets, we are not yet able to analyse the reasons for this. 4.1
In the future we will experiment with different combinations of the features. We also had a lot of problems with memory usage, which led us to remove some feature groups from the final evaluations. We plan to improve on this, thus being able to validate new features and use the full extraction pipeline available. 5
The Know-Center is funded within the Austrian COMET Program under the auspices of the Austrian Ministry of Transport, Innovation and Technology, the Austrian Ministry of Economics and Labor and by the State of Styria. COMET is managed by the Austrian Research Promotion Agency FFG.