This paper aims to describe a TF-IDF approach based on word bigrams and n-grams at a character level used in the Celebrity Profiling competition at PAN CLEF 2019. 1 Copyright (c) 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CLEF 2019, 9-12 September 2019, Lugano, Switzerland.
This paper will take a look at different approaches towards the 2019 Celebrity Profiling competition at PAN CLEF1. The task is defined in the following way: Given a set of tweets for a user, one has to classify the user in four categories: gender, fame, occupation and age. There are three classes for gender and fame, eight for occupation and the age range is from 1940 to 2012.
Previous related work on the subject of gender classification includes an SVM
classifier with different types of word and character n-grams as features, along with
dimensionality reduction using Latent Semantic Analysis (LSA) [
The applied preprocessing involved the following techniques:
Removing all retweets of the user Removing all symbols except for letters, numbers, @, and # Replacing all hyperlinks with a special <url> token Replacing all user tagging with a special <user> token Substituting multiple consecutive spaces with a single one Adding a special <sep> token at the end of each tweet in order to distinguish between each tweet’s beginning and end.
Two additional ways of preprocessing were tested but proved to have poorer results on a baseline model, which used logistic regression, taking TF-IDF vectors with 10,000 features, the first one tried preserving the retweets of the users, the second one substituted happy and sad emojis, such as ‘:-)’, ‘:)’, ‘:D’, ‘:S’, ‘:(‘, ‘:<’ amongst others, with special <happy> and <sad> tokens.
We chose to split the provided data into a training and validation se t in a ratio of 80:20.
Discarding retweets 0.5483 0.3597 and emojis Preserving retweets 0.5237 0.5331 0.3675 0.3600 0.3564 0.3488 Replacing emojis 0.5233 0.5326 0.3670 0.3600 0.3569 0.3491 Table 1 - Different preprocessing results on a baseline tf-idf model with logistic regression Gender F1 score on training set
Gender F1 score on validation set 0.5514
Fame F1 score on training set 0.3818
Fame F1 Occupation score on F1 score on validation training set set 0.3687
Occupation F1 score on validation set 0.3529
The submitted approach used the described above preprocessing on a subset of all tweets per user. The subset for each user was chosen randomly and had a maximum cardinality of 500. The users’ tweets were vectorized with a TF-IDF vectorizer, taking into account the top 10,000 features from word bigrams. A combination of logistic regression and SVM were used as models for each different task. Different class weights were used for each task based on the number of labeled examp les from each class – the more examples a particular class has, the lower the class weight assigned to that class will be. The computed class weights were directly supplied to each tested algorithm as a parameter (class_weights). For the task of identifying the users’ gender, the following class weights were used:
Male – 0.46586179, Female – 1.17494337, Nonbinary – 428.12698413 The class weights used for the fame task are as follows:
Rising – 7.52987159, Star – 0.44780927, Superstar – 1.57703327 For the occupation task the following class weights were used:
The res ults from tests on different models using different hyperparameters are described below.
We can see that logistic regression with hyperparameters multiclass=multi and solver=newton_cg achieved the best results on the test set – 0.71714
Fame training set F1 0.75532 0.71992
For the task of identifying a person’s age, the range of years were divided into eight classes (subranges) and each time the model predicted the mean year for an interval. The intervals were constructed in such a manner, that assuming the true age of a user lied within an interval, predicting the interval’s mean would result in a correct guess due to the amount of error the contestants are allowed when predicting a user’s birthyear.
Interval range 1940-1955 1965-1969 1970-1980 1981-1989 1990-1997 1998-2004 2005-2009 2010-2012
In this section we will discuss other attempted techniques, which did not prove to be as successful as the one mentioned above.
Our team also attempted to represent each user as a TF-IDF vector of the top 10,000 features using n-grams on a character level, the chosen range was 3- and 4grams. Once again, we sampled 500 tweets for each user before attempting to vectorize the user. The achieved results were poorer than those achieved from the TFDF based on word bigrams.
This section examines the multi-layered neural network approaches we attempted.
We tried replacing the Linear SVM and Logistic regression with other models, having more capacity. One such is the multilayer perceptron. In order to protect against overfitting (which could be a serious problem when having 10 000/20 000 features and 27 000 examples) we used a relatively shallow model with 2 hidden layers, PRELU activation and dropout of 0.5, train ed with Adam optimizer. We also used balanced class weights. We experimented with only TF-IDF features and a concatenation of TF-IDF and character n-grams. However, the inability to mid a global optimum seems to have had a more detrimental effect than a po sitive one from the deeper model.
Fame training set F1 Fame validation set F1 T f-idf 0.82197 0.58564 Tf-idf + character n-grams 0.83256 0.59131
Table 7 - Results from experiments on Fame using different MLP models
With the recent advances in natural language processing embeddings and deep learning become increasingly more attractive. However, due to the big size of information per person, their usage is not straightforward. In order to handle that, we used the following model:
First we removed all retweets
In order to equalize the number of tweets per person we selected only the
first 1000 for each
We used GloVes’s default preprocessing and TweetTokenizer
We averaged Glove’s embeddings for each tweet
For each person we constructed a 200 * 1000 matrix (padded with 0 if the
person has less than 1000 tweets) which we used as the input of our model
The model was DPCNN [
We also tried training each task individually and all of them together with a multihead classifier.
Unfortunately, this did not produce satisfactory results. There was no real difference between individual models and the multihead one. Accuracy -wise the models were slightly behind the logistic regression, but their F1 score was underwhelming. Using class weights severely impacted the training process, leading to very low scores.
Fame training set F1 Fame validation set F1 GloVe + DPCNN 0.68427 0.39955
Table 8 - Results from experiments on Fame using GloVe and DPCNN
[2] Potthast, M., Gollub, T., Wiegmann, M., Stein, B.: TIRA Integrated Research Architecture. In: Ferro, N., Peters, C. (eds.) Information Retrieval Evaluation in a Changing World - Lessons Learned from 20 Years of CLEF. Springer (2019)