This paper describes the participation of the Language and Reasoning Group of UAM at RepLab 2014 Author Profiling evaluation lab. This task involves author categorization and author ranking subtasks. Our method for author categorization uses a supervised approach based on the idea that we can use the information on Twitter's user profile, then by means of employing an attribute selection techniques we can extract attributes that are the most representative from each user's activity domain. For the author ranking subtask we use a two step chained method that uses stylistics attributes (e.g. lexical richness, language complexity) and behavioral attributes (e.g. posts' frequency, directed tweets) extracted from the users' profile and the posts. We use these attributes in conjunction with a Markov Random Fields for improving an initial ranking given by the confidence of Support Vector Machine classification algorithm. Obtained results are encouraging and motivate us to keep working on the same ideas.
From its inception in 2006, Twitter has become one of the most important platform for microblog posts. Recent statistics reveal that there are more that 250 million users that write more than 500 million posts every day1, talking about a great diversity of topics. As a consequence, several entities such as companies, celebrities, politicians, etc., are very interested in using this type of platform for increasing or even improving their presence among Twitter users, aiming at obtaining good reputation values.
As an important effort for providing effective solutions to the above problem,
RepLab2 proposes a competitive evaluation exercise for Online Reputation Management
(ORM) systems. For this year RepLab campaign (RepLab 2014 [
The Author Profiling task consisted on two subtasks, namely the Author Categorization and Author Ranking subtasks. On the one hand, the author categorization subtask consists in detecting author’s activity domain, e.g., discovering if certain author is a journalist, an activist, etc. On the other hand, the author ranking subtask consists in discovering those users that could represent an opinion leader among a community, i.e., finding those users who are the most influential (opinion makers) within a community of users.
In recent years an increasing number of methods and systems are been developed to
tackle the author profiling task. Most of these methods deal with profiling long texts, for
instance, posts in a blog [
The author ranking subtask is as new as the social media has been. One of the
challenges here is to identify an author or authors that are influential to a particular
community. For Twitter, some methods [
Our proposed approach for facing the problem of author categorization is based on the idea that information in the user’s profile description is good enough to find the author’s activity domain. To accomplish this task our method uses different attribute selection techniques to find the most representative characteristics, namely words, for each activity domain. A variation for this approach employs a term expansion technique that aims at improving the descriptive terms in each user’s profile.
Additionally, our proposed approach for author ranking problem is based on the idea that the rank (leadership) of an author can be detected by considering its writing style, and its behavior within the Twitter’s community. To accomplish this task, our method uses different stylistics attributes (lexical richness, language complexity, etc), as well as some behavioral features (posts’ frequency, directed tweets, etc).
Accordingly, this paper describes the participation of the Language and Reasoning
research group from UAM-C to the CLEF 2014 RepLab author profiling task (i.e.,
author categorization and author ranking subtasks) [
The rest of this paper is organized as follows. The next section describes all the steps considered in developing and performing all of our experiments for the author categorization subtask. Then, Section 3 describe the proposed approaches for solving the problem of author ranking. Finally, Section 4 presents the conclusions derived from this work and outlines future work directions. 2
The Author Categorization subtask consisted in classifying Twitter profiles by type of
author (i.e, journalist, professional, authority, activist, investor, company or celebrity).
Since we were given the main categories on the training set (i.e., twitter profiles), we
faced the problem as a supervised approach, particularly as a Text Classification (TC)
task. As a first step, all documents contained in the training and test sets were
preprocessed: deleting stop words, URLs and using a Porter stemmer algorithm [
We selected as main classifier method the Support Vector Machines (SVM) approach as implemented in Weka. Results are reported in terms of Precision, Recall and F-score, while as a validation strategy we employed a 10 cross-fold-validation technique. sion is performed, we compute the DF score of all terms and follow a similar process to the one described above.
Finally, as our third and fourth set of experiments we applied as a term selection
strategy one of the methods that has been proved effective in thematic TC, namely the
transition point (ptT ) [
Consequently, for our third and fourth experiments (second and third row from Table 3) we applied the t pT instead of the DF as term selection strategy. Table 3 shows obtained results on the training set when this strategy is employed.
The first column in Table 3 indicates whether we applied or not a terms expansion technique on users’ profiles. As can be observed, better results are obtained when no expansion is performed. 2.1
Based on the results obtained on the training set, we define the following as our official experiments.
UAM-CALYR-AC-1 : This experiment uses as a term selection strategy the DF score.
According with experimental results (Table 1) we represented users’ profiles by means of the 80% terms whit higher DF scores.
UAM-CALYR-AC-2 : Similar to previous experiment, this configuration represents users’ profiles with the 80% of terms. The main difference is that this experiments applies a users’ profiles expansion using WordNet.
UAM-CALYR-AC-3 : The transition point strategy is employed as term selection strategy. No users’ profiles expansion is performed, and only the 40% of terms surrounding the t pT are preserved.
UAM-CALYR-AC-4 : The transition point strategy is employed as term selection strategy. Users’ profiles expansion is performed by means of WordNet, and only the 40% of terms surrounding the t pT are preserved.
As can be observed in Table 4 our proposed experiments are able to achieve a competitive performance. Notice that in terms of the F measure, our methods that use the t pT as a term selection strategy are able to reach a 0.44 score. Generally speaking, proposed approaches are able to obtain good levels of reliability compared to proposed baselines and systems average, indicating to some extent that proposed approaches allow better precision values without decreasing sensitivity values. 3
We approached the Author Ranking problem as two-step chained method. The first phase is a supervised approach, and the later a unsupervised approach that uses a Markov Random Field. The general schema of our proposed method is shown in Figure 1 and a detail description of each steps is as following.
The supervised phase has as input two sets of attributes that aim at capturing stylistics and behavioral characteristics from each user. Each set is composed as follows: – Self-description (SD). Words in the user’s profile plus tags we use for mentions to users (mentions), number of hashtags, URLs, and Twitter ID’s user appearing in the profile (user-info). – Statistics of use (SU). Number of total tweets, number of followers, number of following, average of tweets per followers, ratio of following by followers, ratio of followers by following, and author’s category activity.
Our intuition is that opinion maker users tend to write profile descriptions in a more professional fashion than non opinion makers. Also, they tend to link to external URL that usually contains the user name on their Twitter ID. We also believe that influential users follow fewer people than people following them.
Once we extracted the set of attributes, we tackled the ranking problem as a binary classification task (with classes opinion maker and non opinion maker). We use a SVM to learn the model, and the final ranked list is generated according to the confidence of the SVM classification algorithm.
For our experiments, we wanted to find if the information of only the supervised phase is enough for providing a proper author ranking. In this direction we propose two experiments: i) using the complete set of extracted attributes (self-description and statistics of use), and ii) using only statistics of use. Table 5 shows the results obtained over the training set. According to these results, self descriptions add some information that is relevant to correctly classify a user into opinion maker and non-opinion maker categories so the rank can be assigned correctly.
For the unsupervised phase, the ranking is obtained based on a Markov Random Field (MRF) model that separates the opinion maker users from non-opinion makers, generating a new list by positioning the relevant users (opinion makers) first, and the others after.
MRF are a type of undirected probabilistic graphical models that aim at modeling dependencies among variables of the problem in turn. MRF modeling has appealing features for problems that involve the optimization of a configuration of variables that have interdependencies among them. Accordingly, MRFs allow the incorporation of contextual information in a principled way. MRFs rely on a strict probabilistic modeling, yet they allow the incorporation of prior knowledge by means of potential functions. For those reasons, in this work we employed an MRF model for refining the initial ranking of a set of users previously ordered by a supervised approach (Figure 1).
For our performed experiments, we adopted a MRF configured as described in [
Contrary to [
Additionally, for estimating similarities among users we did not use textual features,
instead we propose a set of stylistic and behavioral features that are extracted from
users’ posts which are described below:
3 Originally, the work showed in [
Our intuitive idea for proposing SB features is that opinion maker users would have similar writing styles as well as similar posting behaviors. Hence, the optimal configuration of the MRF would be able to gather both opinion-maker users and non-opinion maker users.
Finally, the initial configuration of the MRF is obtained by considering the output provided by the supervised phase (Figure 1). That is, the subset of users that were classified as opinion maker by the supervised phase are initialized as true opinion makers within the MRF, and all other users as non-opinion makers. Then, the MRF configuration of minimum energy (MAP) is obtained via stochastic simulation using the ICM algorithm. At the end of this optimization process, a new re-ordered (improved) list is produced.
Our second set of experiments are directed to evaluate the unsupervised phase, thus we can evaluate the contribution of proposed MRF at improving the initial author ranking (i.e., improving the output of the supervised phase). To accomplish this goal, we design three more experiments: i) using the ranked list from experiment SVM: SD+SU as input for the MRF, and, for estimating similarities within the MRF users are represented by means of features SD+SU+SB, ii) using the ranked list from experiment SVM: SU as input for the MRF, and for similarities estimation users are represented by means of SU+SB features, iii) using the ranked list from experiment SVM: SD+SU as input for the MRF, and for similarities estimation users are represented by just SB features.
As can be observed in Table 6 the MRF ranking proposal improves the initial ranking generated by the supervised phase (See Table 5). It is worth to remark that the best result was obtained when just the style and behavior (SB) features, are employed to estimate similarities within the MRF. These results indicate, to some extent, that opinion makers do have similar writing styles as well as similar behavior patterns when using Twitter. In addition, obtained results also indicate that features extracted from profile information, i.e., SD or SU features, are insufficient and even noisy for the MRF configuration, however the use of these attributes on a supervised fashion are a cheap and efficient option to assign an initial ranking for authors. 3.1
Based on the results obtained on the training set, we define the following as our official experiments.
UAM-CALYR-AR-1 : Using the supervised phase with SVM learning method and attributes: self-description and statistics of use. That is, the same configuration of experiment SVM:SD+SU, see Table 5.
UAM-CALYR-AR-2 : Using the complete schema showed in Figure 1. SVM as learning method with SD and SU attributes, generated ranking of these serves as input for the MRF which employs SD+SU+SB for representing users when estimating similarities within the MRF. This experiment uses the same configuration as experiment SVM: SD+SU & MRF: SD+SU+SB from Table 6.
UAM-CALYR-AR-3 : Using the supervised phase only with SVM learning method and statistics of use as attributes. That is the same configuration of experiment SVM:SU, see Table 5.
UAM-CALYR-AR-4 : Using the complete schema showed in Figure 1. SVM as learning method with SU attributes, generated ranking of these serves as input for the MRF which employs SU+SB for representing users when estimating similarities within the MRF. This experiment uses the same configuration as experiment SVM:SU & MRF:SU+SB from Table 6.
UAM-CALYR-AR-4 : Using the complete schema showed in Figure 1. SVM as learning method with SD and SU attributes, generated ranking of these serves as input for the MRF which employs just SB features for representing users when estimating similarities within the MRF. This experiment uses the same configuration as experiment SVM: SD+SU & MRF: SB from Table 6.
Table 7 shows official results obtained for our submitted experiments. The secondlast row is the official baseline proposed by RepLab organizers (i.e. Followers). The last row (All systems average) represents the Macro-average performance obtained by all participant groups.
It is worth mentioning that showed results in Table 7, represent the average MAP performance among the three domains i.e., Automotive, Banking and Miscellaneous, released as test set. As expected, our best result was achieved when the MRF uses just the SB features for estimating similarities among users’ profiles. Hence, if we consider the three test domains, the MAP performance obtained by our experiment UAM-CALYRAR-5 represents the second best system during RepLab 2014 campaign, particularly for the author ranking subtask. In this paper, we have described the experiments performed by the Language and Reasoning group from UAM-C in the context of the RepLab 2014 evaluation exercise. Our proposed system was designed for addressing the problem of Author Profiling that consists of two subtasks, namely author categorization and author ranking. The former deals with detecting author’s activity domain, e.g. journalist, authority, activist, company, celebrity, etc. The later consists in discovering which author have more influence (opinion maker) and which of those are less influential or have no influence at all (non opinion maker) among a community.
We proposed a supervised method for tackling the author categorization task. The main idea of such method was to perform attribute selection techniques based on document frequency scores and the transition point strategy. This method relies only in the information that authors provide on their Twitter profile. In its tuning phase we found that using only the 80% of the total number of attributes is enough for performing well in this task. An interesting conclusion of the obtained results on the test set is that using term expansion by synonyms of the words appearing in the profiles is not useful. This might be because profile’s text are usually not well written, thus the expansion only add noisy data to an already noisy description.
As future work for our proposed method for author categorization we plan to extend it by generating an activity domain model using external resources such as WordNet or Wikipedia. The idea of this modification is to generate enough (quality) information that can be used as a prototype description for each category or domain. Accordingly, we believe that terms expansion may perform better than our previous experiments since, in this case, terms to be expanded would be far less noisy than profile descriptions.
For the author ranking subtask we proposed a two-step chained method that consists of several processes. The first phase is a supervised approach, and the later a unsupervised approach that uses a Markov Random Field. A key element within our chain are the features extraction processes. Computed characteristics aim at capturing stylistic and behavioral features from authors. Our intuitive idea for proposing such features is that opinion maker users would have similar writing styles as well as similar posting behaviors. Some of the advantages of our proposed method is that it represents a fast and not expensive technique to determine an initial rank for authors. Once an initial rank has been provided, the MRF is able to improve the authors’ ranking considering some stylistics and behavioral attributes.
Obtained results indicate, to some extent, that opinion makers do have similar
writing styles as well as similar behavior patterns when using Twitter. In addition, obtained
results also indicate that features extracted from profile information are insufficient and
even noisy for the MRF configuration, however these represent a cheap and efficient
option for assigning an initial ranking for authors. As future directions we plan to define
and include an observation potential as proposed by [