This paper describes our participation at RepLab 2014, a competitive evaluation for reputation monitoring on Twitter. The following tasks were addressed: (1) categorisation of tweets with respect to standard reputation dimensions and (2) characterisation of Twitter profiles, which includes: (2.1) identifying the type of those profiles, such as journalist or investor, and (2.2) ranking the authors according to their level of influence on this social network. We consider an approach based on the application of natural language processing techniques in order to take into account part-of-speech, syntactic and semantic information. However, each task is addressed independently, since they respond to different requirements. The official results confirm the competitiveness of our approaches, which achieve the 2nd place, tied in practice with the 1st place, at the author ranking task; and 3rd place at the reputation dimensions classification tasks.
In recent years, Twitter has become a wide information network, where millions of users share their views about products and services. This microblogging social network is an important source of information for companies and organisations, which aim to know what people think about their articles. In this way, identifying how people relate aspects and traits such as performance, services or leadership with their business, is a good starting point for monitoring the perception of the public via sentiment analysis applications. In a similar line, companies are interested in user profiling: identifying the profession, cultural level, age or the level of influence of authors in an specific domain may have potential benefits when making decisions with respect to advertisement policies, for example.
The RepLab 2014 on Twitter [
The remainder of the paper is structured as follows. Section 2 describes the main features of our methods. Sections 3, 4 and 5 show how we tackle the proposed tasks, illustrating and discussing the official results. Finally, we present our conclusions in Section 6. 2
The major part of our models rely on natural language processing (NLP) approaches which include steps such as: preprocessing, part-of-speech (PoS) tagging and parsing. The obtained syntactic trees act as a starting point for extracting the features which feed the supervised classifier employed for tasks 1 (reputation dimensions classification) and 2.1 (author categorisation). We built different models for each task and for each language considered in the evaluation campaign. With respect to task 2.2 (author ranking), a simple but effective method was used. Differences between tasks and languages are explained in the following sections. We describe below the high level architecture of our NLP pipeline. 2.1
Preprocessing We carry out an ad-hoc preprocessing to normalise some of the
most common features of the Twitter jargon, which may have an influence on
the perfomance of the tasks proposed at RepLab 2014:
– Replacement of URL’s: References to external links and resources are
replaced by the string ‘URL’.
– Hashtags: The use of hashtags may be helpful for classification tasks, since
they are often used to label tweets. In this way, we only delete the symbol
‘#’ in order to give to these elements the same treatment as words.
– Twitter usernames: In this social network, the usernames are preceded by
the symbol ‘@’. In order not to cause confusion at the tokenisation or tagging
steps, we delete that symbol, to then capitalise the first character and give
these elements the same treatment as actual proper names.
Part-of-speech tagging In order to be able to obtain the syntactic structure
of tweets, we first need to label each token of the message with its respective
part-of-speech tag. We used the Ancora [
Dependency parsing Given a sentence S = w1...wn, where wi represents the
word at the position i in the sentence, a dependency parser returns a
dependency tree, a set of triplets (wi, arcij , wj ) where wi is the head term, wj is
the dependent and arcij repr{esents the depe}ndency type, which denotes the
syntactic function that relates the head and the dependent. In this way, the phrase
‘best performance’ could be represented syntactically as (performance, modifier,
best). We rely on MaltParser [
Our classifiers are fed with three different types of features:
– N-grams : This type of features detect the presence of sequences of
contiguous words, where n is the number of concatenated terms. In this paper,
we consider both 1-grams and 2-grams (which make it possible to capture
some contextual information based on word proximity). Simple
normalisation techniques such as converting words to their lowercase form are applied.
In addition to n-grams of words, we also consider n-grams of lemmas3. The
aim is to reduce sparsity and training more accurate classifiers, specially
for Spanish language, where verbs, adjectives and nouns present gender and
number declensions.
– Psycometric properties: The LIWC [
In all cases, we use the number of occurrences as the weighting factor for the supervised classifier. 2.3
We use the WEKA [
The task consisted on relating tweets with the standard reputation dimensions proposed by the Reputation Institute and the RepTrak model4: products&services, innovation, workplace, citizenship, governance, leadership, performance and undefined (if a tweet is no assigned to any of the other dimensions).
Dataset The RepLab 2014 corpus is composed of English and Spanish tweets extracted from the RepLab 2013 corpus, which contained a collection of tweets referring to up to 61 entities. The RepLab 2014 only takes into account those who refer to banking or automotive entities, where each one is labelled with one of the standard reputation dimensions. To create the collection the canonical name of the entity was used as a query to retrieve the tweets which talk about it. Thus, each tweet contains the name of an entity. In addition, the corpus provides information about the author of each tweet, the content of external links that appear in a message and a flag to know if the tweet is written in English or Spanish. 4 http://www.reputationinstitute.com/about-reputation-institute/the-reptrakframework Evaluation metrics This task is evaluated as a multi-class categorisation problem. Thus, precision, recall and accuracy are the official metrics: T P P recision = T P +F P
T P Recall = T P +F N
T P +T N Accuracy = T P +T N+F P +F N (1) (2) (3) where TP and TN refer to the true positives and negatives and FP and FN indicate the false positives and negatives, respectively. The organisers sorted the official results by accuracy.
Runs We sent two runs. For each run, we trained two different LibLinear
classifiers [
The goal of the task was to assign Twitter profiles to one of these categories: journalist, professional, authority, activist, investor, company and celebrity. An additional class undecidable was proposed to place all those users that did not match any of the proposed categories.
Dataset The training and the test set are composed of the authors who wrote the automotive and banking tweets that we mentioned previously. In addition to user information, the organizers included the identifiers of the last 600 tweets of each user at the moment of the creation of the corpus. Due to the lack of time, we decided to download only 100 tweets for each author. In order to obtain these tweets faster, we used the capabilities of the Twitter API to download the timeline of an author instead of downloading the tweets one by one. However, that API method only allows the user to obtain the 3 200 most recent tweets of each author, so we were unable to find the tweets included in the corpus for many of them (the most active ones). More specifically, we could retrieve no tweets for around 1 000 authors.
Evaluation metrics The official results are the average accuracy between the
categories corresponding to automotive and banking. Only the authors
categorised as influential in the gold standard of task 2.2 are taken into account.
Runs This task is addressed as follows: given a set of tweets for an author,
they are collected into a single file, which is used to finally classify the user
according to the proposed categories. Since many of the categories in the training
corpus only contained a few authors, we discarded those classes in order to avoid
confusing machine learning algorithm. We trained two classifiers, one for each
language. After, testing different Support Vector Machine implementations, we
obtained the best performance on the training set (5-fold cross-validation) using
an SMO [
Run
Accuracy
To identify which authors are Spanish and which ones are English, for each author we counted the number of his last 600 tweets included at the corpus that are written in each language, assigning the author to the most frequent one. This information is provided by the RepLab 2014 organisation, without any need to download the tweets. More specifically, as we did in task 1, we sent two runs: – Run 1 : Both the Spanish and the English models use unigrams of lemmas and psychometric properties as features. We selected these features via greedy search on our processed training corpus (where all tweets of a user are merged into a single file). Since we did not have any tweet for many authors, we trained a back-off machine learner: a bag-of-words classifier which categorises these authors according to their profile description. – Run 2 : The only difference with respect to run 1 is the back-off classifier.
Authors for which we have not downloaded any tweet are always assigned to the majority class in the training corpus: undecidable.
Results Table 4 shows the performance of the systems participating in this task. We think that our poor performance is due to the small site of the training corpus that we were able to collect and process. The baseline proposed by the RepLab organisers reinforces our hypothesis, since they used an SVM approach based on a bag-of-words. They also included another baseline which assigns all authors to the majority class in the training corpus.
The task focusses on classifying authors a influential and non-influential, as well as ranking them according to that level of influence.
Dataset It is the same that the employed at task 2.1: Author Categorisation. The proportion in the training set is about 30% of influential users, with the remaining 70% being non-influential.
Evaluation metrics The organisers address the problem as a traditional ranking information problem using the Mean Average Precision (MAP) as standard metric. The experimental results are ordered according to the average of automotive and banking MAP measures.
Runs Classification of influential and non-influential users is made via a LibLinear classifier, following a machine learning perspective. To rank the authors we take as the starting point the confidence factor reported by the classifier for each sample. The confidence is then used to rank the users according to their level of influence. A higher confidence should indicate a higher influence. With respect to non-influential users, we firstly negate that factor, obtaining in this way lower values for the least influential authors. We again sent two models to evaluate this task, although in this case the runs present significant differences: – Run 1 : A bag-of-words model which takes each word of the Twitter profile descriptions to feed the supervised classifier. The weights of the classes were tuned taking 1.8 and 1.3 for influential an non-influential users, respectively. Since the corpus is domain-dependent (automotive and banking tweets) we hypothesise that the brief biography of the user may be an acceptable indicator of influence. We observed that words such as ‘car’, ‘business’ or magazine were some of the most relevant tokens in terms of information gain. – Run 2 : This run follows a meta-information perspective, taking the information provided by the Twitter API for any user. More specifically, we used binary features such as: URL in the Twitter profile, verified account, profile user background image, default profile, geo enabled, default profile image, notifications, is translation enabled and contributors enabled. In addition the following numeric features are taken into account: listed count, favourites count, followers count, statuses count, friends count and following. Results Table 5 illustrates the official results for this task. The baseline of the RepLab organisers ranks the authors by their number of followers. Our run 1 achieved the 2nd place, tied in practice with the 1st place, reinforcing the validity of the proposal for a specific domain. On the other hand, our second run did not work as expected, although it outperformed the baseline. 6
This paper describes the participation of the LyS research group at RepLab 2014. We sent runs for all tasks proposed. The classification for the reputation dimensions task is addressed from a NLP perspective, including preprocessing, part-of-speech tagging and dependency parsing. We use the output obtained by our NLP pipeline for extracting lexical, psychometric and syntactic-based features, which are used to feed a supervised classifier. We ranked 3rd, very close to the best performing system, confirming the effectiveness of the approach.
The author categorisation task is addressed from the same perspective. However, we could not properly exploit the approach due to problems to obtain much of the content of the training corpus.
On the other hand, the author ranking challenge was addressed from a different perspective. We obtained the second best-performing system, tied in practice with the 1st place, by training a bag-of-words classifier which takes the Twitter profile description of the users as features. This model clearly outperformed our second run based on metadata such as the number of favourited tweets or followers.
Research reported in this paper has been partially funded by Ministerio de Econom´ıa y Competitividad and FEDER (Grant TIN2010-18552-C03-02) and by Xunta de Galicia (Grant CN2012/008).