We report on our participation in the reputation dimension task of the CLEF RepLab 2014 evaluation initiative, i.e., to classify social media updates into eight predefined categories. We address the task by using corpus-based methods to extract textual features from the labeled training data to train two classifiers in a supervised way. We explore three sampling strategies for selecting training examples, and probe their effect on classification performance. We find that all our submitted runs outperform the baseline, and that elaborate feature selection methods coupled with balanced datasets help improve classification accuracy.
Products & Services Related to the products and services offered by the company and reflecting customers’ satisfaction.
Innovation The innovativeness displayed by the company, nurturing novel ideas and incorporating these ideas into products.
Workplace Related to the employees’ satisfaction and the company’s ability to attract, form and keep talented and highly qualified people.
Governance Capturing the relationship between the company and the public authorities.
Citizenship The company’s acknowledgement of community and environmental responsibility, including ethic aspects of the business: integrity, transparency and accountability.
Leadership Related to the leading position of the company.
Performance Focusing on long term business success and financial soundness. Undefined In case a tweet cannot be classified into none of the above dimensions, it is labelled as “Undefined.” as she has been successful in moving forward into a. . . http://fb.me/18FKDLQIr” belongs to the “Workplace” reputation dimension, while “HSBC to upgrade 10,000 POS terminals for contactless payments. . . http://bit.ly/K9h6 QW” is related to “Innovation.”
The author profiling task aims at profiling Twitter users with respect to their domain of expertise and influence for identifying the most influential opinion makers in a particular domain of expertise. The task is further divided into two subtasks: (i) author categorization, and (ii) author ranking. The first subtask aims at the classification of Twitter profiles according to the type of author, i.e., journalist, professional, authority, activist, investor, company or celebrity. The second subtask aims at identifying user profiles with the biggest influence on a company’s reputation.
We focus on the reputation dimensions task. Our main research question is how we can use machine learning to extract and select discriminative features that can help us learn to classify the reputation dimension of a tweet. In our approach we exploit corpus-based methods to extract textual features that we use for training a Support Vector Machine (SVM) and a Naive Bayes (NB) classifier in a supervised way. For training the classifiers we use the provided annotated tweets in the training set and explore three strategies for sampling training examples: (i) we use all training examples for all classes, (ii) we downsample classes to match the size of the smallest class, (iii) we oversample classes to match the size of the largest class. Our results show that our runs consistently outperform the baseline, and demonstrate that elaborate feature extraction and oversampling the training data peak classification accuracy at 0.6704.
The rest of paper is organized as follows. In Section 2 we present related work, in Section 3 we introduce our feature extraction approach, in Section 5 we describe our experimental setup, in Section 6 we report on our results. We follow up with an error analysis and reflections in Section 7 and conclude in Section 8.
The field of Online Reputation Management (ORM) is concerned with the development of automatic ways for tracking online content that can impact the reputation of a company. This involves non-trivial aspects from natural language processing, opinion mining, sentiment analysis and topic detection. Generally, opinions expressed about individuals or organizations cannot be structured around a predefined set of features/aspects. Entities require complex modeling, which is a less understood process, and this turns ORM into a challenging field of research and study.
The RepLab campaigns address the task of detecting the reputation of entities on
social media (Twitter). Each year there are new tasks defined by the organizers.
Replab 2012 [
Replab proposes an evaluation test bed made up of multilingual tweets in English
and Spanish with human annotated data for a significant number of entities. The best
systems from previous years addressed the majority of the above presented tasks as
classification tasks by the use of conventional machine learning techniques, and focused on
the extraction of features that encapsulate the main characteristics of a specific
reputation related class. For the filtering and polarity detection tasks, Hangya and Farkas [
The reputation dimensions task stems from the hypothesis that customer satisfaction is easier to measure and manage when we understand the key drivers of reputation that actively influence a company’s success. The Reptrak system was designed to identify these drivers by evaluating how corporate reputation emerges from the emotional connection that an organization develops with its stakeholders. In this scenario, reputation is measured on a scale from 0–100 and considers the degree of admiration, trust, good feeling and overall esteem investors display about the organization. Reptrak defines seven key aspects that define reputation and the reputation dimensions task uses them to define the reputation dimensions that we listed in Table 1 except the “Undefined” category, which is an extra class local to the reputation dimensions task.
In our study we follow [
Classifying tweets by machine learning techniques imposes the need to represent each
document as a set of features based on the presence, absence or frequency of terms
occurring inside the text. Frequency distribution, tf.idf or 2 calculations are common
approaches in this respect. In addition, identifying the semantic relations between
features can capture the linguistic differences across corpora [
In our approach we consider textual features that we extract using corpus-based
methods for frequency profiling. We build on the assumption that more elaborate
feature extraction methods can help us identify discriminative features relevant for
characterizing a reputation dimension class. We hypothesize that frequency profiling using the
log-likelihood ratio method (LLR) [
Given two corpora we want to compare, a word frequency list is first produced for each corpus. Although here a comparison at word level is intended, part of speech (POS) or semantic tag frequency lists are also common. The log-likelihood statistic is performed by constructing a contingency table that captures the frequency of a term as compared to the frequency of other terms inside two distinct corpora. We build our first corpus out of all the annotated tweets for our target class and our second corpus out of all the tweets found in the rest of the reputation dimension classes. For example, for finding discriminative terms for the class “Products & Services,” we compare pairs of corpora of the form: “Products & Services” vs. “Innovation” and “Workplace” and “Governance” and “Citizenship” and “Leadership” and “Performance” and “Undefined.” We repeat this process for each of the eight classes and rank terms by their LLR score in descending order. We only keep terms that have higher frequency in the target class than for all the rest of the classes. This results in using as features only terms expected to be highly discriminative for our target class. 4
Machine learning methods are sensitive to the class distribution in the training set; this
is a well described issue [
Unbalanced strategy. This strategy uses the original class distribution in the training data, and it uses all of the training data.
Downsampling. This strategy downsamples the training examples of each class to match the size of the smallest class. Training examples are removed at random. We evaluate the system using ten fold cross validation on the training data, and we repeat the process ten times. We select the model with the highest accuracy.
Oversampling. This strategy oversamples the training examples of each class to match the size of the largest class. For each class, training examples are selected at random and are duplicated. Similarly as before, we evaluate the system using ten fold cross validation on the training data, we repeat the process ten times, and we select the model with the highest accuracy. 5
We conduct classification experiments to assess the discriminative power of our features for detecting the reputation dimension of a tweet. We are particularly interested in knowing the effectiveness of our extracted textual LLR features for each of the eight reputation dimension classes, and the effect of the three sampling strategies for selecting training data.
We submitted a total of 5 supervised systems where we probe the usefulness of machine learning algorithms for the current task. We list our runs in Table 2. We train our classifiers regardless of any association with a given entity, since there are cases in the training data when not all classes are present for an entity (see Table 3). In UvA RD 1 we choose to train an SVM classifier using all the available training tweets for each reputation dimension class, which implies that our classes are very unbalanced at this stage. In our next runs, UvA RD 2 and UvA RD 3, we randomly sample 214 tweets from each reputation dimension class and train a NB classifier, and respectively an SVM classifier. We also consider that using more training data can help our classifiers become more robust and better learn the distinguishing features of our classes. We explore a bootstrapping approach in runs UvA RD 4 and UvA RD 5, which train an NB classifier and an SVM classifier, respectively, using 7,738 tweets for each reputation dimension class. For under-represented classes we randomly sample from the labeled training data until we reach the defined threshold.
Dataset The Replab 2014 dataset is based on the Replab 2013 dataset and consists of automotive and banking related Twitter messages in English and Spanish, targeting a total number of 31 entities. Crawling the messages was performed in the period June December 2012 using the entity’s canonical name as query. For each entity, there are around 2,200 tweets collected: around 700 tweets at the beginning of the timeline used as training set, and approximately 1,500 tweets collected at a later stage reserved as test set. The corpus also comprises additional unlabeled background tweets for each entity (up to 50,000, with a large variability across entities). We make use of labeled tweets only and do not process messages for which the text content is not available or users profiles went private. The training set consists of 15,562 tweets. Out of these, we can access 15,294 (11,657 English, 3,637 Spanish) tweets. The test set consists of 32,446 tweets, out of we which we make predictions for 32,019 (24,254 English, 7,765 Spanish) non-empty tweets. Table 3 summarizes our training and test datasets. Preprocessing Normalization techniques help to reduce the large vocabulary size of the standard unigram model. Social media posts are known for the lack of language regularity, typically containing words in multiple forms, in upper and lower case, with character repetitions and misspellings. The presence of blogging annotations, abundance of hashtags, emoticons, URLs, and heavy punctuation can be interpreted as possible indicators of the rich meaning conveyed. We apply uniform lexical analysis to English
LLR Unigrams LLR Bigrams and Spannish tweets. Our preprocessing steps are basic and aim to normalize text content: we lowercase the tweets, remove language specific stopwords and replace Twitter specific mentions @user, URLs and numbers with the [USER], [URL] and [NUMBER] placeholder tags. We consider hashtags of interest since users generally supply them to categorize and increase the visibility of a tweet. For this reason we delete hashmarks and preserve the remaining token, i.e., #BMW is converted to BMW, so that Twitter specific words cannot be distinguished from other words. We reduce character repetition inside words to at most 3 characters to differentiate between the regular and the emphasized usage of a word. All unnecessary characters ["#$%&()?!*+,./:;<=>\ˆ{}˜] are discarded. We apply Porter stemming algorithm to reduce inflectional forms of related words to a basic common form.
Feature selection We select our textual features by applying the LLR approach; see Table 4 for the distribution of features over reputation dimensions in the training set. We represent each feature as a boolean value based on whether or not it occurs inside the tweet content. There is a bias towards extracting more features from the “Products & Services” reputation dimension class, since the majority of tweets in the training data have this label. At the opposite end, the “Innovation” and “Leadership” classes are among the least represented in the training set, which explains their reduced presence inside our list of LLR extracted features.
Training We use supervised methods for text classification and choose to train an
entity independent classifier. For our classifiers we consider Naive Bayes (NB) and
a Support Vector Machines (SVM). We motivate our choice of classifiers based on their
performance on text classification tasks that involve many word features [
Evaluation Replab 2014 allowed participants to send up to 5 runs per task. For the Reputation dimension task systems were asked to classify tweets into 7 reputation dimension classes (see Table 1); samples tagged as “Undefined” according to human assessors are not considered in the evaluation. Performance is measured in terms of accuracy (% of correctly annotated cases), and precision, recall and F-measure over each class are reported for comparison purposes. In the evaluation of our system we also take into account the predictions made for the “Unknown” class. The predictions for this class were ignored in the official evaluation, and therefore the absolute numbers between the two evaluations do not match. 6
We list the performance of our official runs in Tables 5 and 6. All our runs perform better than the baseline (0.6221). We highlight the fact that we make predictions for all 8 classes, including the “Undefined” category which is not considered in the official evaluation. We also decide to ignore empty tweets, even though these are taken into consideration by the official evaluation script!
Our most effective run is UvA RD 4, where we train a NB classifier using a
bootstrapping approach to balance our classes. It is followed closely by UvA RD 5, which
suggests that oversampling to balance classes towards the most representative class
is a more sensible decision than using all training data or downsampling classes
towards the least represented one. When we use all training data (UvA RD 1) we provide
the SVM classifier with more informative features than when dropping tweets (in runs
UvA RD 2, UvA RD 3), which confirms the usefulness of our LLR extracted features.
Balancing classes with only 214 tweets per class can still yield competitive results,
which are rather close to our previous approaches, and a lot more accurate in the case
of the SVM classifier. We notice that NB constantly outperforms SVM. NB’s better
accuracy might be due to independence assumptions it makes among features, which is in
line with other research carried on text classification tasks where NB classifiers output
other methods with very competitive accuracy scores [
Looking at the performance of our system per class, we find the following. The “Citizenship” and “Leadership” reputation dimension classes present high precision, followed by “Governance” and “Products & Services.” Recall is very high for the latter class, which comes as no surprise given the large number of features we extract with this label that tend to bias the predictions of our classifier towards “Products & Services.” The F1-measure is remarkably low for the “Innovation” class, since there are only few “Innovation” annotated tweets in the training set.
Detailed statistics of the number of tweets classified per reputation dimension class by our best system are presented in Table 7. In our analysis section, we perform a further experiment to assess how much including empty tweets in the evaluation and making predictions for the “Unknown” class influences results in terms of accuracy. We regenerated our top two best runs excluding the “Undefined” features and removing the 427 empty annotated tweets from the gold standard test file. We report on an almost 3% improvement in accuracy for run UvA RD 4 (from 0.6704 to 0.6897) and a 2% increase in accuracy for run UvA RD 5 (from 0.6604 to 0.6739).
On the one hand, we believe it is difficult to assess the performance of submitted systems and compare methods for the task of detecting Reputation Dimensions on Twitter data among RepLab participants since making predictions for only 7 reputation dimension classes outperforms systems that consider the “Undefined” category. We are not convinced that including empty tweets in the evaluation is a good idea and we were expecting the test corpus to be re-crawled beforehand so as to ignore non-relevant entries from the gold standard file.
Finally, our suggestion is that results could be more reliable and useful if the ratio of classified tweets would actually be considered when establishing a hierarchy of submitted runs. We were surprised to see systems with high accuracy scores ranking high up in the charts despite classifying fewer tweets than other runs with lower accuracy scores and more test set samples considered. It is well-known that accuracy is highly dependent upon the percentage of instances classified. 8
We have presented a corpus-based approach for inferring textual features from labeled training data in addressing the task of detecting reputation dimensions in tweets at CLEF RepLab 2014. Our results show that machine learning techniques can perform reasonably accurate on text classification if the text is well modeled using appropriate feature selection methods. Our unigram and bigram LLR features combined with an NB classifier trained on balanced data confirm steady increases in performance when the classification model is inferred from more example documents with known class labels. In future work we plan to use Wikipedia pages and incorporate entity linking methods for to improve the detection of concepts underlying the reputation dimensions inside tweets. We would also like to probe the utility of some other classifiers, like Random Forests, at an entity level, and consider tweets separately by language.
This research was partially supported by the European Community’s Seventh Framework Programme (FP7/2007-2013) under grant agreements nr 288024 (LiMoSINe) and nr 312827 (VOX-Pol), the Netherlands Organisation for Scientific Research (NWO) under project nrs 727.011.005, 612.001.116, HOR-11-10, 640.006.013, the Center for Creation, Content and Technology (CCCT), the QuaMerdes project funded by the CLARIN-nl program, the TROVe project funded by the CLARIAH program, the Dutch national program COMMIT, the ESF Research Network Program ELIAS, the Elite Network Shifts project funded by the Royal Dutch Academy of Sciences (KNAW), the Netherlands eScience Center under project number 027.012.105, the Yahoo! Faculty Research and Engagement Program, the Microsoft Research PhD program, and the HPC Fund.