This paper describes our participation in the RepLab 2014 Reputation Dimensions task. The task is a multi-class classification task where tweets relating to an entity of interest are to be classified by their reputation dimension. For our participation we investigate two approaches; Firstly, we use a term's gini-index score to quantify the term's representativeness of a specific class and construct class profiles for tweet classification, and secondly, we perform tweet enrichment using a web scale corpus to derive terms representative of a tweet's class, before training a classifier with the enriched tweets. Our tweet enrichment approach performed exceedingly well, showing that this approach is effective for classifying tweets by their reputation dimensions and a promising direction for future work.
This notebook paper describes our participation in the Reputation Dimensions task of
RepLab 2014 [
For a company to be able to respond to changing public opinion in online communications, relating to the company, there are three components of the communication that must be understood. Firstly, the company must be aware of the aspects (Dimensions) of its business the communication is about, for example Products & Services. Secondly, the company must understand the type of author, for example is the author a journalist, and thirdly how influential the author is.
For our participation in RepLab 2014, we participated in the Reputation Dimensions task which addresses the first of these three components. The remainder of this paper is structured as follows. Section 2 gives an overview of the Reputation Dimensions task before Section 3 outlines our classification approaches. Section 4 presents the results of our submitted runs and, finally, in Section 5 we present our conclusions. 2
In this section we give an overview of the Reputation Dimensions corpus and task. The corpus consisted of English and Spanish tweets crawled during the period 1st June 2012 to 31st December 2012, with just over 75% English tweets. Each tweet related to at least one of 31 entities of interest from the Banking and Automotive industries. For each entity, there were at least 2,200 tweets, with at least 700 and 1,500 tweets for the training and test sets respectively. The most recent tweets were used for the test set. Participants were provided the tweet ids and had to download the tweet text directly from Twitter. To retrieve the tweet text, we used the Java tool provided by the RepLab organisers.
The Reputations Dimensions task is a multi-class classification task. Participating systems were to classify tweets as one of the seven reputation dimensions (Innovation, Citizenship, Leadership, Workplace, Governance, Performance and Products & Services) defined in the RepTrak Framework by the Reputation Institute3. The data set also included the “Undefined” class for tweets that were not classified as one of these seven dimensions. Undefined tweets are not included in the evaluation. 3
In this section we give an overview of the approaches we deployed for our participation in the Reputation Dimensions task. The research questions we address in our participation are twofold: (1) Can we use the gini-index of a term as a measure of the terms belonging to a reputation dimension to construct dimension profiles for tweet classification? and 2) Can we identify a tweet’s reputation dimension with greater accuracy by enriching the tweet using a web scale corpus?
The remainder of this section is structured as follows: Section 3.1 outlines our Reputation Dimension Profiling approach, then Section 3.2 gives an overview of our approach for tweet enrichment. 3.1
For our dimension profiling approach we convert the tweets to lower case, remove
nonalphanumeric characters, new lines and URLs, before using the Terrier Information
Retrieval Platform [
The “Undefined” class is included in our gini-index calculations, resulting in scores in the range of 0.125 (least discriminative terms) to 1 (most discriminative terms). For terms with a suitably high gini-index score, we use the term’s class conditional probability to determine the class that the term is most representative of and add the term to the class profile. Appropriate gini-index and class conditional probability thresholds were determined by parameter analysis, resulting in profiles constructed from terms with a gini-index score >= 0.3 and a class conditional probability > 0.1.
We submitted three runs employing this gini-index technique: Firstly, uogTr RD 1 classifies tweets using profiles constructed by the process. Secondly, uogTr RD 2 constructs profiles using this process before enriching the profiles with expansion terms derived from Wikipedia4. Finlay, uogTr RD 2 constructs profiles using this process before enriching the profiles with class specific query expansion terms. 3.2
For our tweet enrichment approach we pre-process tweets using the same approach as in Section 3.1 (converting to lowercase, removing non-alphanumeric characters, new lines and URLs), before enriching the tweets.
To obtain enrichment terms, we use Terrier to submit a raw tweet as a query to
a large contemporaneous web corpus. The top 10 retrieved documents then form a
pseudo-relevant document set. We calculate the entropy of each term within the set
of retrieved documents, and we select the top 20 terms with the highest entropy as the
most informative terms related to the tweet. Then, we enrich the pre-processed tweets by
appending these informative terms. Stop-words are removed from the enriched tweets,
which are further converted into term frequency feature vectors that are used to train
several types of classifiers in Weka [
We submitted two runs employing this technique: Firstly, for uogTr RD 4 we train
an SVM model using Weka and LibSVM [
A total of 30 systems were submitted for the Reputation Dimensions task. The task organisers also reported on a baseline text classification approach that used tweet words as feature vectors to train an SVM classifier. Results were reported ranked by the system’s overall Accuracy.
Table 1 presents the accuracy scores of our runs plus the baseline approach and the mean over all 31 submissions. 4 http://en.wikipedia.org/wiki/Main Page
We see that our Tweet Enrichment approach with SVM model performed excellently being ranked first with an accuracy score of 0.7318. The Tweet Enrichment approach with a Random Forests model also performs well, achieving an accuracy score of 0.6871 markedly above the baseline score of 0.6221.
Our Dimension profiling approach performed less well due to the fact that increasing gini-index and class conditional probability thresholds results in the selection of fewer discriminative terms for a class profile, therefore profiles become smaller as they become more class specific. This makes the task of classifying previously unseen tweets increasingly difficult due to the sparse nature of tweets. Enriching the dimension profiles counteracted this somewhat, as we see increased performance using the enriched profiles from 0.4960 for uogTr RD 1 to 0.6086 and 0.6205 for uogTr RD 3 and uogTr RD 2 respectively.
Table 2 shows the relative frequency of classes for classified tweets calculated as #class predictions/total tweets classif ied∗100 and Table 3 shows the precision and recall for each of the classes. We see that most of the runs are slightly biased towards the largest class “Products and Services” but the runs that performed best achieved notably higher precision on smaller classes such as “Innovation” and “Leadership”. We would expect to be able to further improve our results by achieving higher precision scores for “Workplace” and “Performance”.
In this paper, we described our participation in RepLab 2014 Reputation Dimensions task. We investigated two distinct approaches; firstly we use a term’s gini-index score to identify terms representative of specific classes to build class profiles for classifying tweets, and secondly, we take a tweet enrichment approach using a large contemporaneous web corpus to derive terms representative of the tweet’s class before training a classifier on the enriched tweets.
We found that our tweet enrichment approach performed very well. In particular, we note that when training a SVM classifier our tweet enrichment approach achieved the best overall Accuracy results of the task. This approach also performed markedly above average when training a Random Forests classifier. These results are very encouraging and we intend to explore this methodology further as future work.
ACKNOWLEDGMENTS The authors would like to thank Information Technology as a Utility Network (ITaaU) and the EC co-funded project SMART (FP7-287583).