Brand management has become increasingly di cult in the age of social media as the volume of opinions and discussions surrounding a companies products have swelled beyond the scale at which they can be reviewed manually. In this paper we report details of a system for monitoring Twitter1 to nd tweets relevant to a speci c entity and the classi cation of such tweets based upon their reputational e ect. The system was evaluated as part of the recent RepLab 2012 pro ling task and the results from this evaluation show that our system out performs a number of nave baseline approaches.
Brand management has always been an important part of any companies public relations stratergy, but with the recent explosion in social media it has become more and more di cult to exhaustively monitor the vast amount of information in a timely fashion. The solution is to employ automatic methods to monitor social media for opinions. Such algorithms usually involve two stages. Firstly relevant documents (tweets, forum posts, etc.) have to be identi ed before a second phase can determine the opinions contained within them. The RepLab 2012 pro lling task accurately mirrors this situation using Twitter; the task consists of ltering tweets to determine those which are related to a given entity, and a polarity task in which the tweets are to be labelled according to their e ect on brand reputation.
In this paper we describe the GATE [
Our approach to determining relevance is based upon our recent research into
disambiguation [
Tie-breaks, i.e. candidate URIs with the same overall score, are resolved based on which one has the highest commonness score. If nevertheless more than one candidate remains, the instance which is more speci c according to the LOD ontology is preferred.
Unfortunately, because of the nature of tweets, it often isn't possible to distinguish a noun (the fruit) from a proper noun (either of the two companies) due to the lack of case information etc. Due to this we assume that any reference to the entity, using any case, might be relevant and then disambiguate. This has the unfortunate side e ect of assigning a URI to every mention. Our approach needs to be improved to incorporate the null assignment to handle cases where it is clear that the mention is not related to any of the known options. 3
We treated the polarity task as a standard text classi cation problem and classied every tweet regardless of whether or not we deemed the tweet to be relevant8. 3 http://en.wikipedia.org/wiki/Apple_Inc. 4 http://en.wikipedia.org/wiki/Apple_Corps 5 http://en.wikipedia.org/wiki/Apple_(disambiguation) 6 Hashtags were expanded via http://tagdef.com/ 7 http://linkeddata.org/ 8 This also makes sense given that in RepLab 2012 the two tasks are evaluated separately as well as in combination.
We experimented with two di erent classi cation approaches; k-Nearest Neighbours and Nave Bayes. Both approaches utilised the same GATE pipeline for pre-processing of the tweets. The rest of this section is split into three parts. Firstly we outline the pre-processing pipeline and then we detail the two classiers we built for this task. 3.1
Both approaches to polarity classi cation are based (primarily) upon simple tokenization of the individual tweets. We have implemented a simple GATE pipeline which performs tokenization, taking in to account a number of token types speci c to the way in which tweets are often written. This pipeline consists of the following processing resources (PR): { Document Reset: a standard GATE PR which simply deletes existing annotations, allowing an application to be run multiple times for development purposes. { Tokenizer: the standard GATE UNICODE tokenizer. { Sentence Splitter: As each document contains a single tweet, which we assume to be a single sentence, simply creates a single annotation spanning the entire tweet. When processing the training data features are created on each annotation recording the polarity and language of the tweet. { Hashtag Processor: a simple JAPE grammar which recognises hashtags, ensuring that a single hashtag is represented as a single token and a HashTag annotation. { Emoticon Processor: a combination of a gazetteer and a JAPE grammar which recognises emoticons, normalises them (see below) and ensures that they are represented as a single token and an Emoticon annotation. { Part-of-Speech Tagger: a standard GATE PR which assigns part-ofspeech (POS) tags to token annotations.
The only non-standard GATE component in this pipeline is the emoticon processor. The motivation behind this component is that with limited training data it is vital that we reduce, or eliminate, variation in expression of emoticons which may well be pivitol in conveying polarity. This normalization was performed by building a gazetteer of known emoticons9. Each entry in the gazetteer was paired with a normalized form of the emoticon. For example :-), :), and :] are all normalised to :). The JAPE grammar then ensures a single token annotation spans the emoticon using the normalised form rather than the underlying characters.
The result of running this pre-processing pipeline is a document annotated with a sequence of tokens which can be used by a machine learning algorithm to learn a polarity classi er. The same pipeline is also used to pre-process the unseen tweets before the classi ers are applied. 9 http://en.wikipedia.org/wiki/List_of_emoticons was used as a starting point 3.2
k-Nearest Neighbours Classi cation
The standard GATE distribution provides a number of machine learning tools
which can be used to perform text classi cation10. For these experiments into
polarity classi cation we used the Batch Learning PR con gured to perform
kNearest Neighbours (k-NN) classi cation using an implementation from Weka[
Whilst the machine leanring PRs provided with GATE are easy to use and highly
con gurable we decided to implement a second polarity classi er to allow for
more ne grained control over the entire process. We chose to build a Nave Bayes
classi er following the example in [
Our speci c implementation used lowercased versions of tokens as well as emoticons and hashtags as the input to the learning algorithm. In an attempt to improve classi cation due to the small amount of text present in the training tweets we also made use of the supporting documents (i.e. pages linked to from tweets). Rather than blindly including these pages in to the training set, the algorithm was tweaked to include the word counts but to ignore the document lengths. This simple change to the Nave Bayes algorith was made in an attempt to not skew the distributions, especially given that we classi ed each tweet in the test set using the tweet alone, and these are of a common length (i.e. usually short and never more than 140 characters). 4
We submitted two runs for the pro ling task; each run paired one of the polarity approaches with our disambiguation based approach to relevance ltering; GATE 1 used the k-NN classi er for polarity classi cation while GATE 2 used the Nave Bayes classi er. Unfortunately due to an error in the script used to combine the approaches for GATE 1 all tweets were classi ed as positive with respect to polarity. Whilst we list the results for this run below, we also detail a third submission, GATE 3, which we have evaluated (using the supplied gold standard) 10 see http://gate.ac.uk/userguide/chap:ml for details
Accuracy R S F(R,S) GATE 1 0.44 0 0 0 GATE 2 0.33 0.27 0.28 0.26
GATE 3 0.41 0.25 0.21 0.22 All positive 0.44 0 0 0 All neutral 0.33 0 0 0 All negative 0.23 0 0 0
Table 2. Polarity Classi cation Results
independently. For comparison we have also included the baseline results. For
full details of other submissions and the evaluation metrics used see the RepLab
2012 overview paper [
As we submitted the same relevance judgements for each of our runs Table 1 shows just a single GATE run instead of duplicating the values. Note that the all relevant baseline has an accuracy of 0.71 which shows the large bias within the evaluation set towards relevant tweets. A similar bias can also be found within the training data which results in only a small amount of non-relevant examples. This bias may be a general occurance or may be due to the speci c entities chosen for this evaluation, i.e. many of the entities do not actually require disambiguation. 4.2
Table 2 shows the results of our two attempts at polarity classi cation. Note that as previously mentioned we also report the results of the GATE 3 run, which while not an o cially evaluated run, shows how our GATE 1 run should have performed.
We believe that relatively low performance of our classi ers is due to two things: the small amount of training data used and the di erence in language use when expressing opinions across entities of di erent types. This second problem is probably more relevant than the lack of training data. During development we tested the algorithms using k-fold cross validation in two ways. In both cases we used six folds. One approach used the training data from one entity as a fold, and in the other data from all six entities were randomly split equally between the six folds. The average accuracy of the two approaches showed a di erence of around 25%, with better performance being achieved when the folds were generated randomly. We belive that this is due to the fact that none of the six entities in the training data overlap in the products or services they provide and as such the language used to talk about them di ers greatly. The six entities were: { Alcatel: a provider of backend communications equipment, usually sold to governments or telecommunication companies rather than end-users { Apple: a seller of consumer electronic goods including computers, phones and MP3 players { Armani: a high-end fashion label { Barclays: a British multinational banking and nancial services company { Lufthansa: the largest airline in Europe { Marriott: a large chain of hotels and leisure resorts
As you can imagine complaining about a late ight (Lufthansa) would use very di erent language to complaints about short battery life in a consumer electronics product (Apple). This suggests that when building a classi er the training data should contain tweets about a variety of di erent entities. 4.3
The results of combining our approaches to relevance ltering and polarity classi cation can be seen in Table 3. The combined evaluation was carried out by assuming a four class classi cation: not relevant, relevant and positive, relevant and neutral, and relevant and negative. As you can see from the results tables our combined approach to pro lling out performed any of the three baseline systems included for comparison. While a system with accuracy of 0.35 (our best result obtained by the GATE 1 run) can not be considered a strong performer we feel that it provides an ideal basis for our ongoing work in this area. As noted above we have already highlighted a number of areas where the approach could be improved and work is already going on to move the algorithms forward. This research is partially supported by the EU-funded FP7 TrendMiner11 project. Kalina Bontcheva is supported by the Engineering and Physical Sciences Research Council (grant EP/I004327/1).