BiTeM/SIBtex is a university research group with a strong background in Text Mining and Bibliomics, and a long tradition of participating in large evaluation campaigns. The CLEF RepLab 2014 Track was the occasion to integrate several local tools into a complete system for tweet monitoring and categorization based on instance-based learning. The algorithm we implemented was a k Nearest Neighbors. Dealing with the domain (automotive or banking) and the language (English or Spanish), the experiments showed that the categorizer was not affected by the choice of representation: even with all data merged into one single Knowledge Base (KB), the observed performances were close to those with dedicated KBs. Furthermore, English training data in addition to the sparse Spanish data were useful for Spanish categorization (+14% for accuracy for automotive, +26% for banking). Finally, our best official run was in top five. Yet, performances suffered from an overprediction of the most prevalent category, while we were not able to address this issue of unbalanced labels within the competition time. The algorithm showed the defects of its virtues: it was very robust, but not easy to improve. BiTeM/SIBtex tools for tweet monitoring are available within the DrugsListener Project page of the BiTeM website (http://bitem.hesge.ch/).
BiTeM/SIBtex is a university research group with a strong background in Text
Mining and Bibliomics, and a particular focus on clinical and biological data.
Occasionally, the group is involved in studies with data from the intellectual property
(granted patents) or the social media (tweets and reviews) domains. Finally, the group
has a long tradition of participating in large evaluation campaigns, such as TREC,
NTCIR or CLEF [
BiTeM/SIBtex only took part in the first task: Reputation Dimensions. The goal of the task was to perform text categorization on Twitter, i.e. to design a system able to assign a predefined category to a tweet. This category was one out of eight related to companies’ reputations. All tweets dealt with entities from the automotive (20 entities) or the banking (11 entities) domain, and were in English (93%) or in Spanish (7%). For training and/or learning purposes, participants were provided with approximately 15,000 tweets labeled by human experts (the training set). Additionally, participants were allowed to use provided sets of tweets related to the mentioned companies for incorporating domain knowledge. Then, the systems had to predict the good categories for 32,000 unlabeled tweets (the test set).
In this task, the main difficulty was to efficiently preprocess the text, as standard Natural Language Processing strategies can fail to deal with the short, noisy, and strongly contextualised nature of the tweets. Another difficulty was to efficiently learn from unbalanced classes: indeed, the “Products & Services” category was assigned to 44% of the training tweets, versus only 1% for the “Innovation” category. Finally, this was a multilingual task, but the language distribution also was unbalanced, with less than 10% Spanish learning instances. We applied a simple and robust statistical approach in order to design our system, based on instance-based learning for categorization purposes. Instance-based learning is a kind of machine learning that compares unseen instances with labelled instances contained in a Knowledge Base (KB). The instance-based learning algorithm we chose to implement is k Nearest Neighbors (k-NN).
Three particular questions were investigated during this study: - Q1 : is it better to build one KB for each domain, or to merge automotive and banking into the same KB ? - Q2 : is it better to build one KB for each language, or to merge English and
Spanish into the same KB ? - Q3 : as the labels are unbalanced, is it efficient to use weighting strategies for categorization ? 2
During the training phase, all tweets belonging to the training set were preprocessed by component 1. Component 1 is composed of several standard Natural Language Processing treatments, along with a language detector. Then, they were indexed in one or several indexes by component 2, in order to make the KB. Component 2 is an Information Retrieval platform, which builds indexes for related documents retrieval.
During the test phase, all tweets belonging to the test set also were preprocessed by component 1. Then, for a given test tweet, the component 3 (k-NN) exploited the KB in order to retrieve the most similar tweets seen in the training data, and to infer a predicted category. Official runs were computed with the whole test set. 2.2 Data
A training set of approximately 15,000 labelled tweets was provided by the organizers. There as an average of 511 tweets for an automotive entity, versus 485 for a banking entity. Table 1 shows the average distribution of each category for a given entity.
Category Citizenship Governance Innovation Leadership
Performance Products & Services
Undefined Workplace
TOTAL
The first observation from Table 1 is that classes are unbalanced. For the automotive domain, 66% of training tweets deal with Products & Services, while only 0.8% deal with Leadership. The second observation is that distributions are different for the banking domain (e.g. only 21.4% for Products & Services). The distribution observed in test set (not reported) were consistent with those observed in the training set.
Here is a representative example of a tweet: 208844584137134080: Me and a sexy BMW M3 at last nights shoot <a href="http://t.co/ibW6sdXW" class="twitter-timeline-link" data-pre-embedded="true" dir="ltr" >pic.twitter.com/ibW6sdXW</a>
Tweets often contain metadata within tags, the most frequent being hyperlinks (<a>) and emphasis (<b>). Moreover, they often don’t have proper punctuation.
The goal of the component 1 was to preprocess the tweets in order to have proper and efficient instances to index (for the training phase) or search (for the test phase). For this purpose, a set of basic rules was applied. Tags were first discarded. Contents within an emphasis tag (<b>) were repeated in order to be overweighted. Contents within a hyperlink tag (<a>) also were repeated, and were preceded by the “HREF” mention.
For language detection purposes, we performed simple N-Gram-Based Text
Categorization, based on the Cavnar and Trenkle works [
The goal of the component 2 was to build one or several indexes from the training
data, in order to obtain a related documents search engine. For this purpose, we used
the Terrier platform [
Dealing with Q1 and Q2, we investigated several strategies and built several indexes: - all: a unique index with all the training tweets; - cars: an index with all tweets from the automotive domain; - banks: an index with all tweets from the banking domain; - cars_en: an index with all English tweets from the automotive domain; - banks_en: an index with all English tweets from the banking domain; - cars_es: an index with all Spanish tweets from the automotive domain; - banks_es: an index with all Spanish tweets from the banking domain. 2.5 k-NN
The goal of the component 3 was to categorize tweets from the test set. For this
purpose, we used a k-NN, a remarkably simple algorithm which assigns to a new text
the categories that are the most prevalent among the k most similar tweets contained
in the KB [
E( xi , c) RSV ( xi ) where predcat is the predicted category for a test tweet, c1,c2…cm are the possible categories, K is the set of the k nearest neighbors of the test tweet, RSV(xi) is the retrieval status value given by the component 2 (i.e. the similarity score) of the neighbor xi, and E(xi,c) is 1 when xi is of category c, 0 otherwise.
Dealing with Q3, an additional score computing was tested for handling the issue of
unbalanced labels when using a k-NN. Several studies were conducted for such an
issue [
E( xi , c) RSV ( xi ) W ( xi , k d , c) where d is a parameter and W(xi,k+d,c) is the frequency of training tweets from category c in the set of the k nearest neighbors of xi.
The Q1, Q2 and Q3 issues were addressed with the training data, thanks to a ten-fold cross validation strategy. 3.1 Q1: is it better to build one KB for each domain, or to merge automotive and banking into the same KB ?
First, we investigated Q1, by exploiting the all, cars, and banks indexes. Both languages were merged into the same indexes. Figures 2a and 2b show the performances of the system for different values of k.
Experiments showed that the optimal k for these data was around 10. They also showed that throughout the curves, it was better to use specific indexes (orange curves) versus a unique merged index (blue curves). Yet, the difference between best performances is not significant, with an accuracy of 0.69 for the all and the banks indexes for banks tweets (at k=10), and accuracies of 0.77 versus 0.76 for the cars index and the all index. We can say that, for categorizing tweets from a given domain, data from the other domain do not provide useful information, but do not degrade the optimal performances, thanks to the k-NN robustness. 3.2 Q2: is it better to build one KB for each language, or to merge English and Spanish into the same KB ?
Second, we investigated Q2, especially for the Spanish language that represented less than 7% of the training data. We exploited the cars, banks, cars_es and banks_es indexes. Figures 3a and 3b show the performances of the system for different values of k.
Experiments showed that the optimal k for Spanish data was around 30, significantly higher than the general case. This could be explained by the smaller set of Spanish instances. They also showed that it was better to use both languages indexes (orange curves) versus a Spanish-specific index (blue curves). We can say that, for categorizing tweets from Spanish, an additional amount of English data provides useful information and increases the top accuracy (from 0.69 to 0.79 for cars, from 0.57 to 0.72 for banks).
The same experiments with the English language (not reported) showed no significant differences between the merged and the English-specific indexes. 3.3 Q3: as the labels are unbalanced, is it efficient to use weighting strategies for categorization ?
The last experiments aimed at tuning the k-NN for dealing with unbalanced labels. Results with different values of d (not reported) showed no improvements from the unweighted k-NN. Other strategies need to be investigated fur this issue.
We finally submitted two runs. For both runs, the automotive and banking training tweets were in separate Knowledge Bases. For run 1 (SIBtex_RD_1), we used a merged index for both languages. For run 2 (SIBtex_RD_2), we used specific languages. The best accuracy in the competition was 0.731. SIBtex_RD_1 had an official accuracy of 0.707 and was ranked #4. SIBtex_RD_2 had an official accuracy of 0.704 and was ranked #6. Interestingly, performances were better with the test set.
Official statistics also showed that, in our run, the “Products & Services” category was overrepresented (68% instead of 49% in the gold standard). Although we failed to design an efficient strategy for dealing with unbalanced data, this distribution shows that our k-NN probably suffered from this issue.
We designed a complete system for tweet categorization according to predefined
reputational categories. Dealing with the domain (automotive or banking) and the
language (English or Spanish), we explored a range of representations and wanted to
know if it was better to use separate or merged Knowledge Bases. The experiments
showed that the k-NN was not very affected by the kind of representations: even with
all data merged into one single KB, the observed performances are close to those
observed with dedicated KB. Moreover, English training data were useful for Spanish
categorization (+14% for accuracy for automotive, +26% for banking). Yet, the
unbalanced labels make the k-NN to predict the most prevalent category (“Products &
Services”) more often than necessary (68% instead of 49%); this issue needs to be
investigated in future works. The k-NN showed the defects of its virtues: it was
robust, but not easy to improve. BiTeM/SIBtex tools for tweet monitoring are
available within the DrugsListener Project page of the BiTeM website [