In this workings notes paper for RepLab 2012, we describe our method of using feature selection metrics for polarity analysis. We use the correlation coefficient, a one-sided metric, to assign polarity scores to relevant words within a tweet; we then use aggregate of these scores to determine polarity of the tweet. Our results show a reasonable level of performance compared to other methods.
This paper describes a correlation-coefficient based procedure for determining
polarity of a tweet. Correlation coefficients have been used successfully for text
categorization, and also for sentiment analysis [
Among the many tasks for RepLab 2012, we concentrated on polarity analysis of the profiling task. Also, while there were tweets in both English and Spanish, we chose to focus on only the English tweets. 1.2
The main objective of the experiment was to determine the polarity (positive,
neutral, or negative) of a single tweet. Irrelevant tweets were excluded from the
polarity analysis. Although this task is closely related with sentiment analysis
[
First, we had to preprocess the tweets to extract relevant keywords that we
would use to determine polarity. To do this, we used the Stanford part-of-speech
tagger, and extracted nouns and adjectives [
As part of the RepLab 2012 task, we were given a small set of labeled files that we could use for training. Given the terms that we extracted above in the preprocessing phase, and the three polarity categories (positive, neutral, and negative), we calculated the correlation coefficient for a term t on class ci as CC(t, ci) = √N [P (t, ci)P (t¯, c¯i) − P (t, c¯i)P (t¯, ci)] pP (t)P (t¯)P (ci)P (c¯i) (1) where t¯ denotes other terms and c¯i denotes other classes. 2.3
Using the correlation coefficients that we calculated above (for each term and each polarity class), we can sum the correlation coefficients of a class for all the relevant terms within a tweet. After the summation of coefficients is done for each polarity class (positive, negative, and neutral), we then assign polarity of the tweet to be that of the class corresponding to the largest summation.
While this represents the most basic usage of the correlation coefficients, we can modify the thresholds somewhat to try to achieve better performance. 2.4
One approach is to try to set the thresholds so that the resulting class proportions
on the test set is equal to the class proportions on the training set. This method
usually works best when the training set is bigger than the test set, and the test
set is similar to the training set. Unfortunately for RepLab 2012, the test set
was much larger than the training set and it was also quite different from the
training set [
Using Feature Selection Metrics for Polarity Analysis
Another approach is to set thresholds that corresponded to the ones that achieve best performance in the training set (via a 5-fold testing within the training set). Again, we would expect better performance if the training set is large and similar to the test set - neither of which were met for RepLab 2012. 2.6
One nice result of the correlation coefficient approach is that we can get a measure of confidence on our classifications. For example, our confidence of a tweet being positive on classification whose values are (positive=3.8, neutral=0.2, negative=-2.5) is much larger than our confidence if the values were (positive=0.3, neutral=0.2, negative=-0.7) instead.
We can take advantage of this additional information by introducing human input for the borderline cases where the difference between the top two classes is small (we used 0.5 as a threshold). These are the cases where the automatically generated categorizations would have a high risk of being incorrect. 3
The evaluation on classifications was performed using reliability, sensitivity, and
the corresponding F measure, which are modified recall and precision measures
[
Method (rank of 42)
Best (ranked 1st) Human Input (ranked 10th)
Basic (ranked 15th) Modified Threshold 1 (ranked 26th) Modified Threshold 2 (ranked 28th)
Reliability Sensitivity F(R,S) .369 .350 .348 .364 .275 .285 .265 .280 .260 .230 .194 .198 .241 .184 .194
As we feared, modifying the thresholds resulted in much worse performance, because the training set was small, and unlike the test set. Meanwhile, the basic correlation coefficient based method performed okay, ranking 15th of 42 submitted runs. As expected, our run integrating human input on just the borderline cases led to a marked improvement compared to the other methods. 4
We were able to achieve reasonable results using relatively simple approach using correlation coefficients. Further, we showed that we can markedly improve our results by incorporating human input on cases deemed to be borderline by the correlation coefficients.
As a future direction, we can try exploiting the massive background data that we did not use for our current results. Because the training set in this case was so small, we can expect better results if we can exploit the background data to help expand our training set. Once we have expanded the training set in such a way, we may be able to expect better results from the modified threshold approaches that were not able to perform well with a small training set.