In this paper we propose an adaptation of the KullbackLeibler divergence score for the task of sentiment and opinion classification on a sentence level. We propose to use the obtained score with the SVM model using different thresholds for pruning the feature set. We argue that the pruning of the feature set for the task of sentiment analysis (SA) may be detrimental to classifiers performance on short text. As an alternative approach, we consider a simple additive scheme that takes into account all of the features. Accuracy rates over 10 fold cross-validation indicate that the latter approach outperforms the SVM classification scheme.
In this paper we consider sentiment and opinion classification on a sentence level. Sentiment analysis of user reviews, and short text in general could be of interest for many practical reasons. It represents a rich resource for marketing research, social analysts, and all interested in following opinions of the mass. Opinion mining can also be useful in a variety of other applications and platforms, such as recommendation systems, product ad placement strategies, question answering, and information summarization.
The suggested approach is based on a supervised learning scheme that uses feature selection techniques and weighting strategies to classify sentences into two categories (opinionated vs. factual or positive vs. negative). Our main objective is to propose a new weighting technique and classification scheme able to achieve comparable performance to popular state-of-the-art approaches, and to provide a decision that can be understood by the final user (instead of justifying the decision by considering the distance difference between selected examples).
The rest of the article is organized as follows. First, we present the review of the related literature in Section 2. Next, we present the adaptation of the Kullback-Leibler divergence score for opinion/sentiment classification in Section 3. Section 4 provides a description of the experimental setup and corpora used. Sections 5 and 6 present experiments and analysis of the proposed weighting measure with the SVM model, and additive classification scheme respectively. Finally, we give conclusions in Section 7. 2
Often as a first step in machine learning algorithms, like SVM, na¨ıve Bayes,
k-Nearest Neighbors, one uses feature weighting and/or selection based on the
computed weights. The selection of features allows decrease of the dimensionality
of the feature space and thus the computational cost. It can also reduce the
overfitting of the learning scheme to the training data. Several studies expose
the feature selection question. Forman [
Kennedy et al. [
In the study carried out by Su et al. [
Pang et al. [
A variation of the SVM method was adopted by Mullen et al. [
Paltoglou et al. [
In our experiments we adopted a feature selection measure described in [
Schneider [
KLt(f ) = K˜t(S) − K˜Lt(S) (1) where K˜t(S) is the average divergence of the distribution of ft in the individual training documents from all training documents. The difference KLt(f ) in the Equation 1 is bigger if the distribution of a feature ft is similar in the documents of the same class and dissimilar in documents of different classes.
K˜t(S) is defined in the following way:
K˜t(S) = −p(ft) log q(ft) (2) where p(ft) is the probability of occurrence of feature ft (in the training set). This probability could be estimated as the number of occurrences of ft in all training documents, divided by the total number of features. Let Njt be the number of documents in cj that contain ft, and Nt = P|jC=|1 Njt/|S|. Then q(ft|cj ) = P|jC=|1 Njt/|cj | and q(ft) = Nt/|S|. The second term from 1 is defined as follows:
|C| K˜Lt(S) = − X p(cj )p(ft|cj ) log q(ft|cj )
j=1
where p(cj ) is the prior probability of category cj , and p(ft|cj ) is the probability
that the feature ft appears in a document belonging to the category cj . Using
the maximum likelihood estimation with a Laplacean smoothing, Schneider [
1 + Pdi∈cj n(ft, di)
|V | + P|tV=|1 Pdi∈cj n(ft, di)
(3)
(4)
where |V | is the training vocabulary size or the number of features indexed,
n(ft, di) is the number of occurrences of ft in di. It is important to note that the
afore mentioned average diversion calculations are really approximations based
on two assumptions: the number of occurrences of ft is the same in all
documents containing ft, and all documents in the class cj have the same length.
These two assumptions may turn detrimental for long extract text classification
as noted by the author himself [
In sentence-based classification the pruning of the feature set can turn out quite
detrimental to the classification accuracy. This is true if the size of the training
set is not big enough in order to be sure that some important for classification
features are not discarded. Thus, we propose to modify the KL-divergence
measure for sentiment and opinion classification. In [
KLtP OS (f ) = K˜tP OS (S) − K˜LtP OS (S)
KLtNEG(f ) = K˜tNEG(S) − K˜LtNEG(S) This way, we obtain two sums P KLtP OS (f ) and P KLtNEG(f ) over the features present in the sentence. The final difference of the two sums (denoted further as KL score) can serve as a prediction score of to which category the sentence is most similar. 4
We use the setup with unigram indexing, short stop word elimination (several
prepositions and verb forms: a, the, it, is, of) and the use of the Porter
stemmer [
In our study we use three publicly available datasets that we chose based on
their popularity as benchmark datasets in SA research. The first one is Sentence
Polarity dataset v1.01 [
As a third dataset, that contains newspaper articles, we use the MPQA
dataset2 [
Here is the information on corpus statistics as reported in [
We were interested in evaluating the features selected by our method with the
use of the SVM classifier. As pointed out in [
Joachims [
We also experimented with other types of kernels, namely with the radial basis function kernel. From our experiments, learning of the SVM model with this kernel takes substantially longer time and gives approximately the same level of the performance as the linear kernel. 1 http://www.cs.cornell.edu/people/pabo/movie-review-data 2 http://www.cs.pitt.edu/mpqa/
We prune the ranked features by the score, accounting for at least 60% of the feature set. This is due to the fact that further pruning of features leads to drastic degradation in accuracy. Further elimination of features from the training model leads to the situation when some testing sentences are represented with one or two features only. The pruning of the feature set up to 60% and 80% of top ranked features did not ameliorate the accuracy of the KL score and the SVM model. In the next section, we discuss possible reasons for degradation in accuracy when pruning the feature set for the task of SA classification on a sentence level, and propose a simple additive classification scheme. 6
In text classification, after calculating the scores between every feature and every category the next steps are to sort the features by score, choose the best k features and use them later to train the classifier. For the task of sentiment classification on a sentence-based level the pruning of the feature set may lead to the elimination of infrequent features (several occurrences) and may cause the loss of important information needed for classification of the new instances. Here are some differences in the aspects of use of the feature selection measures in text classification and opinion/sentiment analysis contexts. First, the aim in topic text classification is to look for the set of topic-specific features that describe the classification category. In sentiment classification, though, the markers of the opinion could be carried by both topic-specific and context words that may also have small differences in distributions across categories due to the short text length. If we look at the opinion review domain, the topic-specific features would be movie, film, flick and context words would be (long, short, horror, satisfy, give up).
Second, the usual text classification methods are designed for documents
consisting of at least several hundreds of words, assuming that the features that
could aid in classification repeat across the text several times. The format of a
sentence does not let us make the same assumption. The opinion or sentiment
polarity can be expressed with the help of one word/feature. There is substantial
evidence from several studies that the presence/absence of a feature is a better
indicator than the tf scores [
Thus, for effective classification, the model should identify features that are strong indicators of opinion/sentiment, take into account the relations between the features in each category, and be able to adjust scores of the features that were not frequent enough in order to expand the set of features that are strong indicators of the sentiment.
As a classification model we use a simple additive score of the features in the sentence computed for each category. Our aim is to determine the behavior of the KL score for the task of sentence sentiment and opinion classification in terms of its goodness and priority in feature weighting based on feature distribution across classification categories.
From the results presented in the Table 2, we can see that a simple classification scheme based on computing the sum of the feature scores according to the classification category outperforms the SVM model on the sentence datasets. As we deal with a small number of features, it is advantageous to use all of them when taking a classification decision. Comparing with the results in Table 1, we have achieved an improvement in accuracy for the Polarity and Subjectivity datasets. Nevertheless, the SVM model gives better results for the MPQA corpus. This may be due to the stylistics and opinion annotation and expression differences in movie and newspaper domains. The former is usually much more expressive, containing more sentiment-related words, than the latter. 7
In this article we suggest a new adaptation of the Kullback-Leibler divergence score as a weighting measure for sentiment and opinion classification. The proposed score, named KL score, we use for feature weighting with the SVM model. The experiments showed that the pruning of the feature set does not improve the SVM performance. Taking into account the differences in topical and sentiment classification of short text, we proposed a simple classification scheme based on calculation of sum of the features present in the sentence according to each classification category. Surprisingly, this scheme yields better results than SVM.
Based on the three well-known test-collections in the domain (Sentence Polarity, Subjectivity and MPQA datasets), we suggested a new way of computing feature weights, that could be later used with SVM or other supervised classification schemes that use feature weight computation. The proposed score and classification model were successfully applied in two different contexts (sentiment and opinion) and two domains (movie review and news articles).