This paper presents a set of experiments to address the global polarity classification task of Spanish Tweets of TASS 2015. In this work, we compare the main supervised classification algorithms for Sentiment Analysis: Support Vector Machines, Naive Bayes, Maximum Entropy and Decision Trees. We propose to improve the performance of these classifiers using a class reduction technique and then a voting algorithm called Naive Voting. Results show that our proposal outperforms the other machine learning methods proposed in this work.
Sentiment analysis is the computational study of opinions about entities, events, people, etc.
Opinions are important because they often are taken into account in decision process.
Currently, people use different social networks
to express their experiences with products or
commercial services. Twitter is one of the
biggest repositories of opinions and it is also
used as a communication channel between
companies and customers. The data generated
in Twitter is important for companies, because
- with that information -- they could know what
is been saying about their products, services
and competitors. In recent years, several
researches of NLP have developed different
methods to address the sentiment analysis
problem in Twitter. The vast majority of works
aim to classify a comment, according to the
polarity expressed, in three categories: positive,
negative or neutral
The supervised classification algorithms are the most used methods to classify comments or opinions.
In this paper, we present a comparison of some supervised learning methods which have achieved good results in other research works.
Analyzing the errors of those methods, we propose to use a class reduction technique and a voting algorithm (which take into account the results of supervised classifiers) to improve the classification of opinions in Twitter.
The rest of the paper is organized as follows: Section 2 summarizes the main works in Publicado en http://ceur-ws.org/Vol-1397/. CEUR-WS.org es una publicación en serie con ISSN reconocido sentiment analysis. Section 3 describes our proposal and in Section 4 we describe the results that we have gotten. Finally, in Section 5, the conclusions of this work are presented. 2
There are two general approaches to classify
comments or opinions in positive, negative or
neutral: supervised and unsupervised
algorithms. Supervised classification algorithms
are used in problems which are known a priori
the number of classes and representative
members of each class. The unsupervised
classification algorithms, unlike supervised
classification, do not have a training set, and
they use clustering algorithms to try to create
clusters or groups
The sentiment classification task could be formulated as a supervised learning problem with three classes: positive, negative and neutral. The most used supervised techniques in sentiment analysis are Naive Bayes (NB), Support Vector Machines (SVM), Maximum Entropy, etc. In most cases, SVM have shown great improvement over Naive Bayes.
Cui, Mittal, and Datar (2006) affirm that
SVM are more appropriate for sentiment
classification than generative models, because
they can better differentiate mixed feelings.
However, when the training data is small, a
Naive Bayes classifier could be more
appropriate. One of the earliest researches on
supervised algorithms which classify opinions
is presented in
Most researches are focused for the English
language, since it is the predominant language
on the Internet. There are less works of
sentiment analysis in Spanish opinions;
however, Spanish is playing an important role.
For Spanish comments,
This paper has two major objectives: First, we make a comparison of some of the main algorithms of supervised classification for Sentiment Analysis: Support Vector Machines, Naive Bayes, Maximum Entropy and Decision Trees. The second goal is to use a class reduction technique and then a voting algorithm to improve the accuracy of final results. The architecture of our system can be seen in Figure 1. 3.1
In this paper we compare some classification methods in order to determine the performance of these algorithms in a set of opinions written by Spanish users. For the experiments, we used the four supervised classifiers described previously. The comparison of methods has the Training and Classification Phase. These phases will be explained below.
For each supervised classification methods used in this work, we identified three steps in the training phase: comment preprocessing, vectorization and learning.
Preprocessing: To make a correct comment preprocessing, we apply the following techniques:
Elimination of symbols and special characters. Elimination of articles, adverbs, pronouns and prepositions (stopwords). Processing of hashtags. Correction of words with repeated letters. Filtration of words with ``@" symbol as initial letter. Elimination of the characters ``RT". URLs removal.
Stemming of comments (opinions).
Vectorization: Each comment in the
training data must be represented
mathematically. There are different
mathematical models to represent information.
The most popular models are: boolean model,
term frequency (TF), term frequency-inverse
document frequency (TF-IDF) and Latent
Semantic Analysis (LSA)
Learning: In this step, the classification algorithm receives as parameters the representative vectors of comments with their class labels. The class labels are: positive (P), negative (N), neutral (NEU) and none (NONE).
A classifier is a function that gives a discrete
output, often denoted as class, to a particular
input
In the first evaluation of the machine learning methods, the obtained accuracy results were slightly lower. For this reason, we propose to use two techniques to improve the results of classifiers. The first technique, called class reduction, removes one class label (NEU or NONE) with the aim of improving the margin of error of classifiers and reducing the number of classes to evaluate. The second technique, called naive voting, receives as input parameters the optimized classifiers of the first technique. A more specific description of these techniques will be explained below.
The basic idea of this technique was proposed
in
We decided to train the classifiers considering three classes: Positive-Negative-Neutral and Positive-Negative-None. The classifiers were trained and tested in this way. The new results of classifiers using that simplification were better. Due to the improvement, we decided to join the partial results of these classifiers. With this union, we could classify the comments considering the four classes defined initially.
merge the partial results into single result. We
established a set of rules to address the union of
partial results of this class reduction technique,
this rules are shown in Table 1.
Our final technique presented in this paper was
Voting System. We choose this method because
all classifiers have a margin of error. Due to this
margin of error, classifiers could classify
incorrectly a comment. A voting system tries to
reduce this margin of error. Voting systems are
based on different classification methods. Many
studies have used voting system to classify text.
Naive Voting, Weighted Voting, Maximun Choice Voting and F-Score/recall/precision Voting.
We proposed the Naive Voting technique, which has as input parameters the four classifiers proposed in this paper. Naive Voting is one of the simplest voting algorithms. In this technique, the comment is classified according to the majority agreement, i.e., the class with most votes in each classifier will be the winning class. The rules we applied for Naive Voting are described in Table 2.
Rule 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
P 4 3 2 0 0/1 0/1 0 0/1 0/1 0 2 2 2 0 0 0 2 1
N NEU 0 0 0/1 0/1 0 4 3 2 0 3 2 0 0/1 0/1 2 0 0 2 2 0 0 1 4 0 0 2 0 2 0 2 2 1
NONE
0 0/1 0/1 0/1 0/1
Each row of the Table 2 shows the votes obtained by each of the polarities (P-N-NEUNONE) according to the output of the proposed classifiers. Due to we have 4 classifiers, the largest vote is 4 and the minimum is 0. Then, the class with the highest vote will be the winning class. In the event of a tie, a set of rules were established to determine the winning class. For example, in the case of a draw at 2 between positive and negative classes, a lottery was established to determine the winning. In other cases of a tie, it was chosen the NEU class or NONE class as the winner. 4 4.1
We used the corpora provided by the
organization of TASS 2015. For our purposes,
we used the General Corpus and the Balanced
General Corpus. The first one is composed of
training and test set which contains 7219 and
60798 tweets, respectively. The Balanced
General Corpus is a test subset which contains
1000 tweets only for test. A complete
description of these corpora is explained in
We performed a series of tests to address the Task 1 of TASS 2015, focusing on finding the global polarity of the Tweets corpora for 4 class labels (P-N-NEU-NONE). A general description of the ''RUNs'' that we have made for TASS 2015 are described in Table 3. Tech.
SVM NB ME DT NB II ME II DT II
Run-Id 60798 UCSP-RUN-2 TestNB60000 UCP-RUN-2
ME TestDT60000 UCSP-RUN
1-NB UCSP-RUN
1-ME UCSP-RUN
1-DT SVM II
UCSP-RUN-1
Run-Id 1000 UCSPRUN-2 UCSPRUN-2
NB TestME10
00 TestDT100
00 UCSPRUN-1 UCSPRUN-1
NB UCSPRUN-1
ME UCSPRUN-1
DR UCSPRUN-3
Description
Support Vector
Machine Naive Bayes Max. Entropy Decision Tree SVM + Class
Reduction NB + Class Reduction ME + Class Reduction DT + Class
Reduction Naive Voting
Class reduction techniques improve results because they allow the classifier having to decide between fewer options and then the classifier could reduce its margin of error.
The voting algorithm gives good results because it takes into account the decisions of all the classifiers. This algorithm tries to reach a single decision that might be the best. A voting algorithm is like a consensus between all classifiers. But it is important to take into account that any voting algorithm is good as long as the majority of voters (classifiers) are good, otherwise, the voting algorithm will not have the expected results.
The results we have gotten for the evaluation of our proposal are shown in Table 4 (Evaluation of full test corpus) and Table 5 (Evaluation of 1k test corpus). It can be seen that class reduction techniques and our voting algorithm improve the accuracy of the original supervised classification algorithms.
Table 5: Results of evaluating the 1k-Test
Corpus 5
One of the main goals of this paper was to evaluate some supervised classification algorithms in the task of sentiment analysis. The results of evaluating the classifiers in initials experiments were not satisfactory. Using an optimization stage (class reduction and voting systems), accuracy improved slightly compared to the original techniques. It could be shown that adequate voting algorithms improve the accuracy of classifiers. For proper operation of a voting system it is required to have multiple classifiers with a relatively high rate of efficiency. If a classifier fails, the other could give the correct prediction. But if most of classifiers give low results, then the voting system does not ensure a correct performance.
The research leading to the results has been founded by Programa Nacional de Innovación para la Competitividad y Productividad (Innóvate Perú)