In this paper we present experiments for global polarity classi cation task of Spanish tweets for TASS 2015 challenge. In our methodology, tweets representation is focused on linguistic and polarity features such as lemmatized words, lter of content words, rules of negation, among others. In addition, di erent transformations are used (LDA, LSI, and TF-IDF) and combined with a SVM classi er. The results show that LSI and TF-IDF representations improve the performance of the SVM classi er applied.
In last years the production of textual
documents in social media has increased
exponentially. This ever-growing amount of
available information promotes the research
and business activities around opinion
mining and sentiment analysis areas. In
social media, people share their opinions about
events, other people and organizations. This
is the main reason why text mining is
becoming an important research topic. Automatic
sentiment analysis in text is one of most
important task in text mining. The task of
sentiment classi cation determines if one
document has positive, negative or neutral
opinion or any level of each of them.
Determining whether a text document has a positive
or negative opinion is turning to an essential
tool for both public and private companies
Several methodologies to classify tweets from Task 1, Sentiment Analysis at global level of TASS workshop 2015, are presented in this work. This task is to perform an automatic sentiment classi cation to determine the global polarity (six polarity levels P, P+, NEU, N, N+ and NONE) of each tweet in the provided dataset. With this purpose, several solutions have been proposed in this work.
The paper is organized as follows, a brief overview of related works is shown in Section 2, the proposed methodology is describe in Section 3. Section 4 shows the experimental results and analysis, and nally, Section 5 concludes. 2
Nowadays, several methods have been proposed in the community of opinion mining and sentiment analysis. Most of these works employ twitter as a principal input of data and they aimed to classify entire documents as overall positive or negative polarity levels (sentiment) or rating scores (i.e. 1 to 5 stars).
Such is a case of work presented in
In
Another method to sentiment extraction
and classi cation on unstructured text is
proposed in
In
An approach of multi-label sentiment
classi cation is proposed in
In summary, most of works analyzed classify the documents mainly in three polarities: positive, neutral and negative. Moreover, most of works use social media (mainly Twitter) as analyzed documents. In this work, several methods to classify sentiment in tweets are described. These methods were implemented, according with TASS workshop speci cations, with the purpose of classify tweets in six polarity levels: P+, P, Neutral, N+, N and None. The proposed method are based on several standard techniques as LDA (Latent Dirichlet Allocation), LSI (Latent Semantic Indexing), TF-IDF matrix in tom): combination with the well-known SVM classi er. 3
In this section the proposed solution is detailed. First, a preprocessing step was carried out, later a Pseudo-phonetic transformation was done and nally the generation of Q-gram expansion was employed. 3.1
Preprocessing focuses on the task of
nding a good representation for tweets. Since
tweets are full of slang and misspellings, we
normalize the text using procedures such as
error correction, usage of special tags, part
of speech (POS) tagging, and negation
processing. Error correction consists on
reducing words/tokens with invalid duplicate
vowels and consonants to valid/standard
Spanish words (ruidoooo ! ruido; jajajaaa ! ja;
jijijji ! ja). Error correction uses an
approach based on a Spanish dictionary,
statistical model for common double letters, and
heuristic rules for common interjections. In
the case of the usage of special tags, twitter's
users (i.e., @user) and urls are removed
using regular expressions; in addition, we
classify 512 popular emoticons into four classes
(P, N, NEU, NONE), which are replaced by a
polarity tag in the text, e.g., positive
emoticons such as :), :D are replaced by POS,
and negative emoticons such as :(, :S are
replaced by NEG. In the POS-tagging step, all
words are tagged and lemmatized using the
Freeling tool for Spanish language
With the purpose of reducing typos and slangs we applied a semi-phonetic transformation. First, we applied the following transformations (with precedence from top to botcsjxc ! x
qu ! k guejge ! je guijgi ! ji shjch ! x ll ! y z ! s h ! c[ajoju] ! k c[eji] ! s w ! u v ! b ! !
In our transformation notation, square
brackets do not consume symbols and ;
means for any valid symbols. The idea is
not to produce a pure phonetic
transformation as in Soundex
Along with the placing bag of words representation (of the normalized text) we added the 4 and 5 gram of characters of the normalized text. Blank spaces were normalized and taken into account to the q-gram expansion; so, some q-grams will be over more than one word. In addition of these previous steps, several transformations (LSI, LDA and TFIDF matrix) were conducted to generate several data models for testing phase. 4
The classi er submitted to the competition was selected using the following procedure.
The 7; 218 tweets with 6 polarity levels were split in two sets. Firstly, the tweets provided were shu ed and then the rst set, hereafter the training set, was created with the rst 6; 496 tweets (approximately 90% of dataset), and, the second set, hereafter the validation set, was composed by the rest 722 tweets (approximately 10% of dataset). The training set was used to t a Support Vector Machine (SVM) using a linear kernel2 with C = 1, weights inversely proportional to the class frequencies, and using one vs rest multiclass strategy. The validation set was used to select the best classi er using as performance the score F1.
The rst step was to model the data
using di erent transformations, namely Latent
Dirichlet Allocation (LDA) using an online
learning proposed by
Table 1 complements the information presented on Figure 1.
The table presents the score F1 per polarity and the average (Macro-F1) for di erent con gurations. The table is divided in ve blocks, the rst and second correspond to a SVM with LSI (400 topics) and TF-IDF, respectively. It is observed that TF-IDF outperformed LSI; within LSI and TF-IDF it can be seen that 5-gram and 4-gram got the best performance in LSI and TF-IDF, respectively.
The third row block presents the performance when the features are a direct addition of LSI and TF-IDF; here it is observed that the best performance is with 4-gram furthermore it had the best overall performance in N+. The forth row block complements the previous results by presenting the best performance of LSI and TF-IDF, that is, LSI with 5-gram and TF-IDF with 4-gram. It is observed that this con guration has the best overall performance in P+, N, None and average (Macro-F1). Finally, the last row block gives an indicated of whether the phonetic transformation is making any improvement. The conclusion is that the phonetic transformation is making a di erence; however, more experiments are needed in order to know whether this di erence is statistically signi cant.
Based on the score F1 presented on Table 1 the classi er submitted to the competition is a SVM with a direct addition of LSI using 400 topics and 4-gram and LDA with 5-gram.
This classi er is identi ed as INGEOTECM14 in the competition. The SVM, LSI and LDA were trained with the 7218 tweets and then this instance was used to predict the 6 polarity levels of the competition tweets.
This procedure was replicated for the 4 polarity levels competition.
Table 4 presents the accuracy, average recall, precision, and F1 of INGEOTEC-M1 run using the validation set created, a 10fold crossvalidation on the 7218 tweets and the 1k tweets evaluated by the system's competition. This performance was on the 5 polarity levels challenge. It is observed from the table that the 10-fold crossvalidation gives a much better estimation of the performance
4We also submitted another classi er identi ed as INGEOTEC-E1; however, the algorithm presented a bug that could not be nd out on time for the competition. P P+ N N+ Neutral None
SVM + LSI
Text 0:238 0:549 0:403 0:348 0:025 0:492 4-gram 0:246 0:543 0:404 0:333 0:048 0:533 5-gram 0:246 0:552 0:462 0:356 0:000 0:575
SVM + TF-IDF
Text 0:271 0:574 0:414 0:407 0:103 0:511 4-gram 0:290 0:577 0:477 0:393 0:130 0:589 5-gram 0:302 0:577 0:476 0:379 0:040 0:586
SVM + fLSI + TF-IDFg 4-gram 0:297 0:578 0:471 0:421 0:142 0:578 5-gram 0:307 0:567 0:474 0:391 0:040 0:579
SVM + fLSI with 4-gram + TF-IDF with 5-gramg 4-5-gram 0:282 0:596 0:481 0:407 0:144 0:595
SVM + fLSI + TF-IDF without phonetic transformationg 4-5-gram 0:324 0:577 0:459 0:395 0:150 0:593 0:343 0:351 0:365 of the classi er when tested on 1k tweets of the competition (90% of training and 10% of validation).
In summary, in this work the best result reached was a 0.404 of F1. This result was achieved with a combination of LSI with 4gram + TF-IDF with 5-gram, using a SVM classi er (one-vs-one approach).
Val. 10-fold Comp.
Acc. 0:471 0:443 0:431
Recall 0:428 0:397 0:411
Precision 0:421 0:395 0:398
F1 0:417 0:393 0:404 In this contribution, we presented the approach used to tackle the polarity classi cation task of Spanish tweets of TASS 2015.
From the results, it is observed that a combination of di erent data models, in this case LSI and TF-IDF, improves the performance of a SVM classi er. It also noted that the phonetic transformation makes an improvement; however, more experiments are needed to know whether this improvement is statistically signi cant. As a result, we obtained a 0.404 of F1 (macro-F1) in sentiment classi cation task at ve levels, with the proposed solution. This proposed solution uses a combination of LSI with 4-gram + TF-IDF with 5-gram, and a SVM classi er (one-vs-one approach).
This research is partially supported by the Catedras CONACyT project. Furthermore, the authors would like to thank CONACyT for supporting this work through the project 247356 (PN2014). in neural information processing systems, paginas 856{864.