In this paper, we present the approach proposed by INGEOTEC team for global polarity classi cation at tweet level task of TASS'17 contest. We use B4MSA algorithm, a proposed entropy-based term-weighting scheme and, EvoDAG as an ensemble.
The rapid growth of social media such as review websites, microblogging sites, social networks, etc. has made both researchers and entrepreneurs interested in the analysis of that amount of information to create applications like sentimental analysis and opinion mining.
Sentiment Analysis is used to analyze
people's feelings or beliefs expressed in texts such
as emotions, opinions, attitudes, etc.
Twitter is one of the most used social networking app, and as a result it has received a lot of attention. Twitter is considered as a huge and fast source of information (6; 000 tweets each second).1 Then, due to this important task, many international contests have been launched around the world in several languages. This is the case of the Spanish language handled in TASS (Taller de Analisis de Sentimientos en la SEPLN) contest. The
1https://www.brandwatch.com/blog/44-twitterstats-2016/ TASS workshop is an event of SEPLN Conference, which is a conference in Natural Language Processing for the Spanish language.
Our participation in this contest, is mainly based on Genetic Programming (GP), which is an evolutionary algorithm that can be used to solve many di erent types of problems. Recently, the incorporation of semantic knowledge in GP has improved its performance.
The use of e cient geometric semantic
crossover operators as
This paper describes the approach used in our participation in TASS'17 contest, and it is organized as follows, a brief overview of related works is shown in Section 2, the proposed methodology is described in Section 3.
Section 4 shows the experimental results and analysis, and nally, Section 5 concludes.
Copyright © 2017 by the paper's authors. Copying permitted for private and academic purposes. 2
As we know, several methods have been proposed in the community of opinion mining and sentiment analysis. A lot of these works employ Twitter as a primary source of data due to its easy and fast accessibility. Considering these advantages, most of the text polarity classi cation contests employed this source of data. Nevertheless, the way people write on Twitter provides a very complex task due to the tweets are full of slang and misspellings, new words are generated every day, etc. Therefore, Twitter is an easy source of acquisition, but it is very complex analyzing its content.
Tweet level and entity level polarity
classi cation was treated in
Our participation in TASS'17 is based on an
ensemble of SVM classi ers combined into a
non-linear model created with Genetic
Programming (GP). We used B4MSA
Furthermore, our approach uses two kinds of datasets; datasets labeled by human annotators provided by TASS contest, and also datasets generated by distant supervision approach.
Distant supervision has been used for
tasks such as information extraction
Roughly speaking, our approach uses two
layers. In the rst layer, a set of B4MSA
classi ers are trained with two kind of
datasets; datasets labeled by human
annotators: the InterTASS training set and the
Spanish dataset of
In the case of DS, around 18 million tweets are divided into chunks of 30K items (15K positive and 15K negative tweets). Each chunk produces a B4MSA model that predict a polarity level (from -1 to 1). To speed up the combination of partial results from DS, we rank the 1380 110063 B4MSA classi ers with the a nity between each example and the vocabulary known by each classi er. Therefore, we select only the k classi ers with largest vocabulary intersection. The optimal k (k = 30) was experimentally determined.
Finally, EvoDAG's inputs are the concatenation of all the decision functions predicted by individual B4MSA classi ers. The following subsections detail the parts of our approach. The precise con guration of our benchmarked system is described in Section 4. 3.1
B4MSA
B4MSA2
Figura 1: Prediction Scheme that are weakly linked to language dependent methods.
In B4MSA, the whole process is stated as a combinatorial optimization problem, where the set of con gurations is de ne by the possible solutions. In practice, nding the best text con guration for a particular problem has a high computational cost due to the large con guration space. However, a competitive solution can be found using hyperheuristics.
Each con guration is composed by a set of text transformation functions, tokenizers (ngrams of words, q-grams of characters, and skip-grams), and a term weighting scheme. All of these parts are optimized to maximize the performance of the desired task.
In addition, we enriched the standard text transformation functions of B4MSA with the complement of stemming, i.e., we selected nearly word's in ections identi ed by a Spanish stemmer; in this sense, the clues of words (in ections) stand for the original text. The TASS datasets is transformed, tokenized, and then, the fully B4MSA pipeline is followed. 3.2
Instead of the term weighting found in
B4MSA (TFIDF & TF), here we use the
entropy+b term-weigthing scheme, rstly
reported in
In entropy+b each term is represented by a distribution over the available classes. Instead of using the raw probabilities per class, we weight each term with the entropy+b function, de ned as follows: entropyb(w) = log jCj X pc(w; b) log c2C
1 pc(w; b) ; where C is the set of classes, and pc(w; b) is the probability of term w in class c parametrized with b. More detailed, pc(w; b) =
freqc(w) b jCj + Pc02C freqc0 (w) : Here, freqc denotes the frequency of the given term in the class c. The idea behind entropyb(w) is to weight each term using the entropy of the underlying distribution, that is, large entropy values (terms uniformly distributed along all classes) have a low weight while terms being skewed to some class are close to log jCj. The parameter b is introduced to absorb the possible noise that occurs in low populated terms. 3.3
EvoDAG3
The models evolved by EvoDAG have three distinct node's types; the inputs nodes, that, as expected, received the independent variables, the output node that corresponds to the label, and the inner nodes are the different numerical functions such as: sum, product, sin, cos, max, and min, among others. In order to provide an idea of the type of models being evolved, Figure 2 depicts a model evolved for the polarity classi cation at global level task.
Figura 2: An evolved model for the polarity classi cation at global level task
As it can be seen, the model is represented using a directed acyclic graph (DAG) where direction of the edges indicate the dependency, e.g., cos depends on X3, i.e., cosine function is applied to X3. There are three types of nodes; the inputs nodes are colored in red, the inner nodes are blue (the intensity is related to the distance to the height, the darker the closer), and the green node is the output node. As mentioned previously, EvoDAG uses as inputs the decision functions of B4MSA, then rst three inputs (i.e., X0, X1, and X2) correspond to the decision function values of the negative, neutral, none, and positive polarity of B4MSA model trained with InterTASS dataset, and the later two (i.e., X3 and X4) correspond to the decision function values of two B4MSA systems each one trained with our distant supervision dataset.
It is important to mention that EvoDAG does not have information regarding whether input Xi comes from a particular polarity decision function, consequently from EvoDAG point of view all inputs are equivalent. 4
Table 1 shows the data distribution of
training and test datasets used in our
experiments. For training step, we used the datasets
provided by the organizers (training set of
TASS'16 and training set of InterTASS'17),
extra data annotated by humans described
in
In case of distant supervision (DS) dataset, around 18 million tweets were selected 4The datasets are available at http://hdl. handle.net/11356/1054. from more than 500 million tweets collected along one year. The tweets were classi ed into two classes positive or negative based on the words of the a ective lexicons. Each tweet has no contradictions, i.e., the tweet has only positive or negative words, tweets with negative markers or some discourse markers (no, aunque, sin embargo) were avoided to ensure the class.
DataSet Positive Negative Neutral None Total train-TASS'16 2884 2182 670 1483 7219 train-TASS'17 473 635 202 201 1511 Extra-data 69,571 16,472 54,017 - 140,060 DS-dataset 9M 9M - - 18M Tabla 1: Statistics of Spanish training data.
Regarding the parameters used by B4MSA and EvoDAG, it is important to mention that these parameters where optimized using random search an a hill climbing technique (only used by B4MSA) in the parameter search space. Speci cally, we follow the instructions provided by these developments which consist in: rstly, optimize the algorithm's parameters using the training set, secondly, train the model with the parameters obtained in the previous step; and, nally, use the model to predict the data given. It is important to mention that B4MSA and EvoDAG can be used from command line and we decided to follow that path.
Table 2 shows our results on gold standard of TASS'17, namely, InterTASS, General Corpus 1K and General Corpus 60K. To solve the task with these three test sets we use two models, both using all datasets listed in Table 1. For the InterTASS subtask, EvoDAG creates a model that optimizes the combination of the internal classi ers. For this purpose, we use the provided train and validation partition, the rst one to train and the later to compute the objective function; more detailed, EvoDAG uses the geometric mean per-class of F1 scores as objective function. On the other hand, the Global corpus (both 1K and 60K) is optimized to maximize the geometric mean per-class of F1 scores, over TASS'16. 5
In this paper, we presented the system used to tackle the task of global polarity classi cation at tweet level, in Spanish. From the results, it is observed that our system, Corpus InterTASS General Corpus 1K General Corpus 60K
Tabla 2: Results of INGEOTEC participation in TASS'17 which combined B4MSA algorithm for text representation, a new entropy-based termweighting scheme, and EvoDAG as an ensemble reaches good performance achieving high positions in the three corpora provided in TASS'17.
This research is partially supported by the Catedras CONACyT project.