English. In this paper, we describe our neural network models for a commercial application on sentiment analysis. Different from academic work, which is oriented towards complex networks for achieving a marginal improvement, real scenarios require flexible and efficient neural models. The possibility to use the same models on different domains and languages plays an important role in the selection of the most appropriate architecture. We found that a small modification of the state-of-theart network according to academic benchmarks led to a flexible neural model that also preserves high accuracy.
Italiano. In questo lavoro, descriviamo i nostri modelli di reti neurali per un’applicazione commerciale basata sul sentiment analysis. A differenza del mondo accademico, dove la ricerca e` orientata verso reti anche complesse per il raggiungimento di un miglioramento marginale, gli scenari di utilizzo reali richiedono modelli neurali flessibili, efficienti e semplici. La possibilita´ di utilizzare gli stessi modelli per domini e linguaggi variegati svolge un ruolo importante nella scelta dell’architettura. Abbiamo scoperto che una piccola modifica della rete allo stato dell’arte rispetto ai benchmarks accademici produce un modello neurale flessibile che preserva anche un’elevata precisione.
In recent years, Sentiment Analysis (SA) in
Twitter has been widely studied. Its popularity has
been fed by the remarkable interest of the
industrial world on this topic as well as the relatively
easy access to data, which, among other, allowed
the academic world to promote evaluation
campaigns, e.g.,
In this paper, we report on our experience on adopting academic models of SA to a commercial application. This is a social media and microblogging monitoring platform to analyze brand reputation, competition, the voice of the customer and customer experience. More in detail, sentiment analysis algorithms register customers’ opinions and feedbacks on services and products, both direct and indirect.
An important aspect is that such clients push for easily adaptable and reliable solutions. Indeed, multi-tenant applications and sentiment analysis requirements cause a high variability of the approaches to the tasks within the same platform. This should be capable of managing multi-domain and multi-channel content in different languages as it provides services for several clients in different market segments. Moreover, scalability and lightweight use of computational resources preserving accuracy is also an important aspect. Finally, dealing with different client domains and data potentially requires constantly training new models with limited time availability.
To meet the above requirements we started from
1Key Performance Indicators are strategic factors
enabling the performance measurement of a process or activity.
the state-of-the-art model proposed in
In the following, Section 2 places the current work in the literature. Section 3 introduces the application scenario. Sections 4 and 5 presents respectively our proposal for a flexible architecture and the experimental results. Finally, Section 6 reports the conclusions. 2
Although sentiment analysis has been around for
one decade, a clear and exact comparison of
models has been achieved thanks to the organization
of international evaluation campaigns. The main
campaign for SA in Twitter in English is SemEval,
which has been organized since 2013. A similar
campaign in the Italian language (SENTIPOLC)
Among other approaches, Neural Networks
(NNs), and in particular CNNs, outperformed the
previous state of the art techniques
Our commercial application is a social media and micro-blogging monitoring platform, which is used to analyze brand reputation, competitors, the voice of the customer and customer experience. It is capable of managing multi-domain and multichannel content in different languages and it is provided as a service for several clients on different market segments.
The application uses an SA algorithm to analyze the customers’ opinions and feedbacks on services and products, both direct and indirect. The sentiment metric is used by the application clients to point out customer experience, expectations, and perception. The final aim is to promptly react and identify improvement opportunities and, afterward, measure the impact of the adopted initiatives. 3.1
Industrial applications, used by demanding clients, and dealing with real data tend to prefer easily adaptable and reliable solutions. Major problems are related to multi-tenant applications with several client requirements on the sentiment analysis problem, often requiring variations on task approaches within the same platform. Moreover, high attention is put on scalability and lightweight use of computational resources, preserving accurate performance. Finally, dealing with different client domains and data potentially requires constantly training new models with limited time availability. 3.2
The commercial social media and micro-blogging monitoring platform continuously acquires data coming from several sources; among these, we selected Twitter data as the main source for our purposes.
First, the public Twitter stream was collected for several months without specific domain restriction to build the dataset used for the word embedding training. The total amount of tweets used accounts for 100 million Italian tweets and 50 million English tweets.
Then, a dataset has been constructed from a specific market sector in Italian. The data collection was performed on the public Twitter stream with specific word restriction performed in order to filter the tweets of interest on the automotive domain. Afterward, the commercial platform applies different techniques in order to exclude from these collections the tweets that are not relevant for the specific insight analysis.
The messages were then used to construct the dataset for our experiments. A manual annotation phase has been performed together with the demanding client in order to best suit the insight objective requirement. Even though structured guidelines were agreed upon before creating the dataset and continuously checked against, this approach tended to generate dataset characteristics: in particular, unbalanced distribution of the examples over the different classes has been measured. It makes necessary a flexible model capable of handling such phenomena without the need of costly tuning phases and/or network reengineering. 4
The task of SA in Twitter aims at
classifying a tweet t 2 T into one of the three
sentiment classes c 2 C, where C =
fpositive; neutral; negativeg. This can be
achieved by learning function f : T ! C through
a neural network. The architecture here proposed
is based on
In contrast to Severyn and Moschitti (2015), we adopted a Recurrent Pooling layer that allows the network to learn longer dependencies in the input sentence (i.e. sentiment shifts). This architectural change makes the network less sensible to learn biases from the dataset and therefore generalize better on poorly represented classes.
Embedding: a tweet t is represented as a sequence of words fw1; ::; wj ; ::; wN g. Tweets are encoded into a sentence matrix t 2 Rd jtj, obtained by concatenating its word vectors wj , where d is the size of the word embeddings. Sentence Encoder: it is a function that maps the sentence matrix t into a fixed size vector x representing the whole sentence. Severyn and Moschitti (2015) used a convolutional layer followed by a global max-pooling layer to encode tweets. The convolution operation applies a sliding window operation (with window of size m) over the input sentence matrix. More specifically, it applies a non-linear transformation generating an output matrix ~x 2 RN dconv where dconv is the number of convolutional filters and N is the length of the sentence. The max-pooling operation applies an element-wise max operation to the transformed sentence matrix ~x, resulting in a fixed size vector representing the whole sentence.
In this work, we propose to substitute the
maxpooling operation with a Bidirectional Gated
Recurrent Unit (BiGRU)
Similarly to Severyn and Moschitti (2015), for the CNN, we use a convolutional operation of size 5 and dconv = 128 with rectified linear unit activation, ReLU. For the BiGRU, we use 150 hidden units for both GRU and GR!U obtaining a fixed size vector of size 300.
Word embeddings: for all the proposed models, we pre-initialize the word embedding matrices with the standard skip-gram embedding of dimensionality 50 trained on tweets retrieved from the Twitter Stream.
Training: the network is trained using SGD
with shuffled mini-batches using the Adam
update rule
Datasets: we trained and evaluated our
architecture on two datasets: the English dataset of
Semeval 2015
The Italian dataset was built in-house for the automotive domain: we collected from the Twitter stream as explained in Section 3.2 and divided it into three different splits for training, validation and testing, respectively. Table 2 shows the size of the splits. Due to the nature of the domain, many tweets in the dataset are neutral or objective, this makes the label distribution much different from the usual benchmarks. For example, the neutral class is the least represented in the English dataset (see Table 1) and the most represented in the Italian data. The imbalance can potentially bias neural networks towards the most represented class. One of the features our approach is to diminish such effect.
Evaluation metrics: we used the following evaluation metrics, Macro-F1 (the average of the F1 over the three sentiment categories). Additionally, we report the F 1p;n, which is the average F1 of the positive and negative class. This metric is the official evaluation score of the SemEval competition. 5.2
Table 3 presents the results on the English dataset of SemEval 2015. The first row shows the outcome reported by Severyn and Moschitti (2015) (S&M). CNN+Max is a reimplementation of the above system with Convolution and Max-Pooling but trained just on the official training data without distant supervision. This system is used as a strong baseline in all our experiments. Lastly, we report the results obtained with the BiGRU pooling strategy described in Section 4. The proposed architecture presents a slight improvement over the strong baseline ( 1 point of both F 1 and F 1p;n score on the test). 5.3
Table 4 presents the result on the Italian dataset. Despite that on this dataset the proposed CNN+BiGRU model obtains lower F1 scores, it shows improved performance in terms of F 1p;n (5 points on both validation and test sets). This suggests that the proposed model tends to generalize better on the less represented classes, which, in the case of the Italian training dataset, are the positive and negative classes (as pointed out in Table 2). 5.4
We analyzed the classification scores of some words to show that our approach is less affected by the skewed distribution of the dataset. The sentiment trends, as captured by the neural network in terms of scores, are shown in Table 5.4). For example, the word Mexico classified by CNN+Max produces the scores, 0:06, 0:35, 0:57, while CNN+BiGRU outcome, 0:18, 0:52, 0:30, for the negative, neutral and positive classes, respectively. This shows that CNN+BiGRU is less biased by the data distribution of the sampled word in the dataset, which is, 0, 1, 5, i.e., Mexico appears 5 times more in positive than in neutral messages and never in negative messages.
This skewed distribution biased more CNN+Max as the positive class gets 0:57 while the negative one only 0.06. CNN+BiGRU is able, instead, to recover the correct neutral class. We believe that CNN+Max is more influenced by the distribution bias as the max pooling operation seems to capture very local phenomena. In contrast, BiGRU exploits the entire word sequence and thus can better capture larger informative context.
A similar analysis in Italian shows the same trends. For example, the word panda is classified as, 0:05, 0:28, 0:66, by CNN+Max and 0:07, 0:56, 0:35 by CNN+BiGRU, for negative, neutral and positive classes, respectively. Again, the distribution in the Italian training set of this word is very skewed towards the positive class: it confirms that CNN+Max is more influenced by the distribution bias, while our architecture can better deal with it. 6
In this paper, we have studied state-of-the-art neural networks for the Sentiment Analysis of Twitter text associated with a real application scenario. We modified the network architecture by applying a recurrent pooling layer enabling the learning of longer dependencies between words in tweets. The recurrent pooling layer makes the network more robust to unbalanced data distribution. We have tested our models on the academic benchmark and most importantly on our data derived from a real-world commercial application. The results show that our approach works well for both English and Italian languages. Finally, we observed that our network suffers less from the dataset distribution bias.