This paper details LTG-Oslo team's participation in the sentiment track of the NEGES 2019 evaluation campaign (We make the code available at https://github.com/jbarnesspain/neges_2019.). We participated in the task with a hierarchical multi-task network, which used shared lower-layers in a deep BiLSTM to predict negation, while the higher layers were dedicated to predicting document-level sentiment. The multi-task component shows promise as a way to incorporate information on negation into deep neural sentiment classi ers, despite the fact that the absolute results on the test set were relatively low for a binary classi cation task.
Sentiment analysis has improved greatly over the last decade, moving from models
trained on hand-engineered features [
Explicit negation detection has proven useful to create features for
lexiconbased sentiment models [
More recent approaches to sentiment, however, have concentrated on
learning the e ects of negation in an end-to-end fashion. Current state-of-the-art
approaches employ neural networks which implicitly learn to resolve negation, by
either directly training on sentiment annotated data [
In this paper, we propose a multi-task learning approach to explicitly
incorporate negation annotated data into a neural sentiment model. We show that
this approach improves the nal result on the 2019 NEGES shared task [
In this section, we brie y review previous work that is relevant to (i ) attempts to use negation information in sentiment analysis, (ii ) research on negation detection as a separate task, and (iii ) multi-task learning. 2.1
Negation is a pervasive linguistic phenomenon which has a direct e ect on the
sentiment of a text [
Example 1.
El hotel esta situado en la puerta de toledo, no esta lejos del centro.
The English translation is \The hotel is located at the puerta de toledo, it is not far from the center." A sentiment classi cation model must be able to identify the relevant sentiment words (in this case \lejos del centro"), negation cues (\no"), and resolve the scope in order to correctly predict that this sentence expresses negative polarity. Intuitively, a sentiment model that has access to negation scope information should perform better than a non-informed version.
The rst approaches to detecting negation scope for sentiment used heuristics,
such as assuming all tokens between a negation cue and the next punctuation
mark are in scope [
Later research showed that using machine-learning techniques to detect the
scope of negation could improve both lexicon-based [
Approaches to negation analysis often decompose the task into two sub-tasks, performing (i) negation cue detection, followed by (ii) scope detection.
Much work was done within the biomedical domain [
Traditional approaches to the task of negation detection have typically
employed a wide range of hand-crafted features describing a number of both lexical,
morphosyntactic and even semantic properties of the text [
Multi-task learning (MTL) is an approach to machine learning where a single
model is trained simultaneously on two tasks. By restricting the search space of
possible representations to those that are predictive for both tasks, we attempt
to give the model a useful inductive bias [
Hard parameter sharing [
In this work, we implement a multi-task learning where the lower layers of a deep neural network are shared for the main and auxiliary tasks (in our case sentiment classi cation and negation detection, respectively), while higher layers are allowed to adapt to the main task. 3
We propose a hierarchical multi-task model (see Figure 1) which relies on a BiLSTM to create a representation for each sentence in a document, and a second BiLSTM to aggregate these sentence representations into a full document representation. In this section, we rst describe the negation submodel, then the sentiment submodel, and nally the multi-task model. 3.1
In previous work on negation detection, it is common to model negation scope as a two step process, where rst the negation cues are identi ed, and then negation scope is determined. However, we hypothesize that within a multi-task
Sentiment Classification
Softmax Max Pooling
BiLSTM Max Pooling
Max Pooling
Max Pooling BiLSTM
BiLSTM
BiLSTM
Embedding Layer Sentence 1
Sentence 2
Sentence 3
Negation Scope Detection
CRF Tagger framework, it is more bene cial for a network to learn to both identify cues and resolve scope jointly. Therefore, we model negation as a sequence labeling task with BIO tags. In the cases where there are more than one negation scope in a sentence that overlap, we atten these multiple representations, as shown in Figure 2. The negation model, therefore, attempts to identify all cues and all scopes in a sentence at the same time. Note that scopes can also begin before the negation cue and also be discontinuous. While this is an oversimpli cation of the full negation scope task, we argue that in order to classify sentiment, it is enough for a model to know which tokens are negated.
The negation model is comprised of an embedding layer which embeds the tokens for each sentence. The embeddings pass to a bidirectional Long Short-Term Memory module (BiLSTM), which creates contextualized representations of each word. A linear chain conditional random eld (CRF) uses the output of the BiLSTM layer as features. We use Viterbi decoding and minimize the negative log likelihood of CRF predictions.
Es negation labels: O
BIO labels: O que no
O CUE1 O B_cue nos ayudó , N1 N1 O B_neg I_neg O y luego ni siquera llamó O O CUE2 CUE2 N2 O O B_cue I_cue B_neg Fig. 2: An example of the negation which has been converted to BIO labels. Although the example here shows two negation structures where the cue is at the beginning of the scope, there are also examples where the scope begins before the cue or is discontinuous. 3.2
As mentioned above, the sentiment model uses a hierarchical approach. For each sentence in a document, we rst extract features with a BiLSTM. We take the max of the BiLSTM output as a representation for the sentence. This is then passed to a second BiLSTM layer, after which we again take the max. We use a softmax layer to compute the sentiment predictions for each document and minimize the cross entropy loss. As a baseline, we train a single-task sentiment model (STL) on the available sentiment data. 3.3
For the hierarchical multi-task model (MTL), we train both tasks simultaneously by sequentially training the negation classi cation model for one full epoch and then training the sentiment model. We use Adam as an optimizer, and a dropout layer (0.3) after the embedding layer to regularize the model, as this is common for both the main and auxiliary tasks. 4
Given that neural models are sensitive to random initialization, we perform ve runs for each model on the development data with di erent random seeds and report both mean accuracy and standard deviation across the ve runs. As the nal submission required a single prediction for each document, we take a majority vote of the ve learned classi ers in order to provide an ensemble prediction.
Besides the proposed STL and MTL models, we also compare with a baseline (BOW) which uses an L2 regularized logistic regression classi er trained on a bagof-words representation of the documents. We choose the optimal C parameter on the development data. 4.1
The SFU ReviewSP-NEG dataset [
Previous work [
Given that the classi cation task is performed at document-level, it is often di cult to determine what exactly was the cause of a change in prediction from one model to another. Instead, Figure 3 shows a relative confusion matrix of the development results, where positive numbers (dark purple) indicate that the MTL 1 Note that we do not have access to the gold sentiment or negation labels on the test set, so we cannot perform multiple runs, but must rely on the organizers evaluation. model made more predictions in that square than the STL model and negative numbers (white) indicate fewer predictions. On the development data, the MTL model tends to help with the negative class, while adding little to the positive class. The number of negation structures per class (shown in Table 3) shows that there are more negation structures in documents labeled with negative sentiment in the development set, which seems to corroborate the idea that the MTL model is able to use negation information to improve the results on the negative class.
Predicted label l e b a l e u r T Pos 3 0 Pos In this paper, we have detailed our participation in the 2019 Neges shared task. Our approach, the hierarchical multi-task negation model, did not give a strong performance in absolute numbers on the test set (66% accuracy), but does indicate that multi-task learning is an appropriate framework for incorporating negation information into sentiment models, improving from 71.4 to 72.5 accuracy on the development set.
The hierarchical RNN model used in this participation is similar to strong
performing approaches at sentence-level. However, it is not clear that it is the
most adequate model for document-level classi cation. Convolutional neural
networks [
Additionally, the small training set size for the sentiment task (271 documents)
and number of domains (8) complicates the use of deep neural architectures.
Lexicon-based and linear machine-learning approaches have shown to perform
quite well under these circumstances [
In this work we have only explored using a sequence-labeling approach to
negation scope. It would be interesting to incorporate state-of-the-art negation
scope models [
Finally, the SFU ReviewSP-NEG dataset has several additional levels of annotation, i.e. if a negation structure changes the polarity of the tokens in scope or the nal polarity after negation. Future work should explore the use of this information further.
This work has been carried out as part of the SANT project, funded by the Research Council of Norway (grant number 270908).