This paper describes the negation cue detection system presented by the NLP UNED group for Task A (Tarea A: Deteccio´n de claves de negacio´n) of the second edition of NegES workshop [9]. The task deals with negation cues detection in Spanish reviews in domains such as cars, music and books. The proposed system is an extension of the one proposed in the previous edition by the UNED team. This system consists of a deep learning architecture and the application of a set of rules. The deep learning architecture is based on the use of a Bi-LSTM to process contextual information. The purpose of applying a stack of rules is to correct frequent classification errors. The results obtained improve the performance achieved by our team in the previous edition and are highly competitive compared to the rest of participants, placing second in the global ranking of this edition.
The automatic processing of negation is a very important task in natural
language processing as it allows us to identify negated facts. It is a very interesting
field of study if we consider the influence of the negation in tasks such as
sentiment analysis and relationship extraction [
The proposal for Task A of the NegEs workshop is the same as that presented
in the previous edition [
This work is organized as follows: Section 2 contains both the description of the proposed system and the description of the features and resources used. In Section 3 we report and discuss the results obtained during the evaluation stage. Finally, in Section 4 conclusions and future work are presented. 2
The proposed system consists of two components: a deep learning model and
a post-processing phase. On one hand, the proposed deep learning model is
a revision of the model proposed by Fabregat et al. [
The proposed deep learning model (Figure 2) uses the following embedding
features: words, PoS-tagging and characters. Words are encoded using a pre-trained
Spanish word embedding [
Using the training set, the model has been trained during a total of 50 epochs. We have tried not to give preference to any category or domain by integrating the training data of each category in a single set. Pre-trained resources and model’s parameters are the following: – Pre-trained English Word Embedding dimension: 300 – Embeddings dimension (Characters / PoS-tagging): 50 / 50 1 https://keras.io/layers/embeddings/
XCh
Embedding layer Convolutional layer
Time distributed
Bidirectional Ch : Characters W : Word P : PoS-tagging
C : Casing Max Pooling
XP
XW
Xc Flatten
Embedding layers One-hot vector
Concatenate
Dense Neural network T 1
T2
T3 ...
Tz
In order to avoid any possible over-adjustment of the model over the training set, some dropouts have been applied in both the character level and term level processing.
– Conv1D output dropout: 0.5
– LSTM-Dropout: 0.5
– LSTM Recurrent Dropout: 0.25
Finally, we used the standard IOB labeling scheme [
After the deep learning architecture, the proposed system includes a post-processing phase for the correction of frequent errors detected in the development phase. The stack of rules applied in this phase has consisted of a total of 11 rules focused mainly on processing those cases where the role of a trigger is modified by elements such as conjunctions and modifiers such as tan or siquiera among others. Some of the rules applied are: – The expression “sin embargo” does not correspond to a negation cue. – After “,”, “?”, “¿”, “!”, “¡” or “;” any discovered cue is part of another scope. – The term “no” in expressions such as “no tan”, “hasta que no” or “no [me|le|te] [ha|han|has|he|...] [Verb] [Quantifier]” doesn’t correspond to a negation cue.
All the applied rules have been generated after analyzing the model errors in the development set, trying to avoid possible generalization errors. Even so, although some of the rules developed behave effectively in this corpus, these may not properly represent the complexity of the problem. 3
This section describes the results obtained by the proposed system. The evaluation has been carried out taking into account the evaluation criteria proposed by the organizers.
– Punctuation tokens are ignored. – True positives are counted when the system produces negation elements exactly as they are in the gold set. – Partial matches are not counted as False Positive (FP), only as False Negative (FN). – False negatives are counted either by the system not identifying negation elements present in the gold set, or by identifying them partially. – False positives are counted when the system produces a negation element not present in the gold set.
Using the development set to evaluate, Table 1 shows the improvement obtained before and after taking into account the post-processing phase. Results show that improvements are obtained in all domains except in “Computers” where both precision and recall decrease in a remarkable way. The reason for this behavior comes from a particular rule that is applied incorrectly in some cases, being more evident in this domain. The applied rule incorrectly considers that any enumeration of expressions with some negation cue and separated by “,” correspond to different scopes. However, due to its overall positive influence it has been finally maintained within the set of rules. Since no solution could be identified, it was decided to include it in the set due to the global improvement it implies. The difference in the micro-average F1 values for both cases (89.67% with rules and 87.88% without rules) shows that the addition of a postprocessing phase provides remarkable improvements.
Once we have validated the model, it has been compared with the model presented by the UNED team in the last edition of the workshop. As Table 2 shows, for both with and without rules, the improvements are quite noticeable especially in recall. This improvement may be due both to the addition of the character-based representation model and to the simplification of the casing model (in the previous model this casing model was represented as an embedding).
On the other hand, using the test set, as can be seen in Table 3 the results of our proposed system are compared with those of CLiC team. Although our proposed system does not improve the overall results obtained by the CLiC team, in most cases our system obtains better results in terms of precision, with the exception of the “car” domain. Regarding recall, the greatest differences between these two systems are found in domains such as “Books”, “Computers” and “Phones”. After an analysis of these results, one of the possible conclusions is that our system does not behave properly when processing certain unseen samples.
Domain
Finally, table 4 shows a comparison of the average results obtained in this edition of the task A of NegEs workshop. As can be seen, the obtained results are very competitive, although as we mentioned above, the average recall of the proposed system is low compared with the obtained precision. However, this fact is repeated in most of the reported systems. These results may indicate that systems do not work properly with some kinds of instances unseen during the training phase. A more detailed analysis would be necessary to draw other conclusions. 3.1
After analyzing the output provided by the system for the development set we found that many of the errors detected by the UNED team with this new attempt were minimized, especially those cases where there are typos in the text. The errors detected in multi-term expressions and enumerations have also been reduced. Some of the detected errors in this new version of the system are derived from scenarios not covered by the set of rules or negation triggers not seen during training phase. 4
Due to the importance of automatic negation processing in the field of natural language processing, the detection of negation cues is a very important task. In this paper we present a revision of a system based on deep learning for the detection of negation cues in Spanish. In summary, the system consists of a model based on Deep Learning and a post-processing phase based on the application of a stack of rules. Considering the results shown, the system achieves a quite remarkable improvement compared to its predecessor and results are comparable to those obtained by the rest of the participants, both in this edition and in the previous one.
Taking into account the results obtained, our future work focuses on
improving the stack of rules in order to solve the errors detected in the development
phase. Another improvement could be to explore the replacement of the output
layer of the Deep Learning model by a CRF. The combination of Bi-LSTM and
CRF has achieved very interesting results in similar tasks [
This work has been partially supported by the Spanish Ministry of Science and Innovation within the projects PROSA-MED (TIN2016-77820-C3-2-R) and EXTRAE (IMIENS 2017).