Detection of Negation Cues in Spanish: The CLiC-Neg System

Abstract

This paper describes the system proposed by the CLiC team at the University of Barcelona for detecting negation cues in Spanish. It applies the Conditional Random Field (CRF), a supervised machine learning method, to the identi cation of negative expressions. After carrying out an error analysis, we tried to improve on the results by the CRF adding vocabulary lists and rules. The results obtained show that, contrary to our expectations, neither adding rules nor creating a lexicon of multiword expressions signi cantly improved the performance of the model. Resumen. Este art culo describe el sistema de deteccion de claves de negacion propuesto por el equipo CLiC de la Universitat de Barcelona. Este sistema utiliza el Conditional Random Field (CRF), un metodo de aprendizaje automatico supervisado, para marcar las expresiones negativas. Tras un analisis pormenorizado de los errores, hemos intentado mejorar los resultados an~adiendo listas de vocabulario y reglas. Los resultados nales demuestran que, en contra de nuestras expectativas, an~adir reglas o diccionarios de expresiones a un modelo de aprendizaje automatico no mejora notablemente la deteccion de la negacion.

Keywords

• Negation cues
• Negation detection
• CRF
• Supervised ma- chine learning

-

Detecting negation is a complex but pressing issue in NLP given the importance of correctly processing whether a statement is negated or not ([ 19 ]). This is especially true for tasks like sentiment analysis and areas like biomedicine, in which detecting medical conditions is crucial for a diagnosis. Negation is also a central issue for many other tasks such as information extraction and text mining.

One of the rst and most in uential algorithms for negation detection is NegEx ([ 3 ]), which was developed to identify negation cues in discharge summaries in English. NegEx is a rule-based algorithm and has been adapted to other languages (see [ 2 ]; [ 6 ] for German and [ 5 ] for Spanish). It has been the starting point for other extended algorithms such as ConText ([ 10 ]), pyConTextNLP or DEEPEN ([ 16 ]).

However, in recent years the research community has switched to machine learning techniques, given that this kind of approach has produced good results in other NLP related tasks. Morante & Daelemans ([ 17 ]) applied machine learning to detect negation cues and scope, Goldin & Chapman ([ 9 ]) to detect whether a term is negated and Morante et al. ([ 18 ]) speci cally to detect negation scope. All these models have shown promising results and have motivated the research community to abandon rule-based algorithms in favour of machine-learning techniques. In the case of Spanish, the two best classi ed models in Task 2 at the NEGES 2018 Workshop [ 12 ] applied these kind of systems. Loharja et al. ([ 15 ]) used Conditional Random Fields (CRF) and Fabregat et al. ([ 7 ]) recurrent neural networks (RNN). We chose to apply CRF, following [ 15 ], with the intention of tackling one of its biggest shortcomings: the detection of discontinuous negation cues. In order to do so, we compared the results obtained by a CRF model with the results obtained by adding rules and dictionaries as a previous step to CRF classi cation. We think that our approach is useful for understanding more fully the limits of machine learning and demonstrating whether combining rules with this methodology actually improves results.

This paper describes our approach to solving subtask A at the NEGES 2019 conference ([ 11 ]). The goal of this subtask is to automatically identify all the negation cues in a document, including discontinuous negative expressions which, as we will show in the following sections, is the most challenging aspect in this task. The organizers provide SFU ReviewSP-NEG [ 13 ] as a corpus from which teams can build their systems. This corpus consists of 400 reviews in 8 di erent topics, with 25 positive and 25 negative reviews for each topic. The corpus is divided into training, development and test sets and is presented in the CoNLL format ([ 8 ]), in which each line contains a token and a column that includes grammatical information: the lemma, morphological features, and the annotation regarding negation in the case of the training and development sets.

This paper is organized as follows: Section 2 describes the di erent models evaluated, all of which are based on a CRF classi er and compared with a baseline that uses only a dictionary of words. Section 3 discusses the results obtained, and Section 4 synthesizes our conclusions and de nes lines for future research. 2

The CLiC-Neg System

We developed several methods for predicting negation markers from the data provided. The corpus was already lemmatized and morphologically tagged, and the negation markers were indicated in the training and development sets. We translated these to a representation in IOB format ([ 21 ]), which distinguishes between the beginning and the rest of the cue. Speci cally, the initial item in a complex negation cue or a simple negative expression is identi ed with the tag \B" (Begin), the remaining elements in the cue are identi ed with the tag \I" (Inside) and all the tokens that do not correspond to a negative expression are tagged as \O" (Outside). This is necessary to predict the cues correctly, as it allows one to di erentiate several negation cues in the same sentence. Without the distinction between Begin and Inside, there is no way to say where one cue ends and another starts. The corpus only speci es which negation cues exist in the sentences, but they can be translated directly to IOB labels. 2.1

Baseline System

The rst step in developing our model was to design a simple baseline to compare it with a more sophisticated approach. Therefore, we designed a baseline that only used a dictionary to detect negation cues. We obtained it from the SFU-ReviewSP-NEG corpus provided for the task. We rst built a classi er using logistic regression to detect whether a sentence contains negation. This is a simpler problem for which the negation cues are accurate predictors. This system performed successfully at identifying negative sentences and, using only the lemma and part-of-speech of words as features, obtained an accuracy of 97%.

We took the 12 most predictive features of the logistic model, that is, the 12 words with the highest coe cient, and kept them in a dictionary to be used in our baseline. This model only detects simple negation cues, that is, it only used the tags B and O. Therefore, it was not capable of detecting discontinuous negation cues. However, even with all these limitations, this model obtained an F1 score of 73.07% (see Table 1). 2.2

CRF System

The next step was to choose a model to improve our baseline results. In line with [ 15 ], we developed a system based on a Conditional Random Field (CRF) model. This approach was the most successful in the NEGES 2018 Workshop. CRF is a method used for structured prediction, allowing one to predict sequences of labels e ciently by taking into account the labels of the previous and following words in a sentence. CRF requires less training data to obtain good results than other methods for structured predictions (see [ 14 ]), which made it a suitable choice for our corpus. Agarwal & Yu ([ 1 ]) showed that CRF is very accurate and achieves high F1 scores when detecting negation cues and the scope of negation in clinical notes, and Loharja et al. ([ 15 ]) applied it to the detection of negation cues in Spanish.

We implemented our system using Python 3. The CRF model uses the CRF Suite library (see [ 15 ]), which is available as an extension of the Scikit-Learn library for machine learning. For the training process, our CRF model used a set of features based on the methodology in [ 15 ]: each word feature in a sentence consists of the word and its neighboring words (up to 6 positions before and up to 1 position after), and the same for the POS and its neighboring POS. The model was trained on the training part of the SFU ReviewSP-NEG corpus, using the default parameters of the CRF Suite algorithm. It was evaluated on the development part and it obtained an average F1 score of 84.18% (see Table 2, where this model is labeled CRF).

Our CRF system improved the baseline in all of the topics. Note that this improvement, though, was more signi cant in some areas, such as cars, where results were almost 12% better than the baseline, but the improvement was always around 10%. However, some topics, such as mobile phones (90.29%) or books (85.2%) obtained much better results than areas such as cars (75.86%). It would be interesting, for future research, to look up the speci cities of each area to nd linguistic di erences that could justify these results. 2.3

CRF System with rules and list of words

We observed the false positives and false negatives yielded by the CRF model described in Section 2.2 when evaluated on the development set, in an attempt to correct the main sources of error. We came to the conclusion that most errors could be classi ed in four categories: 1. Errors that were caused because non-negative sentences were detected as negative: `Ya estaba casi todo, no (B)?' 2. Errors contained in multiword expressions: some multiword expressions that include negation cues were not correctly identi ed by the system (`a no ser que', `a excepcion de', `a falta de', etc.). 3. Errors in tagging elements such as `tan', `tanto' and `muy', `mucho', especially in the case of discontinuous cues. 4. Errors in detecting discontinuous negation cues: one particularity of negation in Spanish is that we can nd more than one negation cue as part of the same negative statement.

In order to improve the CRF model, we applied two di erent approaches: 1) we introduced a list of rules that could help to identify multiword expressions and discontinuous cues, a common cause of misclassi cations; 2) we added a dictionary list of negative expressions to reduce the false negatives that are due to unseen negation markers in the training dataset.

Rule-based system and CRF First, we applied a set of rules for correctly assigning a tag to cases where we observed that the CRF of the previous section fails. These rules are useful for detecting not only negation cues but also prevent some words from being erroneously tagged as negation. The rules are the following: { Rule 1: In the sequence \, no?", nothing is marked as a negation cue; this is a case of not negative sentence even though it uses \no", a word generally marked as a negation cue in the CRF model in Section 2.2. { Rule 2: If \no" is followed by \nada mas" in a distance between 0 and 5 words, neither \no" nor \nada mas" are negation markers. { Rule 3: If \ningun" appears in the initial position of a sentence, it is tagged as \B". { Rule 4: If \no", \tampoco" or \sin" are followed by \nada", \ningun" or \nadie" in a distance between 0 and 10 words, the former word is tagged as \B" and the latter as \I", thereby being a discontinuous cue. { Rule 5: if \aun" or \todav a" are immediately followed by \no", the former is tagged as \B" and \no" is tagged as \I". { Rule 6: \tan" or \tanto" are always tagged as \O". This avoids a common false positive in the CRF model in Section 2.2.

Then, we applied the same CRF model presented in Section 2.2 to tag the remaining words, those not classi ed by the rules. That is, the rules' decisions prevail over the CRF model in the cases where they are triggered. Results obtained on the development set for this model are presented in Table 2 under the name Rules+CRF.

List of words and CRF Our second approach uses a list of multiword expressions that were extracted from NewsCom, a corpus developed and annotated by CLiC at the University of Barcelona containing users comments on news. If a word in a sentence appears in the list, it is tagged as a negation marker. Then, we used the same CRF model to tag the remaining words, which were not detected using the list. This means that the prediction made by the CRF model only takes into account cases outside the list.

When ne-tuning this model, we observed that some words in the list also caused false positives, because they were used in other senses than negation. Similarly, we increased the list with cases of negative expressions not learned by the CRF, such as `a no ser',`en absoluto', `en ningun momento' and `sin necesidad de'. In Table 2 we present the evaluation results for the list of words that yielded the best score when tested on the development set, under the label List+CRF. 3

Discussion

Table 2 summarizes the results of our three di erent approaches (CRF, Rules + CRF, List + CRF) tested on the development set. Contrary to our expectations, adding rules worsened the model by 3 points. This means that our rules introduced error because they were triggered in more cases than they should have been, misclassi cating some examples. This contradicts the hypothesis that combining a rule-based method with machine learning techniques can improve the model by helping it to detect cues that are not correctly identi ed. Similarly, adding a lexicon of multiword expressions does not improve the scores either, although the e ect is less signi cant. Despite implementing these approaches in a controlled way, through the manual observation of the main sources of error in the predictions, we increased the number of errors rather than reduced them.

Our best model (CRF) was presented in the NEGES 2019 competition ([ 11 ]), where it was evaluated on their test set, achieving the results shown in Table 3. It was ranked rst among its competitors, but did not improve on the best score obtained at the same competition in the previous year by Loharja et al. ([ 15 ]), whose F1 score is 2% higher than ours. The main di erence is in the feature set used, which is larger in their case and represents more linguistic phenomena. For instance, the use of features at the sub-word level such as pre xes or su xes could help detect cases of morphological negation that are missing from our model. In this paper we have described our proposal for the detection of negation cues based on a CRF classi er, following the approach in [ 15 ] for NEGES 2018. The results obtained show that this supervised learning technique is a promising approach to building a system that automatically detects negation cues in Spanish, with an average F1 score of 84.09% on unseen data. However, our expectations of signi cantly improving on these results by identifying the main sources of error and readjusting the model have not produced the desired outcome. In fact, adding rules designed to assign the right tags to some cues worsened the F1 score, and using a dictionary of negative expressions also failed to improve our results.

For future research, we will analyze the linguistic characteristics of each eld to re ne our CRF model. We have seen that there are signi cant di erences in the precision and recall depending on the topic of the comments. It will be of interest to analyze whether textual di erences, such as register or syntax, have a meaningful impact on the quality of our model. Additionally, the use of features at the sub-word level seems a promising approach to the detection of cases of morphological negation and the reduction of the problems related to learning from a training corpus with a limited set of negative expressions.

Acknowledgements

This work has been carried out in the framework of the MISMIS-Language project (PGC2018-096212-B-C33), funded by the Ministerio de Ciencia, Innovacion y Universidades, and CLiC (2017SGR341), funded by the Agencia de Gestio d'Ajuts Universitaris i de Recerca. Generalitat de Catalunya (AGAUR).

References

1. Agarwal , Shashank, & Hong Yu: Biomedical negation scope detection with conditional random elds . Journal of the American Medical Informatics Association , 17 ( 6 ), 696 { 701 ( 2010 )

2. Chapman , Wendy W., et al.: Extending the NegEx lexicon for multiple languages . Studies in health technology and informatics , 192 , 677 ( 2013 )

3. Chapman , Wendy W., et al.: A simple algorithm for identifying negated ndings and diseases in discharge summaries . Journal of biomedical informatics , 34 ( 5 ), 301 { 310 ( 2001 )

4. Chen , Edwin: Introduction to Conditional Random Fields. http://blog.echen.me/ 2012 /01/03/introduction-to-conditional-random- elds/. Last accessed 20 June 2019

5. Costumero , Roberto , et al.: An approach to detect negation on medical documents in Spanish . International Conference on Brain Informatics and Health . Springer, Cham, 366 { 375 ( 2014 )

6. Cotik , Viviana , et al.: Negation detection in clinical reports written in German . Proceedings of the Fifth Workshop on Building and Evaluating Resources for Biomedical Text Mining (BioTxtM2016) , 115 { 124 ( 2016 )

7. Fabregat , H. et al.: Deep Learning Approach for Negation Cues Detection in Spanish . Proceedings of NEGES 2018. Workshop on Negation in Spanish , 43 { 48 ( 2018 )

8. Farkas , Richrd , et al.: The CoNLL-2010 shared task: learning to detect hedges and their scope in natural language text . Proceedings of the Fourteenth Conference on Computational Natural Language Learning , Shared Task, Association for Computational Linguistics ( 2010 )

9. Goldin , I. , & Wendy W. Chapman : Learning to detect negation with notin medical texts . Proc Workshop on Text Analysis and Search for Bioinformatics , ACM SIGIR ( 2003 )

10. Harkema , Henk , et al.: ConText: an algorithm for determining negation, experiencer, and temporal status from clinical reports . Journal of biomedical informatics , 42 ( 5 ), 839 { 851 ( 2009 )

11. Jimenez-Zafra , Salud Mar a and Cruz D az, Noa P. and Morante, Roser and Mart n-Valdivia, Mar a Teresa: NEGES 2019 Task: Negation in Spanish . Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2019 ), CEUR Workshop Proceedings, Bilbao, Spain, CEUR-WS ( 2019 )

12. Jimenez-Zafra , Salud Mar a and D az, Noa P Cruz and Morante, Roser and Mart nValdivia , Mar a Teresa: NEGES 2018: Workshop on Negation in Spanish. Procesamiento del Lenguaje Natural , 62 , 21{ 28 ( 2019 )

13. Jimenez-Zafra , S. M. , Taule , M. , Mart n Valdivia, M.T., Urea-Lpez , L.A. , & Mart , M.A. : SFU Review SP-NEG: a Spanish-corpus annotated with negation for sentiment analysis. A typology of negation patterns . Language Resources and Evaluation , 52 ( 2 ), 533 { 569 ( 2018 )

14. La erty , John, Andrew McCallum & Fernando CN Pereira: Conditional Random Fields : Probabilistic models for segmenting and labeling sequence data . Proceedings of the 18th International Conference on Machine Learning 2001 (ICML 2001 ), 282 { 289 ( 2001 )

15. Loharja , Henry, Llu s Padro & Jorge Turmo Borras: Negation cues detection using CRF on Spanish product review texts . NEGES 2018 : Workshop on Negation in Spanish: Seville, Spain: September 19-21 , 2018 : proceedings book, 49 { 54 ( 2018 )

16. Mehrabi , Saeed , et al.: DEEPEN: A negation detection system for clinical text incorporating dependency relation into NegEx . Journal of biomedical informatics , 54 , 213{ 219 ( 2015 )

17. Morante , Roser & Walter Daelemans : A metalearning approach to processing the scope of negation . Proceedings of the Thirteenth Conference on Computational Natural Language Learning, Association for Computational Linguistics , 21 { 29 ( 2009 )

18. Morante , Roser, Sarah Schrauwen & Walter Daelemans: Corpus-based approaches to processing the scope of negation cues: an evaluation of the state of the art . Proceedings of the Ninth International Conference on Computational Semantics, Association for Computational Linguistics , 350 { 354 ( 2011 )

19. Morante , Roser & Caroline Sporleder: Modality and Negation: An Introduction to the Special Issue . Computational Linguistics , 38 ( 2 ), 223 { 260 ( 2012 )

20. Prateek , Joshi: Why Do We Need Conditional Random Fields? . https://prateekvjoshi.com/ 2013 /02/23/why-do -we-need-conditional-random- elds/ . Last accessed 20 June 2019

21. Ramshaw , Lance A. , & Mitchell P. Marcus: Text chunking using transformationbased learning . Natural language processing using very large corpora , Springer, Dordrecht, 157 { 176 ( 1999 )