The availability of automated approaches to e ectively detect and characterize negation in textual contents is essential to robustly perform a wide range of Natural Language Processing tasks. The Workshop on Negation in Spanish 2019 provides a forum to share investigations and compare methodologies dealing with the characterization of negation in Spanish texts. In this paper we present our participation to Sub-task A organized in the context of this Workshop, focusing on the detection of negation cues in Spanish product reviews. We consider four negation cues detection approaches based on supervised learning and compare their performance. The best performing approach, based on a Conditional Random Fields sequence labeller, has been evaluated in the context of the Sub-task A of the Workshop on Negation in Spanish 2019 obtaining robust performance and scoring as the most precise system across several text domains, while keeping acceptable recall rates.
Negation represents a core linguistic phenomenon that aims at reversing the
truth value of a statement [
As in previous editions [
The participants of NEGES 2019 have been proposed two sub-tasks concerning the detection of negation cues (Sub-task A) and the assessment of the role of negation in sentiment analysis (Sub-task B). In this paper, we describe our participation to the Sub-task A of NEGES 2019, presenting our approach to detect negation cues. In particular, after providing a brief overview of the Sub-task A in Section 2, Section 3 introduces the four supervised learning approaches to negation cue detection that we have considered in our experiments: these approaches are evaluated and thus compared by relying on the train dataset of the Sub-task A of NEGES 2019. We have chosen the best performing negation detection approach to support our participation to NEGES 2019, whose o cial evaluation results are discussed in Section 4. To conclude, in Section 5 we present our nal remarks and plans for future work. 2
The Sub-task A of NEGES 2019 challenged participants to develop e ective
approaches to automatically identify negation cues in Spanish texts. In particular,
the SFU Review SP-NEG corpus [
The SFU Review SP-NEG corpus includes the text of 400 Spanish review gathered from the web portal Ciao.es and dealing with the following eight domains: movies, books, cell phones, music, hotels, cars, washing machines and computers. Table 1 provides an overview of the number of sentences by domain included in NEGES 2019 train and test datasets as well as the number of annotated negation cues that are present in the train dataset. From Table 1, we can notice that about 81% of negation cues of the train dataset span over a single token. Moreover, the majority of the negation cues of the train dataset (2,616 over 3,098) are expressions that span over one or more contiguous tokens. In particular, as we can see from Table 2 `no` is by far the most common negation cue with a total of 1,824 occurrences as single-token cue in the train dataset.
We modelled the detection of negation cues as a token labeling task. In particular, we assigned to each token of the sentences of the NEGES 2019 train dataset one of the following three labels: B, I or O. By default, all the tokens of a sentences are labelled as O-tokens (tokens outside a negation cue), except the tokens belonging to a negation cue. If the negation cue spans over a single token, it is assigned the label B since represents the beginning of the cue. Otherwise, if a negation cue spans over two or more consecutive tokens, the rst token is labelled as a B -token while the consecutive ones as I -tokens (tokens of a negation cue subsequent to the rst one). In case a negation cue is composed by discontinuous text spans, each one of these spans is treated as a separate negation cue. The following sentence provides an example of the token labeling approach just described: Ah O no B me O esperaba O nadie B demostrando O falta B de I seriedad O . O
We considered four supervised learning approaches and evaluate their ability to learn to predict the label (B, I or O) to assign to each token of a sentence, thus detecting the occurrence of negation cues. Independently from the supervised learning approach adopted, we represented each token by means of the following set of features: { Shallow textual features: number of characters of the token; position of the token in the sentence, obtained by dividing the index of the token in the sentence by the total number of tokens that are present in the sentence, thus generating a number in the interval [0; 1]; lower-cased token; percentage of lowercase characters; percentage of non alphabetic characters. { Lemma features: lemma; if the lemma includes more than one character, rst two characters and last two characters of the lemma. For instance in case of the lemma create, two additional textual features are created: cr and te. { Part of Speech features:
Part of Speech category of the token (one nominal value among adjective, conjunction, determiner, etc.); the complete result of the morphological analysis of the token including information about person, gender, number when appropriate. For instance the label NCFP for a noun (N), proper (P), feminine (F), plural (P). { Dependency tree features: For the considered dependency tree node (i.e. token) and, if any, its parent node we considered the following features: token; lemma; Part of Speech category (one nominal value among adjective, conjunction, determiner, etc.); depth of the token in the dependency tree; number of children and descendent nodes; dependency relation towards the parent, if any.
We relied on the open-source language analysis framework Freeling (version
4.1) [
By exploiting the set of features just described to characterize each token, we evaluated the performance of the following four token labelling / classi cation approaches: { Conditional Random Fields (CRF), a sequence labelling statistical modelling method. We relied on the crfsuite1 implementation of CRF. { Random Forest (RF), an ensamble learning method for classi cation based on decision trees. We relied on the Random Forest implementation provided by SciKit learn2. { Support Vector Machine with linear kernel (SVM-linear), handling multi-class classi cation by means of the one-vs-the-rest scheme. We relied on the linear SVM implementation provided by SciKit learn. { XGBoost (XGB), optimized gradient boosted decision trees. We relied on the XGBoost python package3.
We evaluated the previous token labelling / classi cation approaches by considering the default values of their parameters as speci ed by each algorithm implementation considered. When we applied the RF, SVM-linear and XGB approaches, we characterized each token by relying on the set of features previously mentioned in order to describe both the token and all the tokens occurring in a [ 3; 3] window centered on that token. In this way, also information modelling the context of a token can be considered to predict the most likely label to assign to it.
Table 3 shows the performance of the four token labelling / classi cation approaches considered with respect to a 10-fold cross-validation over the NEGES 2019 train dataset. This Table evaluates also a BASELINE negation cue detection strategy (last column): the negation detection approach of this strategy creates a list of lemmatized negation cues from the train dataset of each fold and marks the occurrences of these cues in the test dataset. From Table 3 we can notice that the CRF is the best performing approach, with a more sensible improvement in performance when we consider the macro F-score of the BIO labels and strict matches of predicted negation cues with a gold standard ones. We have to notice that the BASELINE negation detection strategy, based on simple string match, obtains acceptable performance. Anyway, looking into further details, even if this trend is not evident when we consider the F-scores, the results of the BASELINE negation detection strategy are the ones that present 1 http://www.chokkan.org/software/crfsuite/ 2 https://scikit-learn.org/ 3 https://xgboost.readthedocs.io/ the strongest di erences among precision and recall: low values of precision are balanced by high values of recall. The other approaches based on supervised learning obtain a better balance among precision and recall.
By inspecting the types of classi cation errors of the four supervised learning approaches we considered, we can spot the following trend: the performance of most approaches drastically decreases when they deal with the identi cation of multi-token negation cues and, in particular, when the tokens that occur after the rst one should be spotted. This trend could be related to the greater di culty in characterizing linguistically these tokens and the low number of annotated examples of multi-token negation cues that are available in the NEGES 2019 corpus. In future investigations we plan to analyze in detail this issue by eventually proposing a negation cue detection strategy tailored to improve the performance of the detection of multi-token cues. We chose the best performing supervised learning approach resulting from the experiments described in Section 3 (i.e. the CRF sequence labeller) as the methodology exploited to generate our negation cue predictions for the Sub-task A of NEGES 2019. Our approach scored second in terms of precision and third in terms of recall and F-score. In particular, when we look at the negation detection results across each single domain of the text of the test set of the Sub-task A of NEGES 2019, our approach obtained the highest precision in four domains over eight (movies, mobiles, washing machines and hotels). 5
In this paper we presented the negation cues detection approach we devised in the context of our participation to the Sub-task A of the Workshop on Negation in Spanish 2019, dealing with the automated identi cation of negation cues in Spanish texts. We described in detail the four supervised learning approaches we considered for our participation to NEGES 2019, by comparing their performance on the train dataset of the Sub-task A. We also discussed the results of negation cues detection approach that we selected to participate to NEGES 2019, based on a Conditional Random Field sequence labeller. As future work, we would like to evaluate the performance of a wider range of negation cues detection systems, considering both sequence labeller based on neural network architectures, relying on word embeddings and ensebling methods that combine the predictions of distinct sequence labelling approaches. We plan also to perform a more detailed error analysis to better characterize and try to mitigate the weknesses of the negation cues detection approaches considered.