This article describes the negation cue detection approach designed and built by UPC's team participating in NEGES 2018 Workshop on Negation in Spanish. The approach uses supervised CRFs as the base for training the model with several features engineered to tackle the task of negation cue detection in Spanish. The result is evaluated by the means of precision, recall, and F1 score in order to measure the performance of the approach. The approach was ranked in 1st position in the o cial testing results with average precision around 91%, average recall around 82%, and average F1 score around 86%.
This paper describes the negation cue
detection model approaches presented by UPC's
team for the NEGES 2018 workshop task 2
(negation cues detection)
The article is organized as follows. Section 2 describes the approach used to learn the negation cues detection model. Section 3 describes the system built based on the approach explained in the previous section and the details of the implementation. The results achieved by our approach are presented and brie y analyzed in Section 4. Finally, Section 5 gives conclusion about the work that has been done.
1NLTK { the Natural Language Toolkit (Bird, Klein, and Loper, 2009)
Before describing the approach, let us begin by addressing some de nitions. A negative sentence n is de ned as a vector of words (w1; w2; :::; wn) containing one or more negation cues, where the latter can be a word (e.g. no), a morpheme (e.g.in-capaz) or a multi-word expression (e.g. ya no, todav a no) which inherently expressing negation. The goal of negation cue detection is to predict vector c given the sentence n where c 2 f1; 0gjnj is a vector of length same with the length of n so that ci = 1 if wi is part of the negation cue and ci = 0 otherwise.
It is possible that more than one negation cue can appear inside a sentence. In Spanish, one of the special characteristic of negation cue is that a cue can consist of more than one word, not necessarily consecutive. This special characteristic increases the complexity of detecting whether two words recognized as cue are indeed two separated cues or are actually the same non-contiguous cue. This also makes negation cue detection in Spanish a more challenging task compared to negation cue detection in English because that case is scarce in English.
The approach we use for this work will
be one of the state of the art approach: a
CRF based negation detection. We try to
reproduce the approach used by previous works
1 Z(x)
K expfX X j fj (x; yt; yt 1)g
t j=1
The estimation of weights ( j ) for each feature fj is carried out by maximizing the conditional log likelihood: max l( ) = max
N X p(y(i)jx(i)) i=1 .
where N is the number of observation sequences x(i) and label sequences y(i).
Training CRFs might be time-consuming for some tasks since the time needed for training depends quadratically on the number of class labels and linearly on the number of training instances and the average sequence length. However, state-of-the-art solutions use CRF models for many NLP tasks where time consumption is still tolerable.
As discussed before, the goal of negation cue detection is to obtain vectors which represent the sentence and give information whether the token or words which are part of the sentence is a part of the negation cue in a value. Using the knowledge of named entity recognition, we can infer that negation detection is a type of NER in which we would like to recognize entities that are parts of negation. In other words, we would like to classify whether each words inside a sentence is part of negation cue or not a part of it. From this, we de ne a three-class classication problem for each word which we observe: Begin-Cue(B-C), Inside-Cue(I-C), or Out(O). A word classi ed as Out is not part of a cue. In order to handle cues which consist of more than one word, we give two kind of classi cation for the cues which are BeginCue for the rst words that identify that start of a cue and Inside-Cue for the rest of the words of a cue which are not the rst word but is still identi ed as part of the same cue. 3
For the preliminary, we do some preprocessing to the data in the corpus provided in order to match the input format of the system we built. The corpus provided by this task is using CoNLL format. Each line corresponds to a token or word and each annotation is provided in a column with empty lines indicate end of sentence. We produce two set of data with di erent format with respect to their usage for each step: 1. Data format (Ramshaw and with BIO
Marcus,
tagging 1999) in order to be used as input for training. The annotated token is tagged with "BC" if it is in the beginning of negation cue; tagged with "I-C" if it is part of the negation cue but not the rst word of the cue; and tagged with "O" if it is outside of the cue. One of the examples of sentence in this format is:
El|O coche|O funciona|O estupendamente|O ,|O es|O muy|O manejable|O ,|O por|O cierto|O ,|O casi|B-C no|I-C consume|O gasolina|O algo|O que|O para|O mi|O es|O muy|O importante|O .|O 2. Raw data format without any tagging in order to be used for testing input.
Las ruedas a los 15000 kms las tuve que cambiar , todas , las cuatro , por ser de una marca coreana , que no da mucho resultado .
After the preprocessing is done, the documents is ready to be used as input for the next respective steps. This preprocessing part did not alter any important information contained in the data as the purpose is only to change the format in order to make it easier to be used in the following steps.
Before implementing the approach we have explained before in the system, we developed a baseline system to be used as starting point and a comparison. Our aim is to see whether the approach we have will perform better than a baseline approach which used simple techniques. To reach this, we use the baseline system as comparison with the system we develop using the approach we propose. The baseline system we developed uses simple techniques which are common such as dictionary lookup combined with some rules for detecting negation cues.
The rst thing we did in this baseline was to create a dictionary based on the training dataset from the corpus. We collected all the words which are tagged as negation cues from all the documents in the training dataset together with their frequency. After having sorted the collected negation cues based on the frequency from the most to the least, we chose the top 25 words with the most frequency. These 25 words became the dictionary of negation cues in our baseline system. We also developed several rules to capture the characteristics of negation cues in Spanish which we have explained before. These rules are used to decide whether more than one cues which appear in a sentence is actually part of a cue or separated cues. The rule checked whether a word is in a list of special word which we created and then check if it ful ll condition of having another cues that precede it. Here are the algorithm from baseline system which describe the rule: if word in DICTIONARY then if word in SPECIAL then if exist cue before then
word is part of cue end if else
word is new cue end if end if
After having implemented the combination of dictionary lookup and rules we developed, we then use the baseline system to tag the documents from development testing dataset. We use the result as the preliminary result to be later compared with the result from the system we developed using our proposed approach. By doing this, we could see whether the approach we have can give more advantage compared to using simple techniques.
The system we built use a toolkit named
NLTK which is a Python based toolkit for
building Python programs to work with
human language data. NLTK provides
easyto-use text processing libraries for classi
cation, tokenization, stemming, tagging,
parsing, and semantic reasoning. One of the
modules in NLTK is an CRF tagger which can
be used for the tagging of text using Python
CRFSuite2 as it's core. This module are what
we mainly used in our approach for
negation detection by adapting a point of view
2Python CRFSuite -Python bindings to CRFSuite
By using the features mentioned above, we do the training using the given data and CRF module in NLTK to produce the model which can be used to detect the negation cue in Spanish. The parameters for training the CRF are the default parameters used in NLTK toolkit. This model will be used as one of the input for the next step which is testing. Testing is the process of detecting negation from the testing data (data in which negations are not annotated or raw data) by using the model which we get from the training process as the knowledge base. The result of the testing process is an annotated version of testing data in which words in each sentence are classi ed into either part cue or outside of them. The result we obtain after the testing process will be in the format of BIO tagged since this is the format which we use to represent our data. Related to that, we do some post-processing to change the format of the result into the same original format as the input (training data). We use the original data format of CoNLL and then add the information of negation cue which we obtain from the testing. 4
After nished with the testing process, we will obtain the result of the negation cue detection as annotated documents of testing data we provide as input. In order to evaluate the performance of the approach used in the system, we will use recall, precision, and f1 score measurement. We use the evaluation script provided by the organizers to make sure that our output match the requirement. In the rst phase, we use the development testing data which is provided in order to measure the performance of our system. We perform the testing on each document in the development testing dataset which are divided based on the domain. Each document is processed separately and also evaluated separately. To give a general view of the performance, we also calculate the micro average of the whole result from the development testing. Table 1 shows the result of baseline system we have obtained using the development testing data meanwhile Table 2 shows the result of system based on our proposed approach using the same development dataset. which use simple techniques. The result also gives a fairly high value of performance with most of them reach over 80%. Especially in precision, the average reach more than 90%. This is possible due to a fairly simple task of detecting negation cue detection. Most of the cues consist of word such as "no","ni","nada" and several other words which describe negation with little variability of vocabulary. This leads to a fairly easy detection of cues and the small number of false positives. On the other hand, the recall have much lower result with some reach even lower than 80%. This happens due to the higher number of false negatives caused by the di culty of detecting non-contiguous multi-token cues. In most of the cases of false negative, our system has di culties to detect such cases, for example:
No es cosa del paralelo ni del equilibrado.
In the example, no...ni is a negation cue meanwhile our system recognize them as two separated cues. Another kind of false negative is the opposite, where two separated cues is recognized as one cue. Those cases contribute to most of the false negatives in the development testing result.
The o cial testing result measurement can be observed in Table 3. This result is obtained using the model we have and the o cial testing dataset provided by the organizers. The evaluation is done directly by the organizers and we receive the measurement result as can be seen in Table 3 after we submit our testing result.
Based on the evaluation from organizers, our result is ranked rst compared to other participants in the same task. As can be seen on the table, the o cial testing result follows the same pattern as the development testing result with higher precision and lower recall. Even though we can't observe the cases happening in o cial testing result, we can infer that similar cases in development testing probably also can be found by looking at the result. The percentage also have almost similar value with precision reach around 91%, recall around 82%, and F1 score around 86%. The average of result in o cial testing has slightly lower value compared to the one in development testing but the di erence is not signi cant. 5
In this article we have described the approach and system we built for the participation in NEGES 2018: Workshop on Negation in Spanish task 2 of negation cues detection for Spanish product review texts. Our approach to detect the negation cues consisted of a supervised approach combining CRF and several features for negation cue detection in Spanish for training the model. The model will then be used to classify whether a word in the observed data or testing data is a part of negation cue or not. This approach was ranked in 1st position in the o cial testing results with average precision around 91%, average recall around 82%, and average F1 score around 86%.
This works has been partially funded by the Spanish Goverment and by the European Union through GRAPHMED project (TIN2016-77820-C3-3-R and