This paper describes the model used by the Aspie96 team in the subtask A of NEGES 2019 (part of IberLEF 2019), aimed at the automatic detection of negation cues in Spanish. The approach is based on a neural network which uses character-level features to compute tokenlevel features.
In the shared task NEGES, described in [
Each negation may have been an individual word, or composed of multiple (even non-contiguous) words. In the provided training and testing datasets, documents were already tokenized. Tokens were words non words (for instance, in the case of punctuation). For each document, PoS-tags and lemmas were also provided.
The data used in the task was from the SFU ReviewSP-NEG corpus,
presented in [
The main characteristic of the presented model, described in the next section, is that it is not based on word-level features, nor on any kind of knowledge which isn't strictly automatically extracted from the provided training dataset. 2
The model used by the Aspie96 team was based upon the system presented in
[
It had been completely developed from scratch by the author of this paper
(and of [
It constitutes the basis of the system used in the subtask A of NEGES, thus it is crucial to understand it rst. A representation is provided in Figure 1.
Its input was represented as a list with xed length (leading to padding, added to the left, or truncation, to the right, where needed) of sparse vectors. Each vector of the list represented an individual character of the tweet and contained ags whose values were either 0 or 1. Most of the ags were mutually exclusive and were used to identify a character among a list of known ones. Additional ags were used to represent additional properties of the character (such as being uppercase). 1 Under Expat License: https://github.com/Aspie96/ThesisValentinoGiudice2018.
A series of three unidimensional convolutional layers, each with a width of 3
and an output depth of 8, convolving along the length of the tweet, was used to
reduce the size of such a representation and to encode characters in a way which
accounted for the context provided by the neighbouring characters. The output
of the last convolutional layer, still a temporal sequence of xed-size vectors, was
used as input to a bidirectional GRU layer, originally described in [
The described neural network, as presented in [
The full list of known characters in the used representation was the following: Space ! " # $ % & ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; = ? @ [ ] \_ a b c d e f g h i j k l m n o p q r s t u v w x y z | ~
Emojis were represented similarly to their Unicode name (in English), with additional ags.
The full list of additional ags, as described in [
Emoji Indicating whether the character is part of the Unicode name of an emoji.
Emoji start Indicating whether the character is the rst in the unicode name of an emoji.
Letter Indicating whether the character is a letter.
Number Indicating whether the character is a numerical digit. Inverted Indicating if the character is an opening inverted question mark or exclamation mark.
Tilde Indicating whether the character is an N with virgulilla.
The neural network presented in [
The subtask A of NEGES was much di erent from tweets classi cation: the texts were much longer than individual tweets and they were not to be classi ed: instead, elements within the text were to be recognized.
The problem was considered by the Aspie96 team as a problem of classi cation of words within the text: each word had to be classi ed as either: { Not part of a negation cue, at all (118441 instances in the training set). { The rst word of a new negation cue (2490 instances in the training set). { Part of the latest started negation cue (597 instances in the training set).
This assumed a negation cue containing multiple words, although possibly not contiguous, could not contain, between its rst and last words, any part of other negation cues. This was mostly the case in the training set: only 26 words constituted exceptions to this rule as they were non beginning parts of negations cues and they were not part of the last began negation cue.
In the input texts, already tokenized, each token was marked as either a word or not a word (according to a simple match with a regular expression): words were the main focus of the task as they were the ones which had to be classi ed.
The representation of the text was built as follows.
In the text, spaces (which were not provided) were inserted back, simply by putting one between every two tokens. This might not have been accurate (it often is not in the case of punctuation as, for instance, commas are not usually preceded by a space), but, because the network was not pretrained and all the knowledge about the language had to be gained from scratch, this was irrelevant.
Then each word was represented as a xed-size (of length 50) list of vectors, each of which representing an individual character. The word being represented was centered in its representation and, because of the length of the representation of each word being xed, the neighbouring characters, on the left and on the right, whether belonging to other words or not (in the case of spaces or nonword tokens) were used as padding.
To each vector representing an individual character a ag was added, the token ag whose value speci ed if the character was part of the word being represented or the padding (its name comes from the fact that its value is 1 if the character is part of the token being represented).
Lorem ipsum dolor sit amet, consectetur adipisci elit, sed do eiusmod tempor incidunt ut labore et dolore magna aliqua.
The 5th word (amet) has a length of 4 and will therefore need 14 4 = 10 characters of padding: 10=2 = 5 on the left and 10=2 = 5 on the right.
It will, therefore, be represented as:
sit amet, con
marked characters has value 1.
The whole text has 19 words and will thus be represented as 19 lists of 14 vectors: each list representing a word (and its neighbouring characters) and each vector representing an individual character.
Because the representations of each token already encodes the neighboring characters as well there is no need to represent non-word tokens as individual elements.
This results in a representation in which each element corresponds to a word and is a xed-size list of vectors of binary ags, where each individual vector represents a character.
Because the representations of each token already encoded the neighboring characters as well there was no need to consider anything other than a word as a token.
Based on this representation and on networkA (which, given a short text as input, represented as a list of vectors of ags, outputs an individual vector representation of such text), a model to solve the task could be built.
networkA was used to convert the representation of each word into an individual vector: in the model presented by the Aspie96 team for the subtask A of NEGES, networkA convolves trough the text, word by word.
The result is a list of as many xed-size vectors as there are words in the original text and such that each one of them represents the corresponding word in the original text, with its context provided by the neighbouring characters, thus taking into account the di erent meanings a word could have depending on its context.
The representation obtained in this way is the input to the following layers of the neural network: a recurrent layer. All outputs of the recurrent layer, each of which being a vector, are considered (also as many as the words in the text), not just the last one. The role of the additional recurrent layer is to consider each word not one by one, but according to its context, provided by the other words, reading the text fully.
A simple classi er (a simple neural network, using the softmax function as output) is then applied to each such vectors individually to get, for each word (corresponding, in position, to the vector), its classi cation in one of the three classes.
The resulting model is represented in Figure 2.
It must be noted that in the presented model PoS-tags and lemmas are not unused.
In order to reduce the amount of false positives, a copy of the model was used to distinguish, simply, between words that were not part of negation cues and words that were (whether as the rst word or not), resulting in two copies of the model, trained separately (one for classi cation among all the three considered classes and one for classi cation among the rst and the conjunction of the other two). In the nal classi cation, words were marked as not part of negation cues if that was the result of either one of the copies of the model, otherwise one of the remaining two classes was selected. 3
Models have been evaluated by the task organizers using precision, recall and
F1score, using the same script developed for [
The average precision, recall, and F1-score for the Aspie96 team were, respectively, 0.1880, 0.2834 and 0.2258. As a comparison, the best average precision, recall and F1-score were, respectively, 0.9182 (NLP UNED team), 0.7940 (CLiC team) and 0.8409 (CLiC team).
The results do suggest the unsuitability of the model for the task.
The reasons for this needs to be investigated. The copy of the model which simply classi es words as part of a negation cue or not, although a seemingly unnecessary source of complexity (as the problem is of classi cation among three classes, which is dealt with by the other copy of the model) is not the source of the problem as it causes very few false negatives on its own. In a test, it resulted in 9480 true negatives, 365 false positives (which can be dealt with by the main model, which classi es among all three classes), 560 true positives and just 54 false negatives. As its purpose was simply to prevent at least some false positives, without causing too many false negatives, this was its intended behavior.
Another apparent cause of the poor results could seem to be the main assumption of the model: that three classes are enough to solve the problem, although it not being a problem of classi cation by itself. However, it must be noted that in the training set only 26 words could not possibly have been properly handled by the system as a result of this assumption: in every other case, no negation cue contained, between its starting and ending word, any part of other negation cues.
The root of the problem must, therefore, be the structure of the model itself, perhaps inapplicable due to the very high unbalance of the classes. 4
The poor results obtained with the proposed model suggest the unsuitability of the approach for the task at hand, while the good results obtained by other teams suggest much room for improvement.
The main disadvantages of the proposed model are, arguably, its most basic characteristics: it only works on what can be strictly and automatically learned from the text contained in the documents, but a lot more could be used, such as, to name just a few, PoS-tags and lemmas, which were provided in the datasets, pre-computed word-level features, such as word embeddings and common knowledge about negations and the Spanish language.
An extremely similar model was employed in the FACT (Factuality Analysis
and Classi cation Task) task for factuality classi cation, also part of IberLEF
2019 as described in [
This means this kind of model is indeed suitable for the classi cation of
words within a text, while the results got in [
Further research may include the creation of a unique structure of networkA, without alterations or multiple versions, to be applied to di erent tasks, resulting in a more stable neural network, and the exact same structure, based on networkA, to be used for both the FACT task and the NEGES task, in order to allow for a better comparison between the results.
Any persistent di erence in the result will be due, strictly, to the di erences between the tasks.
The main di erence is that the subtask A of NEGES isn't of classi cation of speci c words within a text but, rather, it requires the recognition of elements, some of which composed of multiple, even non contiguous, words. It is very possible that, although this can be indeed converted to a task of classi cation of every word within the text (which already is di erent than only trying to classify some, already marked, ones), the two kinds of task are not to be seen as similar ones. Alternatively, the problem might have been caused by the highly unbalanced classes: the negative class is much bigger than the other ones, as it includes every word in the text which isn't part of a negation cue. In FACT, the thee classes had 2918, 1171 and 255 words each: most words did not belong to any class and they were not to be classi ed.
The main point of the model presented in [
It is, therefore, needed to create a more general model than the one used in the FACT task, able to work properly in di erent tasks. Because the results obtained in this paper are so poor, the NEGES dataset can still be used for this purpose, with the aim of producing a model still able to work on FACT, but also on NEGES.
Whether this is possible will require more research and a more stable structure for networkA, upon which to build a model for classi cation of words within a text.