We present here FANCY (FActivity, Negation, Common-sense, hYpernimy), a new dataset with 4000 sentence pairs concerning complex linguistic phenomena such as factivity, negation, common-sense knowledge, hypernymy and hyponymy. The analysis is developed on two levels: coarse-grained for the labels of the Natural Language Inference (NLI), that is to say the task of determining whether a hypothesis is true (entailment), false (contradiction), or undetermined (neutral) and finegrained for the linguistic features of each phenomenon. For our experiments, we analyzed the quality of the sentence embeddings generated from two transformerbased neural models, BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019b), that were fine-tuned on MNLI and were tested on our dataset, using CBOW as a baseline. The results obtained are lower than the performance of the same models on benchmarks like GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) and allow us to understand which linguistic features are the most difficult to understand.
Nowadays it has become more and more important to understand how much neural models applied to Natural Language Processing can understand about language features.
The probing task methodology is a simple but
effective approach to address this issue
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). task and then the representations are passed to a classifier. The performance of the classifier is evaluated with a dataset constructed to test the understanding of specific linguistic phenomena. If the classifier performs well, then it can be deduced that the neural embeddings have stored syntactic and semantic knowledge relative to those specific linguisitc phenomena.
One of the most widely used tasks for this approach is Natural Language Inference, in which the model must decide whether a hypothesis is an entailment, a contradiction, or simply neutral with respect to the premise.
Another approach consists in using benchmarks, i.e. datasets relating to various types of tasks, which are able, on the basis of the results obtained, to provide a general judgment on the performance of the model. Although benchmarks are very useful in evaluating the average performance of models, they are less effective in representing a wide range of linguistic phenomena that the models are able to deal with.
It is in this context that the challenge sets are
born, (also called adversarial sets, stress sets or
diagnostic sets) such as the SNLI (Stanford
Natural Language Inference)
For our research we built a diagnostic dataset that addresses key aspects of the human knowledge of lexical and compositional meaning, in order to test the deep semantic abilities of the latest computational models.
In this paper, we introduce FANCY, a dataset with 4,000 different hand-annotated sentence pairs with inference relation between them. In Section 3 we will briefly present the linguistic phenomena we decided to analyze. In Section 4 we will present the methods of dataset construction and in Section 5 we will discuss the results of the experiments conducted on FANCY. 2
Despite the progress made in recent years in the study of vector representations, it is still difficult to understand exactly what kind of linguistic properties they capture. The main approaches used in this area are probing tasks and diagnostic datasets.
A probing task is a classification problem
focused on the simple linguistic properties of
sentences
On the other hand, the importance of challenge sets is demonstrated by the fact that some traditional benchmarks have been equipped, in addition to the standard datasets, with challenge sets dedicated entirely to the NLI task. In fact, both GLUE and SuperGLUE have a diagnostic dataset, consisting of about 1000 pairs of manually constructed sentences involving 30 linguistic phenomena, including anaphora, factivity, negation, redundancy, hyponymy, etc. Similar challenge sets have been developed and described in the publications of Naik et al. (2018), a dataset in which the errors committed related to negation, antinomies and numerical reasoning are also investigated, Glockner et al. (2018), a challenge set created with particular reference to common knowledge and McCoy et al. (2019), an evaluation dataset that contains 30,000 specific examples on which neural models perform incorrect classifications, such as lexical overlap, subsequence, constituent, etc.
We selected four different kinds of linguistic phenomena to analyze: (1) the factivity, which address the truthfulness and the factuality of the events mentioned inside the phrases, (2) the negation, which in the English language can be expressed by several terms and situations, (3) hierarchical relations, i.e. semantic relations like hypernymy between a general term and a more specific term, and (4) the common-sense knowledge, which relates to the shared knowledge among speakers about events and facts concerning the real world. 3.1
Factivity is a linguistic phenomenon related to the truthfulness of events or concepts that are mentioned and expressed in a sentence: each event, based on the elements contained in the sentence, can assume a certain degree of certainty. a. John thinks it’s raining. b. John knows it’s raining.
When a speaker reads the non-factive verb think (a.), he understands that the event mentioned in the sentence (it’s raining) is just a possibility, while he deduces that it’s a fact when the factive verb know is used (b.).
When we talk of situations and events that
occur, have occurred or will surely occur in the
world, we present them as facts, while we usually
complete our tales using approximations in cases
where we do not know whether the things we are
talking about have actually happened and we are
not completely sure of their certainty. It is in this
context that we can observe the phenomenon of
factivity
Negation is a complex phenomenon that
characterize human language among all
Negation may be implicit, such as forget
meaning not remember, or affixal in such terms as
illegal or dis-agreement. It could be related to
quantifiers, in cases such as not all veggies are
tasty which contradicts all veggies are not tasty.
Some sentences can occur with double negative
markers, such as John called neither his father
nor his mother. Moreover, we can observe
contrastive negation
In many cases, the entailment relations can
occur not only at a sentence level but also at a word
level, if we consider the meaning relations that
exist between words: these kinds of relations are
deifned as lexical entailment
The concept of common-sense is hard to define
because it is strictly entangled with the way we
humans reason. Even though its definition is
controversial, we adopt here what Feldman called The
Standard View
Starting from the types suggested by LoBue and Yates (2011), we grouped common-sense into vfie macro-categories.
Causal Relations The categories in which the statement of the premise causes the hypothesis statement, e.g. the man had a bath entails the man got wet: here we can see how the fact that the man took a bath is the cause for him of being wet, hence there is a Cause/Effect relation. At the same time the fact that Mary was married to John automatically implies John was married to Mary, therefore the relation is of Simultaneous Condition.
Spatial Relations This category includes sentences that specify the physical position of an agent or an object with respect to someone or something, e.g. the fact that John is inside his home contradicts the sentence John is close to his home because: in this case, the spatial prepositions inside and close to cannot subsist at the same time.
Temporal Relations In this category are included texts that specify the time of en event with respect to someone or something, e.g. the fact that Julius Caesar was assassinated in 44 B.C. implies that Julius Caesar died before the birth of Christ. In this example the reader is supposed to know that B.C. indicates the birth of Christ, which is not trivial.
World Knowledge Relations All the categories that suppose a previous knowledge of the phenomenal or human world, for example all the sentences that suppose a geographic knowledge to be correctly tagged, e.g. Charles Dickens is buried in Westminster Abbey implies that Charles Dickens rests in London only if we know that Westminster is in London.
Other Relations In this set we put all the categories which are not included in the previous ones (e.g., arithmetic relations and mutually exclusive relations). For example, On the train, there are 340 passengers and 40 employees implies that On the train, there are 380 people because we know that if there are 340 + 40 people on the train then the total of the people will be 380.
1The categories that we know, such as the past, morality, science etc.
2He individuated six different sources of knowledge such as perception, memory,reasoning etc.
The dataset created for the experiments consists in 4000 pairs of sentences that were built manually by the authors, and this is because we decided to only include sentences that were as simple and clear as possible, in order to specifically focus on the linguistic features of the phenomena and to exclude other external factors of complexity that could have affected the performance of the neural models. For the construction of FANCY, we followed the diagnostic dataset schema provided with the SuperGlue3 benchmark for models evaluation, so all the data were inserted in a tabular framework and tagged with the following columns and labels.
columns of the dataset and indicate which sentence is the premise and which is the hypothesis.
FW and BW These two columns point out which one of the sentences should be used as the premise. For instance, if we find the sentence Granada is in Spain as the premise, and Granada is in Europe as the hypothesis in the database, the column FW (forward) considers the first as the premise and the second as the hypothesis while the columns BW (backward) considers the second sentence as the premise and the first as the hypothesis. In both of the columns we inserted the correct output: in the example above, the column FW would contain the tag entailment, because the first sentence implies the second one, while the column BW would contain the tag neutral because the second sentence does not imply the first one but does not contradict that either.
important for this study because it specifies which kind of feature regarding a particular phenomenon is represented by the sentence pairs.
Phenomenon Factivity Negation Hierarchical Common-sense
In Table 1 we can see that FANCY is composed of 1406 pairs of sentences that lead to an entailment, 1622 sets of neutral sentences and 968 contradictions.
In this section, we report the results of the experiments conducted using our dataset FANCY. We tested state-of-the-art models for NLI on the four different linguistic phenomena in the dataset. We selected bert-base-uncased-MNLI and robertalarge-mnli, both of which were finetuned on the MNLI dataset, and also a baseline model based on CBOW. The BERT and RoBERTa models are based on the Transformer architecture and are available on the Hugging Face web page.4 For what concerns the CBOW model, it was built using the tensorflow library, 5 with the word embeddings generated by GloVe pretrained with 840 Billions tokens, a vocabulary of 2.2 millions cased words and the resulting word vectors with 300 dimensions.6 The model was then trained on the MultiNLI dataset, so that all three models were trained on the same data.
Set MNLI Factivity Negation Hierarchical Common-sense
BERT 84.6 65.2 70.0 49.7 57.0
RoBERTa 90.2 74.6 82.0 60.4 68.0
CBOW 65.2 45.1 45.0 37.8 41.0
We tested every model on the examples of FANCY. The results in Table 5 show how the models struggled to address these kind of phenomena, if compared with the results on the MNLI. We can see that the baseline model performed quite poorly on all the subsets of our data. RoBERTa is the best performing one, even though it showed poor performances on linguistic phenomena such as common-sense and hierarchical relations while performing better on factivity and negations.
Label Possibly Fact Possibly Counterfact Fact Counterfact
In Table 3 we can see the errors that RoBERTa made in labeling examples regarding factivity. Most of the errors concern examples where the hypothesis gave place to a Possible fact and therefore should be tagged as neutral.
In Table 5 it is evident that the largest number of errors belongs to the Negation macro-category. In this case, the sentences contained elements such as quantifiers, modals, temporal adverbs and relative pronouns. Therefore, it appears that the comprehension of negation is more difficult when it is related to these elements.
In Table 6 we can see the errors made by the RoBERTa in dealing with hierarchical relationships. Most errors relate to Downward Monotone and Non-Monotone sentences.
In Table 7 we show only the most relevant categories for what concerns the errors committed by the model dealing with common-sense and common-knowledge.
As we can see, Temporal Relation, Preconditions and Spatial Relation are the most difficult categories for the model to label correctly.
As illustrative examples, in Table 4 are four sentences mislabelled by RoBERTa. We note that the
The man was born in 1950. No arrow hit the target.
Bob believes that Twin Peaks is the best tv show ever. All seagulls fly.
The man was 18 in 1968. Not all arrows hit the target. Twin Peaks is the best tv show ever.
All birds fly.
E C N N
C
E
E
E
Following a large number of recent studies
We have shown how the models can better handle phenomena such as factivity and negation if compared with the results obtained on hierarchical relation and common-sense knowledge. More in particular, we were able to stress how the stateof-the-art models struggle in dealing with linguistic phenomena that are essential for a correct understanding of the language such as the possibility generated by a statement, temporal relations between entities, the negation when there is a presence of temporal adverbs and relative pronouns and cases of downward monotone sentences. In future developments of our work we could use FANCY in order to perform fine tuning on Transformer-based models with the aim of increasing model performance and inferential capabilities. To do this it would be useful to produce more data, possibly annotated by different people, to test the models developed on different types of natural language. At the same time, the dataset could be implemented with other languages, such as Italian.