Different aspects and approaches to commonsense reasoning have been investigated in order to provide solutions for the Winograd Schema Challenge (WSC). The vast complexities of natural language processing (parsing, assigning word sense, integrating context, pragmatics and world-knowledge, ...) give broad appeal to systems based on statistical analysis of corpora. However, solutions based purely on learning from corpora are not currently able to capture the semantics underlying the WSC - which was intended to provide problems whose solution requires knowledge and reasoning, rather than statistical analysis of superficial lexical features. In this paper we consider the WSC as a means for highlighting challenges in the field of commonsense reasoning more generally. We begin by discussing issues with current approaches to the WSC. Following this we outline some key challenges faced, in particular highlighting the importance of dealing with pragmatics. We then argue for an alternative approach which favours the use of knowledge bases where the deep semantics of the different interpretations of commonsense terms are formalised. Furthermore, we suggest using heuristic approaches based on pragmatics to determine appropriate configurations of both reasonable interpretations of terms and necessary assumptions about the world.
The Winograd Schema Challenge
it was made of [steel/styrofoam]. What was made of [steel/styrofoam]? Answers: The ball/the table.1 (1) The pronoun ‘it’ refers to either the ball or the table depending on whether ‘steel’ or ‘styrofoam’ is used. In both cases the syntactic structure remains the same and, supposing that clear semantic preferences relating ‘steel’ and ‘crashing through things’ or ‘styrofoam’ and ‘being crashed through’ cannot be easily learned from mining a large corpus, it is hoped that any system which resolves the pronoun must use some sort of genuine understanding.
In the literature discussing the WSC and its motivation as
a benchmark we see example reasoning processes
incorporating detailed semantics of the language involved
In order to further the symbolic approach we investigate
some problems faced, mainly pragmatics. It is hoped that this
sort of analysis helps to shed light on what kind of reasoning
is needed where; and that heuristic methods will remove a
large portion of the burden of reasoning about natural
language. Along similar lines, a partial solution is provided in
In this paper we first explore what kind of reasoning capabilities we expect a system to display when solving the WSC and we analyse how some of the proposed approaches compare to this. We then consider some key challenges for solving the WSC using reasoning we consider appropriate; in particular, that pragmatics and context are very difficult to capture and semantics are hard to formalize due to vagueness. Finally, we show how pragmatic considerations can help in solving the WSC, specifically we consider how prototype theory and heuristic methods can be used to support symbolic approaches.
We first consider the example above (1) given in
Levesque then considers what should be the outcome if
we change styrofoam to XYZZY, where XYZZY is some
material that we are given some facts about, one of the facts
being ‘It is ninety-eight percent air, making it lightweight and
buoyant’. Given this fact, humans would be able to reason
that the table is made of XYZZY. This is a part of intelligent
behaviour that we would like to replicate, and is clearly
dependent on having and being able to reason about detailed
knowledge. Further, it has been suggested as a possible
extension to the test to add a requirement for the solution to provide
a simple explanation of its choice
The WSC was conceived as a new benchmark for artificial
intelligence; as such, we hope that solutions to the WSC will
provide tools for tackling a broader range of question
answering tasks and commonsense challenges. In this way, solutions
to the challenge should display versatility as well as making
advances in the WSC specifically, thus representing genuine
progress towards truly intelligent machines. Solutions which
are over-specific to the WSC and only provide insight into
this narrow set of coreference resolution problems are not
likely to be ‘engaging in behaviour that we would say shows
thinking in people’
The situations described in Winograd sentences (WS) are
generally common/normal occurrences; however, it is
desirable for AI systems to be able to reason about out-of-place
objects and strange scenarios. The ability to do this displays
a genuine understanding of what is going on. Levesque gives
the example ‘Can a crocodile run a steeplechase?’
Having briefly considered the kind of solutions we are aiming for, we now look at how some existing approaches compare to this.
Since the inception of the WSC there has been some
theoretical discussion on the purpose of the challenge
Formalizing the necessary aspects of reasoning to tackle the WSC (spatial, temporal, causal, epistemic, ...) and integrating them in one system is notoriously hard. Therefore, it is not surprising that the space of genuine proposed solutions is sparse, and that existing approaches are mostly based on statistical methods, that circumvent the need for a precise understanding of the semantics of the schemas by learning likely answers from text corpora.
In this section we analyse some of the solutions proposed along these lines. We consider both their performance and success on the challenge and also their achievements and relevance for broader commonsense reasoning, which is the ultimate aim of the WSC as a benchmark.
Machine learning methods for anaphora resolution have been
used extensively over the past two decades
The team that came first in the 2016 WSC challenge2 used
‘Commonsense Knowledge Enhanced Embeddings’
Though achieving a good performance on the challenge, it would be down to chance whether it correctly answers the XYZZY problem given by Levesque, whether it could be used to solve the crocodile-steeplechase problem, or in future how it could be developed to explain how it comes to the conclusion.
Work by
Rather than applying reasoning to knowledge, these techniques are geared towards mining what we may call commonsense rules. We discuss the nature of such rules in the following section.
It is clear that, in the WSC, it appears possible to resolve pronoun ambiguity through an appeal to normality — heavy things cannot be lifted, younger people are fitter, useless objects go in the bin while useful tools are kept in storage etc... Hence, a large part of the suggested approaches to the WSC have been about ways of finding and/or incorporating such ‘commonsense rules’. We believe, however, that this is a rather crude view of commonsense reasoning and outline some problems of these approaches below.
One proposed approach is that we reduce some of the
implied causation in WSs to correlation
This is not to say that conventions do not exist or form an
important part of commonsense reasoning. Natural language
is full of conventions that we may rely upon to communicate.
For example, considering the sentence ‘Sam chopped down
the tree’ there is a default assumption that the chopping is
done with an axe. This kind of convention can be considered
as part of linguistic knowledge
The tactic for many approaches is to begin by learning
commonsense knowledge from large text corpora or by
integrating natural language knowledge bases. Part of the appeal
of this is that knowledge can be exploited without having to
translate between formal and natural language. However, the
methods for extracting commonsense knowledge from the
Web can be problematic. Language is used in an efficient way
and commonsense knowledge is often left implicit
Even if we were able to overcome some of the problems of mining commonsense, do we want to use reasoning that relies solely on these correlations and rules? Though they may be helpful for certain applications, the reasoning mechanisms need to incorporate less crude knowledge. Regarding the desire for versatility and considering some of the problems listed on the Common Sense Problem Page3, it is clear that this approach is over-specific to the WSC. It would also clearly be hard to mine relations between crocodiles and steeplechases in this way! Moreover, any explanation of the disambiguation given by such a system would not be very enlightening. Considering schema (1) with ‘steel’; explaining why ‘it’ refers to the ball by saying that ‘steel things are more likely to crash through things than to be crashed through’ is not a reasonable explanation. Even the ability to cite a salient property of steel like ‘steel is hard’ would be an important improvement.
The approaches outlined above at best only incorporate
shallow semantic features and do not appear to exhibit the
3www-formal.stanford.edu/leora/
commonsense/
kind of intelligent behaviour the challenge was designed to
test. We believe that, in order to carry out complex
inferences and really understand the world, some definitions of
the natural language in terms of more refined primitives is
often necessary. It is necessary to have genuine world
knowledge of entities, as well as their physical, social/historical
and functional attributes, as in
This section outlines some particular problems that need resolving in order to tackle the WSC and for commonsense reasoning systems more generally.
A large part of the complexity of the WSC comes from
pragmatic considerations. There are varying positions on the
definition of pragmatics
Semantic considerations are clearly essential but they are
generally not enough in order to reach a conclusion about
the disambiguation for a WS. This is an example of semantic
underdeterminacy — that from only considering the literal
meanings of terms in a sentence and not accounting for the
intended meaning, we do not obtain a truth-evaluable
proposition. For example, the sentence ‘Tom threw his school bag
down to Ray after he reached the top of the stairs’ does not
contain much information if we only consider the semantics.
We also need to consider the intention of the speaker and we
may infer this from the decisions the speaker takes regarding
the specific choice of language, what information is omitted,
what is left ambiguous, the phrasing of the sentence etc...
Indeed, Kempson argues that ‘the articulation of semantics
[does not alone] provide the full propositional content/logical
form/truth conditions expressed by a sentence’
To evidence this view, we can see that for many WSs wrongly disambiguating the pronoun does not necessarily violate world knowledge. For example, when dealing with the sentence:
The trophy does not fit into the suitcase because itx is too large1 (2) there are various interpretations of ‘large’ which give no definite disambiguation. If we imagine a trophy and suitcase to be vase-shaped, with a wide base, narrow stem and wide top, and that the trophy fits into the suitcase, it is possible that making the suitcase larger via a scale projection would make the trophy no longer fit. It is in part by making pragmatic considerations that we can assign appropriate interpretations to these terms and thus disambiguate the pronoun.
Moreover, even in the sentences where each term can be
precisely and appropriately defined we can still have
semantic underdeterminacy. Is it often the case that an utterance is
not totally explicit and leaves the reader to fill in the gaps
with available assumptions and inferences
In the next section we consider some particular examples of this sort of inference when addressing a WS.
When facing any WS there are multiple commonsense principles that apply which allow us to create an accurate model of the situation. What we aim to achieve is some guidance on how to choose these principles and when they apply. To this end we examine the following WS:
reached the [top/bottom] of the stairs. Who reached the [top/bottom] of the stairs? Answer: top: Tom. bottom: Ray.1 (3) We will use this example to help elucidate some of the complexities faced, including the initial position of objects and relevant objects.
The main idea of this sentence is that to throw something down to someone, that person must be below you. We then use the idea of what it means to be at the top of something, i.e. that if Ray is at the top of the stairs then he cannot be below Tom. This is however not as clear as it seems. Initial position It is possible that Tom is on some balcony above the stairs and waits for Ray to reach the top of the stairs before throwing the bag down to Ray. So why do we like the answer ‘Tom’? It appears we assume that Tom and Ray are initially in a similar location, or to be more precise, that they both have the same relation to any given landmark — in this case the stairs. Character x reaching the top of the stairs implies that x has moved upwards. Not given any information on the other character, y, we assume they have not moved and so x is likely to be above y.
Alternatively, x may have been walking along a corridor to reach the top of the stairs. In this scenario we have two locations to consider, the corridor and the stairs. We suppose that Tom and Ray are on the stairs or in the corridor. In this case it would make no sense for Ray to be at the top of the stairs, as then Tom would not be able to throw anything down to him (from the corridor or the stairs); so we suppose that it must be Tom who walks along the corridor to reach the top of the stairs and throw the bag down to Tom.
We appeal to a rule that in some narrative, unless we have reason to infer otherwise, characters are nearby/in the same place. This idea can be explained by Grice’s quantity maxim i.e. there is no pertinent difference in the positions of either Tom or Ray; if there were then the quantity maxim says it should be made known.
This rule however does not always hold. Imagine we replace ‘stairs’ with ‘swimming pool’:
the top of the swimming pool. Who reached the top of the swimming pool? Answer: Ray.
In this scenario x reaches the top of the swimming pool, breaking the surface of the water. x is then not in a position to throw something like a school bag downwards, as it is pretty hard to throw textile objects through water. Hence, we imagine that x is not Tom, but Ray, and that Tom must be stood somewhere above the swimming pool.
Relevant objects In general in the WSC to come to a conclusion we only need to reason about entities that are explicitly mentioned. In the school bag example we reason about the two characters in the narrative, Tom and Ray, the staircase and the school bag itself. Combining knowledge of actions like ‘throwing’ ‘reaching the top of’ etc.. with knowledge of these objects. In general then, we do not need to appeal to the existence of extra entities in order to come to a conclusion. This can also be explained by the quantity maxim, the sentence should provide the necessary objects for the reader to make sense of the sentence.
However, as previously discussed, certain words or
phrasings indirectly suggest the existence of certain entities, as
in the ‘Sam chopped down the tree’ example. We can in
part account for these entities by encoding into a lexicon
To conclude our discussion about assumptions about the world, we see that appropriate assumptions need to be made in order to reach the right conclusion. Further, we believe that, to varying extents, these kinds of considerations arise when analysing most WSs appearing in the collection maintained by Davis1. However, the assumptions are dependent on the specific situation and we need to discern somehow when the assumptions are appropriate. Deciding when to accept these assumptions should include pragmatic considerations. For example, it is lexical and semantic knowledge that suggest the existence of an axe in the sentence ‘Sam chopped down the tree’, however it is a pragmatic task to actually infer this. This motivates a heuristic process which incorporates pragmatics and gives preference to default assumptions, we will discuss this idea later.
An important issue is to recognize the level of semantics that
one believes is appropriate for a solution to the WSC. Our
discussion so far has motivated a detailed level of knowledge.
Further, there is evidence that, even for coreference
problems that would be considered easy with respect to the WSC,
incorporating shallow semantic features is not enough
On the one hand, due to the variety and scope necessary,
mining commonsense knowledge is appealing; however, as
previously discussed, the available methods and nature of
text corpora pose limitations to obtaining deep knowledge,
which is complex and commonly not explicit. On the other
hand, hand crafted knowledge bases such as CYC
Beyond the problem of its acquisition, it is well known that
commonsense knowledge is hard to formalize, particularly if
the required level of detail involves the semantics of natural
terms to be preserved. Vagueness and ambiguity are inherent
to natural language and, for that reason, it is problematic
to prescribe single strict interpretations to natural terms. To
illustrate this, consider the WS (3) and imagine the case of
a naive definition of a relation at the top of (x; y) x is on
y and for any z which is part of y, x is not below z. We see
that this fails for multiple reasons.
1. If Tom were one step below the very last one, it could still
be considered that he is at the top of the stairs, particularly
if Ray were well below him. We call it sorites vagueness
when there is a the lack of a clear threshold of application
of a term.
2. If we change ‘stairs’ to ‘building’ we might say that
Tom is at the top of a building because he is on the top
floor, rather than on the roof. In that case we are
shifting the interpretation of the predicate to something like
at the top of (x; y) z is the top part of y and x is on
z. There may also be many admissible interpretations of
what it means for z to be the top part of y. We call the
multiplicity of conceptually distinct interpretations of
natural terms conceptual vagueness. Further discussion on
the multiple interpretations of natural language terms and
their role in knowledge bases and ontologies can be found
in
Much of the work done in acquiring commonsense
knowledge circumvents vagueness in different ways, such as
using shallow semantics or microtheories that do not need to
be consistent with one another. Various theories, however,
have been proposed for dealing with vagueness. Fuzzy logic
So where do all these considerations lead us? In order to reach the kind of solution we desire, we must be able to deal with semantic underdeterminacy — part of which involves deciding when to use appropriate commonsense assumptions — and also make use of a vast amount of detailed knowledge while dealing with the associated problems of vagueness.
With these issues in mind, we now consider some avenues for further work.
In the previous sections we have highlighted how current approaches, regardless of their success in solving schemas, have provided limited support for the kind of intelligent behaviour that we would like to replicate. Here, in an attempt to account for some of the key challenges, we propose an alternative approach, favouring the use of knowledge bases where the deep semantics of the different interpretations of commonsense terms are formalised. Furthermore, we suggest using heuristic approaches based on pragmatics to determine, in the context of each particular schema, appropriate configurations of both reasonable interpretations of the terms and necessary assumptions about the world.
For this purpose we first motivate the use of prototypes for categories and relations and then develop how heuristic methods can provide a manageable way of using pragmatic knowledge for the disambiguation of WSs.
There is various work in pragmatics and cognitive science
highlighting the importance of using prototypes: in utterance
interpretation defaults are assigned before contextual and
pragmatic considerations are taken into account
Some of the previously discussed approaches work along
similar lines, using general commonsense rules and a notion
of correlation which appeal to a sense of typicality. However,
we believe that this should be more nuanced and that the deep
semantics of different interpretations should be preserved.
Hence, we propose an approach using ideas from prototype
theory
Much work has been done on how to pinpoint prototypical
members of categories, mainly using vector analysis or
conceptual spaces to find the centroid of a concept
Suppose we have a vague term, like ‘smaller’. How can we
decide on prototypical instances of this relation? Adopting
the supervaluation approach we would have a collection of
precise interpretations of its meaning. Following motivation
from
a
b 9s(s > 1 ^ b = Scale(a; s))
In this scenario, there are certainly pairs of objects that fall into all four categories (e.g. a sphere of radius 1 is smaller than a sphere of radius 2 in all the above senses). Hence, it would be appropriate to take the conjunction of all four definitions as a requirement for an instance to be considered a prototypical case of ‘smaller’. However, in certain scenarios it may be inappropriate to take the conjunction in this way, as some definitions may be conflicting. In this case different metrics can be proposed for selecting prototypes that satisfy most of the interpretations.
Finally, our main claim in this section is twofold. On the one hand, we consider that an understanding of typicality is necessary for commonsense reasoning — that by default we should consider prototypes. On the other hand, a process which can only reason over prototypical definitions is clearly flawed in many respects as it creates over-simplification. Humans often use context to help narrow definitions, for example defining ‘smaller’ in a particular way makes sense when talking about ‘fitting in’. Hence we believe that a good approach should reflect the diversity of possible interpretations of vague terms and that an engine based on pragmatics should guide the selection of appropriate alternatives when the prototype is not suitable.
In this paper we have discussed some approaches proposed
for the WSC relying on heuristic methods in different ways
Instead, as has been argued, we believe that a good solution to the WSC should disambiguate the pronoun by considering the most plausible configuration of the scenario described, and the process of finding it should incorporate rich syntactic, semantic and pragmatic considerations. However, although advocating deeper semantics and symbolic based approaches that allow for the kind of reasoning that we want (see section above), we propose that heuristic methods have a key role in the WS resolution: that of simplifying the space of possibilities and estimating reasonably good configurations of precisifications and necessary assumptions about the world.
As we have highlighted above in order to carry out
satisfactory reasoning we believe a system should give preference
to both commonsense assumptions about the world as well as
prototypical interpretations of the terms involved. These
however should only be preferences rather than concrete rules.
When to accept or reject these default assumptions requires
knowledge and pragmatic understanding. The ability for this
complex mix of pragmatics and world knowledge to
contradict itself means that possible solutions or configurations
of a described scenario are not unique. For example, when
discussing the issue of throwing a school bag in a
swimming pool above, the implausibility of throwing a school
bag through water outweighed the assumption of Tom and
Ray being in the same place. However, we may also consider
that the assumption of characters being in the same place
outweighs the usual interpretation of ‘throw down’ and ‘top’:
supposing Tom and Ray are both stood in the swimming pool,
we may interpret ‘throw down’ as ‘throw horizontally away
from the end of the swimming pool’ and ‘top of the
swimming pool’ to denote the end of the swimming pool. The
result would then be to disambiguate the pronoun as ‘Tom’
rather than ‘Ray’. This second interpretation is not wrong,
however when ‘throw down’ and ‘top’ are interpreted in their
usual way there is a plausible inference that Tom and Ray
are not both located in the swimming pool. This would then
be an example of a ‘conversational implicature’
Being able to leverage these kinds of inferences is an
important and difficult task in commonsense reasoning. Along
these lines, one avenue
To conclude this section, it is our claim that this use of heuristics is much more in keeping with the nature of the WSC. That what should be simplified in order to keep the task manageable is not so much the deep semantics of natural terms, but the process of selecting and integrating relevant interpretations and background knowledge in the particular context of the resolution of each sentence.
In this paper we have discussed the nature of the WSC as a benchmark, highlighting the shortcomings of several current approaches and providing motivation for a more detailed level of knowledge. We have also analysed some of what we consider to be key challenges, in particular drawing attention to the need to take account of pragmatic considerations. To begin addressing these challenges, we have suggested using frameworks able to support the detailed semantics of natural terms while accounting for its vagueness. Moreover, that their complexity can be manageable with the use of prototypes, which should be identified and used by default, and, finally, that heuristic methods can be used to incorporate varying semantic interpretations as well as assumptions about the world, which maintain the pragmatic principles of cooperative communication.
In conclusion, it is our view that, while heuristic mechanisms are necessary to deal with natural language and to reduce the complexity of commonsense reasoning, they should not be used to over-simplify the semantics of natural terms. Instead, we believe that applications along the lines of theoretical studies in pragmatics can play a significant role in the selection of good interpretations of natural terms and to enrich the provided descriptions of the world with the appropriate implicit knowledge.
Thanks to Brandon Bennett for helpful discussion and to the anonymous reviewers for their useful feedback.