Winograd Schemas are sentences where a pronoun must be linked to one of two possible entities in the same sentence. Deciding correctly which entity should be linked was proposed as an alternative to the Turing test. Knowledge is a critical component of solving this challenge and Linked Data resources promise to be useful to that end. We discuss two example Winograd Schemas and related knowledge that can be discovered by manual search in WikiData, DBPedia, BabelNet, freebase, WordNet, VerbNet, and the Component Library. We nd that these resources are di cult to leverage because (i) they mix named entities with expert jargon and generic ontological knowledge, (ii) annotation tools are lacking, and (ii) commonsense knowledge is kept implicit.
The Winograd Schema Challenge (WSC) [20, 12] was proposed as a more practical alternative for the Turing Test. An example is the following Winograd Schema (WS): [The sculpture]a rolled o [the shelf ]b because [it]X wasn't anchored. [The sculpture]a rolled o [the shelf ]b because [it]X wasn't level. (ScAnchor) (ScLevel) Each sentence in such a schema poses a coreference ambiguity problem between the phrases marked in square brackets. This example has two candidate solutions: X = a and X = b. An important property of WSs is, that the correct solution is di erent in both sentences, and that the sentences di er only by a single word (`level' vs. `anchored'): the correct solution of (ScAnchor) is X = a while for (ScLevel) it is X = b. Because of this property it has been argued, that purely statistical approaches will be insu cient for beating the WSC and that methods of (symbolic) knowledge representation and reasoning will be necessary [12].
This work has been supported by Scienti c and Technological Research Council of Turkey (TUBITAK) Grant 114E777.
Reasoning requires knowledge, the biggest repository of knowledge is arguably the Internet, however it is mostly unstructured information. The Linked Data e ort [4] structures data in a way that it becomes machine readable, hence it can be used for automated reasoning. Therefore using the Semantic Web as a knowledge resource for tackling the WSC is highly suggestive. But knowledge is more than data, so how far can we get with existing resources?
In this work we discuss two examples of WSs and attempt to resolve them using data repositories typically considered part of the Semantic Web and other resources. We show that repositories like WikiData, DBPedia, BabelNet, and freebase are necessary but not su cient by themselves: they contain mostly taxonomic knowledge and (historical) facts about named entities.
On the contrary, most existing Winograd Schemas [7] do not refer to historical events or entities, they can be understood out of the blue (i.e., without additional context) using Commonsense knowledge [14] that is shared by humans because they live in a similar world as other humans.1 In the above WS such knowledge is that anchoring/ xing an object (usually) prevents its movement.
But is such Commonsense knowledge represented in Semantic Web resources?
In this work we rst outline how to perform reasoning, following the idea that many schemas can be resolved using correlation [2].2 Then we give an | in our opinion | representative part of background knowledge obtained from existing resources by manual search
Our contribution is to point out potential use of Semantic Web resources towards tackling Winograd Schemas and to show problems that become apparent while doing so. The main issues we point out are as follows.
Misinterpreting the topic of a sentence causes annotation of many wrong entities, in particular if knowledge about named entities or expert jargon is preferred over generic concepts. Therefore tools that annotate text with the correct links (concepts or entities) are crucial. A high level of detail in textual descriptions, or a varying detail of such descriptions, can mislead reasoning. Missing Commonsense knowledge is a limiting factor, but annotating Web content with links to common vo1We will disregard the question where commonsense knowledge ends and where culture-dependent knowledge starts. 2Note that such reasoning need not be based on symbolic logic, we can similarly envision to realize it statistically. cabularies has the potential to enable future work that mines such knowledge from the (annotated) Web.
Why is correlation [2] a possibility for resolving coreferences in the WSC?
If we split (ScAnchor) and (ScLevel) into two sentences, including all possible resolutions of the pronoun, we obtain the following simple statements.
In the original schema, the word `because' raises an
expectation in the reader, namely that the second sentence is
a plausible reason for the rst sentence. One way to handle
this plausibility is to reduce it to correlation: For example
by checking whether (
But what kind of correlation do we use in this case?
The topic of all three sentences is related with movement
or its impossibility. Hence the correlation can be about
properties within that topic. (
Note that we deal with (ScLevel) in the next section. Our second example is the following schema.
(EnvyBecause) (EnvyAlthough) [Pete]a envies [Martin]b because [he]X is very successful. [Pete]a envies Martinb although [he]X is very successful.
Pete envies Martin.
Pete is very successful.
Martin is very successful.
Note that this time the same three sentences are used for resolving (EnvyBecause) and (EnvyAlthough). The only difference between (EnvyBecause) and (EnvyAlthough) is the connective between the sentences: `because' (again) raises an expectation of positive correlation, however `although' raises an expectation of an exception: although we would normally assume that Pete is not envious (because he is successful), he actually is.
Intuitively, the object of `to envy' is correlated with being successful, and the subject of `to envy' is correlated with not being successful. Therefore the solution for (EnvyBecause) can be found by maximizing correlation, while we need to minimize correlation for (EnvyAlthough).
Theoretical Justi cation. The expectation of correlation
is explained by linguistic theories about discourse structure
(
We now investigate how to obtain the required knowledge from resources integrated into the Semantic Web and from similar resources built by other communities. We consider WikiData [19], which is a language- and presentation-independent annotated data backend for Wikipedia, DBPedia [10], which contains RDF triples extracted from Wikipedia infoboxes, freebase [5], which is a community-built knowledge graph repository, and BabelNet [17], which connects several Wikipedia projects with the linguistic resources WordNet [15]. Furthermore we use the linguistic resources WordNet, VerbNet [9], and the Component Library Clib [3], which is a Commonsense knowledge resource.
For WikiData, freebase, WordNet, VerbNet, and Clib, we lookup single words. We also search in the Falcons Semantic Web search engine [18] which performs search and ranking results in most of the above resources.
Additionally we perform annotation of the whole schemas using the annotation engines DBPedia Spotlight3 [6] and the Babelfy [16] annotation engine for linking to BabelNet.4 3.1
(ScAnchor) can be disambiguated if we have knowledge that
(i) sentences (
We will take for granted that we have linguistic knowledge and mechanisms that allow us to identify subject and object of `rolled' and how to handle the predicate `wasn't': these are research areas on their own.
Useful Knowledge. WikiData has de nitions for `sculpture' as well as `shelf', classifying them as `three-dimensional work of art' and as `furniture', respectively, and both are a subclass of `arti cial physical object'.
VerbNet contains an entry classifying `roll o 'as `move'.
Babelfy produces several correct annotations: `rolled' is linked to `roll' which is a kind of `move'; `anchored' is linked to `anchor' which is a kind of ` x' which is a kind of `attach'.
Falcons nds WordNet and WikiData concepts for `sculpture', `to roll', and `shelf', ranking correct data high but not in rst place. Results for `anchored' are not helpful, but searching for `anchor' reveals the correct WordNet entry.
While attaching intuitively prevents moving, this knowledge is cannot be found easily.
Clib contains nearly enough information to infer this causal relation: `Move' has a precondition that the object to be moved does not have the property `Be-Restrained' which inherits from `Be-Obstructed'. The problem is that obstruction or restraint is caused by `Move' and the separate 3http://spotlight.dbpedia.org/rest/annotate 4http://babelfy.org/index event `Attach' is not causally related to restraining (`BeRestrained' contains the linguistic annotation ` xed' but xing and attaching are modeled as di erent concepts).
While it seems that enough knowledge is available, automatically linking that knowledge in the right way is not trivial. Next we show that there is also additional knowledge that could be linked and that is counterproductive towards the goal of reasoning about the intuition of our example. Misleading knowledge. freebase provides several entries for `sculpture', mostly about the art form of sculpturing, and few about (very speci c) physical objects. The closest helpful entry is `statue' which, according to its de nition, `. . . is a sculpture representing one or more people or animals . . . .' For `shelf' the rst hit is the correct entry, however in its de nition we nd that `A shelf is a at horizontal plane . . . to hold items . . . It is raised o the ground and usually anchored/supported on its shorter length sides by brackets.'
The de nition of sculpture (or statue) does not contain information about anchoring, hence a system that heuristically evaluates correlation will conclude that shelves are more likely to be anchored than sculptures. Therefore (ScAnchor) will be disambiguated in the wrong way, even though we have only truthful evidence. The problem is that the heuristic fails because (by chance) one de nition contains misleading information. (Actually searching the web yields many do-it-yourself forums with information about proper ways to anchor shelves in the wall, and not nearly as much for anchoring statues, hence correlation of anchoring seems to be higher for shelves than for statues, however we need to consider correlation between rolling an object and not anchoring that object.)
Babelfy provides the following annotations: `sculpture' is a `three-dimensional gure' which is a `shape' which is a `mathematical object', moreover `sculpture' is subclass of several art forms; `shelf' is a `furniture' which is a `decorative art' which is a `perceptible object', moreover `shelf' is a `support' which is a `machine' and a `tool'. In summary `sculpture' is classi ed as an intangible abstract concept, and `shelf' is classi ed also as a tool which can be misleading.
DBPedia Spotlight annotates `sculpture' with a particular species of sea snail, `shelf' with `Shelf life' (shelf here means shallow coastal area of the sea). While `rolled' and `anchored' are not associated with any entity, a search on the web reveals more potentially misleading information: there are `roll anchors' for anchoring ships in the shelf zone, moreover rolling is a speci c movement of ships induced by wind.
While the presence of this (expert jargon) knowledge in DBPedia is no problem, its usage is a problem: it should be linked only when signi cant evidence suggests that the text is about anchoring ships near the coast. It seems that Babelfy performs better than DBPedia Spotlight, although this can be a random e ect due to the limited number of examples we are looking at. While Falcons contains a possibility to choose between `object' and `concept' in the search, this does not seem to provide the required distinction: expert jargon is always contained in search results.
Note that we mainly discussed `sculpture', because for other content words, misleading knowledge cannot be found to such an extent. Due to the amount of available knowledge, separating useful from irrelevant knowledge is crucial. 3.2
For (ScLevel) we do not need correlation: if we can show that
among the two candidates (
To show this we require the following knowledge: (i) a shelf is usually a at entity, (ii) a sculpture is usually not at, and (iii) `level' is a potential property of at entities.5 Useful knowledge. freebase's entry for `shelf' states `A shelf is a at horizontal plane [. . . ]' and most of its entries for `level' refer to `horizontal' or `plane' in their de nitions. This allows us to infer that `level' is a more likely property of `shelf' than of `sculpture', yielding the solution X = b. Misleading knowledge. DBPedia Spotlight wrongly links `level' to `deck of a ship', again using the wrong topic area.
Babelfy wrongly links `level' to `level of a game'.
Falcons provides many pages of search results, but the order of results is misleading: the rst ve entries are related with `level of visibility'. 3.3
(EnvyBecause) and (EnvyAlthough)di er only in the rhetorical relation, therefore the same knowledge is relevant. Useful knowledge. DBPedia Spotlight correctly links `envies' to an emotion which `occurs when a person lacks another's superior quality, achievement, or possession and either desires it or wishes that the other lacked it'.
VerbNet does not contain an entry for `to envy' but for `success' which is a potential property of humans according to several of its free-text de nitions.
WikiData provides as rst results for `envy' the same entry as DBPedia Spotlight. Additionally for `success' it contains an entry for `achievement of a goal' and one for `victory'. (For `successful' there are only entries related to arts pieces.)
Babelfy links `envies' to `to envy' which is a subclass of `to admire' which is a subclass of `to think', moreover `successful' is linked to the entry of the same name but this entry does not contain any classi cation.
Falcons provides useful results, linking `successful' to the corresponding WordNet entry, and `to envy' to `jealousy'.
These pieces of knowledge can be su cient for our purpose, in particular the de nition of `to envy' in connection with linking success to `achievement'.
Misleading knowledge. However there is also misleading and missing knowledge.
freebase provides many results for `envy', `envies', `success', and `successful', most of them names of arts pieces.
DBPedia Spotlight links `Pete' and `Martin' to TV programs and characters, respectively. Note that interpreting these names is not useful for disambiguating this schema.
Clib does not contain any information about envy or success, it does not even contain the concepts of feeling, emotion, attitude, or thinking.
In summary, (EnvyBecause) and (EnvyAlthough) can be disambiguated automatically with existing resources, if we manage to ignore irrelevant search results. 4.
Authors of content in the Web rarely describe how the world works, mostly they give an e cient account of what happened, why, when, and how it happened. Such an e cient 5The words `usually' and `potential' point out that this knowledge is default knowledge and can be defeated by more speci c knowledge (to account for atypical cases). use of natural language omits certain content that can be inferred by the reader, therefore computers must interpret natural language to reason with it. Similarly, if data is published in non-annotated unstructured form, humans can often guess which part of that data is a name or a location. Computers cannot do that, therefore the Linked Data initiative aims to annotate data with type information in common ontologies and information about its relation to other data.
Linked Data, as the name indicates, is about data, annotated with its type and further meta information. However, various Semantic Web resources do not only contain data about named entities and events, they also contain a bit of (mostly taxonomic) commonsense knowledge. This knowledge is used to organize the meta information and is often mixed with the other knowledge.
Using Linked Data for reasoning requires to connect it with additional commonsense knowledge that is currently not contained in existing resources. Moreover, connecting Linked Data to natural language texts (i.e., with unstructured data), requires annotation tools like Babelfy or DBPedia Spotlight that annotate words and phrases in a text with appropriate URIs to resources in the Semantic Web. Such tools are often based on (or supported by) machine learning.
In this work we saw that only Babelfy provides reasonable automatic annotations, so its ranking scheme seems to be superior to DBPedia Spotlight. Babelfy distinguishes between knowledge about concepts and named entities, and internally uses coherence measures. Our examples of misleading knowledge consider only the correct POS, however NER detection could help to separate between common nouns and names. Falcons distinguishes between `concepts' and `objects', however it returns expert jargon in both result types and the t df ranking [18] produces misleading search results.
About the issue of expert jargon (e.g., `sculpture' as a certain type of mollusc) we note that already in the CYC project [11] there were `microtheories' for separating more speci c from more generic knowledge. However, in none of the resources discussed in this work we found a method of distinguishing between these types of knowledge.
RDFa allows Web authors to annotate parts of their website content (i.e., words or phrases) with type information and links to common vocabularies such as WikiData. This eliminates the need for disambiguation and can make machine reading more feasible on these websites. Therefore we think that widespread usage of RDFa could be a crucial enabler for mining commonsense knowledge from the web, in e orts similar to [8].
We conclude that Linked Data can be used for reasoning, but we need better tools that automatically annotate a given text with the most suitable Semantic Web URIs. Additionally, only if we manage to integrate Linked Data with commonsense knowledge, we can use this data as knowledge.