While more and more semantic data is published on the Web, the question of how typical Web users can access this body of knowledge becomes of crucial importance. Therefore there is a growing amount of research on interaction paradigms that allow end users to profit from the expressive power of Semantic Web standards while at the same time hiding the complexity behind an intuitive and easy-to-use interface. These paradigms range from keyword search, faceted browsing and exploration to natural language question answering. Keyword-based semantic search (e.g. provided by Swoogle1 and Watson2) suffice for simple information lookup. However, keyword queries constitute an ambiguous and impoverished representation of an information need and do not fully exploit the expressive power of Semantic Web datamodels and query languages. Explicit formal queries, on the other hand, allow the user to exploit the full power and expressiveness of the Semantic Web standards. But they require schema knowledge and query language expertise, thus are not suitable for naive users. Faceted browsing interfaces (e.g. Ontogator [16], mSpace [17], and BrowseRDF [14]), in contrast, do not require domain knowledge or expertise in Semantic Web languages, rather they allow users to navigate through the dataset and thereby explore its content. However, faceted browsing is often domain-dependent. Also, in case the user has a clear information need, it is tedious to search for the answer manually, not knowing where exactly to look for it. Similarly, graph visulatization (e.g. IsaViz3, [13]) provides an intuitive access to understanding semantic data, but often does not scale to large datasets and is not suitable for searching for an answer to a particular question. In order to precisely answer particular questions, a lot of research has been conducted on natural language interfaces (see for instance [4], [5, 6], [7], and [3]), which allow users to express arbitrarily complex information needs in an intuitive fashion. They combine progress in the areas of question answering from textual data [8] and natural language interfaces to databases [9]. The key challenges lie in translating these information needs into a form such that they can be evaluated using standard Semantic Web query processing and inferencing techniques, as well as in scaling question answering approaches to Linked Data. Despite different goals and different kinds of interaction, the main challenge involved in interacting with Linked Data is the same for all approaches: dealing with a heterogeneous, distributed and very large set of highly interconnected data. The availability of such an amount of open and structured data has no precedents in computer science and approaches that can deal with the specific character of Linked Data are urgently needed.
The main goal of the challenge was to get a picture of the strengths, capabilities and current shortcomings of question answering systems, as well as to gain insight into how question answering approaches can deal with the fact that the amount of RDF data available on the Web is huge and that this data is distributed and heterogeneous with respect to the vocabularies and schemas used. Five question answering systems participated in the test phase of the challenge, all reporting on the DBpedia question set (where Alexandria used German translations of the questions and extracted the answers from Freebase, thus suffering from data mismatches). The results of our online evaluation are given in Table 1. Alexandria, SemSeK, and QAKiS are described in papers in these proceedings; MHE was developed by Marek Ciglan5 at the Institute of Informatics at the Slovak Academy of Sciences.
total answered right partially right precision recall f-measure SemSeK 100 Alexandria 100 MHE 100 QAKiS 100
With the ILD workshop and the accompanying open challenge we hope to establish a series of workshops that aim at coupling current research on interaction paradigms for accessing Linked Data with open challenges that benchmark question answering approaches and thereby evaluate their success in providing an easy and intuitive interface to the Semantic Web.
May 2012 5 http://ups.savba.sk/~marek/ Christina Unger Philipp Cimiano
Vanessa Lopez Enrico Motta
Paul Buitelaar
Richard Cyganiak (Organizing Committee ILD 2012)