This paper reports on the participation of the system squall2sparql in the QALD-3 question answering challenge for DBpedia. squall2sparql is a translator from SQUALL, a controlled natural language for English, to SPARQL 1.1, a standard expressive query and update language for linked open data. It covers nearly all features of SPARQL 1.1, and is directly applicable to any SPARQL endpoint.
Data (LOD), and covers nearly all features of SPARQL 1.1, for both queries and
updates. The advantage of CNLs is to provide a natural language syntax while
retaining the precision and lack of ambiguity of formal languages like SPARQL [
The objective of our participation in the QALD-3 question answering task was to evaluate the capability of SQUALL to express English questions in a natural and precise way, and the capability of squall2sparql to correctly translate SQUALL questions into SPARQL. Therefore, the measures of precision and recall are not enough to evaluate our approach, and the naturalness of SQUALL questions has also to be assessed. 2
Our system squall2sparql, as its name suggests, is a translator from SQUALL
to SPARQL. Given a SQUALL sentence, the system rst translates it into an
intermediate logical representation using a Montague grammar [
We now brie y describe SQUALL as a controlled natural language. Content words are nouns, verbs, and proper nouns. Nouns (e.g., Person) and intransitive verbs are interpreted as class URIs or built-in unary predicates. Relation nouns (e.g., child, birthPlace) and transitive verbs (e.g., matches) are interpreted as property URIs or built-in binary predicates. Proper nouns (e.g., res:Germany) are interpreted as entity URIs. Grammatical words are determiners (e.g., a. the, every, at least 10), auxilliary verbs (e.g., is, has), prede ned verbs (e.g., shares, relates), imperative verbs (e.g., give me, return), comparative and superlative adjectives (e.g., higher, later, most), aggregation nouns and adjectives (e.g., number, average), interrogative determiners and pronouns (e.g., what, which, how many), coordinations (e.g., and, or, not), and others. Boolean coordinations can be applied to most types of phrases: noun phrases, verb phrases, relative propositions, and sentences. Every proposition has a subject, a verb, and also an object if the verb is transitive. A sentence can be an open question (e.g., starting with Which or What), a closed question (e.g., starting with Whether or using auxiliary verbs and inversion), an imperative-style question (e.g., starting with Give me or Return), or an assertion (for updates). 2 Web forms, examples, and source code can be found from the SQUALL homepage: http://www.irisa.fr/LIS/softwares/squall.
Complete examples of SQUALL questions are given in the following sections.
More details and examples about the SQUALL language and its translation to
SPARQL can be found in previous papers [
The use of squall2sparql in QALD-3 assumes that English questions are reformulated in SQUALL, i.e. Controlled English. Its syntax is regular and sufciently similar to English so that it can be learned without too much e ort. Many examples are available on the SQUALL's Web page. Its vocabulary (i.e., nouns and verbs) is made of URIs because there is so far no lexical treatment in squall2sparql. This has the obvious drawback that SQUALL queries look less natural, and that URIs have to be known or discovered manually. However, the advantage is that squall2sparql is directly applicable to any LOD dataset, because no linguistic resource is required. If such linguistic resource is available, like those produced by the lexicon task of the QALD-3 challenge, it could be combined in SQUALL, using words instead of URIs.
From the training phase, we already learned some of the DBpedia vocabulary, and other URIs were found manually with Google searches and DBpedia browsing. We spent on average a few minutes per question for the reformulation phase. The automatic translation to SPARQL takes much less time than SPARQL query evaluation, and is therefore not an issue.
For illustration purposes, we list below a few original questions along with their SQUALL reformulation. The full list of SQUALL questions can be found in the o cial results of the QALD-3 open challenge.
1 Which German cities have more than 250000 inhabitants?
Which Town that has country res:Germany has a populationTotal greater than 250000? 2 Who was the successor of John F. Kennedy?
Who is the successor of res:John F. Kennedy? 3 Who is the mayor of Berlin?
Who is the leader of res:Berlin? 4 How many students does the Free University in Amsterdam have?
What is the numberOfStudents of res:Vrije Universiteit? 5 What is the second highest mountain on Earth?
Which Mountain has the 2nd highest elevation? 7 When was Alberta admitted as province?
What is the dbp:admittancedate of res:Alberta? 9 Give me a list of all trumpet players that were bandleaders.
Give me all Person-s whose instrument is res:Trumpet and whose occupation is res:Bandleader. 12 Give me all world heritage sites designated within the past ve years.
Give me all WorldHeritageSite whose dbp:year is between 2008 and 2013. 15 What is the longest river?
Which River has the highest dbp:length?
Out of the 99 questions, we got the right answers for 80 questions (including the three OUT OF SCOPE questions), and partial answers for 13. Recall is 0.88, precision is 0.93, and the F-measure is 0.90. Errors come: { from heterogeneity in data (12 errors, questions 1, 6, 17, 19, 29, 33, 39, 60, 63, 72, 93, 96), { from the user reformulation in SQUALL (2 errors, questions 14, 43), { from squall2sparql (2 errors, questions 49, 59), { from the gold standard (2 errors, question 16, 75), { from the endpoint (1 error, question 92).
Looking at heterogeneity errors in detail, it appears that most of them could be solved simply by: either adding generic super-properties in the DBpedia ontology, or by expanding common words (e.g., location, date) into UNION graph patterns. For example, in question 39 \Give me all companies in Munich.", the implicit relation \has location" can be translated in any of the three RDF properties: dbo:location, dbo:headquarter, dbo:locationCity. This explains why our reformulation in SQUALL \Give me all Company-es whose location is res:Munich." has recall 0.6 only (the default pre x was used for DBpedia ontology, so that location stands for dbo:location). If location, or another property, was de ned as a super-property of the other properties, the same SQUALL question would have recall 1. Alternatively, assuming linguistic knowledge, the word \location" could be mapped to the graph pattern { ?x dbo:location ?y }
UNION { ?x dbo:headquarter ?y }
UNION { ?x dbo:locationCity ?y } where ?x and ?y respectively stand for the subject and object of the relation. Such graph patterns could easily be exploited in the translation from the intermediate representation to SPARQL without the need to change the SQUALL language and its parsing.
Another problem related to heterogeneity is that some expected domain and range axioms are not veri ed in some cases. For example, in question 19 \Give me all people that were born in Vienna and died in Berlin.", 2 out of the 6 expected answers are not instances of the class Person. This is why our reformulation \Give me all Person-s whose birthPlace is res:Vienna and whose deathPlace is res:Berlin." missed 2 answers, even though it is arguably equivalent to the original formulation.
The errors coming from the user reformulation of questions are due to misspelling or misunderstanding of URIs. In question 14, \res:Prodigy" was used instead of \res:The Prodigy". In question 43, the property \dbp:breed" was used in the wrong direction.
The errors coming from squall2sparql are due to an incorrect translation of the special verb \share". For example, Question 49 \Which other weapons did the designer of the Uzi develop?" was reformulated as \Which Weapon shares the dbp:designer with res:Uzi?", which returns \Uzi" itself as an answer. Another possible reformulation is \Which Weapon has the same dbp:designer as res:Uzi?", but it exhibits the same error.
The error from the endpoint is because the BIND construct of SPARQL is not (yet) supported by the QALD-3 endpoint. It is possible to write the SPARQL query to avoid it, but squall2sparql relies on it to simplify the translation from SQUALL. Note that the correct answers are returned when using the o cial DBpedia endpoint.
Regarding the naturalness of SQUALL sentences, most of them are not much longer than the original ones, and can be understood without learning SQUALL. Most di erences fall into three categories: 1. reformulating the question to make it agree with SQUALL's grammar, 2. replacing a word by another (e.g., movie ! Film), 3. making explicit some relations (e.g., \is a chemist" ! \has profession res:Chemist"). 5
We here discuss a few directions to go in order to improve the usability and performance of our approach.
Lexicons. The data independence of SQUALL is valuable as it allows to query all LOD with neither preparation nor linguistic resources. However, when such linguistic resources are available [12], it is a shame not to use them, as they could improve recall, and make SQUALL sentences much more natural at the lexical level. The useful format of lexicons for squall2sparql would be mappings from words to graph patterns. Nouns and intransitive verbs would be mapped to RDF classes or mono-dimensional graph patterns (one free variable), and relational nouns and transitive verbs would be mapped to RDF properties or bi-dimensional graph patterns (two free variables). Such lexicons may be extended to adjectives, adverbs, and prepositions by corresponding extensions of SQUALL's syntax. A candidate format of lexicons is lemon (lexicon model for ontologies) [10].
Multilinguality. This is an aspect of the QALD-3 challenge that we did not address. A priori, it su ces to de ne a di erent concrete syntax for each language, keeping unchanged the intermediate representation. However, for some languages, it may be more di cult than for English whose morphology is less complex than many other languages.
Guidance. Writing questions in a controlled natural language is easier than in a
formal language like SPARQL. However, it is still error-prone, and may be
frustrating for users. A possible solution that has already been used for CNLs is an
auto-completion mechanism that suggests possible completions for the sentence,
based on the grammar (e.g., Ginseng [