In this paper, we present the POMELO system developed for participating in the task 2 of the QALD-4 challenge. For translating natural language questions in SPARQL queries we exploit Natural Language Processing methods, semantic resources and RDF triples description. We designed a four-step method which pre-processes the question, performs an abstraction of the question, then builds a representation of the SPARQL query and nally generates the query. The system was ranked second out of three participating systems. It achieves good performance with 0.85 F-measure on the set of 25 test questions.
Biomedical knowledge is disseminated in knowledge bases which become
increasingly available on the Web. These knowledge bases usually focus on a given type
of information: chemical, pharmacological and target information on drugs in
Drugbank [
For instance, it has been shown that for querying the knowledge bases and
the Semantic Web data, the use of full and standard sentences is preferred to
the use of keywords, menus or graphs [
In relation with such research problems, the Question Answering over Linked Data (QALD-4) campaign proposes a task dedicated to the retrieval of precise biomedical information in linked knowledge bases according to questions in natural language. We present in this paper the methodology we propose to translate natural language questions in SPARQL queries and the system we developed for our participation to the challenge.
We start with the de nitions of the main terms used in the proposed presentation (Sect. 2). Then, we describe the semantic resources available and developed for enriching the questions (Sect. 3). The methodology and the system are described in Sect. 4. The evaluation of the system on the QALD-4 queries is presented in Sect. 5. 2
The main terms are used with the following meaning: Question The questions are the natural language expressions uttered by human users in order to formulate their information need.
Query The queries respect the SPARQL syntax and semantics. They are created automatically on the basis of (natural language) questions. Semantic type Semantic types are indicative of the word meaning. The semantic types are de ned from URI pre xes such as disease for rickets, drug for Cetuximab, etc.
Frame Frames are de ned as linguistic representations of RDF schema. Usually the frames contain one predicate and at least two elements with associated semantic types.
Linguistic annotation The linguistic annotations are obtained with the Natural Language Processing (NLP) tools and provide lemma, part-of-speech categories, and terms.
Semantic annotation Semantic annotations are obtained by tagging questions with semantic resources. The main challenge of the proposed work is to design the optimal methodology for an easy and reproducible rewriting of (natural language) questions in SPARQL queries. 3
Some resources used are provided by the challenge organizers (Sect. 3.1), others are collected and built speci cally for the challenge (Sect. 3.2) to support the method. We use these resources for rewriting the questions in queries. 3.1
Three datasets are provided by the QALD challenge:
{ Drugbank7 is dedicated to drugs [
On the basis of the RDF triples, we build frames from the RDF schema where the RDF predicate is the frame predicate, and subject and object of the RDF triples are the core frame elements. This also includes the OWL sameAs triples. Several types of frame entities are isolated: { As indicated, subject, object and predicate become semantic entities. They may occur in questions: in this way, the frames are the main resource for rewriting questions in queries. 7 http://www.drugbank.ca 8 http://www.drugbank.ca/documentation 9 http://diseasome.eu 10 http://sidee ects.embl.de { The vocabulary speci c to questions is also built. It covers for instance aggregation operators, negation and types of questions. { RDF literals, issued from named entity recognizer or term extractor, complete the resources. The RDF literals are detected with speci cally designed automata that may rely on the source knowledge base documentation. These entities are associated with the expected semantic type, which allows creating the queries and rewriting the RDF triples in the SPARQL queries. In that respect, we can consider IRI, strings, common datatype or regular expressions when literals are expected.
Most of the entities are considered and processed through their semantic type, although some ambiguous entities (e.g. interaction or class) are considered atomically. For these, the rewriting rules will be applied contextually to generate the semantic entities corresponding to the frames (see Sect. 4.2). When using the queries, the semantic types are variables and are used for connecting the edges of queries. 4
We design a four-step process based on NLP methods, semantic resources and RDF triple description (see Fig. 1): 1. We pre-process the questions in order to enrich them with linguistic and semantic information (Sect. 4.1). 2. We perform a question abstraction (Sect. 4.2). 3. We use the abstracted question to construct the corresponding SPARQL query representation (Sect. 4.3); 4. We generate the SPARQL query (Sect. 4.4).
The process is implemented as a module within the NLP platform Ogmios [
The pre-processing of the questions is the same for the training and test sets. The
annotation of the questions consists in recognition of numerical values (such as
numbers and solubility values), word segmentation, part-of-speech tagging and
lemmatization of the words with TreeTagger [
However, as we realized during our preliminary experiments on the training
set, the coverage of the terminological entities that appear in questions is not
su cient. We also apply the term extractor YATEA12 [
question
SPARQL query generation
sameAs description resources in blue) test set because it is time consuming.
test set because it is time consuming. and lemmatized text by chunking it according to syntactic frontiers (pronouns, and lemmatized text by chunking it according to syntactic frontiers (pronouns, conjugated verbs, typographic marks, etc.) in order to identify noun phrases. conjugated verbs, typographic marks, etc.) in order to identify noun phrases.
13 Then, parsing patterns 13that take into account the morpho-syntactic variation,
Then, parsing patterns that take into account the morpho-syntactic variation, are recursively applied and provide parsed terminological entities, usually noun are recursively applied and provide parsed terminological entities, usually noun phrases relevant for the targeted domain. Each term is represented as a syntactic phrases relevant for the targeted domain. Each term is represented as a syntactic tree, and sub-terms are also considered as terms in the current con guration.
tree, and sub-terms are also considered as terms in the current configuration. illustrates the linguistic and semantic annotation of a question.
illustrates the linguistic and semantic annotation of a question.
No semantic types are associated to the terms extracted by Y T A. Figure 2 No semantic types are associated to the terms extracted by AYATEEA. Figure 2 On the training step, we observed that the lemmatizer and POS-tagger Genia
On the training step, we observed that the lemmatizer and POS-tagger Genia
Tagger [
me drugs VB PRO
N [drug] in the
P
DT gaseous
A [drugbank/state/Gas] state
.
N [state] bobxoexsesrerperpersesnetnt htehewworodrdanadndsesmemanatnitcicenetnitietse.s.TThehesusubbscsrcirpiptttetxextstsaarereththeePPaartr-to-of-f
ThTihsisstsetpepaiami mssatatidiednetnitfyifiynigngthtehererleelveavnatnteleelmemenentstswwitihthininththeeqquueestsitoionnssaanndd bubiuldilidnigngthtehererperperseesnetnattaitoinonofofthtehseeseeleelmemenetnst.s.AAt tththisisstsetpep,,wweeuuseseththeelilningguuisistticic anadndsesme manatnicticanannontoattaitoinosnsasassoscoicaitaetdedtotoththeeququesetsitoinonwwoordrdssininththeepprerevvioiouussstsetepp. .
BeBfeofroerethteheidiednetniticfiactaitoinonofofrerleelveavnatnteleelmemenentst,s,ththeeddisiasammbbigiguuaatitoionnooffththeeaann-fofrotrhtehenenxetxsttsetpesp.sF.oFrorinisntsatnacnec,ew,weekekeepeplalragregrertetremrms swwhhicihchddoonnoottininccluluddeeooththeerr sesme manatnicticenetnititeise.s.
TThuhsu,sw,weededenfiendedrerwe wrirtitnigngrurluelsesininorodrderertotommooddifiyfyoorrddeleelteeteththeesesemmaanntticic mmodoidf yifyorordedlelteetethteheenetnitiytyacaccocrodridnigngtotoththeecocnontetxext.t.FFoorrininstsatannccee,,ththeesesemmaann-cocnotnetxetxtcocnotnatianisnsa atetremrmwwitihththtehesesmemanatnitcictytpypeefofodod..OOnnththeewwhhoolele,,wweeddeefinneedd 4444cocnotnetxetxutaularlerwewrirtitnigngrurluelsesbabsaesdedononththeevovcoacbabuulalrayryuusesdedininththeeqquueestsitoionnssaanndd 1313 dedpeepnednednecnycyinintetremr minionlolgoigciaclaelnetnititeise.s. on the documentation of knowledge bases, mainly the one from Drugbank14. Besides the rewriting rules, additional disambiguation of the annotations is also performed during the query construction step when the arguments of the predicate are connected by selecting the correct semantic types.
For performing the abstraction of questions, we identify information related to the query structure: 1. De nition of the Result form: the question is scanned for identifying words expressing the negation, e.g. no, and its scope, the aggregation operation on the results, e.g. number for count, mean for avg or higher for max, and speci c result form such as boolean queries (ASK). The information concerning the presence of the negation and aggregation operators or of speci c result form is recorded in data structures to be used at the end of the query construction step or during the query generation step. 2. Identi cation of the Question topic: we consider the rst semantic entity with a given expected semantic type to be the question topic. The expected semantic types are those provided by the RDF subjects and objects in the Drugbank, Diseasome and Sider. This information will be used during the query construction step. 3. Identi cation of Predicate and Argument: according to our internal framebased representation of the three resources, the potential predicates, subjects and objects are identi ed among the semantic entities and described into a symbol table.15 At this step, the subjects and objects are fully described in the symbol table. Concerning the predicates, only the semantic types of their arguments are instantiated in the symbol table within the RDF schema. The subjects and objects can be URI, RDF typed literals (numerical values or strings) and extracted terms (these are considered as elements of regular expressions). The objective of the query construction step is to connect previously identi ed elements and to build the SPARQL graph pattern (introduced by the keyword WHERE). Thus, the symbols of the predicate arguments are instantiated by either URI associated with the subjects and objects, variables, numerical values or strings.
For each question, we perform several connections: 14 http://www.drugbank.ca/documentation 15 Similarly to a compiler, RDF schema of the predicates and semantic types of their arguments are considered as entries of the symbol table. Information associated with the symbols (in ected and lemmatized form of the word or term, corresponding
drug (sider/drugs)
drugbank/drugs state STRING
drugbank/state/Gas gaseous STRING/Gas 1. The question topic is connected to the predicate(s). A variable is associated with the question topic and the predicate arguments that matched the semantic type of the question topic. Note that at the end of this stage, the question topic may remain non-associated with any predicate; 2. The predicate arguments are connected to the subjects and objects identi ed during the question abstraction: they concern elements referring to URI; 3. The predicates are connected between them through their subjects and objects. The connection between two predicates is then represented by a variable; 4. The predicates from di erent datasets are connected. We use the sameAs description to identify URI referring to the same element. New variables are de ned to connect two predicates; 5. The remaining question topic is connected to arguments of the sameAs predicate; 6. The arguments are connected to the string type to extracted terms annotated in the question. We assume these arguments will be related to the string matching operator REGEX. Thus, terms are considered as string expressions.
At this point, the predicate arguments which remain unassociated are replaced by new variables in order to avoid empty literals. Finally, the negation operator is processed: the predicates are marked as negated and the arguments within the negation scope are included in a new predicate rdf:type if required.
At this stage, the question is fully translated into data structures representing the SPARQL query. Figure 4 illustrates the construction of the query corresponding to the question 22 of the test set.
?v0 state STRING/Gas
drugbank/state/Gas gaseous STRING/Gas
This nal step aims at generating the SPARQL query string based on the data structures built during the query construction step. The output of this step is the string corresponding to the SPARQL query. It is composed of two parts: 1. The generation of the result form which takes into account the expected type of the result form (ASK or SELECT), the presence of aggregation operators and the variable associated to the question topic; 2. The generation of the graph pattern. Basically, the part of the query generation consists in generating the strings representing each RDF triple and lter if predicates are negated. But when aggregation operators are used, we also need to recursively generate sub-queries computing the subsets of expressions, before their aggregation.
The SPARQL queries have been submitted without retrieving the answers. We let this task to the evaluation tool. Figure 5 presents the generated query which corresponds to the question 22 of the test set.
SELECT DISTINCT ?v0 WHERE { ?v0 <http://www4.wiwiss.fu-berlin.de/drugbank/resource/drugbank/state> "Gas". }
The automatically generated SPARQL queries are evaluated with the online evaluation tool16. The answer of each query q is compared with the gold standard. The evaluation measures (F-measure, precision and recall) are computed. The system results are evaluated with macro-measures. The challenge provides 25 questions for the training and 25 questions for the test. 5.2
In Tab. 1, we present the overall results on the training and test sets. Our system was ranked 2nd out of 3 submissions. We can observe that our results are similar for these two sets of queries (training and test sets).
We can observe that in both cases, for 19 questions, the system exactly provides the expected answers. For 4 questions from the training set and 3 questions from the test set, we obtain partial answers. In the training set, 2 questions receive no answer while in the test set, we have 3 such questions. 16 http://greententacle.techfak.uni-bielefeld.de/~cunger/qald/index.php?x=evaltool&q=4 Error Analysis To our opinion, the reference SPARQL query for the question 19 of the training set is not correct: the expected result of the question is a list of drugs, while the reference SPARQL query returns a list of diseases. On the test set, we can propose two observations on the limitations of our system: { In question 1,17 the contextual rewriting rules cannot be correctly applied because the semantic entity (gene ... associated) is discontinued; { In question 18,18 the system correctly detects the semantic entities and the predicates, including the sameAs predicate. The remaining problem is that the system assumes that the sameAs predicate is re exive while in the resources provided, the instances of this predicate do not encode the re exivity of the relation. We analyzed the system performance on a standard computer (2.7GHz dual-core CPU and 4 Gb of memory). Figure 6 presents the running time for each query ac17 Which genes are associated with Endothelin receptor type B? 18 List the number of distinct side-e ects of drugs which target genes whose general function involves cell division. ) s d n o c e s ( e m i t n u r 3 2 1 0 5 10 15 20
25 number of questions cording to the pre-processing sub-steps (named entity recognition, word and sentence segmentation, POS tagging, semantic entity tagging and term extraction) and the question translation into SPARQL queries (Question2SPARQLQuery). Most of the processing time is dedicated to the TermTagger which aims at recognizing the semantic entities. With the internal Ogmios processing (i.e. mainly the input/output), each question is processed in 1.94 seconds on the average on the training set and 1.97 seconds on the average on the test set. We proposed a four-step process based on Natural Language Processing methods, semantic resources and RDF triples description. The system achieves good performance with 0.85 F-measure on the test set of 25 questions. It is ranked 2nd out of 3 submissions. Further work includes optimization of the running time for the question processing and the extension of the SPARQL syntax by taking into account the operators on collections of sets. We also plan to investigate the integration of other biomedical resources such as Dailymed or RxNorm, and the use of the developped system for the text mining applications. This work was partly funded through the project POMELO (PathOlogies, MEdicaments, aLimentatiOn) funded by the MESHS (Maison europeenne des sciences de l'homme et de la societe) under the framework Projets Emergents.