In the Web of Data, the pieces of information obtained by querying distributed SPARQL endpoints may provide di erent results for the same query. Moreover, the combination of these query results may lead to an inconsistent set of information about the same topic. In particular, the problem of reconciling information obtained by distributed SPARQL endpoints is encountered in question-answering systems over linked data, where di erent SPARQL endpoints are queried to retrieve the answer to the user's question. In this paper, we propose to address this problem by adopting argumentation theory to reason over inconsistent information sets, and provide nevertheless a unique and motivated answer to the user. We implement and evaluate our approach on QAKiS (Question Answering WikiFramework-based system), that exploits multilingual chapters of DBpedia as RDF data sets to be queried using a natural language interface.
In the Web of Data, the combination of the information items concerning a
single real-world object coming from di erent data sources, e.g., the results of
a single SPARQL query on di erent endpoints, may lead to an inconsistent
results set. This is an open problem for consuming in the Web of Data since
these inconsistencies mine the overall quality of the data itself. In particular,
this problem arises while querying the multilingual chapters of DBpedia [
In this paper, we answer the following research question: How to reconcile
information provided by the multilingual chapters of DBpedia to obtain a
consistent results set? This issue is particularly relevant to Question Answering
(QA) systems over DBpedia [
First, we need to detect the semantic relations which relate each piece of information to the others returned by the di erent sources, i.e., SPARQL endpoints. In particular, two kinds of relations can be highlighted: a con ict relation such that the information items represent contradicting answers to a single question (e.g., given the question \Which is the capital of Italy? ", the answers \Paris" and \Rome" con ict), and a support relation such that the information items are semantically connected in a way that one item may be derived from the other by means of an ontology (e.g., given the question\In which place was William Shakespeare born? ", the answers \Stratford-upon-Avon" and \England " do not actually con ict because Stratford-upon-Avon is located in England).
Second, we adopt abstract argumentation theory [
Third, the overall argumentation framework together with the acceptability
degree of each argument is used to motivate to the user the answer the system
returns. We evaluate our approach through its integration in QAKiS [
The remainder of the paper is as follows: Section 2 provides the basic notions of argumentation theory. Section 3 presents our argumentation-based framework for inconsistencies detection. In Section 4 our approach is evaluated on QAKiS. Section 5 compares the existing research with the proposed approach. 2
A Dung-style abstract argumentation framework [
However, associating a crisp label, i.e., in or out, to the arguments is
limiting in a number of real life situations where a numerical value expressing
the acceptability degree of each argument is required [
This section describes our
framework to detect the
inconsistencies among the
results obtained querying the c a b c a
SPARQL endpoints of
multilingual DBpedia chapters: i) (a) d
how we assign an attack or a
support relation between two
pieces of information
depending on the semantic relation
linking them (Section 3.1),
and ii) how compute a fuzzy evaluation of the arguments (Section 3.2).
Given a set of answers to a certain factual query, each provided by a di erent
endpoint, we de ne and apply our algorithm to assign the relations of support
and attack between two arguments (where an argument is an answer
independently provided by a speci c SPARQL endpoint). Multilingual chapters of
DBpedia [
Given a certain query, if two endpoints provide identical answers, we identify a positive relation between them (that we call identity ) which is then translated into a support relation between such arguments. For instance, in Figure 2a both the French and the English DBpedia SPARQL endpoints provide Italy as answer to Where is the Colosseum located?. The algorithm assigns therefore a support relation between such arguments (double arrows in the gures), and merges them into a unique argument with value Italy. We do not consider these arguments as independent for two reasons: rst, every other answer con icting with one of them is also con icting with all the others, and second, given that this argument is shared among several sources then it is highly acceptable, i.e., reliable. The con dence score of this new argument is calculated as the arctangent of the con dence scores of the endpoints providing such answer (max value = 1). We bene t from the sameAs links between the translation of the same word in DBpedia multilingual chapters, and we consider also the case reported in Figure 2b as identity, since both answers contain the same value expressed in di erent languages (i.e., Italia in Italian, and Italy in English).
Figure 2 (c,d) reports another positive relation between arguments, i.e., what
we call subsumption. This case arises when one of the obtained answers is more
speci c than the other, both in terms of i) spacial/geographical relation, e.g.,
x is located in y (Figure 2d: Rome is located in Italy ); ii) hyperonymy (is a
relation) as in Figure 2c (Gibson J-160E is a Guitar : this example shows two
possible answers to the question Which instruments did John Lennon play? ).
Our algorithm exploits external sources of semantic knowledge to detect the
relations between the arguments (e.g., GeoNames1 for geographical entities, and
the DBpediaYAGO class hierarchy2 [
Once such relations are found and veri ed, the algorithm sets a support relation between the two arguments following the direction from the most speci c argument toward the more general one (double arrows in Figure 2).
In case the answers provided by two di erent endpoints for a certain query are di erent, and no subsumption relation between them is identi ed, then the algorithm assigns an attack relation between such arguments. For instance, an attack arises between the arguments visualized in Figure 3a, since the answers provided by the Italian and by the English DBpedia endpoints to the query concerning the location of the Colosseum are contradictory. It should be noted that the attack relation is always bidirectional in our framework (i.e., the two arguments con ict with each other, as shown by simple arrows in Figure 3).
A subtler case of attack is reported in Figure 3b. Here we have a support relation between the arguments Roma and Italy provided respectively by the Italian and the English endpoints (subsumption relation, as in the example visualized in Figure 2). Then, a third endpoint, i.e., the French one, provides the value France as answer to the same query. Our algorithm assigns an attack relation between this argument and the argument Italy, and it automatically adds an additional attack from France to the argument Roma which is supporting the argument it attacks (i.e., Italy ). This additional attack results from the assumption that if an argument attacks Italy, it attacks also Rome since Rome is in Italy.
The last case we consider in our algorithm is when each endpoint provides a list of values (i.e., a list ) as answer to a certain query (e.g., non-functional properties in DBpedia). For instance, in the example reported in Figure 4 the English DBpedia endpoint provides both Guitar and Voice as instruments played by J. Lennon, while the French endpoint provides both Guitare and Harmonica. In this case, our algorithm assigns a support relations between Guitar and Guitare, and it merges them in a unique value (see identity in Figure 2). Moreover, dif
ferently from the case illustrated in Figure 3, in case of lists we do not consider arguments of the same list as con ictual. For this reason, we limit the assignment of the attack relation to those arguments which are not provided together in the same list. As illustrated in Figure 4, no attack relation is assigned between, e.g., Harmonica and Guitar, since both answers are provided by the French endpoint.
In this section we have described the algorithm that, given the answers provided by the endpoints to a single query, highlights the support/attack relation among them. Next section explains how such list of arguments and relationships is sent to the argumentation module that calculates the arguments' acceptability degree (i.e., the arguments that will be proposed to the user as more reliable). 3.2
Since we need to take into account the con dence associated to an information
source when computing the set of accepted arguments and their own
acceptability degree, we go beyond standard argument semantics introduced by Dung [
We now extend the de nition of fuzzy labeling for standard abstract argumentation into a bipolar fuzzy labeling. Note that in this paper we assume that the following two constraints hold: an argument cannot attack and support at the same time another argument, and an argument cannot support an argument attacking it, and vice versa. These constrains are not on the computation of the acceptability degree of the arguments, but they underlie the construction of the argumentation framework itself.
De nition 1. Let hA; !; )i be an abstract bipolar argumentation framework where A is a fuzzy set of (trustful) arguments, ! A A and ) A A are two binary relations called attack and support, respectively. A bipolar fuzzy labeling is a total function : A ! [0; 1].
Such an may also be regarded as (the membership function of) the fuzzy set
of acceptable arguments where the label (A) = 0 means that the argument is
outright unacceptable, and (A) = 1 means the argument is fully acceptable. All
cases inbetween provide the degree of the acceptability of the arguments which
may be considered accepted at the end, if they overcome a given threshold. In [
The rationale behind this constraint is the following: if the set of supports is empty, then the standard fuzzy labeling is used; otherwise, the support of an argument towards another is intended to augment the acceptability degree of such supported argument. As we will show by means of examples, the acceptability degree of an argument which is both supported and attacked depends on the con dence assigned to the sources proposing the supporter and the attacker of such argument. Using the above constraint, we obtain the following de nition De nition 2. (Bipolar Fuzzy Labeling) Let that is a bipolar fuzzy labeling (A) = avgfminfA(A); 1 maxB:B!A (B)g; maxC:C)A (C)g.
be a bipolar fuzzy labeling. We say
i , for all arguments A,
da Costa Pereira et al. [
When the argumentation module receives the couples of arguments linked by the appropriate relation and the degree of con dence associated to each source, the bipolar fuzzy labeling algorithm is raised on the argumentation framework to obtain the acceptability degree of each argument. This step returns also the overall bipolar argumentation framework where each argument is linked to its source, and the acceptability degree is associated to the arguments. This overall view is then used to explain to the user how the QA system comes to nd this answer, thus its motivations. This step is necessary to prevent the user from seeing the QA system as a black box, but to understand the reasons behind correct, and more importantly, erroneous answers.
The fact that an argumentation framework
can be used to provide explanations [
To evaluate the argumentation-based framework to detect inconsistencies that
we described in Section 3 in a real setting, we integrate it in an existing
QuestionAnswering system, QAKiS [
QAKiS3 (Question Answering wiKiFramework-based System) [
to the answer through one property of the ontology, as Which river does the Brooklyn Bridge cross?. Such questions match a single pattern, i.e., one relation.
We embed into the QAKiS architecture the argumentation module (Figure 5), that collects the answers obtained from the di erent endpoints and applies the algorithm described in Section 3.1 to identify the arguments and the relations among them, and then it runs the bipolar fuzzy labelling algorithm (Section 3.2) to detect inconsistencies and return the more reliable arguments. 4.2
To run our experiments, we extract from the reference data set of QALD-2 (Question Answering over Linked Data challenge)4 the questions that the current version of QAKiS is built to address (i.e. questions containing a NE related to the answer through one property of the ontology), corresponding to 26 questions in the training and 32 in the test sets. The discarded questions require either some forms of reasoning (e.g., counting or ordering) on data, aggregation (from datasets di erent from DBpedia), involve n-relations, or are boolean questions. We consider these 58 questions as the reference data set for our experiments. 4.3
To evaluate the validity of the proposed approach, we run the questions contained into our reference datasets on the English, German and French chapters of DBpedia (con dence scores: 0.7, 0.6 and 0.5 respectively). We chose these three chapters based on i) their dimensions and the robustness of their endpoints, and ii) the presence of the SameAs relation among the translations of the words in the di erent languages. Since the questions of QALD-2 dataset were 4 http://greententacle.techfak.uni-bielefeld.de/~cunger/qald/index.php?x= challenge&q=2 created to query the English chapter of DBpedia only, it turned out that only in 25 cases out of 58 at least two endpoints provide an answer (in all the other cases the answer is provided by the English chapter only, not useful for our purposes). For instance, given the question List the children of Margaret Thatcher the English DBpedia provides Mark and Carol Thatcher as answers, while the French one provides only the answer Mark Thatcher. Or given the question How many employees does IBM have?, the English and the German DBpedia provide 426751 as answer, while the French DBpedia provides 433362 and 2010.
We evaluate our approach with two sets of experiments: in the rst case, we start from the answers provided by the di erent DBpedia endpoints to the 25 questions, and we run our argumentation-based algorithm on it. In the second case, we add QAKiS in the loop, meaning that the data we use as input for the argumentation module are directly produced by the system, as explained in Section 4.1. In this second case, the input are the 25 natural language questions. Table 2 reports the results we obtained for the two experiments over 24 questions, one question timed out. We evaluated both the ability of the argumentation module to correctly identify and (if necessary merge) the answers from the di erent endpoints (1st row), and its ability to assign the correct relations among such arguments (2nd row), w.r.t. a manually annotated goldstandard of arguments and their relations (in total, over the 25 questions, 90 items should be recognized as arguments-answers, and 219 attack and 3 support relations should be detected). For instance, given the examples before, for the question How many employees does IBM have? the algorithm generates 4 arguments (where the answer 433362 provided by both the English and German DBpedia is merged and its con dence score augmented, see Section 3) and 12 relations (attacks among the di erent values). Applying the fuzzy labeling algorithm, the answer provided with the highest con dence is 433362 (in future work, we could allow some approximation, merging values 433000 to this value). For List the children of Margaret Thatcher, the algorithm identi es two arguments (i.e. Mark and Carol ), and no relations, since the DBpedia property child allows for list answers. Both answers are provided to the user, where Mark has a highest con dence score.
Most of the errors in identifying the arguments are due to the missing SameAs links in DBpedia: the algorithm is therefore not able to merge translations of the same answer, considering them as di erent. Wrong relation assignments are mainly due to missing attacks among arguments (in particular for numerical values, or for arguments from the same endpoint). Concerning the second column of Table 2, since QAKiS performances are about 50%, the results are obtained accordingly (the argumentation-based module is biased by QAKiS mistakes in submitting the query to DBpedia). The average computation cost of the argumentation-based algorithm is high ( 124 seconds), mainly due to the n-answers questions. Considering only 1-answer question, the computation cost drops to 5 seconds. The complexity is quadratic, at least one SPARQL query is sent for each couple of answers. This could be improved by importing data for each answers individually and then process it locally. We are currently investigating this solution to optimize the algorithm.
Task Argumentation module QAKiS+Argum. module
Precis. Recall F-meas. Precis. Recall F-meas.
Argument identi cation 0.95 1 0.97 0.71 0.75 0.73
Relation assignment 0.71 0.73 0.72 0.54 0.56 0.55
State of the art QA systems over Linked Data generally address the issue of
question interpretation mapping a natural language question to a triple-based
representation (see [
Several works address alignment agreement based on argumentation theory.
More precisely, Laera et al. [
In this paper, we propose an automatic framework to detect the inconsistencies which may arise into a set of answers provided by a QA system over linked data. These inconsistencies are due to i) heterogeneous answers provided by the same information source, e.g., a single SPARQL endpoint, or ii) heterogeneous answers from distributed information sources. Our framework adopts bipolar abstract argumentation to provide an overall view of the information items provided by the single sources. This argumentation framework is built thank to the semantic relations among the information items automatically extracted using ontological knowledge, e.g., DBpedia ontology. The framework is then evaluated to assign the acceptability degree of the arguments using the bipolar fuzzy labeling algorithm. We have shown the feasibility of our approach on the QAKiS question answering system on the multilingual chapters of DBpedia.
There are several points to be addressed as future work. First, we assign to each information source a con dence degree which is derived from the coverage of such DBpedia version. However, another possibility is to leave the user itself to assign the con dence degree to the information sources. We plan to embed this feature in the QAKiS interface. Second, we have to extend the set of ontologies we consider in order to be able to detect further relations (positive and negative) among the information items. Finally, the user evaluation should not be underestimated and we will perform an evaluation campaign to verify which kind of visualization of the motivations is more usable by consumers.