In this study, we propose a method to validate knowledge candidates using the Web to minimize the false positive rate in a knowledge base (KB). Our approach assesses the objectivity and corroborativeness of a triple, which is the basic form of knowledge, using diverse Web resources. Compared to the state-of-the-art baseline of the Defacto framework, our approach demonstrates superior false positive rates, enabling more e ective ltering of false triples in the construction of a KB.
Knowledge validation that identi es the truth value (true or false) of facts is a vital step for constructing a reliable, usable knowledge base (KB). To the greatest extent possible, true facts should be distinguished and stored in a KB while incorrect or unreliable facts are ltered out as the existence of false triples in a KB can cause serious problems for those applications that use the KB. For instance, a wrong answer can be generated from a Q & A system that references a KB implanted with false knowledge. Therefore, techniques for validating knowledge are essential for the use of KB-based systems.
In literature, several studies have attempted to validate facts. Defacto
demonstrated a new approach that allowed testing whether a given fact (i.e., an RDF
triple) [
In this paper, we present a new technique to utilize the objectivity and
corroborativeness of Web resources for RDF truth validation. For validating an
RDF triple, we identify con rming sources on the Web as introduced in Defacto
[
Our primary purpose is to determine whether an RDF triple (s; p; o) given is true
or false. We deal with this problem as a binary classi cation. In this section, we
describe how we estimate the objectivity and corroborativeness of webpages.
Figure 1 shows a complete picture of our strategy for estimating the proposed
features of webpages retrieved for a given triple t (s; p; o). The estimated
objectivity features are replaced with the trustworthiness features proposed by Defacto
[
Ffobjsum(t) =
X (f(w) scw(w) obj(w)) Ffobjmax(t) = wm2sa(xt)(f(w) scw(w) obj(w))
w2s(t)
f (w) is instantiated by three trustworthiness scores: topic majority (tmweb),
topic majority in search results (tmsearch), and topic coverage (tc) of w. scw(w)
is the proof score on a webpage w. The proof, one of the supporting evidence on
webpage that the triple given is true, is de ned as a textual occurrence between
s and o within a certain token distance. The details for the trustworthiness and
proof score are described in [
Moreover, for validation of RDF triples, we utilize various web resources: images, videos, and news. We counted the number of hits of images, videos, and news. Our rationale is that true triples are more likely to have more supporting evidence in multiple forms (i.e., images, videos, and news) than the false ones because diverse types of information about the triples is accumulated on the Web. For example, if we have an RDF triple such as (Maroon 5, Song, Sugar) and search the Web with the \AND" operator (e.g., \Maroon 5" AND \Song" AND \Sugar") using the Bing Search API2, the API returns 14200 images, 833000 videos, and 16800 news as of June 2015; however, with a false triple (\Maroon 5", \Song", \Blank Space"), the API returns 4530, 103000, and 4290, respectively. (\Blank Space" is a song by Taylor Swift.) For corroborativeness, we employ the following three features: Num Images, Num Videos, and Num News. 3
We collected 570 RDFs from DBpedia as true triples on the top 57 most frequently used properties in DBpedia3. We randomly selected triples containing the property. We derived the false triples from the true triples with the following restriction. A triple (s , p , o ) is generated, where s and o are randomly selected resources, and p is a randomly selected property from the de ned properties. We applied the Defacto as a baseline, which is the only Web-based RDF true=false validation framework, to our best knowledge. To measure the objectivity of sentences in webpages, we used a sentiment analysis tool4 in the Stanford CoreNLP. The classi cation of the two classes (true and false) was conducted using ten-fold cross-validation. We used random forest (RF), which have been widely adopted for classi cation, in the Weka toolkit5 with the default parameter values given in Weka. We report on three measures: precision (P), recall (R), F-1 score (F1) (micro-averaged), and false positive rate (FP rate).
The classi cation results are shown in Table 1. Note that the performance was measured by replacing the six objectivity features with those proposed by Defacto (e.g, from TM Web Sum to TM Web Obj Sum) and adding the corrobrativeness features. A 5.9% and 6.7% decrease in the FP rate with RF was achieved using the objectivity and corroborativeness features, respectively.
Mean objectivity score of the retrieved websites from the true triples was
0.871 (StdDev: 0.202) and score from the false was 0.804 (StdDev: 0.262). This
indicates that the websites from the true triples have less sentiments on their
texts than those of the false. The overall performance decrease in the FP rate
and the di erent distribution of objectivity score in the two classes con rm the
e ectiveness of objectivity in judging the trustworthiness of the fact. In addition,
our results are in general agreement with the results of a prior study, in which
users admitted that sentiment analysis for a fact is useful in fact validation [
In this paper, we presented an approach for applying the objectivity and corroborativeness of webpages retrieved by the RDF triples in the true/false validation of given RDF triples, showing the e ectiveness of these concepts in knowledge validation. For future work, we are planning to extend the current study to perform a deeper analysis about the di erences of sentiment distributions for true and false fact validation as well as the con rmation of our approach even in political triples that could have limitations. 5
This work was supported by Institute for Information & communications Technology Promotion(IITP) grant funded by the Korea government(MSIP) (No. R0101-15-0054, WiseKB: Big data based selfevolving knowledge base and reasoning platform)