Knowledge bases are in widespread use for aiding tasks such as information extraction and information retrieval. However, knowledge bases are known to be inherently incomplete. As a complimentary data source, embedded entity markup based on Microdata, RDFa, and Microformats have become prevalent on the Web. RDF statements extracted from markup are fundamentally different from traditional knowledge graphs: entity descriptions are flat, facts are highly redundant and of varied quality, and, explicit links are missing despite a vast amount of coreferences. Therefore, data fusion is required in order to facilitate the use of markup data for KBA. We present a novel data fusion approach which addresses these issues. We perform a thorough evaluation on a subset of the Web Data Commons dataset and show significant potential for augmenting existing knowledge bases. A comparison with existing data fusion baselines demonstrates superior performance of our approach when applied to Web markup data.
Knowledge bases (KBs) such as Freebase [
Building on standards such as RDFa, Microdata and Microformats, and driven by initiatives such as schema.org, a joint effort led by Google, Yahoo!, Bing and Yandex, markup data has become prevalent on the Web. Through its wide availability, markup lends itself as a diverse source of input data for KBA. However, RDF statements extracted from markup are fundamentally different from traditional knowledge graphs: entity descriptions are flat, facts are highly redundant and of varied quality, and, explicit links are missing despite a vast amount of coreferences.
In this work, we introduce KnowMore, an approach based on data fusion techniques which exploits markup crawled from the Web as source of data to aid KBA. Our approach consists of a two-fold process, where first, candidate facts for augmentation of a particular KB entity are retrieved through a combination of blocking and entity matching techniques. In a second step, correct and novel facts are selected through a supervised classification approach and an original set of features. We apply our approach to the WDC2015 dataset and demonstrate superior performance compared to state-ofthe-art baselines. We also demonstrate the capability for augmenting three large-scale knowledge bases, namely Wikidata, Freebase and DBpedia through markup data based on our data fusion approach. The main contributions of our work are threefold: – Pipeline for data fusion on Web markup. We propose a pipeline for data fusion that is tailored to the specific challenges arising from the characteristics of Web markup. – Model & feature set. We propose a novel data fusion approach consisting of a supervised classification model, utilising an original set of features geared towards validating correctness and relevance of markup facts. – Knowledge base augementation from markup data. As part of our experimental evaluation, we demonstrate the use of fused markup data for augmenting three wellestablished knowledge bases. 2
Motivation. For a preliminary analysis of DBpedia, Freebase and Wikidata, we randomly select 30 Wikipedia entities of type Movie and Book and retrieve the corresponding entity descriptions from all three KBs. We select the 15 most frequently populated properties for each type and provide equivalence mappings across all KB schemas as well as the schema.org vocabulary manually. Since all vocabulary terms and types in the following refer to schema.org, prefixes are omitted. Figure 1 shows the proportion of instances for which the respective properties are populated. We observe a large amount of empty slots across all KBs for most of the properties, with an average proportion of missing statements for books (movies) of 49.8% (37.1%) for DBpedia, 63.8% (23.3%) for Freebase and 60.9 % (40%) for Wikidata.
DBpedia
Freebase
Wikidata
DBpedia
Freebase
Wikidata cea itred100 rp 90 lua 80 c itra 70 pg 60 inn 50 i tan 40 co 30 irsn 20 o itp 10 cs 0 yedn itt e f o % namdeescription authinoLranguagepublisher genbroeoknFuomrmbaetrOfPdaagteesPublished isbntransclaotpoyrrightYebaorokEdition award editor
name abodutescriptiionnLanguage directorproducer actor authorduratiopnromduucsticioBnyComcpoapnyyrightdYaeteaPrublished genre awards (a) Book (b) Movie Fig. 1: Proportion of book and movie instances per KB that include selected popular predicates.
In addition, coverage varies heavily across different properties, with properties such
as editor or translator being hardly present in any of the KBs. Tail entities/types as well
as time-dependent properties which require frequent updates, such as the award of a
book, are prevalent in markup data [
There exist n 0 subjects s1; s2; :::; sn , and consequently, n entity descriptions ei = hsi; pi; oii 2 E which represent a particular query entity q in M . Here, E is the set of all entity descriptions which (co)refer to entity q. We define a property-value pair hp; oi describing the entity q as a fact of q. Note that we explicitly consider multivalued properties, i.e. a particular predicate p might be involved in more than one fact for a particular entity q. We define the task of augmenting an entity description eq, representing a query entity q within a particular KB from data in a markup corpus M as follows: Definition 1. KBA task: For a query entity q that is represented through an entity description eq in a KB, we aim at selecting a subset Fnov from M , where each fact fi 2 Fnov represents a valid fact which augments the entity description eq for q.
Fnov represents the final output of the KnowMore pipeline. We consider a fact valid for augmentation, if it meets the following criteria: – A fact is correct with respect to query entity q, i.e. consistent with the real world regarding query entity q according to some ground truth (Section 4). – A fact represents novel, i.e. not duplicate or near-duplicate, information with regard to the entity description eq of q in a given KB. – The predicate pi of fact hpi; oii should already be reflected in a KBs given schema. 3 3.1
The first step, KnowM orematch, aims at obtaining candidate facts fi 2 F by collecting the set E of coreferring entity descriptions ei 2 E from M which describe q and corefer to the entity description eq in a given KB. We use a three step approach in order to efficiently achieve high accuracy results.
Data Cleansing. This step aims at (i) resolving object references and (b) fixing common
errors [
Blocking. We implement the blocking step through entity retrieval using the BM25 model to reduce the search space. We created an index for each type-specific subset using Lucene, and then use the label of eq to query the field name within a type-specific index. This result in a set of candidate entity descriptions ei0 2 E0 that potentially describe the same entity as eq .
Entity Matching. This step is for the validation of each entity description ei0 2 E0 in the result of the blocking step. We use supervised classification on the similarity vector between ei0 and eq . In order to compute the similarity for each property, we consider all properties as attributes of the feature space !A = fa1; a2; :::; ang, so that each entity description e can be represented as a vector of values !v = foa1; oa2 ; :::; oan g which represent the objects of the considered hp; oi tuples. We construct a similarity vector si!m(vK!B ; !v) between eq and each entity description ei0 2 E0 as in Equation 1. si!m(vK!B ; !v) = f a1 ; a2 ; :::; an g ai = sim(oaKiB ; oai ) (1) (2)
In order to compute sim(oaKiB ; oai ), we employ datatype-specific similarity metrics, i.e., we implemented one similarity measure for each schema.org datatype, and automatically select the appropriate metric. We then train a supervised classification model, to make the decision whether or not ei0 is a match for eq . We experimented with several state-of-the-art classifiers (SVM, Logistic Regression and Naive Bayes). Since Naive Bayes achieves a F 1 score that is 0.08 higher than the best SVM (linear kernel), and 0.123 higher than the Logistic Regression (LR), throughout the remaining paper we rely on a trained Naive Bayes classifier unless otherwise stated. 3.2
Data fusion During the data fusion step, KnowM oreclass, we aim at selecting a subset Fnov F that fulfills the criteria as listed in Section 2. More specifically, we introduce data fusion techniques based on supervised classification to ensure the correctness and two deduplication steps to ensure novelty, namely deduplication with respect to M (KnowM oreded) and deduplication with respect to the KB (KnowM orenov ). Correctness - Supervised Classification. The first step (KnowM oreclass) aims at detecting correct facts by learning a supervised model that produces a binary classification for a given fact f 2 F into one of the labels f‘correct’, ‘incorrect’g. For the classification model, we have experimented with several different approaches. We rely on a Naive Bayes classification since our experiments have shown superior performance over other classifiers. The features used are listed in Table 1.
While we aim to detect the correctness of a fact, we consider characteristics of the source, that is the Pay-Level-Domain (PLD) from which a fact originates, the entity description, the predicate term as well as the fact itself. From the computed features we train the classifier for classifying the facts from F into the binary labels f‘correct’, ‘incorrect’g. The ‘correct’ facts form a set Fclass that is the input for the next steps. Novelty. A fact f is considered to be novel with respect to the KBA task, if it fulfills the conditions: i) is not duplicate with other facts selected from our source markup corpus M , ii) is not duplicate with any facts existing in the KB. Each of these two conditions corresponds to a deduplication step.
Deduplication with respect to M (KnowMoreded). We detect near-duplicates via clustering. For each predicate p, all the facts f = hp; oii corresponding to p are clustered into n clusters fc1, c2, ,cng. Each cluster ci; i = 1; :::; n contains a set of nearduplicates. To fulfill i), we select only one fact from each cluster by choosing the fact that is closer to the cluster’s centroid. This results in the fact set Fded that is the input for next deduplication step.
Deduplication with respect to KB (KnowMorenov). We compute the similarity sim(fi; fKB) between a fact fKB in a respective KB for a particular predicate p and a fact fi for the same (mapped) predicate p in Fded with the datatype-specific similarity metrics. If sim(fi; fKB) is higher than a threshold , we remove the fact. We explain and its configuration during the experimental Section 4.3. The facts selected from Fnov in this step are the final result for augmenting the KB.
Note that our deduplication step considers and supports multi-valued properties. By relying on the clustering features, computed during the fusion step, we select facts from multiple clusters (corresponding to multiple predicates) as long as they are classified as correct. As documented by the evaluation results (Section 5), this does not negatively affect precision while improving recall for multi-valued properties. 4 4.1
We use the WDC2015 dataset5, where we extracted 2 type-specific subsets consisting of entity descriptions of the schema.org types Movie and Book. As input for the KBA task, we randomly select 30 entities for each type Book and Movie. We evaluate the performance of our approach for augmenting entity descriptions of these 60 entities obtained from three different KBs: DBpedia (DB), Freebase (FB) and Wikidata (WD). To simplify the schema mapping problem between WDC data and the respective KBs while at the same time taking advantage of the large-scale data available in our corpus, we limit 5 http://webdatacommons.org/structureddata/index.html#toc3 the task to entities annotated with the http://schema.org ontology for this experiment. We manually create a set of schema mappings that maps the schema.org vocabularies to the DB, FB, WD vocabularies.
Data Fusion - Correctness. We used crowdsourcing to build a ground truth for the correctness of facts fi 2 F . For the valid entity descriptions in E, we acquire labels for all distinct facts, as either correct or incorrect with respect to q. We acquired 5 judgments from distinct workers for each entity and corresponding facts through Crowdflower. Data Fusion - Novelty. We built ground truths for validating (i) deduplication performance within M , as well as (ii) novelty with respect to the different KBs. Authors of this paper acted as experts and designed a coding frame to decide whether a fact is novel. After resolving disagreements on the coding frame on a subset of the data, every fact was associated with one expert label through manual deliberation. 4.2
Metrics We consider distinct metrics for evaluating each step of our approach. – KnowM oreclass. We evaluate the performance of the approaches through standard precision P , recall R and F 1 scores, based on our ground truth. – KnowM oreded. We evaluate the performance of deduplication with respect to M using Dist% - the percentage of distinct facts within the respective result set. We compare between Dist% (Fded) and Dist% (Fclass), that is, before and after the deduplication within M . – KnowM orenov. For evaluating the performance of deduplication with respect to a given KB, we measure the novelty as N ov - the percentage of novel facts - and compare between N ov (Fded) and N ov (Fnov), that is, the novelty before and after this step. We also measure the recall R - the percentage of distinct and accurate facts in Fded that have been selected by KnowM orenov into Fnov.
Furthermore, we demonstrate the potential of our approach for augmenting a given KB by measuring the coverage gain. The coverage gain of predicate p is computed as the percentage of entity descriptions having p populated through the KnowM ore approach (i.e. after step KnowM orenov) with at least one fact hp; oi, out of the ones that did not have statement involving property p within the KB before augmentation. 4.3
Configuration & Baselines Configuration. For the entity matching step, we use Lucene for indexing and BM25 retrieval with the Lucene default configuration where k1 = 1:2, b = 0:75. For the deduplication with respect to KBs, we report the evaluation result of KnowM orenov using different = f0:3; 0:5; 0:7g in Section 5.
Baselines. We compare (KnowM oreclass) with P recRecCorr that is proposed by
Pochampally et al. [
Correctness - Data Fusion. The results for KnowM oreclass and the baselines are
shown in Table 2. Our chosen configuration, i.e. using a Naive Bayes classifier achieves
highest F 1 scores among all the different configurations. The presented F1 score of the
P recRecCorr baseline is the best possible configuration for our given task, where we
experimented with different thresholds ([
KB Fded Movie DBpedia 0.631
Freebase 0.527
Wikidata 0.412 Book DBpedia 0.736
Freebase 0.639 Wikidata 0.705 Novelty with respect to KB. The results before (N ov (Fded)) and after (N ov (Fnov)) the deduplication for specific KBs using different similarity thresholds ( ) are presented in Table 4. Since our approach is not aware of the total number of novel facts for a particular entity description on the Web a priori, in this evaluation, we consider all the novel facts in Fded as the gold standard, and compute the recall of Fnov after applying the KnowM orenov accordingly. We evaluate the performance of KnowM orenov using in f0.3, 0.5, 0.7g. As shown in Table 4, even though there is a trade-off between novelty and recall, different values of do not have a strong influence on the evaluation metrics. One of the reasons is that, a large proportion of facts have non-literal (e.g. numeric) values. While our datatype-specific similarity computes a binary (0 or 1) score in these cases, it is not influenced by the selection of .
DB
FB
WD
DB
FB
WD 100 in80 a g60 e g rea40 v o c20 0 100 (a) Movie (b) Book
Fig. 2: Proportion of augmented entity descriptions with KnowM ore.
Coverage Gain. Figure 2 shows the coverage gain on the previously empty slots as shown in Figure 1 per predicate and KB for our selected entities. Based on the result, the KnowM ore pipeline shows a coverage gain of 34.75% on average across different properties for DBpedia, 39.42% for Freebase and 36.49% for Wikidata. We observe that the obtained gain varies strongly between predicates and entity types, with a generally higher gain for book-related facts. For instance, within the Movie case, for property actor we were able to gain 100% coverage in both DBpedia and Freebase, while the property award shows a coverage gain of 10% or less for all three KBs. Reasons behind low coverage gain for a particular property are 2-fold: 1) the lack of data in the Web markup data corpus, and 2) the lack of true facts in the real world for a particular attribute, e.g. only a small proportion of movies have won an award. On average, we obtained 2.8 (6.8) facts for each movie (book) entity in our experimental dataset. 6
We have introduced KnowM ore, an approach towards knowledge base augmentation from large-scale Web markup data. We apply it to the WDC2015 corpus and augment three established KBs. Evaluation results suggest superior performance of our approach with respect to novelty as well as correctness compared to state-of-the-art data fusion baselines. Our experimental results indicate comparably consistent performance across a variety of types, whereas the performance of baseline methods tends to vary strongly.