We are currently investigating methods to triplify the content of Wikipedia's tables. We propose that existing knowledge-bases can be leveraged to semi-automatically extract high-quality facts (in the form of RDF triples) from tables embedded in Wikipedia articles (henceforth called \Wikitables"). We present a survey of Wikitables and their content in a recent dump of Wikipedia. We then discuss some ongoing work on using DBpedia to mine novel RDF triples from these tables: we present methods that automatically extract 24.4 million raw triples from the Wikitables at an estimated precision of 52.2%. We believe this precision can be (greatly) improved through machine learning methods and sketch ideas for features that should help classify (in)correct triples.
the entities that are mentioned, and the table is embedded in an article that
provides a clear context for the table (potentially a protagonist [
By targeting Wikitables, we (mostly) avoid having to disambiguate entities
and relations based on textual labels, where we can (often) directly map table
cells to DBpedia entities and subsequently see what pre-existing relationships
exist between them. We can thus often avoid the di cult problem of resolving
entities (as tackled by, e.g., Limaye et al. [
In this paper, we rst provide a motivating example. We then discuss related works in the area of recovering the semantics of HTML tables and mapping relational content to RDF (§ 2). Next, we survey Wikitables available in a recent Wikipedia dump (§ 3). Thereafter, we discuss some of our ideas for using DBpedia as a reference knowledge-base to extract triples and sketch ideas for machine learning methods to further classify correct/incorrect triples (§ 4). We then wrap-up with discussion of future plans (§ 5).
Example 1. The table in Figure 1 is taken from the Colorado Wikipedia article (http://en.wikipedia.org/wiki/Colorado) and shows a list of the ve executive o cers of the state of Colorado. Some table cells are links to Wikipedia articles and others are plain string literals. The table contains a caption, header types O ce, Incumbent, Party and Term; and instances for each type Governor, John Hickenlooper, Democrat, 2011{2015, Lieutenant Governor, Joseph Garcia, etc.
This table constitutes quite a complex example. First of all, there are obvious relationships between the politicians listed as Incumbent and the elements of the Party column. Also, there are some implicit relationships that hold between many of the politicians and the subject of the article (Colorado), such as residence, etc. Furthermore, as per an attribute{value table, there are explicit relationships between the incumbents and Colorado, where the attributes are given in the O ce column (e.g., Walker Stapleton is State Treasurer of Colorado) and a temporal context is provided in the Term column.
In this paper, we survey the corpus of all such tables in Wikipedia. We furthermore propose some initial ideas on how to triplify some of the content of the table using DBpedia as a reference knowledge-base. The core idea is to map the elements of the cells with wiki-links to their respective DBpedia entities and to then look for existing relationships in DBpedia between entities on the same row: for example, we can nd that dbr:John Hickenlooper has the relation dbo:party to dbr:Democratic Party (United States), where we can suggest that such a relation might hold between entities in the respective columns on other rows. Furthermore, we can look for DBpedia relations from entities in a given column to the article body, where we nd, e.g., the relation dbp:residence from dbr:Joseph Garcia (United States politician) to dbr:Colorado.
We can also use features to classify triples extracted thusly as (in)correct; for example, we could consider that the more rows for which a given relation is found, or the closer the predicate label matches the column header text (e.g. dbo:party), the higher the con dence in the associated triples. Similarly, we could use the co-occurrence of certain properties for entities in di erent rows of the same column to adjust the con dence of a candidate relation. Later, we sketch some features that we plan to investigate (in future work) for classi cation. tu 2
Processing Web Tables. Many works have looked at identifying, parsing,
normalising, categorising and interpreting Web tables (e.g., [
Wikipedia editors can choose from three classes of tables to add to an article: (1)
toc: table of contents; (2) infobox: attribute-value tables embedded in the
topright of the article; and (3) wikitable: relational tables embedded in the article's
body (as per Figure 1). Each class of table is directly identi able by the HTML
attribute class for the associated table tag in the Wikipedia page. Since toc tables
represent article layout and extracting RDF from infobox tables has already been
studied [
The structure and content of Wikitables can be quite complex: there is no standard to de ne/design/publish Web tables. Instead, users are informally guided by look-and-feel. Instead of duplicating the content of contiguous cells, tables often contain spans: cells that span multiple positions, including column-spans (colspans) and row-spans (rowspans). To avoid ugly narrow long tables, or tables wider than the standard display width, users often employ split tables. Oftentimes, cells may also contain (i) multiple values, (ii) prose-text alongside the primary value giving justi cation or context for that value, (iii) superscript references to sources, (iv) images or other embedded content, (v) empty cells in \optional" columns. To simplify matters, we rst normalise Wikitables by \squaring" them such that they can be represented as a matrix of cells. columns, respectively. We extract Wikitables from Wikipedia's HTML content, where an individual Wikipedia article represents a source T (a set) of tables. Creating Matrices. Table headers are very common in tables, and can be simple (one row) or hierarchical (multiple rows). We check for two levels of headers. If T contains table headers (<TH>), we say that MT (h; :) represents those headers, where h is the bottom row with TH tags. It is also common that tables contain their caption embedded in the rst row: we discard that row when generating the matrix. For each table, we must also deal with colspans and rowspans, where we divide such cells and copy the original content into each division. For instance, if the table contains a cell <TD colspan='2'>180</TD> with a colspan of 2, the cell is replicated as <TD>180</TD><TD>180</TD>. Well-formed/Ill-formed. If the result of the pre-processing of a table T 2 T can be represented as a matrix MT (n; m) (i.e., is dense and rectangular), we call the table well-formed. Otherwise we call it ill-formed. The most common cause of ill-formed tables are jagged rows, which we found to be prevalent in Wikipedia due to idiosyncrasies of the wiki-markup used by editors. However, many jagged tables contain rich content. Thus, we extended TARTAR to `repair' jagged tables by simply completing empty cells with empty strings. Finally, we do not consider tables MT (n; m) where m = 1 or n = 1; tables must be larger than 2 2. This leads us to the nal structural taxonomy of tables that we extract from Wikipedia, as illustrated in Figure 2. 3.3
We extract our corpus of Wikitables from the January 2013 Wikipedia dump. We rst apply the Bliki engine parser1 to convert wiki-markup to the HTML pages for articles. Each HTML page is then cleaned and canonicalized ( xing syntax mistakes) using CyberNeko2 before all HTML tables (including nested tables) are extracted. The size of the resulting corpus is summarised in Table 1.
In total, 17.3% of Wikipedia articles contain at least one table (vs. 75%
reported in [
101 102 Tables per article 105 s leb103 a t f o 101 rows columns internal external 100 101 102 Links per table 103 { The article3 with the highest number of well-formed tables contains 623 (most of them nested inner-tables). { Considering articles with at least one table, there are 1.66 tables per article. { The maximum number of detected rows in a table is 250, commonly found in List of * articles (likely due to a Wikipedia formatting constraint). The average number of rows per table is 12.44. 3 http://en.wikipedia.org/wiki/Winners and runners-up in the legislative elections of Nepal 1994 and 1999 { The maximum number of columns in a table4 is 250 (due to an erroneous colspan for a caption in a table with 3 columns). The average number of columns per table is 5.55. { The highest number of internal links in a single table is 2,774, and 594 for external links. The averages are 1.93 and 0.35 for internal and external links, respectively. Internal links are important for us since they ensure that we can map cell entries to Wikipedia articles (and thus to DBpedia entities). { 19.4% of tables do not contain column headers. 79.9% contain headers only in the rst row, 7.4% contain headers in the second row, and the remaining tables contain captions in further rows. { Only 5.5% of tables contain non-empty captions.
We thus view this corpus of tables as a rich source of structured data that can be exploited for information extraction and ultimately for tripli cation, with many tables containing a high number of internal wiki-links (as per the bottom plot in Figure 3), which can be used for entity disambiguation. 4 4.1
To extract RDF from these Wikitables, we propose to use a reference 104 knowledge-base to mine relations, where tscea 103 we use English-language data from DB- id pedia v3.8, describing 9.4 million entities. rep102 The overall corpus consists of 465 mil- fo lion unique RDF triples, and contained №101 57,985 unique RDF relations (i.e., RDF 100 predicates), 48,293 of which were in the 100 102 104 106 108 DBpedia property namespace, 1,397 of Triples per predicate which were in the curated DBpedia ontology namespace, and 28 of which were Fig. 4: Distribution of number of from external vocabularies.5 As per Fig- triples per predicate (relationship) in ure 4 (log=log), we can see that the num- DBpedia knowledge-base. ber of triples in which di erent DBpedia relationships appear follows a long-tailed distribution, where many relationships appear in few triples and few relationships appear in many triples. 4.2
In Section 3, we described how we extract a set of tables from Wikipedia, where each table is represented as a matrix and each element (cell) of the matrix 4 http://en.wikipedia.org/wiki/2007--08 QMJHL season 5 We also found examples of noise where 8,267 DBpedia resource URIs referring to
Wikipedia templates appeared as predicates.
MT (i; j) can contain a variety of content. However, we are primarily interested in internal links (class=internal) present in the cells since they can be directly mapped to DBpedia entities. We represent the set of internal links in a cell as R(i; j). Each link r 2 R(i; j) is further ltered by removing superscript footnote links, media-links with pre x Image: or File:, and by pruning fragment identi ers.
Then, for each r 2 R(i; j) after ltering, we map the corresponding Wikipedia URL to a DBpedia entity URI by following redirects and replacing the namespace http://en.wikipedia.org/wiki/ of the URL with http://dbpedia.org/resource/. For optimisation purposes, we cache the set of redirects involved. We additionally lter entities corresponding to DBpedia categories and list pages. We denote by E(i; j) the set of DBpedia entities extracted for table cell MT (i; j). Furthermore, column headers in tables contain plain strings that identify types of data, which we do not map to DBpedia resources; header cells are not mapped to resources, but could potentially be matched with DBpedia relationships at a later phase. 4.3
For discovering relations within tables, we query the DBpedia knowledge base for relationships that exist between entities in di erent cells of the same row. For each row, we look for relations between all pairs f(ei;j ; ei;k) j ei;j 2 E(i; j); ei;k 2 E(i; k); 1 i m; 1 j < k ng. Any relations found for (ei;j ; ei;k) are suggested as candidates for relating (eh;j ; eh;k) for 1 h m, h 6= i.
For discovering relations from entities in the table and p the protagonist entity of the article, we query the knowledge-base for pairs f(ei;j ; p) j ei;j 2 E(i; j); 1 i m; 1 j ng. Any relations found for (ei;j ; p) are suggested as candidates for relating (eh;j ; p) for 1 h m, h 6= i.
To query a pair, we use a SPARQL query as follows over the DBpedia knowledge-base, looking for relationships that hold in either direction: SELECT DISTINCT ?p1 ?p2
{ { <e1> ?p1 <e2> } UNION { <e2> ?p2 <e1> } } Example 2. With respect to relations within tables, if we consider table T in Figure 1, we have:
E(2; 1) = fdbr:Governor of Coloradog, E(2; 2) = fdbr:John Hickenlooperg, E(2; 3) = fdbr:Democratic Party (United States)g, and
E(2; 4) = ; (in the example MT (2; 4) is a literal).
If we query DBpedia indexes looking for relations in this row, between columns 1 and 2, or sets E(2; 1) and E(2; 2), we nd two relations in DBpedia: dbp:incumbent from 1 to 2 and dbp:title from 2 to 1. We suggest this relation may hold across entities in, e.g., E(3; 1), E(3; 2) and so forth. The principle is similar for the protagonist and elements of the table. For example, we nd the relation dbp:residence from dbr:Joseph Garcia (United States politician) to dbr:Colorado and suggest that relation from all entities in column Incumbent (S8x E(x; 2)) to dbr:Colorado. Of course, this method is approximate and may produce incorrect triples (as discussed later in Sections 4.5{4.6). 4.4
We extract all candidate relationships found for all Wikitables as RDF triples: Example 3. Consider applying the dbp:title relation found in Example 2 to row 3 (from E(3; 2) to E(3; 1)). The resulting RDF triple is then: dbr:Joesph_Garcia_(United_States_politician) dbp:title dbr:Lieutentant_Governor .
We use local indexes of the DBpedia knowledge-base for answering queries, and for each pair, we perform two atomic on-disk lookups for relations in either direction. We also use in-memory LRU caching to catch repeat queries (as caused by spans). Still, given that tables can potentially contain hundreds of entities, and that the number of entity-pairs to consider is quadratic for a given row, each table may require a large amount of lookups for relations. However, the extraction can be e ectively partitioned on a per-table basis. For example, given a cluster of shared-nothing commodity servers, if we make the full index for the reference knowledge-base available to each machine (in our scenario, a 7.7GB le), individual tables can then be assigned arbitrarily to each machine, and the extraction of triples run in an embarrassingly parallel manner.
We adopt this parallel approach and using six machines (bought ca. 2005) with 4GB of RAM, 160GB SATA hard-drives, 2.2GHz single-core processors, assigning an even work-load of input Wikipedia articles to each, the full process of extracting and normalising Wikitables from the articles and computing the candidate triples took approximately 16 days. We extracted a total of 27.9 million candidate RDF triples. However, from this set, we further lter triples that, from initial inspection, we found to be often incorrect: we lter 3.4 million re exive RDF triples (with the same subject and object resource), and a further 12 thousand triples with the predicate dbo:wikiPageDisambiguates. Afterwards, 24.4 million candidate triples remain. 4.5
Many of the extracted triples are incorrect. Aside from incorrect source data (be it the table or the relations in DBpedia), imprecision can be due to a number of reasons. Multiple entities or additional text in table cells can cause problems: e.g., in many tables referring to international sports results, the sports-person and their nationality appear in the same cell and are not currently distinguished by our approach; thus, we might say that countries have participated in sports events. Oftentimes a relation that holds between two entities in a given row (or from an entity to the protagonist) do not apply for analogous entities on a di erent row; referring to the running example, although Walker Stapleton was born in Colorado (the table protagonist), many of the other politicians were not and extracted triples suggesting otherwise would be incorrect.
To investigate this, we created an initial gold standard. We randomly selected 250 candidate triples from the total set of 24.4 million. Each of the three authors manually labelled each triple as: correct, incorrect or unknown. The information given was the triple, a link to the original Wikipedia article containing the source table and a number to identify the particular table. We did not pre-agree on any judging strategies, where interpretation of the validity of triples was left to the discretion of individual judges. After judging, we found few unknown verdicts, which we mapped to incorrect. The results are given in Table 2.
To measure the inter-rater agreement, we computed a Fleiss' coe cient of 0.43 (0 indicates no agreement, 1 indicates perfect agreement), which by convention is considered as \moderate agreement" between judges. Looking at cases of disagreement, predicates of DBpedia are often not well-de ned and their meaning generally has a subjective dimension. Temporality also caused disagreement: for example, would stating that dbr:Bill Clinton has the relation dbp:president to dbr:United States be correct or incorrect? The answer would seem open to interpretation. If we de ne consensus by a majority vote, we see that in the shared labelling, 52% of the raw extracted triples are considered correct and 48% considered incorrect without any further classi cation. If instead we only consider those triples with unamimous correct/incorrect verdicts, then 48% of the extracted triples are deemed correct while 52% are deemed incorrect. 4.6
We are currently investigating machine learning methods to classify correct/incorrect triples and to improve upon the 52% precision of our raw extraction process. For this, we are currently investigating the following main features to use for binary classi cation (amongst a variety of others): Extraction type: Whether the triple is a table-row or a protagonist relation. Ratio of rows held: The ratio of rows for which we could nd the extracted relation in the original table (we assume that higher ratios are better). Predicate label: The string similarity between the predicate label and the subject/object column header (we assume that more similar is better). Predicate multiplicity: For the triple predicate, we measure the ratio of unique subjects and unique objects to unique DBpedia triples with that predicate to indicate whether or not the triple is 1 1, 1 , 1, , etc. For example, if dbp:governor is deemed 1 1 (high ratio), we should not extract multiple such relations to a common protagonist (e.g. Colorado). Cell content: Some cells contain multiple internal links, additional text content, bullets, and so forth (we assume such noise lowers con dence).
Such features (and many more besides) can be automatically associated with candidate triples during the extraction process and we are currently investigating machine learning methods|such as SVM, Nave Bayes, Decision Trees, etc.| to train on labelled examples and to build classi ers that boost precision by ltering incorrect triples. Evaluation of such methods is subject to future work. 5
In this paper, we have discussed ongoing work in the area of extracting triples from the relational tables of Wikipedia. We have discussed related works, given a survey of Wikitables in a recent Wikipedia corpus, and discussed methods we are exploring that leverage existing knowledge-bases to guide tripli cation. We have applied our methods to one million Wikipedia tables and extracted 24.4 million raw triples with an estimated precision of 52%. To boost precision further, we are currently evaluating machine-learning methods that take a feature-set and classify raw triples as correct/incorrect. In the short term, we also hope to compare other knowledge-bases, such as YAGO2 and Freebase, for extraction.
In the longer term, since our automatic extraction is often incomplete (e.g. does not consider text/numeric cells), we are also considering a method to detect common table structures in Wikipedia that are candidates for manual mapping, where, for example, we note that climate tables for cities are often copy/pasted). If we could cluster structurally similar tables (that contain similar content in an \isomorphic" schema) with high accuracy, we could write a single mapping to triplify all such tables in the cluster. This would be similar to DBpedia, where common info-box templates are manually mapped to the core ontology terms: we would similarly have a higher-quality mapping for common table structures, and lower-quality automatic extraction (as presented) for the \long tail". Integrating our methods with DBpedia's extractors could then be possible.
An even more ambitious direction would be to generalise our methods to
enrich existing knowledge-bases from generic HTML tables. For this, we could
investigate use of existing entity recognition tools (e.g., [
Acknowledgements: This paper was funded in part by Science Foundation Ireland under Grant No. SFI/08/CE/I1380 (Lion-2).